Kafka Replication Explained Part - 3

Kafka Replication Explained: Leaders, Followers and Broker Failures<p>So far, the <code>orders</code> topic has survived every example because its partitions were assumed to exist. Now remove one machine.</p><p>Suppose partition P1 has a replication factor of 3. Broker 1 hosts its leader replica, while Brokers 2 and 3 host other replicas. An <code>OrderCreated(order-123)</code> event is written to P1—and then Broker 1 crashes.</p><p>Does P1 disappear with it? No. Kafka's answer is replication: other replicas already exist before the failure. Apache Kafka replicates at the topic-partition level specifically so another replica can take over when a server fails.</p><img src="https://media.hyive.com/users/1d40ca1f-8d27-4361-baca-b0c668644889/5f47567a-2f6d-4d76-9d9e-0ff24f587f9a.png" alt="image"><h2>Why Kafka Replicates Partitions</h2><p>If P1 has <code>replication.factor=3</code>, Kafka maintains <strong>three replicas total</strong>—not one leader plus three backups. The current leader is itself one of those replicas.</p><p>Conceptually:</p><p><code>Broker 1 → P1 Leader</code><code>Broker 2 → P1 Replica</code><code>Broker 3 → P1 Replica</code></p><p>Replica placement can differ from partition to partition. The purpose is to keep live copies on multiple brokers so one broker failure does not automatically remove the partition.</p><p>This is different from consumer groups. Consumer groups answer, “Which consumer processes this partition?” Replication answers, “Which brokers hold copies of it?”</p><h2>Leaders, Followers and the ISR</h2><p>One replica is the partition leader. Producer writes for that partition go to the current leader. Follower replicas copy the leader's log so they can stay synchronized and potentially take over later. Kafka 4.3 also supports reads from followers in appropriate configurations, so “all reads always go to the leader” is no longer a safe universal rule.</p><p>Kafka also tracks an <strong>ISR</strong>, or in-sync replica set. A configured replica is not automatically current enough forever. If a follower dies or falls too far behind, Kafka can remove it from the ISR; <code>replica.lag.time.max.ms</code> contributes to that decision.</p><p>A replication factor of 3 might begin with:</p><p><code>Replica set: B1, B2, B3</code><code>ISR: B1, B2, B3</code></p><p>If B3 falls behind, the configured replica set can still contain all three brokers while the ISR temporarily becomes:</p><p><code>ISR: B1, B2</code></p><p>So “configured replica” and “currently in sync” are not the same thing. Kafka's monitoring documentation likewise notes that ISR can shrink when a broker goes down and expand after its replicas catch up.</p><h2>What Happens When the Leader Broker Fails?</h2><p>Return to P1. Broker 1 is the leader and suddenly goes offline.</p><p>Kafka detects broker failure, and leadership for affected partitions moves to another eligible replica. The KRaft control plane manages partition-leadership metadata; clients do not elect the replacement leader themselves.</p><p>A simplified transition might be:</p><p><strong>Before:</strong> <code>B1 → Leader</code>, <code>B2 → Replica</code>, <code>B3 → Replica</code><strong>Failure:</strong> <code>B1 → Offline</code><strong>After:</strong> <code>B2 → New Leader</code>, <code>B3 → Replica</code></p><p>The exact broker chosen is not guaranteed by this example.</p><img src="https://media.hyive.com/users/1d40ca1f-8d27-4361-baca-b0c668644889/d062cfaf-36ae-4f41-b3aa-d2f4c008fd66.png" alt="image"><p>Failover works because followers were replicating before Broker 1 failed. Kafka does not wait for a crash and then create a backup.</p><p>When Broker 1 returns, Kafka's operations documentation says it initially returns as a follower for its partitions. Leadership may later rebalance toward preferred replicas.</p><h2>Why Kafka May Reject Writes Instead</h2><p>Replication factor alone does not determine when a write is safe enough to acknowledge.</p><p>Kafka also has <code>min.insync.replicas</code>. With a producer using <code>acks=all</code>, the setting defines the minimum number of in-sync replicas required for the write to succeed. Apache gives the common example of replication factor 3 with <code>min.insync.replicas=2</code>.</p><p>If the ISR falls below that minimum, Kafka can reject the write instead of acknowledging it under weaker durability conditions. That is an availability-versus-durability trade-off.</p><h2>Modern Kafka Also Has Eligible Leader Replicas</h2><p>Older explanations often say a replacement leader must always come from the ISR. That is no longer complete.</p><p>Kafka 4.0 introduced <strong>Eligible Leader Replicas (ELR)</strong>, and ELR is enabled by default for new clusters starting with Kafka 4.1. The KRaft controller can track replicas outside the ISR that are nevertheless considered safe leader candidates under ELR's rules. Current election logic checks the ISR first, then ELR where applicable.</p><p>Kafka can also permit unclean leader election as a last resort. Its current default is <code>false</code>, and Apache warns that enabling it may result in data loss.</p><h2>Where KRaft Fits</h2><p>Modern Kafka separates broker and controller responsibilities. Brokers store and serve partition data; KRaft controllers manage cluster metadata and control-plane decisions. Kafka 4.x no longer uses ZooKeeper mode.</p><p>The controller does not store <code>OrderCreated</code> as an application-data replica. The brokers holding P1 do that. KRaft manages the metadata describing which replica currently leads P1.</p><h2>The Mental Model to Remember</h2><p>Replication factor tells Kafka how many replicas of a partition to maintain. One is the current leader; followers replicate its log. Kafka tracks replica synchronization and eligibility, while the KRaft control plane manages leadership changes after failures.</p><p>The key idea is: <strong>failover works because replication happens before failure, not after it.</strong></p><p>Replication does not make data loss impossible. Durability still depends on replica state, acknowledgement settings and election rules.</p><p>A later article in this series can answer the consumer-side follow-up: how committed offsets and delivery semantics determine where processing continues.</p><h2>Official References</h2><ul><li><p><a href="https://kafka.apache.org/43/design/design/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Design</a></p></li><li><p><a href="https://kafka.apache.org/43/operations/basic-kafka-operations/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Basic Kafka Operations</a></p></li><li><p><a href="https://kafka.apache.org/43/operations/eligible-leader-replicas/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Eligible Leader Replicas</a></p></li><li><p><a href="https://kafka.apache.org/43/configuration/topic-configs/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Topic Configs</a></p></li><li><p><a href="https://kafka.apache.org/43/operations/kraft/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — KRaft</a></p></li><li><p><a href="https://kafka.apache.org/43/getting-started/zk2kraft/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — KRaft vs ZooKeeper</a></p></li></ul>