Kafka Consumer Groups Explained Part - 2

Kafka Partitions and Consumer Groups Explained<p>In the first Kafka mental model, one Fulfillment Service read events from the <code>orders</code> topic. Now scale that service. Suppose <code>orders</code> has four partitions and you start a second, third, or sixth consumer process.</p><p>Who gets which partition?</p><p>That is the problem Kafka consumer groups solve. Partitions are Kafka's units of parallel consumer work, and a consumer group lets several instances of one logical application divide those partitions among themselves.</p><img src="https://media.hyive.com/users/1d40ca1f-8d27-4361-baca-b0c668644889/4c3dc763-2db0-4293-a6cc-6626d4cca684.png" alt="image"><h2>From One Consumer to Many</h2><p>Keep the topic simple:</p><p><code>orders</code> → P0, P1, P2, P3</p><p>A single Fulfillment Service instance could read all four partitions. If engineers start another instance, both consumers can join the same group by using the same <code>group.id</code>, for example <code>fulfillment-service</code>.</p><p>Apache Kafka's current <code>KafkaConsumer</code><a href="https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html" target="_blank" rel="noopener noreferrer ugc"> documentation</a> defines consumer groups as a way for a pool of processes to divide record consumption and processing. Consumer instances sharing the same <code>group.id</code> belong to the same group when Kafka's group-management APIs are used.</p><p>Together, those processes form one logical subscriber whose partition work Kafka coordinates.</p><img src="https://media.hyive.com/users/1d40ca1f-8d27-4361-baca-b0c668644889/c72c57af-0ebf-4daa-b889-db2ca8c73797.png" alt="image"><h2>How a Consumer Group Divides Partitions</h2><p>The rule that matters most is scoped to one consumer group: <strong>each assigned partition is consumed by exactly one consumer in that group at a time</strong>. Kafka's current consumer documentation states this directly.</p><p>With four partitions and two consumers, one possible assignment is:</p><p><code>C1 → P0, P1</code><code>C2 → P2, P3</code></p><p>The exact mapping is not guaranteed; it depends on the active group protocol and assignment strategy. But the invariant is that P0 is not simultaneously assigned to both C1 and C2 inside the same normal consumer group. Kafka 4.3 supports both <code>classic</code> and <code>consumer</code> group protocols, with different assignment mechanics.</p><p>That is unrelated to broker replication. Replicas concern copies of partition data on brokers. Consumer-group assignment concerns which consumer process handles a partition.</p><p>With four consumers, a possible assignment becomes:</p><p><code>C1 → P0</code><code>C2 → P1</code><code>C3 → P2</code><code>C4 → P3</code></p><p>Now all four partitions can be consumed in parallel by different members of the group.</p><h2>Why More Consumers Do Not Always Mean More Parallelism</h2><p>Add two more consumers without adding partitions:</p><p><code>C1 → P0</code><code>C2 → P1</code><code>C3 → P2</code><code>C4 → P3</code><code>C5 → no assignment</code><code>C6 → no assignment</code></p><p>Kafka allows those extra consumers to exist, but there is no fifth or sixth <code>orders</code> partition to assign. For this single topic, four members can simultaneously own partition work. That follows from Kafka's one-consumer-per-partition-within-a-group assignment rule.</p><p>A consumer can own several partitions when consumers are fewer than partitions. When the counts match, an assignment may give one partition to each consumer. When consumers outnumber the relevant partitions, some members can receive no partition assignment.</p><p>Partition count therefore affects group parallelism. It is not a sizing formula: groups may subscribe to multiple topics, and partition-count decisions involve other workload and operational trade-offs.</p><h2>Multiple Groups Can Read the Same Topic</h2><p>Now add an Analytics Service. It should not normally join <code>fulfillment-service</code>, because that would make analytics instances share the Fulfillment Service's partition work.</p><p>Instead, give it another group identity:</p><p><code>Fulfillment Service → group.id = fulfillment-service</code></p><p><code>Analytics Service → group.id = order-analytics</code></p><p>The same <code>orders</code> partitions can then be consumed independently in each group. Fulfillment owning P0 does not stop Analytics from reading P0 in its own group.</p><p>This gives the useful contrast described in Kafka's consumer documentation: <strong>within one group, work is divided; across separate groups, each group behaves as an independent logical subscriber.</strong> Kafka supports multiple groups for the same topic without creating another physical copy of the topic's data for each group.</p><h2>What Happens When a Consumer Fails?</h2><p>Assignments are not permanent. Suppose the group starts as:</p><p><code>C1 → P0, P1</code><code>C2 → P2, P3</code></p><p>If C2 shuts down or is detected as failed, its partitions must be reassigned. Conceptually, C1 might temporarily end up with all four. If a replacement C3 joins later, Kafka may redistribute the partitions again. Kafka documents joins, failures, shutdowns and partition changes as events that can cause group assignments to change.</p><p>Kafka calls this redistribution a <strong>rebalance</strong>.</p><p>The exact coordination mechanics are version- and configuration-dependent. Kafka 4.3 supports both the older <code>classic</code> group protocol and the newer <code>consumer</code> protocol; the newer protocol uses broker-side assignment machinery. For a beginner mental model, the stable idea is simpler: Kafka coordinates group membership and distributes subscribed partitions across members.</p><p>A rebalance changes <strong>consumer-to-partition ownership</strong>. It is not a broker leader election and does not mean Kafka is moving the partition's stored data between brokers.</p><h2>The Mental Model to Remember</h2><p>For one topic and one normal consumer group, partitions are the pieces of work Kafka assigns. One consumer may handle several partitions, but one partition is assigned to only one consumer in that group at a time. Add consumers until the available partition work is covered; beyond that, extra members may sit without an assignment.</p><p>Separate consumer groups solve a different problem. They let independent applications—such as Fulfillment and Analytics—read the same retained topic independently.</p><p>That leaves one important question: when a partition moves from C2 to C1, how does the group know where processing should continue? Kafka uses committed offsets to preserve group progress per partition. We'll examine that separately later in the series.</p><h2>Official References</h2><ul><li><p><a href="https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3.1 — KafkaConsumer API</a></p></li><li><p><a href="https://kafka.apache.org/43/configuration/consumer-configs/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Consumer and Share Consumer Configs</a></p></li><li><p><a href="https://kafka.apache.org/43/operations/consumer-rebalance-protocol/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Consumer Rebalance Protocol</a></p></li><li><p><a href="https://kafka.apache.org/43/design/design/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Design</a></p></li><li><p><a href="https://kafka.apache.org/43/implementation/distribution/" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Distribution</a></p></li></ul>