Technology · ssh / techDive
Kafka Consumer Groups Explained Part - 2
Understand how Kafka distributes partitions across consumers in a group, what happens when consumers join or fail, and why partitions define the limit of parallel consumption.
In the first Kafka mental model, one Fulfillment Service read events from the orders topic. Now scale that service. Suppose orders has four partitions and you start a second, third, or sixth consumer process.
Who gets which partition?
That is the problem Kafka consumer groups solve. Partitions are Kafka's units of parallel consumer work, and a consumer group lets several instances of one logical application divide those partitions among themselves.
From One Consumer to Many
Keep the topic simple:
orders → P0, P1, P2, P3
A single Fulfillment Service instance could read all four partitions. If engineers start another instance, both consumers can join the same group by using the same group.id, for example fulfillment-service.
Apache Kafka's current KafkaConsumer documentation defines consumer groups as a way for a pool of processes to divide record consumption and processing. Consumer instances sharing the same group.id belong to the same group when Kafka's group-management APIs are used.
Together, those processes form one logical subscriber whose partition work Kafka coordinates.
How a Consumer Group Divides Partitions
The rule that matters most is scoped to one consumer group: each assigned partition is consumed by exactly one consumer in that group at a time. Kafka's current consumer documentation states this directly.
With four partitions and two consumers, one possible assignment is:
C1 → P0, P1C2 → P2, P3
The exact mapping is not guaranteed; it depends on the active group protocol and assignment strategy. But the invariant is that P0 is not simultaneously assigned to both C1 and C2 inside the same normal consumer group. Kafka 4.3 supports both classic and consumer group protocols, with different assignment mechanics.
That is unrelated to broker replication. Replicas concern copies of partition data on brokers. Consumer-group assignment concerns which consumer process handles a partition.
With four consumers, a possible assignment becomes:
C1 → P0C2 → P1C3 → P2C4 → P3
Now all four partitions can be consumed in parallel by different members of the group.
Why More Consumers Do Not Always Mean More Parallelism
Add two more consumers without adding partitions:
C1 → P0C2 → P1C3 → P2C4 → P3C5 → no assignmentC6 → no assignment
Kafka allows those extra consumers to exist, but there is no fifth or sixth orders partition to assign. For this single topic, four members can simultaneously own partition work. That follows from Kafka's one-consumer-per-partition-within-a-group assignment rule.
A consumer can own several partitions when consumers are fewer than partitions. When the counts match, an assignment may give one partition to each consumer. When consumers outnumber the relevant partitions, some members can receive no partition assignment.
Partition count therefore affects group parallelism. It is not a sizing formula: groups may subscribe to multiple topics, and partition-count decisions involve other workload and operational trade-offs.
Multiple Groups Can Read the Same Topic
Now add an Analytics Service. It should not normally join fulfillment-service, because that would make analytics instances share the Fulfillment Service's partition work.
Instead, give it another group identity:
Fulfillment Service → group.id = fulfillment-service
Analytics Service → group.id = order-analytics
The same orders partitions can then be consumed independently in each group. Fulfillment owning P0 does not stop Analytics from reading P0 in its own group.
This gives the useful contrast described in Kafka's consumer documentation: within one group, work is divided; across separate groups, each group behaves as an independent logical subscriber. Kafka supports multiple groups for the same topic without creating another physical copy of the topic's data for each group.
What Happens When a Consumer Fails?
Assignments are not permanent. Suppose the group starts as:
C1 → P0, P1C2 → P2, P3
If C2 shuts down or is detected as failed, its partitions must be reassigned. Conceptually, C1 might temporarily end up with all four. If a replacement C3 joins later, Kafka may redistribute the partitions again. Kafka documents joins, failures, shutdowns and partition changes as events that can cause group assignments to change.
Kafka calls this redistribution a rebalance.
The exact coordination mechanics are version- and configuration-dependent. Kafka 4.3 supports both the older classic group protocol and the newer consumer protocol; the newer protocol uses broker-side assignment machinery. For a beginner mental model, the stable idea is simpler: Kafka coordinates group membership and distributes subscribed partitions across members.
A rebalance changes consumer-to-partition ownership. It is not a broker leader election and does not mean Kafka is moving the partition's stored data between brokers.
The Mental Model to Remember
For one topic and one normal consumer group, partitions are the pieces of work Kafka assigns. One consumer may handle several partitions, but one partition is assigned to only one consumer in that group at a time. Add consumers until the available partition work is covered; beyond that, extra members may sit without an assignment.
Separate consumer groups solve a different problem. They let independent applications—such as Fulfillment and Analytics—read the same retained topic independently.
That leaves one important question: when a partition moves from C2 to C1, how does the group know where processing should continue? Kafka uses committed offsets to preserve group progress per partition. We'll examine that separately later in the series.
Conversation
Comments
Sign in to join the conversation.