Technology · ssh / techDive
Kafka Consumer Offsets Explained Part - 4
Understand how Kafka consumers remember their position, why crashes can cause duplicates or skipped work, and what exactly-once processing really requires.
A Fulfillment Service reads OrderCreated(order-123) from orders partition P1, creates the fulfillment job, and then crashes before recording its progress. Another consumer takes over P1.
Should Kafka return that record again?
That small gap between doing the work and recording progress is where Kafka consumer offsets become operationally important. Kafka maintains both a consumer's current position and durable committed progress, and the timing of that progress update determines whether failure can cause repeated or skipped processing.
Kafka Tracks Two Kinds of Position
Imagine P1 contains:
40 → OrderCreated(order-123)41 → OrderCreated(order-124)42 → OrderCreated(order-125)
Each record has a partition-scoped offset. Kafka's consumer API also distinguishes the consumer's current position from its committed position. The current position is the offset of the next record that would be returned and advances as records are handed to the consumer through poll(). The committed position is durable progress: after failure or restart, it is the position the consumer recovers to.
That distinction prevents a common off-by-one mistake. If the application successfully finishes offset 40, the committed value should normally be 41 — the next record it intends to read. Kafka's current KafkaConsumer documentation explicitly defines committed offsets this way.
What Committing an Offset Really Means
Suppose fulfillment-service commits:
P1 → 41
That does not delete record 40, modify it, or mark it consumed for every Kafka application. It records progress associated with that consumer group. Another group can independently maintain a different position for the same partition, while Kafka record retention remains a separate concern.
Kafka persists group offset commits through its coordination infrastructure. Current Kafka implementation documentation describes the special compacted internal topic __consumer_offsets, to which the group coordinator appends offset-commit information.
If older records are still retained, a consumer can also deliberately move its position and re-read earlier data rather than treating a commit as deletion.
Why At-Least-Once Can Produce Duplicates
Return to OrderCreated(order-123) at offset 40.
The consumer reads it, creates the fulfillment job, and then crashes before committing 41. When another consumer takes over P1, Kafka can recover only the last durable committed position. Offset 40 can therefore be returned again, causing the application to attempt the business action a second time. Kafka's own consumer documentation describes this exact class of failure: process first, crash before the commit, then repeat the last work after recovery.
That is the classic at-least-once failure window:
read → process → commit
Failure between processing and committing can produce duplicate processing. Kafka has not necessarily stored another copy of the original record; the retained record is being consumed again because durable progress had not advanced.
Applications using this model therefore commonly need business operations that can tolerate repeated attempts, although the correct idempotency or deduplication strategy depends on the application.
Automatic commits do not automatically mean at-most-once. Kafka's current Java consumer documentation says automatic offset commits can still provide at-least-once behavior if all records returned by each poll() are fully processed before a subsequent poll or consumer close can move committed progress ahead. Manual commitSync() or commitAsync() instead gives the application more explicit control over when progress is recorded.
How At-Most-Once Can Lose Processing
Reverse the sequence:
read → commit → process
The consumer reads offset 40 and records 41 as its durable resume position, then crashes before creating the fulfillment job. A replacement consumer can recover at 41. Offset 40 is now behind the recorded progress even though its business work never finished. This illustrates an at-most-once failure window: avoiding replay can come at the cost of skipped processing after failure.
It is a conceptual model, not a claim that every at-most-once implementation must be structured exactly this way.
What Exactly-Once Actually Requires
“Exactly once” needs a boundary. Kafka producer idempotence is enabled by default when producer configuration does not conflict with its requirements, and it prevents supported producer retries from writing duplicate copies of the same producer record. It does not make an unrelated database update or HTTP request execute exactly once.
Kafka transactions cover a broader Kafka-native boundary. In a consume-transform-produce flow, a transactional producer can include produced records and consumed offsets in the same transaction; those offsets become committed only if the transaction itself commits. A consumer configured with isolation.level=read_committed filters aborted transactional records. Neither feature automatically makes arbitrary external side effects atomic.
Kafka's consumer documentation also describes another approach when results live in an external transactional system: store the processing result and its corresponding offset in that same system atomically, then restore the Kafka consumer position from the externally stored offset. This is possible only when the chosen external system can provide that shared transaction boundary.
Kafka Streams provides exactly_once_v2 as an exactly-once processing mode for Kafka-native stream-processing applications.
The Mental Model to Remember
A record offset identifies a position within one partition. A consumer's position says what it would return next. A committed position is durable progress used after restart or reassignment.
Then the crash window determines the trade-off:
process before commit → possible replay → at-least-once
commit before process → possible skipped work → at-most-once
Exactly-once behavior requires the processing result and progress to be coordinated inside a transactional boundary that actually covers both.
That completes the basic Kafka architecture chain: partitions organize the log, consumer groups divide the work, replication protects broker-side copies, and committed offsets tell consumers where durable processing progress resumes.
Conversation
Comments
Sign in to join the conversation.