Kafka Consumer Offsets Explained Part - 4

Kafka Offsets and Delivery Semantics Explained<p>A Fulfillment Service reads <code>OrderCreated(order-123)</code> from <code>orders</code> partition P1, creates the fulfillment job, and then crashes before recording its progress. Another consumer takes over P1.</p><p>Should Kafka return that record again?</p><p>That small gap between doing the work and recording progress is where Kafka consumer offsets become operationally important. Kafka maintains both a consumer's current position and durable committed progress, and the timing of that progress update determines whether failure can cause repeated or skipped processing.</p><img src="https://media.hyive.com/users/1d40ca1f-8d27-4361-baca-b0c668644889/fc25da3d-fcae-4ee6-b8e7-42733e2e2220.png" alt="image"><h2>Kafka Tracks Two Kinds of Position</h2><p>Imagine P1 contains:</p><p><code>40 → OrderCreated(order-123)</code><code>41 → OrderCreated(order-124)</code><code>42 → OrderCreated(order-125)</code></p><p>Each record has a partition-scoped offset. Kafka's consumer API also distinguishes the consumer's current <strong>position</strong> from its <strong>committed position</strong>. The current position is the offset of the next record that would be returned and advances as records are handed to the consumer through <code>poll()</code>. The committed position is durable progress: after failure or restart, it is the position the consumer recovers to.</p><p>That distinction prevents a common off-by-one mistake. If the application successfully finishes offset 40, the committed value should normally be <strong>41</strong> — the next record it intends to read. Kafka's current <code>KafkaConsumer</code> documentation explicitly defines committed offsets this way.</p><h2>What Committing an Offset Really Means</h2><p>Suppose <code>fulfillment-service</code> commits:</p><p><code>P1 → 41</code></p><p>That does not delete record 40, modify it, or mark it consumed for every Kafka application. It records progress associated with that consumer group. Another group can independently maintain a different position for the same partition, while Kafka record retention remains a separate concern.</p><p>Kafka persists group offset commits through its coordination infrastructure. Current Kafka implementation documentation describes the special compacted internal topic <code>__consumer_offsets</code>, to which the group coordinator appends offset-commit information.</p><p>If older records are still retained, a consumer can also deliberately move its position and re-read earlier data rather than treating a commit as deletion.</p><h2>Why At-Least-Once Can Produce Duplicates</h2><p>Return to <code>OrderCreated(order-123)</code> at offset 40.</p><p>The consumer reads it, creates the fulfillment job, and then crashes <strong>before</strong> committing 41. When another consumer takes over P1, Kafka can recover only the last durable committed position. Offset 40 can therefore be returned again, causing the application to attempt the business action a second time. Kafka's own consumer documentation describes this exact class of failure: process first, crash before the commit, then repeat the last work after recovery.</p><p>That is the classic <strong>at-least-once</strong> failure window:</p><p><strong>read → process → commit</strong></p><p>Failure between processing and committing can produce duplicate <em>processing</em>. Kafka has not necessarily stored another copy of the original record; the retained record is being consumed again because durable progress had not advanced.</p><p>Applications using this model therefore commonly need business operations that can tolerate repeated attempts, although the correct idempotency or deduplication strategy depends on the application.</p><p> </p><img src="https://media.hyive.com/users/1d40ca1f-8d27-4361-baca-b0c668644889/7e7d026e-61f8-4fb2-8519-e55735b025b9.png" alt="image"><p>Automatic commits do not automatically mean at-most-once. Kafka's current Java consumer documentation says automatic offset commits can still provide at-least-once behavior if all records returned by each <code>poll()</code> are fully processed before a subsequent poll or consumer close can move committed progress ahead. Manual <code>commitSync()</code> or <code>commitAsync()</code> instead gives the application more explicit control over when progress is recorded.</p><h2>How At-Most-Once Can Lose Processing</h2><p>Reverse the sequence:</p><p><strong>read → commit → process</strong></p><p>The consumer reads offset 40 and records 41 as its durable resume position, then crashes before creating the fulfillment job. A replacement consumer can recover at 41. Offset 40 is now behind the recorded progress even though its business work never finished. This illustrates an <strong>at-most-once</strong> failure window: avoiding replay can come at the cost of skipped processing after failure.</p><p>It is a conceptual model, not a claim that every at-most-once implementation must be structured exactly this way.</p><h2>What Exactly-Once Actually Requires</h2><p>“Exactly once” needs a boundary. Kafka producer idempotence is enabled by default when producer configuration does not conflict with its requirements, and it prevents supported producer retries from writing duplicate copies of the same producer record. It does <strong>not</strong> make an unrelated database update or HTTP request execute exactly once.</p><p>Kafka transactions cover a broader Kafka-native boundary. In a consume-transform-produce flow, a transactional producer can include produced records and consumed offsets in the same transaction; those offsets become committed only if the transaction itself commits. A consumer configured with <code>isolation.level=read_committed</code> filters aborted transactional records. Neither feature automatically makes arbitrary external side effects atomic.</p><p>Kafka's consumer documentation also describes another approach when results live in an external transactional system: store the processing result and its corresponding offset in that same system atomically, then restore the Kafka consumer position from the externally stored offset. This is possible only when the chosen external system can provide that shared transaction boundary.</p><p>Kafka Streams provides <code>exactly_once_v2</code> as an exactly-once processing mode for Kafka-native stream-processing applications.</p><h2>The Mental Model to Remember</h2><p>A <strong>record offset</strong> identifies a position within one partition. A consumer's <strong>position</strong> says what it would return next. A <strong>committed position</strong> is durable progress used after restart or reassignment.</p><p>Then the crash window determines the trade-off:</p><p><strong>process before commit → possible replay → at-least-once</strong></p><p><strong>commit before process → possible skipped work → at-most-once</strong></p><p>Exactly-once behavior requires the processing result and progress to be coordinated inside a transactional boundary that actually covers both.</p><p>That completes the basic Kafka architecture chain: partitions organize the log, consumer groups divide the work, replication protects broker-side copies, and committed offsets tell consumers where durable processing progress resumes.</p><h2>Official References</h2><ul><li><p><a href="https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html?utm_source=chatgpt.com" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3.1 — KafkaConsumer API</a></p></li><li><p><a href="https://kafka.apache.org/43/implementation/distribution/?utm_source=chatgpt.com" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Distribution</a></p></li><li><p><a href="https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html?utm_source=chatgpt.com" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3.1 — KafkaProducer API</a></p></li><li><p><a href="https://kafka.apache.org/43/configuration/producer-configs/?utm_source=chatgpt.com" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Producer Configs</a></p></li><li><p><a href="https://kafka.apache.org/43/configuration/consumer-configs/?utm_source=chatgpt.com" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Consumer Configs</a></p></li><li><p><a href="https://kafka.apache.org/43/streams/developer-guide/config-streams/?utm_source=chatgpt.com" target="_blank" rel="noopener noreferrer ugc">Apache Kafka 4.3 — Kafka Streams Configuration</a></p></li></ul>