Apache Kafka Architecture Explained Part - 1

How Apache Kafka Works: Topics, Partitions and Brokers

A checkout service finishes an order and emits OrderCreated for order-123. A fulfillment service needs that event now; another application may want to read it later. Kafka sits between those applications so they do not have to call each other directly.

Apache Kafka describes itself as a distributed event-streaming platform: producers publish events, Kafka stores them durably in topics, and consumers read them. Reading a record does not inherently delete it; retention is controlled separately.

The Simple Kafka Mental Model

Follow one event:

Checkout Service → Producer → orders topic → partition → broker → Fulfillment Consumer

An event—also called a record or message in Kafka documentation—can carry a key, value, timestamp and optional headers. Our example might use order-123 as the key and OrderCreated as the value. Those are application choices, not a Kafka-required schema.

The producer is the client writing the event. The consumer is the client reading it. The interesting part is what sits between them.

Topics Organize Events

Applications publish records to named topics such as orders, payments or inventory-events. A topic is a logical stream, not one file sitting on one machine.

Kafka topics can have multiple producers and consumers. More importantly, a topic is divided into partitions. When the checkout service publishes our OrderCreated record to orders, that record is ultimately appended to one partition of the topic.

That separation is a useful Kafka architecture idea: applications name a logical stream; partitions give Kafka units it can distribute.

Partitions Make Topics Distributed

Think of each partition as an ordered append-only log:

orders partition 0:Offset 0 → Event AOffset 1 → Event BOffset 2 → Event C

orders partition 1:Offset 0 → Event DOffset 1 → Event E

Partitions let Kafka spread a topic's storage and traffic across brokers, allowing reads and writes in parallel. Kafka's ordering guarantee is within a topic-partition. A multi-partition topic is not one globally ordered sequence.

Keys help with placement. A producer can use an identifier such as order-123 so related records are routed consistently under a given partitioning setup. But changing the partition count or partitioning logic can change assignment.

Brokers Store the Partitions

A Kafka cluster contains servers performing the broker role. Brokers form Kafka's storage layer and serve client data requests. Topic partitions are distributed across those brokers.

A simplified picture might put orders P0 on Broker 1, P1 on Broker 2 and P2 on Broker 3. That is only an illustration: a broker can host many partitions, and production Kafka also replicates partitions for fault tolerance.

There is no single “master broker” through which every producer record must pass. Clients communicate with brokers responsible for the relevant partition data. Kafka's design documentation describes producers pushing data to brokers and consumers pulling data from them.

Producers Write; Consumers Read

Suppose Kafka routes order-123 to orders partition 1. The broker serving that partition accepts the record, and the record gets an offset within that partition.

Later, a Fulfillment Service consumer subscribed to orders can fetch it. Kafka consumers pull data from brokers. That read does not automatically erase OrderCreated. Kafka retains records according to the topic's configured retention policy, allowing retained data to be read again or by other consumers.

That is why calling Kafka only a message queue is incomplete. Kafka can support messaging-style workloads, but durable event storage and replay are part of its model.

What an Offset Means

An offset is a record's position inside one partition. It is not a global event ID.

If partition 1 contains offsets 0, 1 and 2, a consumer uses a position in that partition to track where it is reading. Kafka's current consumer API also allows applications to move backward and re-consume retained records. Committed offsets and delivery guarantees add more detail, but they belong in a deeper discussion.

Where KRaft Controllers Fit

Modern Kafka also has controllers. Kafka 4.0 and newer are KRaft-only; ZooKeeper mode was removed. In current Kafka 4.3 documentation, a server can have the broker role, controller role, or both, though separate roles are recommended for critical deployments.

At a high level, brokers handle data-related client work, while controllers manage cluster metadata and control-plane responsibilities. Controllers are not an extra hop that every OrderCreated event travels through.

Putting the Flow Together

Our checkout example now has a complete path. The producer publishes OrderCreated to orders; Kafka appends it to one partition hosted by a broker; the record receives a partition offset; the Fulfillment Service reads it; and Kafka retains it according to the topic's policy.

The compact mental model is: topics name streams, partitions provide ordered distributed logs, brokers store and serve them, producers write, and consumers read.

In the next part of this series, we'll look more closely at partitions and consumer groups and see how Kafka divides work across multiple consumers.

Official References

• Apache Kafka 4.3 — Introduction

• Apache Kafka 4.3 — Design

• Apache Kafka 4.3 — KRaft

• Apache Kafka 4.3 — KafkaConsumer API

• Apache Kafka 4.0 Release Announcement