KRaft & Cluster Architecture

So far we’ve treated “Kafka” as a thing that stores partitions and replicates them. This lesson zooms out to the cluster: what the pieces are, how they coordinate, and the significant architectural change of recent years — KRaft, which removed ZooKeeper and simplified running Kafka considerably. If you’re deploying Kafka today, this is the architecture you’re deploying.

Before and after: goodbye ZooKeeper

For most of Kafka’s life, a cluster was actually two distributed systems: the Kafka brokers, plus a separate ZooKeeper ensemble that held the cluster’s metadata.

ZooKeeper’s job was to store cluster metadata — the list of brokers, topics and partitions, which broker leads each partition, ACLs, and configs — and to coordinate leader election. It worked, but it had real costs: you had to run, tune, secure and monitor a second distributed system; metadata operations went through ZooKeeper and became a scaling bottleneck (limiting how many partitions a cluster could handle); and the two systems’ failure modes interacted in confusing ways.

KRaft (Kafka Raft) removes ZooKeeper entirely. The metadata now lives inside Kafka itself, managed by a built-in Raft consensus quorum of controllers, stored as a replicated Kafka log (an internal metadata topic). One system, not two.

The benefits are concrete:

• Operational simplicity — one system to deploy, secure, upgrade and monitor, not two. This alone is a major win for anyone who’s operated ZooKeeper.
• Faster metadata operations and failover — metadata changes are Kafka log appends, and brokers cache the metadata log, so leader election and broker recovery are much faster.
• Far more partitions — removing the ZooKeeper bottleneck lets clusters handle millions of partitions, well beyond the old limits.

KRaft has been the default since Kafka 3.3, and ZooKeeper is removed entirely in Kafka 4.x. So if you’re starting today, you’re on KRaft — ZooKeeper is history.

What a KRaft cluster is made of

A modern Kafka cluster has two roles:

• Brokers store partition data (the segment files from Lesson 7) and serve client requests — producers’ appends (produce) and consumers’ reads (fetch). You scale throughput and storage by adding brokers and spreading partitions and their replicas across them. Brokers are where the data lives and the work happens.
• Controllers form a Raft quorum (typically 3, for an odd number that tolerates 1 failure) that owns the cluster metadata: which topics and partitions exist, the replica assignments, who leads each partition, ACLs and configs. One controller is the active leader of the quorum; the others are hot standbys. Because the metadata is a replicated Raft log, the controllers agree on it consistently and survive failures.

How they relate at runtime:

• In small clusters, a node can play both roles (broker and controller) — simple to deploy. At scale, you run dedicated controller nodes separate from brokers, so metadata management and data serving don’t contend.
• When a broker dies, the active controller notices, and for every partition that broker was leading, it elects a new leader from that partition’s ISR (Lesson 6) and tells the other brokers and clients. This happens in milliseconds, and clients transparently reconnect to the new leaders. The cluster self-heals.
• Clients (producers/consumers) talk to brokers, discover metadata (which broker leads which partition) via the cluster, and route their requests accordingly — you connect to any broker (a “bootstrap server”) and the client learns the rest.

Why the architecture matters to you

Even if you never configure a controller by hand (managed Kafka services handle it), the architecture explains the behaviour you rely on:

• Availability: RF=3 + automatic leader election from the ISR means a broker can die with no data loss and a sub-second blip — the controller orchestrates the failover.
• Scaling: add brokers for more throughput/storage; the controller rebalances partition replicas across them (with tooling). Partition count (Lesson 2) and broker count together set your ceiling.
• Simplicity: one system (KRaft) to reason about, not a Kafka-plus-ZooKeeper tangle — fewer moving parts, fewer failure modes, easier upgrades.

The practical guidance: use KRaft (you have no choice on 4.x, and it’s better anyway); run an odd number of controllers (3 is standard) so the Raft quorum has a majority; separate controllers from brokers once you’re past a small cluster; and lean on a managed Kafka offering if you don’t want to operate the cluster yourself — the architecture here is exactly what they manage for you. With the cluster understood, the next lessons move up the stack to the ecosystem and stream processing that turn the cluster into a data platform.

Pitfalls to Avoid

1. Deploying new clusters on ZooKeeper. It’s removed in Kafka 4.x and deprecated before that. Start on KRaft — don’t build new infrastructure on the legacy path.
2. An even number of controllers. A Raft quorum needs a majority; 2 controllers tolerate 0 failures (a split can’t reach majority). Use an odd number — 3 is standard.
3. Mixing broker and controller roles at scale. Fine for small/dev clusters, but at scale metadata work and data serving contend. Separate them once the cluster is large.
4. Assuming failover is instant and free. It’s fast (milliseconds) but only works if replicas are in the ISR (Lesson 6). RF=1 or an empty ISR means no safe failover — watch under-replicated partitions.
5. Not understanding bootstrap vs the full cluster. Clients connect to a bootstrap broker and then discover the rest. Pointing a client at one broker’s address only isn’t a single point of failure for discovery, but do list several bootstrap servers for resilience.
6. Operating Kafka yourself without needing to. The cluster (controllers, rebalancing, upgrades, failover) is real operational work. If it’s not your core competency, a managed Kafka service runs exactly this architecture for you.

Quick Recap

• A Kafka cluster = brokers (store partition data, serve produce/fetch — add them to scale) + controllers (own the cluster metadata).
• KRaft removed ZooKeeper: metadata now lives inside Kafka as a replicated Raft log managed by a controller quorum. One system, not two — simpler to run, faster failover, millions of partitions. Default since 3.3, ZooKeeper gone in 4.x.
• Run an odd number of controllers (3); a node can be both roles in small clusters, separate them at scale. On broker failure, the controller elects a new leader from the ISR in milliseconds.
• You scale with brokers (throughput/storage) and partitions (parallelism); RF + ISR + controller failover give availability. Consider managed Kafka to avoid operating it yourself.

Next lesson: the tools built around the broker that turn it into a platform — the Kafka ecosystem (Connect, Schema Registry, Streams, ksqlDB).