Design a Distributed Message Queue (Kafka)

A message queue decouples producers from consumers: services publish events without knowing who reads them, and consumers process at their own pace. “Design Kafka” (or a generic distributed message queue / commit log) tests whether you understand append-only logs, partitioning for parallelism, replication for durability, and the subtle trade-offs of delivery guarantees. Let’s build one that moves millions of messages per second without losing data.

Understanding the Problem

Functional Requirements

1. Publish — Producers send messages to a named topic.
2. Subscribe — Multiple independent consumers read messages from a topic.
3. Durability — A committed message survives broker crashes.
4. Ordering — Messages with the same key are delivered in order.
5. Replay — A consumer can rewind and re-read old messages.
6. Retention — Messages are kept for a configured window (e.g. 7 days), independent of whether they were consumed.

Non-Functional Requirements

Scale Estimation

• 1 trillion messages/day ≈ ~12M messages/sec average, ~30M/sec at peak.
• Average message 1 KB → ~12 GB/sec of ingest, ~1 PB/day.
• With 7-day retention, that is ~7 PB of stored log — so storage and replication strategy dominate the design.

These numbers immediately tell you: a single machine cannot hold a topic. You must partition (to spread load and storage) and replicate (to survive failures). Those two ideas are the whole design.

Core Entities and APIs

The core abstraction is the commit log: an ordered, append-only sequence of messages. A topic is a logical stream, split into partitions. Each partition is an independent log, and every message in a partition has a monotonically increasing offset.

Data Model

Topic:  "orders"  →  Partition 0: [m0 m1 m2 m3 ...]   (offsets 0,1,2,3,...)
                     Partition 1: [m0 m1 m2 ...]
                     Partition 2: [m0 m1 m2 m3 m4 ...]

Message = {
    offset:     int64      # position in the partition (assigned by broker)
    key:        bytes      # determines the partition (hash(key) % N)
    value:      bytes      # the payload
    timestamp:  int64
    headers:    map        # optional metadata
}

# Consumer offset tracking (stored in a special internal topic)
ConsumerOffset = {
    group_id:    string
    topic:       string
    partition:   int
    offset:      int64      # next message to read
}

API Design

# Producer
POST /produce
  body: { topic, key?, value, acks: "all" }
  response: { partition, offset }     # where it landed

# Consumer
POST /subscribe
  body: { group_id, topics: ["orders"] }

GET  /poll
  body: { group_id, max_bytes }
  response: { records: [{ partition, offset, key, value }] }

POST /commit
  body: { group_id, offsets: { "orders-0": 1053, "orders-1": 980 } }

# Admin
POST /topics
  body: { name, partitions: 32, replication_factor: 3 }

Notice the producer chooses nothing about placement except the key — the broker maps hash(key) % partition_count to a partition. All messages for one key land in one partition, which is how ordering is guaranteed per key.

High-Level Design

• Producers serialize a message, pick a partition (by key hash, or round-robin if keyless), and append it to the partition’s leader broker.
• Brokers are the servers that store partitions on disk. The cluster spreads a topic’s partitions (and their replicas) across many brokers.
• Topic/partition is the unit of parallelism and ordering. More partitions = more throughput and more consumer parallelism.
• Consumers belong to a consumer group. The group collectively reads all partitions, with each partition assigned to exactly one member.
• Offsets are stored by the cluster itself (in an internal compacted topic), so a consumer that restarts resumes exactly where it left off.
• Coordination (which broker leads which partition, group membership) is handled by a metadata/consensus layer (ZooKeeper historically, KRaft/Raft in modern Kafka).

The design cleanly separates storage (brokers, partitions) from consumption (groups, offsets), which is why one topic can feed a real-time fraud detector, a data-warehouse loader, and an email service — each as its own group, each at its own speed, none interfering with the others.

Deep Dive: The Partitioned Append-Only Log

The single most important design decision is that a partition is a sequential, append-only file — not a queue with per-message delete.

Why append-only is fast

A spinning disk or SSD is dramatically faster at sequential writes than random ones. By only ever appending, the broker turns all writes into sequential I/O. Reads are sequential too (consumers scan forward). Combined with the OS page cache, recently written messages are served to consumers straight from memory without a disk seek, and Kafka can use zero-copy (sendfile) to ship bytes from the page cache to the network socket without copying through user space. This is how a single broker sustains hundreds of MB/sec.

Partitioning for parallelism and ordering

A topic is split into N partitions. This buys two things at once:

• Throughput and storage scale — partitions live on different brokers, so writes and storage spread across the cluster. Want 2x throughput? Double the partitions and brokers.
• Ordering within a key — because partition = hash(key) % N, all messages for user_42 go to the same partition, and a partition is a single ordered log. So per-key order is guaranteed, while different keys process in parallel.

The cost is that there is no total order across partitions. If you need global ordering, you need a single partition — which caps throughput. In interviews, state this trade-off explicitly: ordering is per-partition (per-key), and that is almost always the right granularity.

Choosing the partition key

The key choice is a design decision, not an afterthought:

• Key by user_id or order_id when you need per-entity ordering (all of a user’s events in sequence).
• A bad key concentrates traffic on one partition — a hot partition (the celebrity problem again). If one key is disproportionately busy, consider a composite key or salting.
• Keyless messages round-robin across partitions for maximum spread when order doesn’t matter.

def pick_partition(key, num_partitions):
    if key is None:
        return round_robin_counter() % num_partitions
    return murmur2(key) % num_partitions   # stable: same key → same partition

Deep Dive: Consumer Groups and Scaling Consumption

A single consumer can’t keep up with a firehose topic. Consumer groups are how you parallelize reading while preserving ordering.

The one-partition-per-member rule

Within a group, each partition is assigned to exactly one member. So:

• With 32 partitions and 8 consumers, each consumer handles 4 partitions.
• Scale consumers up to 32 and each handles 1 partition — maximum parallelism.
• Add a 33rd consumer and it sits idle — there’s no partition left to give it.

This is why the number of partitions is the ceiling on consumer parallelism. Pick the partition count up front with future scale in mind (it’s painful to increase later because it changes key→partition mapping).

Independent offsets per group

Each group tracks its own offset per partition, stored in the cluster. Two consequences:

• Different applications (groups) read the same topic independently — fraud detection, analytics, and notifications each at their own pace.
• A group can replay by resetting its committed offset backwards (reprocess the last day after a bug fix) — the data is still on disk thanks to retention.

Rebalancing

When a consumer joins or dies, the group rebalances: partitions are reassigned among the surviving members. A group coordinator detects membership changes (via heartbeats) and redistributes partitions. Rebalances briefly pause consumption, so you want them rare — long-lived, healthy consumers with sensible heartbeat/session timeouts.

# Consumer loop: poll, process, commit
while running:
    records = consumer.poll(max_bytes=1_000_000)
    for r in records:
        process(r)                      # your business logic
    consumer.commit()                   # advance offset AFTER processing

Committing the offset after processing (not before) is the difference between at-least-once and at-most-once — covered next.

Deep Dive: Durability and Delivery Semantics

Partition replication

Each partition has a leader and a configurable number of follower replicas on other brokers. All reads and writes go through the leader; followers continuously fetch from it. The set of replicas that are caught up is the in-sync replica set (ISR).

Durability is tuned with the producer’s acks setting:

• acks=0 — fire and forget. Fastest, can lose data.
• acks=1 — leader has written it. Loses data if the leader dies before a follower copies it.
• acks=all — all in-sync replicas have it. Safe: a committed message survives any single broker (and up to ISR−1 broker) failures.

With replication factor 3 and acks=all, a message is only acknowledged once it’s on multiple brokers, so a leader crash triggers a failover to a follower that already has the data — no loss.

Delivery semantics

The guarantee a consumer gets depends on when it commits its offset relative to processing:

In practice, at-least-once + idempotent consumers is the winning combination for almost every real system. You accept that a message might be delivered twice, and you make processing idempotent (e.g. upsert keyed by message_id, or track processed IDs). This gives you “effectively exactly-once” without the overhead of distributed transactions.

def handle(record):
    msg_id = record.headers["id"]
    if seen.contains(msg_id):      # idempotency check
        return                      # already processed — safe to skip
    apply(record.value)
    seen.add(msg_id)

True exactly-once (Kafka’s transactional producer + consumer) ties the read, the processing, and the offset commit into one atomic transaction. It works, but it’s heavier and only worth it for strict pipelines (e.g. financial aggregation). Reach for idempotency first.

Backpressure and lag

The health metric to watch is consumer lag — how far behind the latest offset each group is. Rising lag means consumers can’t keep up; the fix is more consumers (up to the partition count), faster processing, or more partitions. Because the log persists messages regardless of consumption, a slow consumer never blocks the producer — the queue naturally absorbs bursts (the write-buffering pattern from the Scaling Writes lesson).

Key Takeaways

A message queue is one of the highest-leverage components in system design — it turns up as a sub-component in feeds, notifications, payments, and crawlers. Interviewers probe two things hardest: how partitioning gives you both parallelism and ordering, and why at-least-once plus idempotency beats chasing exactly-once. Have the log diagram and the delivery-semantics table ready to draw.