The Log: Topics, Partitions & Offsets — Apache Kafka — The Distributed Log, End to End

Lesson 1 said Kafka is a log. This lesson is the data model that turns “a log” into a system that scales to millions of events per second: topics, partitions, and offsets. It’s a small vocabulary — three words — but the relationships between them, especially the ordering rule, are the foundation everything else in the course builds on.

Topic → partitions → offsets

Topic — a named stream of events, like orders or clicks. It’s the logical thing you produce to and consume from. A system typically has many topics, one per kind of event.
Partition — a topic isn’t a single sequence; it’s split into N partitions. Each partition is a separate, strictly-ordered, append-only log that lives on a broker. Partitions are the unit of parallelism and scale: splitting a topic across partitions lets many producers and consumers work on it at once, and spreads the data and load across brokers. A topic with 12 partitions can be consumed by up to 12 consumers in parallel (Lesson 4).
Offset — within a partition, each message gets a sequential offset — 0, 1, 2, 3, … — its position in that partition’s log. Offsets only ever increase; Kafka never renumbers or reuses them. A consumer tracks an offset per partition to remember how far it has read, which is how it resumes after a restart and how it replays (move the offset back).

So the full picture: a topic is a set of partitions; each partition is an ordered log of messages identified by offset. Producing a message means appending it to the end of one partition; consuming means reading forward from your current offset.

The one rule: order is per-partition

Here is the single most important consequence of this model, and the thing newcomers most often get wrong:

Kafka guarantees ordering within a partition, not across partitions.

Within one partition, messages are strictly ordered by offset — offset 5 was appended before offset 6, always. But Kafka makes no guarantee about the relative order of messages in different partitions. A message at P0:offset 100 and one at P1:offset 5 have no defined ordering between them; they may be processed in either order.

This is not a limitation to work around — it’s the fundamental trade-off that lets Kafka scale. Global ordering across everything would mean a single sequence, which means a single writer and no parallelism. By ordering per partition instead, Kafka gets both ordering where it matters and parallelism across partitions. The art is making sure the things that need to be ordered land in the same partition — which is what keys are for.

Keys: how you control ordering and placement

A message can carry a key (e.g. user_42, order_99). The key determines which partition the message goes to: Kafka hashes the key and maps it to a partition, so all messages with the same key land in the same partition — and are therefore strictly ordered relative to each other. (Producers and the partitioner are Lesson 3.)

This is the mechanism that reconciles ordering with scale:

Key by user_id, and all of one user’s events are in one partition, in order — while different users spread across partitions for parallelism. You get per-user ordering and throughput.
Key by order_id, and every event for an order (created → paid → shipped) stays ordered, even though the topic as a whole is processed in parallel.
Use no key, and messages spread round-robin across partitions for maximum parallelism — but with no per-entity ordering.

So choosing the partition key is really choosing your unit of ordering, and it’s one of the most consequential design decisions in a Kafka system. (It’s also where skew can bite — a hot key sends everything to one partition; we measure that in Lesson 3.)

How many partitions?

Partition count is a sizing decision you should make deliberately, because it sets your ceiling:

More partitions = more parallelism (more consumers can work at once, Lesson 4) and more even spread across brokers.
But more partitions = more overhead (open files, memory, longer failover, more metadata) and, past a point, diminishing returns.
Crucially, you can add partitions later but it’s disruptive: adding partitions changes the hash(key) % N mapping, so existing keys may move to different partitions — breaking the per-key ordering guarantee for data already in flight. So choose with headroom for growth rather than resizing constantly.

A reasonable default is “enough partitions for your expected peak consumer parallelism, with room to grow” — often a dozen to a few dozen for a busy topic, far more for very high throughput. We’ll return to sizing in production (Lesson 11). For now, the key takeaway is the model itself: topics split into ordered partitions addressed by offsets, with keys controlling placement and ordering. Everything else — producers, consumers, replication, retention — operates on this structure.

Pitfalls to Avoid

Expecting global ordering across a topic. Order holds only within a partition. If two events must be ordered, they must share a key (and thus a partition). Designing as if the whole topic is one ordered sequence is the classic Kafka mistake.
Partitioning by a key that doesn’t match your ordering need. If you need per-order ordering but key by user_id, an order’s events can split across partitions and reorder. Key by the entity whose order you must preserve.
Too few partitions. The partition count caps consumer parallelism (Lesson 4). Set too low and you can’t scale consumers no matter how many you add.
Way too many tiny partitions. Thousands of partitions add real overhead (memory, failover time, metadata). Size for your throughput, not “more is always better.”
Assuming you can freely re-partition. Adding partitions remaps hash(key) % N, breaking per-key ordering for existing keys. Plan partition count with headroom instead of resizing often.
Treating offsets as global IDs. An offset is only meaningful within its partition. “Offset 500” means nothing without saying which partition.

Quick Recap

Topic = a named event stream; it’s split into partitions (the unit of parallelism/scale); each partition is a strictly-ordered, append-only log; each message has an offset (its position in its partition, ever-increasing).
Order is guaranteed within a partition, not across — the fundamental trade-off that enables parallelism. Make things that must be ordered land in the same partition.
A message’s key picks its partition (same key → same partition → ordered), so the key is your unit of ordering. No key → round-robin spread, no per-entity order.
Partition count sets your parallelism ceiling and is disruptive to change (it remaps keys) — choose with headroom.

Next lesson: how events actually get in — producers: keys and partitioning, acks, batching, and the idempotent producer (with a measured look at partition skew).