Streaming with Kafka — Data Engineering: Building Data Pipelines — From Raw Events to a Trustworthy Warehouse

When a use case genuinely needs low latency (Lesson 2), you reach for streaming — and the backbone of streaming data infrastructure is Apache Kafka. Kafka is deceptively simple at its core (it’s a log) but its model — topics, partitions, offsets, consumer groups — and its delivery guarantees are essential knowledge for any data engineer. This lesson is that model, and the delivery-semantics reality that shapes how you build correct streaming pipelines.

The Kafka Model: Topics, Partitions, Offsets

Kafka’s core abstraction is a topic: a named stream of events (say, orders). But a topic isn’t a single sequence — it’s split into partitions, and partitions are the key to everything:

Each partition is an ordered, append-only log. Events are appended to the end, and each gets an offset — its position number in that partition (0, 1, 2, …). Within a partition, order is strict and guaranteed.
Order is per-partition, not across partitions. This is the subtlety that trips people up: Kafka guarantees ordering within a partition, but says nothing about order across partitions. Event at P0:offset5 and P1:offset5 have no defined ordering relative to each other.
Partitions give parallelism. Splitting a topic into N partitions lets N consumers read in parallel — that’s how Kafka scales to millions of events per second.

Reading is done by consumer groups. A consumer group is a set of consumers that together read a topic, with each partition assigned to one consumer in the group — so the work is divided and parallelised. Each consumer tracks its offset (how far it has read) per partition, so it knows where to resume. Add consumers to a group (up to the partition count) to scale reading; if one dies, its partitions are reassigned to the others.

The property that makes Kafka special — and different from a traditional message queue — is retention. Kafka keeps events for a configured time (days, or forever), rather than deleting them when they’re read. This makes a Kafka topic a replayable log, not a one-shot queue:

A consumer can replay from any past offset — reprocess the last hour after fixing a bug, or rewind to rebuild state.
A brand-new consumer can read the entire retained history from offset 0 — add a new downstream system and backfill it from the stream.
Many independent consumers read the same topic at their own pace, each tracking its own offset — the analytics pipeline, the fraud detector, and the search indexer all consume orders without affecting each other.

This decoupling — durable, replayable, many-readers-at-their-own-pace — is why Kafka became the central nervous system of streaming data platforms (and why CDC, Lesson 3, streams change events through it).

Keys, Partitioning, and Skew

How does an event get assigned to a partition? By its key. A producer attaches a key (e.g. customer_id) to each message, and Kafka hashes the key to pick the partition — so all messages with the same key land in the same partition, and are therefore strictly ordered relative to each other. This is how you get ordering where it matters: all of one customer’s events, in order, even though the topic as a whole is parallelised across partitions.

Choosing the partition key is exactly like choosing a shard key (the Database Internals and Distributed Systems courses cover this): you want a key that spreads load evenly while keeping related events together. And the warning is the same as Lesson 6: watch for skew. If one key is a hot spot (one massive customer = 70% of events), its partition gets overloaded while others idle, and the consumer for that partition becomes the bottleneck. A hot key in a Kafka topic has the same 5.9×-the-average effect we measured for partitioning — choose keys with enough cardinality to spread evenly.

Delivery Semantics: Design for Duplicates

The last essential piece is what guarantee you get about each event being processed — and the honest answer shapes every correct streaming pipeline.

Delivery semantics: at-least-once is normal — what happens when the network retries and why you design for duplicates. At-most-once: send and forget; a failure means message LOST; no duplicates, but data loss; rarely acceptable. At-least-once: retry until acked; a retry means DUPLICATES possible; no loss — the common default; you must DEDUP downstream. Exactly-once: no loss AND no dups; needs idempotent writes plus transactions / dedup keys; possible but costly and narrow. The practical stance: distributed messaging almost always gives you at-least-once delivery — to guarantee nothing is lost, the system retries on uncertainty, and retries produce duplicates; exactly-once exists but is narrow, costly, and often really at-least-once plus idempotent processing underneath; so the robust design is: ACCEPT that duplicates and retries will happen, and make your processing IDEMPOTENT — dedup on a stable event id (measured: a 4.8% duplicate rate deduped cleanly by event_id), make writes upserts keyed by business id; build for repeats, and exactly-once becomes a property you achieve, not a setting you toggle.

There are three possible guarantees:

At-most-once — send and forget. If something fails, the message is lost. No duplicates, but data loss — rarely acceptable for data you care about.
At-least-once — retry until acknowledged. Nothing is ever lost, but a retry after an uncertain failure can deliver the same message twice — duplicates are possible. This is the common default, because “never lose data” is usually the priority.
Exactly-once — no loss and no duplicates. The ideal, but it’s narrow and costly: it requires idempotent writes plus transactions or dedup keys, works only within specific boundaries, and under the hood is often “at-least-once + idempotent processing” rather than true magic.

The practical reality: distributed messaging almost always gives you at-least-once, because to guarantee nothing is lost, a system must retry when it’s unsure whether a message got through — and those retries produce duplicates. So the robust engineering stance is not to chase a perfect “exactly-once” setting, but to accept that duplicates and retries will happen and make your processing idempotent:

Dedup on a stable event id. Give every event a unique id and drop ones you’ve already seen. (In the experiments for this course, a stream with a 4.8% duplicate rate deduped cleanly by event_id into exactly the right unique count — Lesson 11 shows this.)
Make writes upserts keyed by a business id, so applying the same event twice is a no-op.

Do that, and “exactly-once” becomes a property you achieve through idempotent design, not a fragile setting you toggle and hope. This is the same idempotency principle from Lesson 3 (ingestion) and from the distributed-systems and idempotency material on the site — it is the backbone of correct data movement everywhere, and streaming is where it matters most, because retries are constant. Build for repeats, and your streaming pipeline is correct regardless of the messy delivery underneath.

Pitfalls to Avoid

Assuming global ordering across a topic. Kafka orders events only within a partition, not across them. If you need two events ordered, they must share a key (and thus a partition). Don’t rely on cross-partition order.
A hot partition key. Keying by a low-cardinality or skewed field (one giant customer) overloads one partition and its consumer while others idle. Choose keys that spread load (Lesson 6’s skew, again).
Treating Kafka like a queue that deletes on read. It’s a retained log — consumers track offsets and can replay; it doesn’t delete on consume. Use retention/replay deliberately (reprocess, backfill new consumers), and set retention sensibly.
Chasing exactly-once instead of idempotency. “Turn on exactly-once and forget it” is fragile and narrow. Assume at-least-once, dedup on event id, and upsert — make processing idempotent so duplicates are harmless.
Not tracking/committing offsets correctly. Committing offsets before processing risks losing events on a crash; committing after, with idempotent processing, is the safe pattern. Understand your consumer’s offset-commit semantics.
Too few or too many partitions. Too few caps parallelism (consumers can’t scale past partition count); too many adds overhead and tiny files downstream. Size partition count to your throughput and consumer parallelism.

Quick Recap

Kafka is a replayable log. A topic splits into partitions; each partition is an ordered append-only log with per-event offsets. Order is guaranteed per-partition, not across partitions; partitions give parallelism.
Consumer groups divide partitions among consumers, each tracking its offset. Retention makes it a log, not a one-shot queue: replay from any offset, new consumers read history, many readers at their own pace — the backbone of streaming (and CDC).
Messages with the same key go to the same partition (ordered together) — choose the key like a shard key and watch for skew (a hot key overloads one partition).
Delivery semantics: at-least-once is normal (retries → duplicates), so design for duplicates: make processing idempotent — dedup on a stable event id, upsert by business id. “Exactly-once” is something you achieve through idempotency, not a magic setting.

Next lesson: running all of this reliably, on schedule, in the right order — orchestration and scheduling, with incremental processing measured at ~200× less work.