Apache Kafka — The Distributed Log, End to End

Apache Kafka runs the event backbone of a huge fraction of modern systems — payments, logistics, clickstreams, change-data-capture, metrics, and the stream processing on top of all of it. But most Kafka explanations are either a shallow “it’s a message queue” or a wall of configuration options with no mental model underneath. This course takes the middle path: it builds Kafka up from the one idea everything rests on — the distributed log — and then works outward to producers, consumers, durability, storage, architecture, the ecosystem, and production operation, with the important claims grounded in numbers rather than adjectives.

We start by getting the core idea right: Kafka is a durable, replayable, ordered-per-partition commit log, not a queue that deletes on read — and that one difference is why it became the event backbone it is. From there we build the data model (topics → partitions → offsets), then producers (how a key picks a partition, how acks and batching trade durability and latency against throughput, and the idempotent producer), and consumers and consumer groups (how work is shared, why the partition count caps parallelism, and what rebalancing costs).

Then the properties that make Kafka trustworthy at scale: delivery semantics (at-least-once by default, and how exactly-once actually works), replication and durability (leaders, ISR, and the acks + min.insync.replicas combination that prevents data loss), and storage, retention and compaction (how Kafka keeps data on disk, and how to size it). We cover the modern KRaft architecture (life after ZooKeeper), the ecosystem (Connect, Schema Registry, Streams, ksqlDB), stream processing with Kafka Streams, operating Kafka in production (consumer lag, monitoring, sizing, tuning), and finally the honest patterns — where Kafka shines and where it’s the wrong tool.

It’s written in plain Indian English with a working engineer in mind, and it’s grounded in reproducible measurements I ran: a high-cardinality key spreading across 12 partitions within 1.04× (even) while one hot key creates 7.6× skew and a lagging consumer; a 12-partition topic feeding 1→12 consumers linearly but leaving a 16th consumer idle; a 5%-duplicate stream deduplicated cleanly by event id; the acks/min.insync.replicas durability table on RF=3; and disk sizing that turns 50k msg/s × 1 KB × 7 days × RF 3 into 92.9 TB. Every lesson ends with pitfalls and a quick recap.

What This Course Covers

#	Lesson	What you will learn
1	What Is Kafka & Why	The distributed commit log; why a log beats a queue; what it’s for
2	The Log: Topics, Partitions & Offsets	The data model; order per partition; parallelism
3	Producers	Keys → partitions, acks, batching & compression, idempotence
4	Consumers & Consumer Groups	Offsets, commits, rebalancing, the partition-count parallelism cap
5	Delivery Semantics	At-most/least/exactly-once; dedup (measured); transactions
6	Replication, ISR & Durability	Leaders/followers, RF, acks + min.insync.replicas (measured)
7	Storage, Retention & Compaction	Segments, retention vs compaction, disk sizing (measured)
8	KRaft & Cluster Architecture	Controllers, metadata, life after ZooKeeper
9	The Ecosystem	Connect, Schema Registry, Streams, ksqlDB/Flink
10	Kafka Streams & Stream Processing	Stateful processing, state stores, windows, exactly-once
11	Operating Kafka in Production	Consumer lag, monitoring, sizing, tuning, rebalancing
12	Patterns & When NOT to Use Kafka	Event sourcing, CDC, CQRS; anti-patterns; alternatives

Who Is This For?

• Backend engineers building event-driven services who need Kafka’s model, not just its API
• Data engineers running ingestion, CDC and streaming pipelines on Kafka
• Anyone who’s used Kafka through a library but wants to understand partitions, consumer groups, durability and exactly-once properly
• Engineers deciding whether to adopt Kafka — and wanting the honest “when not to”

This course pairs naturally with the other backend and data courses on the site — Data Engineering: Building Data Pipelines (where Kafka is the streaming backbone), Distributed Systems in Practice (the replication, consensus and time ideas under Kafka), Observability & Production Debugging (operating it), and Go for Backend Engineers (a great language for writing producers and consumers).

How to Use It

Read it in order the first time — it builds from the log up through producers, consumers, durability, storage, architecture and the ecosystem to production and patterns, and each lesson leans on the last. After that, keep it as a reference: when you’re choosing a partition key (Lessons 2–3), sizing a consumer group (Lesson 4), setting durability (Lessons 5–6), provisioning disk (Lesson 7), or deciding whether Kafka fits at all (Lesson 12), come back to that lesson. The measured numbers are there so you can reason about Kafka with real figures, and the pitfalls sections are where the hard-won operational lessons live.

Let’s start with the idea the whole system is built on: Kafka as a distributed commit log.