menu_bookGyanByte
arrow_backBACK TO APACHE KAFKA — THE DISTRIBUTED LOG, END TO END
Lesson 01Apache Kafka — The Distributed Log, End to End4 min read

What Is Kafka & Why

October 30, 2026

TL;DR

Apache Kafka is a distributed COMMIT LOG, not a message queue. A traditional queue deletes a message once a consumer reads it — one reader, one shot. Kafka is a log: producers APPEND events, the events are RETAINED (days, or forever), and each consumer tracks its own position (offset). That one difference is everything: MANY independent consumers read the same events at their own pace (analytics, fraud, search all read 'orders' without affecting each other); any consumer can REPLAY from an old offset to reprocess after a bug; a brand-new consumer can read all of history; and producers are fully DECOUPLED from who reads, when, or how often. This is why Kafka became the event backbone for microservices, log/metric/clickstream pipelines, change data capture, and stream processing — a durable, ordered, replayable spine that connects systems without wiring them directly together.

Before any configuration or API, the single most important thing to understand about Kafka is what it is — because almost everyone’s first mental model (“it’s a message queue like RabbitMQ”) is wrong in a way that makes the rest confusing. Kafka is a distributed commit log, and that one idea explains everything else in this course. Get it right first, and partitions, consumer groups, replay and retention all follow naturally.

A log, not a queue

Here’s the distinction that matters. A traditional message queue delivers a message to a consumer and then deletes it — one message, one reader, consumed and gone. Kafka is a log: producers append events to the end, the events are retained on disk (for a configured time, or forever), and each consumer tracks its own position in the log, called an offset.

What Kafka is: a distributed commit log. Producers (orders service, payments, clicks) append events to Kafka, the log — an append-only, ordered sequence with offsets, where events are retained. Many consumers (analytics, fraud, search index) read from the log at their own position. Why a log beats a traditional message queue: a classic queue deletes a message once read (one reader, one shot); Kafka retains events so many independent consumers read the same events at their own pace, any consumer can replay from an old offset to reprocess after a bug, a new consumer reads all of history, and producers are decoupled from who reads, when, or how often.

That difference sounds small and changes everything:

  • Many independent consumers, same events. Because reading doesn’t delete, any number of consumers can read the same events independently, each at its own pace. The analytics pipeline, the fraud detector, and the search indexer all consume the orders stream without touching each other. In a delete-on-read queue, the first reader takes the message and the others never see it.
  • Replay. A consumer can rewind to any past offset and reprocess — rerun the last hour after fixing a bug, or rebuild a downstream store from scratch. The data is still there; you just move your pointer back.
  • History for newcomers. A brand-new consumer can start from offset 0 and read all of retained history — add a new downstream system and backfill it from the stream, no special export needed.
  • Decoupling. Producers don’t know — and don’t care — who reads their events, when, or how many times. They just append. Consumers appear, disappear, and replay without the producer ever being involved. This decoupling is what lets large systems evolve without everything being wired directly together.

So Kafka is less like a postbox (deliver and remove) and more like a ledger or a transaction log: an ordered, durable, append-only record that many parties can read and re-read. That’s the mental model to hold for the entire course.

Why this became the backbone

Those properties — durable, ordered, replayable, read by many — turn out to be exactly what modern systems need for events, which is why Kafka ended up at the centre of so many architectures:

  • Event-driven microservices. Services emit events (“order placed,” “payment captured”) to Kafka, and other services react. The emitter doesn’t call the reactors directly, so you can add, remove or change consumers without touching the producer.
  • Log, metric and clickstream pipelines. High-volume streams of events flow through Kafka into analytics, dashboards and data lakes — it buffers spiky load and feeds many sinks.
  • Change Data Capture (CDC). A database’s changes are streamed into Kafka (via Connect/Debezium) so other systems get a live, ordered feed of every insert/update/delete — the backbone of keeping systems in sync.
  • Stream processing. Because the data is an ordered, replayable log, you can compute continuously over it — running aggregates, joins, windows — with Kafka Streams or Flink (Lessons 9–10).

The common thread: whenever you have a high-volume stream of events that many systems need, durably and replayably, Kafka is the natural spine. It’s the “central nervous system” for events — everything publishes to it and subscribes from it, instead of point-to-point integrations that multiply into an unmaintainable mess.

What it costs — and what it isn’t

Kafka earns those superpowers with real operational weight: it’s a distributed system you run as a cluster (brokers, controllers, replication, retention), and that is not free to operate. It is also, emphatically, not a general-purpose message queue and not a database — points we’ll keep returning to, and settle honestly in the final lesson. For request/response you want RPC; for a simple job queue a managed queue is lighter; for key lookups you want a database or Redis. Kafka is brilliant at exactly one thing — being a distributed log of events — and the whole course is about using that one thing well.

With the core idea in place, the next lesson makes it concrete: the data model of topics, partitions and offsets that turns “a log” into a system that scales.

Pitfalls to Avoid

  1. Thinking of Kafka as a queue that deletes on read. It’s a retained log — reading doesn’t consume. This single misconception breaks your mental model of consumer groups, replay and retention. Think ledger, not postbox.
  2. Assuming one consumer “takes” a message. Many consumers (and many groups) read the same events independently. If you expect queue-style single delivery, you’ll be confused by how consumer groups work (Lesson 4).
  3. Forgetting data is retained, not permanent by default. Retention is time- or size-bounded (Lesson 7). “Replay from the start” works only within the retention window unless you’ve configured otherwise.
  4. Using Kafka for request/response. It’s a one-way log, not an RPC mechanism. Expecting synchronous replies leads to awkward, fragile designs — use HTTP/gRPC for that.
  5. Reaching for Kafka for a tiny or simple workload. Its operational cost only pays off for genuine high-volume, many-consumer event streams. For a small job queue, something lighter is the right call (Lesson 12).

Quick Recap

  • Kafka is a distributed commit log, not a delete-on-read queue: producers append, events are retained, and each consumer tracks its own offset.
  • That enables the four superpowers: many independent consumers of the same events, replay from any offset, history for new consumers, and producer/consumer decoupling.
  • It became the backbone for event-driven microservices, log/metric/clickstream pipelines, CDC, and stream processing — anywhere you have a high-volume event stream many systems need.
  • It’s operationally heavy and is not a queue or a database — use it for what it’s uniquely good at (Lesson 12).

Next lesson: the data model that makes the log scale — topics, partitions and offsets.

kafkadistributed-logevent-drivenpub-submessaging
Next arrow_forwardThe Log: Topics, Partitions & Offsets