Apache Kafka — The Distributed Log, End to End
Apache Kafka runs the event backbone of a huge fraction of modern systems…
Tag
#kafka
24 posts
Patterns & When NOT to Use Kafka
We’ve built Kafka up from the log to production. The last and most valuable…
Operating Kafka in Production
A Kafka cluster that’s been designed well (good keys, right durability, sensible…
Kafka Streams & Stream Processing
A log of events is only half the story; the value is in processing it…
The Ecosystem
Everything so far has been the Kafka broker — the distributed log that stores…
KRaft & Cluster Architecture
So far we’ve treated “Kafka” as a thing that stores partitions and replicates…
Storage, Retention & Compaction
Kafka’s superpower — retaining and replaying events — has a physical cost: all…
Replication, ISR & Durability
Lesson 5 said at-least-once “never loses data” — but why doesn’t it? Because…
Delivery Semantics
“Does Kafka deliver each message exactly once?” is the question everyone asks…
Consumers & Consumer Groups
Producers append; consumers read. But the clever part of Kafka consumption isn’t…
What Is Kafka & Why
Before any configuration or API, the single most important thing to understand…
The Log: Topics, Partitions & Offsets
Lesson 1 said Kafka is a log. This lesson is the data model that turns “a log…
Producers
Events get into Kafka through a producer — a client that appends messages to a…
Data Engineering: Building Data Pipelines — From Raw Events to a Trustworthy Warehouse
Every company that runs on data has an invisible layer of engineering underneath…
Streaming with Kafka
When a use case genuinely needs low latency (Lesson 2), you reach for streaming…
Messaging and Delivery Guarantees
Every company I’ve worked with has had at least one “duplicate” incident from a…
In the Wild: Spotify's Event Delivery
Open Spotify and every interaction — pressing play, skipping a track, liking a…
In the Wild: Slack's Job Queue
Most of what a Slack message triggers does not happen in the request that sent…
Advanced: Change Data Capture (CDC)
Your data lives in a database, but that is rarely the only place it needs to be…
Key Technologies: Pick the Right Tool
“I’ll use Kafka” is not an answer. “This needs a durable, replayable log to…
Design an Ad Click Aggregator
“Design the system behind an advertising dashboard” is a favourite analytics…
Stream Processing
Most of the designs so far store data and answer queries against it. Stream…
Design a Distributed Message Queue (Kafka)
A message queue decouples producers from consumers: services publish events…
Message Queues & Event-Driven Architecture
Every system starts synchronous. User clicks a button, server does the work…