Patterns & When NOT to Use Kafka

We’ve built Kafka up from the log to production. The last and most valuable thing is judgment: knowing when Kafka is the right tool — and, just as important, when it isn’t. Kafka is powerful and fashionable, which means it gets used for jobs it’s bad at, bringing operational weight to problems that didn’t need it. This lesson is the honest picture: where Kafka genuinely shines, where it’s the wrong tool, and what to reach for instead.

Where Kafka shines — and why

Every great-fit pattern leans on Kafka’s actual superpowers (durable, ordered-per-partition, replayable, scalable, read by many):

• Event-driven microservices — the backbone use case. Services emit events; others react. Producers and consumers are decoupled (Lesson 1), so the architecture evolves without point-to-point wiring.
• Log, metric and clickstream pipelines — high-volume event streams feeding analytics, dashboards and lakes, with Kafka buffering and fanning out.
• Change Data Capture (CDC) — stream a database’s changes into Kafka (Connect/Debezium) so many systems get a live, ordered feed (Lesson 9).
• Stream processing & real-time analytics — compute continuously over the log (Streams/Flink, Lesson 10).
• Event sourcing & CQRS — store state as a log of events: the Kafka log becomes the authoritative source of truth, and you derive read models from it. Kafka’s durable, ordered, replayable log is a natural fit for the event store, and compacted topics (Lesson 7) materialise current state.
• Buffering spiky load — a burst of events lands in Kafka and downstream consumers drain it at their own pace, so a traffic spike doesn’t overwhelm a slower system.
• Fan-out — one event, many independent consumers (Lesson 4’s groups), each doing its own thing with the same stream.

The common thread: a high-volume stream of events that many systems need, durably and replayably. When that’s your problem, Kafka’s operational cost is well spent.

Where Kafka is the wrong tool

Equally important — and more often ignored — is where Kafka is a poor fit. Using it here brings all its operational weight for none of its benefit:

• Request/response → use HTTP/gRPC. Kafka is a one-way log, not an RPC mechanism. If service A needs a reply from service B, that’s a synchronous call. Bolting request/response onto Kafka (correlation ids, reply topics) is awkward and fragile — use the right tool.
• A simple job/task queue (per-message ack, redelivery, priorities, delayed delivery) → use a message queue like SQS, RabbitMQ, or a DB-backed queue. Kafka has no per-message acknowledgement or deletion, no priorities, no native delayed delivery — it’s a log, not a queue. For “process these tasks, retry failures, with priorities,” a real queue is simpler and fitter.
• A tiny app with a handful of events → a database table or the outbox pattern. Kafka’s operational cost (a cluster to run) isn’t worth it for low volume. A table you poll, or an outbox in your existing DB, is far less to operate.
• Low-latency key lookups / a datastore → Redis or a database. Kafka is a log, not a lookup store. You can’t efficiently “get the current value for key X” from a Kafka topic (you’d scan). Pair Kafka with a DB or Redis that serves lookups; don’t make it your database.
• Strict total ordering across everything → you can’t, by design. Kafka orders per partition only (Lesson 2). If you truly need one global order over all events, a single partition (no parallelism) is the only way — and if you need that, Kafka probably isn’t the right architecture.

For each of these, the symptom is the same: you find yourself fighting Kafka’s model (adding reply topics, simulating acks, scanning for lookups, forcing one partition). That’s the signal you’ve picked the wrong tool.

The honest test

So how do you decide? One question cuts through it:

Do you have a high-volume stream of events that many systems need, durably and replayably?

• Yes → Kafka earns its operational cost. The decoupling, replay, fan-out and scale are exactly what you’re buying, and nothing else does them as well.
• No → something lighter will serve you better. A queue for tasks, a database for state and lookups, HTTP/gRPC for request/response, a table or outbox for low volume.

And remember Kafka is not free to operate — it’s a distributed system (brokers, controllers, replication, retention, rebalancing) that needs real care, or a managed service that costs money. That operational weight is justified when the problem is genuinely a high-volume event backbone, and wasted when it isn’t. Many teams adopt Kafka because it’s the fashionable answer, then carry its complexity for a workload a simpler tool would have handled — don’t be one of them.

That’s the course. You now have the full picture of Apache Kafka: the distributed log it’s built on, the data model of topics/partitions/offsets, producers and consumers and the parallelism the partition count sets, delivery semantics and exactly-once, replication and the durability trio, storage and retention and compaction, the KRaft architecture, the ecosystem and stream processing, production operation, and the judgment to know when to use it — every major claim grounded in a reproducible number. Use Kafka for the event streams it’s brilliant at, operate it by watching lag and replication, and reach for something lighter when the problem doesn’t need a distributed log.

Pitfalls to Avoid

1. Using Kafka for request/response. It’s a one-way log; simulating RPC with reply topics and correlation ids is fragile. Use HTTP/gRPC for synchronous calls.
2. Using Kafka as a job queue. No per-message ack, priorities or delayed delivery. For task processing with retries and priorities, a real queue (SQS/RabbitMQ) is simpler and fitter.
3. Treating Kafka as a database. It’s a log, not a lookup store — you can’t efficiently get “the current value for key X.” Pair it with a DB/Redis for lookups; don’t store state you need to query in Kafka alone.
4. Expecting global ordering. Kafka orders per partition only. Needing strict total order forces a single partition (no parallelism) — usually a sign Kafka isn’t the right fit.
5. Adopting Kafka for low volume. Its operational cost isn’t worth it for a handful of events. Use a table, outbox, or lightweight queue until you genuinely have a high-volume event backbone.
6. Underestimating the operational cost. Kafka is a distributed system to run (or pay for). Adopt it when the event-backbone benefit justifies that weight — not because it’s fashionable.

Quick Recap

• Great fit (leans on durable/ordered/replayable/scalable/many-readers): event-driven microservices, log/metric/clickstream pipelines, CDC, stream processing, event sourcing & CQRS, buffering spiky load, fan-out.
• Wrong tool → use instead: request/response → HTTP/gRPC; simple job queue → SQS/RabbitMQ; tiny/low-volume → DB table/outbox; key lookups/datastore → Redis/DB; strict global order → not Kafka (per-partition only).
• The honest test: a high-volume stream of events many systems need, durably and replayably → Kafka earns its cost; otherwise go lighter.
• Kafka is not free to operate — justify its weight with the event-backbone benefit, don’t adopt it for fashion.

That’s the course — go build on the log, and use it where it’s brilliant.