Most of what a Slack message triggers does not happen in the request that sent it. Delivering push notifications, unfurling a link, firing webhooks, updating search indexes, running reminders — all of it is asynchronous work pushed onto a job queue and processed later. At Slack’s scale that is billions of jobs per day, with bursty, spiky load. This case study follows how Slack’s job queue evolved, and why it is a clean illustration of the message queue and managing long-running tasks lessons.
The original design, and how it broke
Slack’s first job queue was built on Redis. Web app servers enqueued jobs into Redis; a fleet of workers dequeued and executed them. Simple, fast, and fine at first.
Under growth and spikes it developed serious failure modes:
- Redis is in-memory, so the queue is bounded by RAM. When job production outran consumption — a traffic spike, or workers falling behind — the queue grew until Redis ran out of memory. A full Redis is a catastrophic failure: new jobs can’t be enqueued and the incident cascades.
- Head-of-line blocking. A slow or failing job type could clog the queue and starve everything behind it.
- Weak durability and duplication under failure. The enqueue/dequeue dance did not cleanly guarantee that a job was executed exactly the intended number of times when workers crashed mid-job; jobs could be lost or double-run.
These are the exact pitfalls the course warns about: an unbounded in-memory buffer with no back-pressure, and no durable record of accepted work.
The redesign: Kafka in front of Redis
Slack’s rebuilt system keeps Redis for what it is good at (fast, per-type work queues that workers poll) but puts Kafka in front of it as a durable, append-only buffer. The flow becomes:
- Enqueue to Kafka. Web app servers write every job to Kafka. Because Kafka is a durable, disk-backed log, the job is safely persisted the moment it is accepted. Kafka’s sequential writes and huge retention let it absorb enormous spikes that would have blown up an in-memory queue — the log soaks up the burst and the rest of the system drains it at its own pace.
- Dispatchers move jobs into Redis. A pool of dispatcher processes consume from Kafka and place jobs into the appropriate per-type Redis queues. Because each job type has its own queue, a slow type no longer blocks the others — head-of-line blocking is gone.
- Workers execute, at-least-once. Workers pull from the Redis queues and run the jobs, with retries and backoff on failure. Delivery is at-least-once, so jobs are made idempotent — running one twice has the same effect as running it once.
The key architectural insight — and the one to carry into an interview — is decoupling durability from dispatch. Kafka provides durability and spike absorption; Redis provides fast, flexible, per-type scheduling to workers. Each layer does the one thing it is best at.
The patterns underneath
| Course idea | Slack’s implementation |
|---|---|
| Buffer writes with an append-only log | Kafka in front absorbs spikes; the queue is no longer RAM-bounded |
| Durability of accepted work | A job is persisted in Kafka the moment it is accepted |
| Avoid head-of-line blocking | Per-type Redis queues so a slow job type doesn’t starve others |
| At-least-once + idempotency | Workers retry; jobs are idempotent so duplicates are safe |
| Back-pressure | The durable log absorbs the burst instead of overflowing |
This is the core lesson of Scaling Writes and Managing Long-running Tasks: when writes are bursty, put a durable log between producers and workers so a spike or a slow consumer can never drop accepted work. The failure of the original design — an unbounded in-memory queue — is one of the most common real-world outages, and the fix is almost always “add a log in front.”
Taking it to the interview
If asked to design a job/task system, structure your answer around exactly this evolution:
- Start with “producers enqueue, workers consume.”
- Then ask the hard question: what happens when production outruns consumption, or a worker dies mid-job? — and answer with a durable buffer (Kafka) for spike absorption and durability, at-least-once delivery with idempotent workers, retries + dead-letter queues, and per-type isolation so one slow job type can’t block the rest.
Naming the in-memory-queue failure mode and reaching for a durable log is a strong senior signal — it shows you have seen the pattern fail, not just drawn it.
Key Takeaways
- An in-memory queue is bounded by RAM — under a spike or a slow consumer it fills and drops work. This is a classic real outage, not a theoretical risk.
- Put a durable, append-only log (Kafka) in front to absorb spikes and guarantee that accepted work is never lost. Decouple durability (the log) from dispatch (fast per-type queues).
- Per-type queues prevent head-of-line blocking — a slow job type should not starve every other job.
- At-least-once + idempotency is the practical delivery contract. Assume a job can run twice and make that safe, rather than chasing true exactly-once.
- Retries, backoff, and dead-letter queues turn transient failures into eventual success and quarantine the permanently broken jobs.