Open Spotify and every interaction — pressing play, skipping a track, liking a song, finishing an album — produces an event. Multiply that by hundreds of millions of users and you get hundreds of billions of events per day. Those events are the raw material for recommendations, royalty payments to artists, the famous year-end Wrapped, A/B tests, and every operational dashboard. Getting them reliably from a phone to the systems that consume them is one of Spotify’s foundational problems, and their event delivery pipeline is a beautiful real-world example of the stream processing and change data capture lessons.
The problem: collect everything, lose nothing, serve two masters
Event delivery has a deceptively hard set of requirements:
- Massive, uneven volume — hundreds of billions of events a day, spiky with usage.
- Reliability — events drive royalties and recommendations, so losing them has real cost; delivery must be at-least-once and durable.
- Two very different consumers — some systems want all the data, cheaply, for batch analysis (train a recommender overnight); others want fresh data, fast, for real-time aggregation (how many plays is this new release getting right now?).
Serving both the “complete and cheap” and the “fresh and fast” consumers from one pipeline is exactly the tension the lambda/kappa discussion in the stream-processing lesson is about.
The architecture
Spotify’s pipeline (which evolved from a Kafka-based system to one built on Google Cloud Pub/Sub and Dataflow) follows a shape you will recognise:
- Clients emit events to an event-delivery gateway. The client batches and buffers locally so a flaky network does not lose events, then ships them.
- The gateway writes events to a durable log / message bus (Pub/Sub, conceptually a Kafka-like durable, partitioned log). This is the linchpin: the moment an event lands in the log it is durable and replayable, and the log absorbs volume spikes that would overwhelm any synchronous consumer. Events are partitioned by type so each stream scales independently.
- Fan-out to two paths:
- Batch path → the events are written to a data lake on Google Cloud Storage, partitioned by event type and time, where cheap, massive batch jobs (and the warehouse, BigQuery) process them for recommendations, royalties, and Wrapped.
- Stream path → stream-processing jobs (Dataflow / Flink-style) consume the same log for real-time aggregations — live play counts, trending, anomaly detection.
Because both paths read from the same durable log, they stay consistent, and the log can be replayed to reprocess history when a job changes or a bug is fixed — the kappa idea of “reprocess by replaying the log.”
The patterns underneath
| Course idea | Spotify’s implementation |
|---|---|
| Durable, partitioned log | Pub/Sub (Kafka heritage) buffers every event durably and absorbs spikes |
| Batch + stream (lambda/kappa) | Data lake for cheap batch; stream jobs for real-time aggregates |
| Replayability | The log can be re-read to reprocess history after a change |
| At-least-once + dedup | Events are delivered at-least-once; consumers dedupe by event id |
| Partition by key | Events partitioned by type/time for independent scaling and cheap range scans |
This is the same backbone as the Ad-Click Aggregator design — a firehose of events into a durable log, split into a fast approximate stream path and a slower exact batch path — just generalised to Spotify’s entire analytics foundation.
Taking it to the interview
Whenever a question involves “collect a huge stream of user events and do analytics on them” — ad clicks, metrics, activity feeds, recommendation signals — reach for this template:
- A durable log in the middle (Kafka / Pub/Sub) for durability, spike absorption, and replay.
- A batch path (data lake + warehouse) for cheap, complete, historical processing.
- A stream path (Flink / Dataflow) for fresh, real-time aggregates.
- At-least-once delivery with idempotent/deduped consumers, and partitioning by key so it scales.
The senior insight is that you do not choose batch or stream — you put a durable log in the middle and serve both from it, which also gives you free reprocessing by replay.
Key Takeaways
- Model user activity as a stream of events and put a durable, partitioned log at the center — it gives you durability, spike absorption, and replay in one component.
- Serve batch and stream consumers from the same log. Cheap/complete batch (data lake + warehouse) and fresh/fast stream (Flink/Dataflow) are two readers of one source of truth.
- Replayability is a superpower. Reprocessing history after a bug fix or a model change is just re-reading the log — the kappa architecture advantage.
- Buffer on the client too. Batching and local buffering on the device keeps events from being lost on a flaky network before they ever reach the gateway.
- At-least-once + dedup, partition by key. The same reliability and scaling contract as every other high-throughput pipeline — the Ad-Click Aggregator design is this pattern in miniature.