Kafka Streams & Stream Processing

A log of events is only half the story; the value is in processing it — computing running totals, joining streams, detecting patterns, materialising state. Kafka Streams is the library (Lesson 9) that does this, and stream processing has a few core ideas — state, windows, event time, exactly-once — that apply whether you use Streams, ksqlDB or Flink. This lesson is those ideas, grounded in how Streams implements them.

The pipeline and its state

A Streams app reads from input topics, processes, and writes to output topics — a continuous dataflow. The operations split into two kinds, and the difference is everything:

• Stateless operations — filter, map, flatMap. Each record is transformed independently; no memory of past records needed. Easy, cheap, and trivially parallel.
• Stateful operations — count, sum, aggregate, join. These need to remember something across records: a running total per key, the last value seen, the other side of a join. That memory is state, and managing it correctly under failures is the hard part of stream processing.

Streams handles state elegantly: it keeps it in a local state store (embedded RocksDB, on the instance’s disk) for fast access, and — critically — backs that state up to a compacted changelog topic in Kafka (Lesson 7’s compaction, used here). So if an instance crashes, a new one replays the changelog to rebuild the exact state and resumes where it left off. You get fault-tolerant local state for free — fast local reads, durable Kafka-backed recovery. This is why Streams can do heavy stateful work without an external database.

Windows: bounding the infinite

Aggregating a never-ending stream raises an obvious problem: “total revenue” over an infinite stream never finishes. Windows bound aggregates in time: “revenue per 1-minute window,” “unique users per hour.” The stream is sliced into windows, and each window produces a result.

But windows surface the hard reality of streaming: data arrives late and out of order. A mobile app was offline and uploads an event from 10 minutes ago; a retry reorders events. So two rules matter:

• Process on event time, not arrival time. Each event carries a timestamp of when it actually happened; windows should be assigned by that event time, not when the event reached Kafka. Otherwise your “last minute” window is wrong whenever anything is delayed.
• Allow a grace period for late data. Keep a window open a little past its end to admit stragglers, then finalise. Events later than the grace period are dropped or sent to a side channel. (This is the same late-data discipline as the data engineering course.)

Getting event-time and late-data handling right is most of what separates a correct streaming aggregate from a subtly-wrong one.

Exactly-once, for one setting

Here’s where Kafka’s pieces pay off together. Stream processing is a classic read-process-write loop: consume from input, update state, produce to output. Doing that exactly once — so a crash never double-counts or loses — is exactly what Kafka’s transactions (Lesson 5) provide. And Streams wires it up for you: set processing.guarantee=exactly_once_v2, and the consume, the state update (changelog write), and the produce all happen atomically within Kafka transactions. One setting gives you exactly-once stream processing — something that’s genuinely hard to build by hand. (The caveat from Lesson 5 still holds: it’s exactly-once within Kafka; external sinks still need idempotent writes.)

KStream vs KTable: events vs state

The last core idea is the abstraction Streams gives you, and it maps directly to Lesson 7’s retention-vs-compaction distinction:

• KStream — a stream of events, where every record matters. “User clicked,” “order placed.” Backed by a normal (retention) topic. You process each event.
• KTable — the latest value per key — a table of current state. “User’s current cart,” “product’s current price.” Backed by a compacted topic, so it’s the up-to-date value for each key.

The power is in combining them: join a KStream of events to a KTable of state to enrich events (“this click, for this user’s current segment”); aggregate a KStream into a KTable (“stream of orders → table of revenue-per-customer”). An aggregation turns events into state; reading a compacted topic gives you state directly. Most real stream processing is exactly this — joining and aggregating streams and tables into derived topics that downstream systems consume, each derived topic being a continuously-updated materialised view over your event data.

Put together: Streams lets you build continuously-running, fault-tolerant, exactly-once processing over Kafka, with fast local state and clean event/state abstractions — and the same concepts (state, windows, event time, exactly-once) carry over to ksqlDB and Flink. Next we turn from building to running: operating Kafka in production.

Pitfalls to Avoid

1. Processing on arrival time instead of event time. Late/out-of-order events make arrival-time windows wrong. Assign windows by the event’s own timestamp and allow a grace period.
2. Ignoring late data. Without a grace period, late events are silently dropped or land in the wrong window, corrupting aggregates. Decide explicitly how late is too late.
3. Putting state in an external DB unnecessarily. Streams’ local RocksDB + compacted changelog gives fast, fault-tolerant state without an external store. Reaching for an external DB adds latency and a failure mode — use the built-in state store.
4. Expecting exactly-once to cover external sinks. exactly_once_v2 covers Kafka-to-Kafka; a write to an external system still needs idempotency (Lesson 5). Don’t assume the setting makes the whole pipeline perfect.
5. Confusing KStream and KTable. Treating a changelog (state) as an event stream, or vice versa, gives wrong results — double-counting state updates, or missing events. Model events as KStream, current-state as KTable.
6. Unbounded aggregations with no window. Aggregating an infinite stream without a window grows state forever. Window time-based aggregates; bound or compact key-based state.

Quick Recap

• Stateless ops (filter/map) are easy; stateful ops (count/sum/join) need state, which Streams keeps in a local RocksDB store backed by a compacted changelog topic — so a crash rebuilds state and resumes (fault tolerance for free).
• Windows bound infinite aggregates; handle reality with event time (not arrival time) and a grace period for late/out-of-order data.
• Exactly-once is one setting (processing.guarantee=exactly_once_v2) because it rides Kafka transactions — Kafka-to-Kafka only; external sinks still need idempotent writes.
• KStream = a stream of events; KTable = the latest value per key (state, from a compacted topic). Most processing joins/aggregates streams and tables into derived topics.

Next lesson: running it for real — operating Kafka in production (consumer lag, monitoring, sizing, tuning).