Batch vs Streaming

The first design decision in any pipeline is how data flows through it: in scheduled chunks, or continuously as it arrives. This batch-versus-streaming choice shapes the tools you use, the complexity you take on, and the latency your consumers see. It is also where teams most often over-engineer — reaching for streaming because it sounds modern when batch would do the job. This lesson is the honest comparison.

Two Processing Models

Batch processes a big, bounded chunk of data on a schedule — every hour, every night. The input has a known start and end (“all of yesterday’s orders”), which makes batch simple to reason about and gives it high throughput (you process millions of rows in one efficient pass). Its latency is minutes to hours — a nightly job means data is up to a day old. Its superpower is that reprocessing is trivial: if a job was wrong, you just re-run it on the same bounded input. Batch is Spark jobs, dbt models, scheduled warehouse SQL — and the large majority of analytics is batch.

Streaming processes each event as it arrives, from an unbounded, never-ending input. Its superpower is low latency — data is processed within seconds of being produced. But that continuous, infinite nature makes it genuinely harder: you must manage state (running aggregates that never “finish”), windows (how do you sum “the last 5 minutes” of an infinite stream?), ordering (events can arrive out of order), and late data (an event from 10 minutes ago showing up now). And exactly-once processing — no lost events, no double-counting — is famously difficult (Lesson 9). Streaming is Kafka feeding Flink or Spark Structured Streaming.

So the two models trade latency against complexity: batch is simpler and higher-throughput but slower; streaming is fresher but carries real engineering burden.

Choose by the Latency You Actually Need

The decision should be driven by one question: what latency does the business genuinely require? Not “what sounds impressive,” not “what’s modern” — what the use case actually needs.

• Nightly revenue report, weekly cohort analysis, monthly finance close → batch. Nobody needs these to the second; a scheduled job is simpler, cheaper and completely sufficient.
• Fraud detection, live operational dashboards, real-time personalisation, alerting → streaming. Here seconds genuinely matter — a fraud decision an hour late is useless — so the complexity of streaming earns its keep.

The common failure is reaching for streaming by default because it feels more advanced, and taking on all of streaming’s complexity (state, windowing, ordering, late data, exactly-once, a Kafka+Flink stack to operate) for a use case where a nightly batch job would have been perfectly fine. A huge amount of real, valuable production data engineering is “just” well-run batch — scheduled, idempotent, tested jobs that move and transform data reliably. There is nothing second-rate about that; it is the right tool for most analytical work. Add streaming when, and only when, the latency requirement demands it.

The Hybrid Reality

In practice, many organisations run both, because different consumers of the same data want different trade-offs. A classic pattern (historically called the Lambda architecture) runs:

• a streaming path that gives fresh-but-approximate results within seconds (good enough for a live dashboard), and
• a batch path that reprocesses the same data more thoroughly and produces the complete, correct, authoritative numbers a few hours later (the source of truth for finance).

The streaming layer answers “roughly what’s happening now,” and the batch layer answers ”exactly what happened,” and the two reconcile. Maintaining two code paths for the same logic is a real cost, which is why the newer Kappa approach argues for doing everything as a stream (a single path, replayed when you need to reprocess), and why modern streaming frameworks keep getting better at correctness so you can trust the streaming numbers alone. You don’t need to pick a camp today — just know that “batch vs streaming” is often “batch and streaming,” matched to what each consumer needs.

The practical takeaway for the rest of the course: we treat batch as the default (most lessons apply to both, but the examples lean batch because that’s most of the work), and give streaming its own lesson (Lesson 9, Kafka) because its concepts — partitions, offsets, delivery semantics, windowing — are distinct and important. Start by asking “how fresh does this data really need to be?” and let the answer, not fashion, pick your model.

Pitfalls to Avoid

1. Defaulting to streaming because it sounds modern. You take on state, windowing, ordering, late-data and exactly-once complexity for a use case a nightly batch job would serve. Choose by required latency; most analytics is fine as batch.
2. Using batch where seconds matter. The opposite error: running fraud detection or live alerting as an hourly batch job, so the answer is useless by the time it lands. Match the model to the latency need.
3. Underestimating streaming’s correctness burden. “Just process events as they come” hides hard problems — out-of-order events, late arrivals, exactly-once. If you choose streaming, budget for them (Lessons 9, 11), don’t discover them in production.
4. Forgetting batch’s reprocessing superpower. With bounded input, re-running a fixed job is trivial; streaming reprocessing (replaying from offsets, rebuilding state) is harder. Factor “how will I fix and backfill this?” into the choice.
5. Running two full pipelines when one would do. Lambda’s dual batch+streaming paths double the code and the bugs. Only take that on when you genuinely need both fresh-approximate and complete-authoritative views; otherwise pick one.

Quick Recap

• Batch: process a bounded chunk on a schedule — high throughput, simple, trivially reprocessed (re-run), latency minutes–hours. Spark, dbt, warehouse SQL. Most analytics is batch.
• Streaming: process each event as it arrives from an unbounded input — low latency (seconds) but genuinely harder (state, windows, ordering, late data, exactly-once). Kafka + Flink/Spark.
• The trade-off is latency vs complexity. Choose by the latency the business actually needs, not by fashion — nightly report → batch; fraud/live dashboard → streaming. Don’t take on streaming’s complexity unless the requirement demands it; well-run batch is most of the job.
• Many shops run both (Lambda: streaming for fresh-approximate, batch for complete-authoritative); Kappa argues for stream-only. “Batch vs streaming” is often “batch and streaming.”

Next lesson: getting data in — ingestion patterns and change data capture, including how to do it without hammering the source or missing deletes.