Orchestration & Scheduling — Data Engineering: Building Data Pipelines — From Raw Events to a Trustworthy Warehouse

A pipeline is never one script — it’s many tasks (ingest this, clean that, join, aggregate, model, publish) with dependencies between them, that must run in the right order, on a schedule, with retries when things fail and alerts when they stay broken. Coordinating all that is orchestration, and it’s what turns a pile of transformation logic (Lesson 7) and processing jobs (Lesson 8) into a reliable, repeatable pipeline. This lesson is orchestration, plus the two ideas — idempotency and incrementality — that make it robust and cheap.

DAGs, Schedules, and What an Orchestrator Gives You

You express a pipeline as a DAG — a directed acyclic graph of tasks, where an edge from A to B means “B depends on A, so run A first.” (Acyclic because dependencies can’t loop.) The orchestrator reads the DAG and gives you:

Correct ordering. It runs tasks in dependency order — extract before clean, clean before model, model before serve — and runs independent branches in parallel. You declare the dependencies; it works out the schedule.
Scheduling. It triggers the pipeline on a cadence (nightly at 2 AM, hourly) and tracks the state of every run — succeeded, failed, running, retrying.
Retries and alerting. When a task fails (a source is down, a query errors), it retries per your policy and, if it stays broken, alerts a human. This is the reliability layer — a pipeline that silently fails is worse than no pipeline.
Observability and backfills. It shows the history of runs, lets you re-run failed tasks, and lets you backfill — run past dates (to fill history, or re-run after fixing a bug).

The standard tools are Apache Airflow (the long-time default, Python-defined DAGs), Dagster (asset-oriented, strong on data-awareness and testing), and Prefect (lightweight, Pythonic). They differ in philosophy but all provide the same core: dependency-ordered, scheduled, retried, observable task execution.

Idempotency Makes Backfills Safe

Here’s the idea that makes orchestration robust rather than fragile: every task must be idempotent — running it again produces the same result as running it once, with no duplicates or corruption. This matters because in orchestration, tasks get re-run constantly: a retry after a transient failure, a manual re-run after a fix, a backfill across 90 past days. If re-running a task appends its output, then every retry and backfill double-counts, and your data silently corrupts.

The standard pattern that achieves idempotency is partition by date and overwrite the partition:

Parameterise each run by the date it’s processing (2026-09-15).
One task run = one partition. The task for 2026-09-15 reads that day’s input and writes only the dt=2026-09-15 partition.
It overwrites (replaces) that partition rather than appending. So re-running the task for a date replaces that date’s output — run it once or ten times, the result is identical.

With this, backfilling becomes safe and trivial: to rebuild 90 days of history (after a logic fix, say), you just run the DAG for each of those 90 dates, and each run cleanly replaces its own partition. No duplicates, no manual cleanup. This is the same idempotency principle from ingestion (Lesson 3) and streaming (Lesson 9) — re-running must be safe — applied to scheduled batch tasks, and it’s what separates a pipeline you can confidently re-run from one you’re afraid to touch.

Incremental Beats Full (measured: ~200×)

The second idea is about cost and speed: process only what changed, not everything, every time. It’s tempting to write a nightly job that reprocesses the entire table from scratch — it’s simple and always correct. But it’s enormously wasteful once the table is big, because you re-read all of history to account for one day of new data.

The arithmetic is stark. Consider a 1,000,000,000-row table that grows by 5,000,000 rows per day:

Full reprocessing every night reads the whole table: 1,000,000,000 rows/day.
Incremental processing reads only the new partition: 5,000,000 rows/day.
That’s 200× less work — and it only grows as the table does (a two-year-old table might be 10× bigger, making full reprocessing 2000× more wasteful).

Incremental processing means each run handles only the new or changed data — typically the new date partition — using a watermark (Lesson 3) to know what’s new, and the idempotent date-partition pattern above to write it. The orchestrator runs one task per new partition each day; history, already processed and unchanged, is left alone. This is how pipelines over huge tables stay fast and cheap: you never re-read what you’ve already processed. (It’s also a direct cost lever — Lesson 12 — since warehouses bill by data scanned, and 200× less scanning is 200× less cost for that work.)

Combine the two ideas and you have the shape of a well-built batch pipeline: idempotent, incremental, date-partitioned tasks, wired into a DAG that an orchestrator runs on schedule, retries on failure, and can safely backfill. That’s the reliability-and-efficiency backbone under everything — and it’s mostly “boring,” well-run batch (Lesson 2), which is exactly what most production data engineering should be.

Pitfalls to Avoid

Non-idempotent tasks that append. A task that appends its output double-counts on every retry and backfill — and retries/backfills are routine. Make each run overwrite its own (date) partition so re-running is safe.
Full reprocessing of a big table every run. Re-reading a billion rows nightly to add one day’s data is ~200× wasteful here and worsens as the table grows. Process incrementally — only the new/changed partition.
No retries or alerting. A pipeline that fails silently leaves stale data that consumers trust blindly. Configure retries for transient failures and alerts for persistent ones — the reliability layer is the point of an orchestrator.
Hidden dependencies not expressed in the DAG. If task B secretly needs A’s output but the DAG doesn’t say so, they can run out of order and B reads stale/missing data. Declare every real dependency.
DAGs that can’t be backfilled. Hardcoding “today” instead of a run-date parameter means you can’t cleanly re-run history. Parameterise by execution date so backfills just replay dates.
One giant monolithic task. A single task doing everything can’t be retried granularly, parallelised, or partially backfilled. Break the pipeline into dependency-linked tasks (ideally one partition each).

Quick Recap

A pipeline is a DAG of dependent tasks; an orchestrator (Airflow, Dagster, Prefect) runs them in dependency order, schedules runs, retries failures, alerts on breakage, and enables backfills — the reliability layer over your logic.
Idempotency makes it robust: design each run to be safe to repeat — partition by date, one task = one partition, overwrite it — so retries and backfills replace (never duplicate) output. Same principle as ingestion (L3) and streaming (L9).
Incremental beats full: process only what changed. Measured (arithmetic): a 1B-row table growing 5M rows/day costs 1B rows/day to fully reprocess vs 5M/day incrementally — ~200× less work (and it’s a cost lever, L12).
The backbone of good batch: idempotent, incremental, date-partitioned tasks in a scheduled, retried, backfillable DAG — mostly well-run “boring” batch, which is the right default.

Next lesson: making the data itself trustworthy — data quality and reliability: deduplication (measured), schema evolution, data contracts, and late data.