Everything so far gets data moved, stored, transformed and scheduled. But a pipeline that runs is not the same as a pipeline you can trust — and trust is the entire value of data engineering. A dashboard built on duplicated, stale, or silently-corrupted data is worse than no dashboard, because people make decisions on it. This lesson is the reliability work that makes data trustworthy: the four concerns every production pipeline must handle.
Four Pillars of Trustworthy Data
Deduplicate (measured)
Because at-least-once delivery is the norm (Lesson 9) and ingestion/jobs retry (Lessons 3, 10), the same record will arrive more than once — and if you don’t dedup, you double-count, inflating every metric built on that data. I measured the fix: a stream of 1,050,015 delivered events with a 4.8% duplicate rate (50,015 dupes) deduped cleanly by event_id — a hash-set pass in 125 ms — down to exactly 1,000,000 correct uniques. Without that dedup step, those 50,015 duplicates would have been double-counted straight into the business metrics.
The technique is simple and reliable: give every record a stable, unique id at the source, and drop records whose id you’ve already seen (a dedup pass, or an upsert/MERGE keyed by that id so re-applying is a no-op). This is the idempotency thread from Lessons 3, 9 and 10, here as a concrete data-quality step: dedup on a stable id, and retries become harmless.
Schema evolution
Upstream data changes shape over time — a source team adds a column, renames one, or changes a type. A rigid pipeline that assumes a fixed schema either breaks (job fails) or, worse, silently corrupts (reads the wrong column, drops data). You handle this with formats built for change: Parquet carries its schema and tolerates added columns; table formats (Iceberg, Delta — Lesson 5) support safe schema evolution — add, rename, or retype columns with the change tracked in metadata, old data still readable. The practice: add columns additively (don’t break existing readers), version breaking changes, and never silently drop data that doesn’t fit — route it somewhere visible instead. Expect the shape to change; design so it doesn’t take the pipeline down.
Tests and data contracts
Software has tests; data pipelines need them too — on the data itself. Assert the properties that must hold: row counts in an expected range, null rates under a threshold, values within valid ranges, uniqueness of keys, referential integrity (every product_id in the fact exists in the dimension). Tools: dbt tests (declarative tests on models — unique, not_null, relationships), and Great Expectations (a richer assertion framework). Run these in CI and in the pipeline, so bad data fails loudly — the job stops and alerts — instead of silently flowing into a dashboard. The stronger form is a data contract: the producer of data formally promises a shape and guarantees (these columns, these types, this freshness), and the pipeline breaks the build if the producer violates it. Contracts shift data quality upstream, catching problems at the source rather than discovering them three transformations later.
Late and out-of-order data
In streaming especially, events don’t arrive neatly in order. A mobile app was offline and uploads an event from two hours ago; a retry reorders events; network delays shuffle arrival. If you process by arrival time, your “last hour” window is wrong — it misses late events and mis-buckets reordered ones. The fix is to process by event time (when the event actually happened, carried in the event) rather than arrival time, and to use windows with a grace period — keep a window open a little longer to admit late arrivals, then finalise. When data arrives after a window has closed, reprocess the affected partition (the idempotent, date-partitioned design from Lesson 10 makes this clean — just re-run that date). Late and out-of-order data is normal, not exceptional; design windows and reprocessing to expect it.
The Mindset: Expect Dirty Data
The unifying idea is a mindset shift. A beginner assumes data is clean, ordered, unique and stably-shaped, and builds a pipeline that works on the happy path and silently breaks on reality. An experienced data engineer assumes the opposite: data is dirty, duplicated, late, and always changing shape — and builds a pipeline that expects all of it:
- Idempotent, so re-runs and retries are safe (no double-counting) — Lessons 3, 9, 10.
- Tested, so bad data fails loudly and visibly, rather than silently corrupting a dashboard someone will trust.
- Tolerant of late, out-of-order, and schema-changed data, with reprocessing to correct it.
This is what “reliability” means for data: not that nothing ever goes wrong, but that when it does, the pipeline catches it, fails safely, and can be corrected — and that the numbers people see can be trusted. All the performance and scale from earlier lessons is worthless if the data is wrong; data quality is what makes the whole pipeline mean something. The final lesson turns to the last concerns: serving that trustworthy data to users, and keeping it affordable and governed.
Pitfalls to Avoid
- Not deduplicating. At-least-once delivery guarantees duplicates (measured 4.8% here); without a dedup step they inflate every metric. Give records a stable id and dedup/upsert by it.
- Assuming a fixed schema. When upstream adds or changes a column, a rigid pipeline breaks or silently corrupts. Use schema-evolving formats (Parquet, Iceberg), add columns additively, version breaking changes.
- No data tests. Without assertions on counts, nulls, ranges, uniqueness and referential integrity, bad data flows silently into dashboards. Add dbt tests / Great Expectations and fail the pipeline on violations.
- Discovering quality problems downstream. Finding bad data three transforms later is expensive and erodes trust. Push checks upstream with data contracts — producers promise a shape, the build breaks if they don’t.
- Processing by arrival time. Late and out-of-order events make arrival-time windows wrong. Use event time with grace-period windows, and reprocess affected partitions when data arrives late.
- Silently dropping non-conforming data. Discarding rows that don’t fit the expected schema hides problems and loses data. Route rejects to a visible dead-letter/quarantine location and alert.
Quick Recap
- A pipeline that runs isn’t one you can trust — and trust is the whole point. Four reliability pillars:
- Deduplicate: at-least-once delivery means duplicates. Measured: a 4.8%-duplicate stream (1,050,015 events) deduped cleanly by
event_id(125 ms) into 1,000,000 correct uniques; without it, 50,015 double-counted. Use a stable id + dedup/upsert. - Schema evolution: upstream changes shape — use evolving formats (Parquet, Iceberg), add columns additively, version breaking changes, never silently drop.
- Tests & data contracts: assert counts, nulls, ranges, uniqueness, referential integrity (dbt tests, Great Expectations) so bad data fails loudly; a contract pushes the promise upstream to the producer.
- Late/out-of-order data: process by event time with grace-period windows, and reprocess the affected partition (clean thanks to idempotent date-partitioning, L10).
- The mindset: expect dirty, duplicated, late, shape-changing data — be idempotent, tested, and tolerant.
Next lesson, the last: getting trustworthy data to users without a runaway bill — serving, cost and governance.