Data Engineering: Building Data Pipelines — From Raw Events to a Trustworthy Warehouse
A practical, measurement-grounded course on building data pipelines: what data engineering is (and OLTP vs OLAP), batch vs streaming, ingestion and change data capture, file formats and compression (row vs columnar — measured), lakes/warehouses/the lakehouse, partitioning and file layout (pruning and skew — measured), ETL vs ELT and dimensional modelling, distributed processing and the shuffle, streaming with Kafka, orchestration and incremental processing (measured), data quality and reliability (dedup — measured), and serving, cost and governance. Grounded in real measurements: Parquet column scans 6.8x faster than CSV, partition pruning 8x, data skew 5.9x, a 4.8% duplicate stream deduped cleanly, DuckDB aggregating 31M rows/sec.
Every company that runs on data has an invisible layer of engineering underneath the dashboards, the ML models and the analysts’ queries: the pipelines that reliably move data from where it is created to where it answers questions. Data engineering is building and running that layer — and it is its own discipline, with its own hard problems (scale, reliability, cost, dirty and duplicated and late-arriving data) that are different from both application engineering and data science. This course teaches it the way you will actually practise it, and it backs the important claims with numbers I measured rather than adjectives.
We start from first principles — what a data pipeline is (ingest → store → transform → serve) and the OLTP vs OLAP split that explains why the whole discipline exists — then work through the two processing models (batch vs streaming), getting data in (ingestion and change data capture), and the storage foundations that make analytics cheap and fast: file formats and compression (why columnar Parquet beats row formats — measured), the lake, warehouse and lakehouse, and partitioning and file layout (partition pruning and the skew that wrecks distributed jobs — both measured).
From there it is transformation and scale: ETL vs ELT and dimensional modelling (the star schema, dbt), how distributed processing engines work and why the shuffle is the expensive part, streaming with Kafka (topics, partitions, offsets, delivery semantics), orchestration (DAGs, idempotency, and why incremental processing does ~200× less work — measured), data quality and reliability (deduplication, schema evolution, data contracts, late data), and finally serving, cost and governance — getting modelled data to users without a runaway warehouse bill.
It is written in plain Indian English with a working engineer in mind, and it is grounded in real measurements I ran: Parquet answering a single-column aggregate 6.8× faster than CSV (and in half the disk space), partition pruning cutting a scan to one file in thirty (8×), a single hot key making one worker do 5.9× the average work, a 4.8% duplicate event stream deduplicated cleanly by id, and DuckDB aggregating ~31 million rows per second on one machine. Every lesson ends with pitfalls and a quick recap.
What This Course Covers
| # | Lesson | What you will learn |
|---|---|---|
| 1 | What Is Data Engineering | The pipeline (ingest→store→transform→serve); OLTP vs OLAP |
| 2 | Batch vs Streaming | The two processing models; latency vs throughput; choosing |
| 3 | Ingestion & Change Data Capture | Snapshots, incremental pulls, CDC, idempotent loading |
| 4 | File Formats & Compression | Row vs columnar, Parquet, compression (measured) |
| 5 | Lakes, Warehouses & the Lakehouse | Object storage, warehouses, table formats (Iceberg/Delta) |
| 6 | Partitioning & File Layout | Partition pruning, skew (measured), the small-files problem |
| 7 | ETL vs ELT & Modelling | The ELT shift, dbt, star schemas, medallion layering |
| 8 | Distributed Processing | How Spark works, the shuffle, skew, single-node engines |
| 9 | Streaming with Kafka | Topics, partitions, offsets, consumer groups, delivery semantics |
| 10 | Orchestration & Scheduling | DAGs, idempotency, backfills, incremental (measured) |
| 11 | Data Quality & Reliability | Dedup (measured), schema evolution, contracts, late data |
| 12 | Serving, Cost & Governance | Serving layers, cost levers, lineage, governance |
Who Is This For?
- Backend and software engineers moving into data engineering or who own a pipeline
- Analysts and data scientists who want to understand the infrastructure under their data
- Anyone working with Spark, Kafka, dbt, Airflow, Snowflake/BigQuery, or a data lake
- Engineers who want the storage, scale and reliability concepts explained with real measurements
This course sits alongside the other data and backend courses on the site — Database Internals for Backend Engineers (the OLTP side of the OLTP/OLAP split), Distributed Systems in Practice (the failure, time and consensus ideas under distributed processing and Kafka), Observability & Production Debugging (operating pipelines in production), and Go for Backend Engineers (a great language for building ingestion and streaming services).
How to Use It
Read it in order the first time — it follows the pipeline from raw data in to modelled, served data out, and each lesson builds on the last (storage before processing, processing before orchestration, everything before cost). After that, keep it as a reference: when you’re choosing a file format (Lesson 4), laying out a table (Lesson 6), deciding ETL vs ELT (Lesson 7), wiring up Kafka (Lesson 9), making a pipeline idempotent (Lessons 10–11), or facing a warehouse bill (Lesson 12), come back to that lesson. The measured numbers are there so you can reason about data systems with real figures, and the pitfalls sections are where the hard-won lessons live.
Let’s start with the foundation: what data engineering actually is, and the OLTP-vs-OLAP split that the whole discipline is built on.