Apache Kafka — The Distributed Log, End to End
Apache Kafka runs the event backbone of a huge fraction of modern systems…
Tag
data-engineering
14 posts
Data Engineering: Building Data Pipelines — From Raw Events to a Trustworthy Warehouse
Every company that runs on data has an invisible layer of engineering underneath…
Serving, Cost & Governance
The pipeline ends where it began — with the business. Modelled, trustworthy data…
Data Quality & Reliability
Everything so far gets data moved, stored, transformed and scheduled. But a…
Orchestration & Scheduling
A pipeline is never one script — it’s many tasks (ingest this, clean that, join…
Streaming with Kafka
When a use case genuinely needs low latency (Lesson 2), you reach for streaming…
Distributed Processing
Some transforms (Lesson 7) run over data far too big for one machine’s memory…
ETL vs ELT & Modelling
Data has landed (Lesson 3) and is stored well (Lessons 4–6). Now it must become…
Partitioning & File Layout
Columnar format (Lesson 4) decides how much of each row a query reads; file…
Lakes, Warehouses & the Lakehouse
Lesson 4 settled what format analytical data should be in (columnar Parquet…
File Formats & Compression
Where the last lesson got data in, this one is about storing it well — and the…
Ingestion & Change Data Capture
Data engineering starts with getting data in. It sounds mundane — “copy data…
Batch vs Streaming
The first design decision in any pipeline is how data flows through it: in…
What Is Data Engineering
Before any tool or technique, it helps to be clear about what data engineering…