Lakes, Warehouses & the Lakehouse

Lesson 4 settled what format analytical data should be in (columnar Parquet). This lesson is about where it lives — the storage architecture around those files. There are three answers, and understanding how the third (the lakehouse) emerged from the first two is one of the most useful mental models in modern data engineering.

Three Architectures

Data lake. A data lake is cheap, near-infinite object storage (Amazon S3, Google Cloud Storage, Azure Blob) holding files in whatever format — Parquet, JSON, images, logs. Its virtues are cheapness and flexibility: store anything, decide its structure when you read it (schema-on-read), keep raw data forever for pennies. Its vice is the lack of guard rails: a plain pile of files has no transactions, no safe concurrent writes, no easy updates or deletes, and no enforced schema. Without discipline, a lake degrades into a “data swamp” — a sprawl of files nobody trusts, with half-finished writes, duplicate copies, and no idea what’s current. Cheap and flexible, but dangerous left ungoverned.

Data warehouse. A warehouse is a managed, columnar, SQL system — Snowflake, BigQuery, Redshift — that gives you exactly what the lake lacks: transactions, fast SQL over huge data, a managed service that handles the hard parts, and enforced structure. Its costs are that it’s pricier (you pay for the managed convenience and often for compute-by-the-query), your data is somewhat locked into the vendor’s system (harder to point other tools at), and it’s less flexible for raw, semi-structured, or ML-feature data. Reliable and fast, but costlier and more closed.

For years you had to choose, or run both (a lake for cheap raw storage + ML, a warehouse for governed SQL analytics), and copy data between them. The lakehouse ends that choice.

The Lakehouse: Table Formats on Top of Files

Almost everything now sits on cheap object storage holding Parquet files. The question the lakehouse answers is: can we make a pile of Parquet files behave like a warehouse table — with transactions and updates — without giving up the open, cheap storage? The answer is table formats: Apache Iceberg, Delta Lake, and Apache Hudi.

A table format is a metadata layer on top of your Parquet files. The data is still ordinary Parquet on S3/GCS; the table format adds a set of metadata files that track which Parquet files make up the table right now, what the schema is, and a transaction log of changes. That metadata is enough to add exactly the warehouse features the lake was missing:

• ACID transactions — a write either fully commits or doesn’t; readers never see a half-written table. Multiple writers can work concurrently without corrupting each other.
• Updates and deletes — you can UPDATE/DELETE/MERGE rows (needed for CDC, GDPR deletes, corrections), which raw Parquet alone can’t do cleanly.
• Schema evolution — add, rename or retype columns safely over time, with the format tracking the change (Lesson 11).
• Time travel — query the table as of a past point (“as it was last Tuesday”), and roll back a bad write. The metadata log makes this possible.

This is the lakehouse: your data stays in open files on cheap object storage (no vendor lock-in — the files are just Parquet), any engine can read it (Spark, Trino, DuckDB, Flink, and increasingly the warehouses themselves all read Iceberg/Delta), and you still get warehouse-grade correctness (ACID, updates, schema evolution, time travel). You get the cheapness and openness of a lake with the reliability of a warehouse — which is why the industry is rapidly converging on this architecture, and why Iceberg in particular has become a de-facto open standard that even the big warehouses now support.

Choosing, Today

Practical guidance for where to put data:

• Object storage (S3/GCS) + Parquet + a table format (Iceberg/Delta) is the modern default for a new analytical platform. Open, cheap, correct, and engine-agnostic. Start here unless you have a reason not to.
• A managed warehouse (Snowflake/BigQuery) is still an excellent, lower-operational-burden choice — especially for a team that wants SQL and doesn’t want to run infrastructure. And the line is blurring: these warehouses increasingly read and write the open table formats too, so “warehouse vs lakehouse” is less either/or than it was.
• A raw landing zone on the lake (Lesson 3’s “land raw first”) plus table-format tables for the cleaned/modelled data is a common, sensible layout: cheap raw storage for everything, governed tables for what analysts query.
• Avoid the ungoverned swamp. The one architecture to not run is a bare data lake with no table format and no discipline — it’s the cheapest to start and the most expensive to live with. The guard rails (a table format, a catalog, Lesson 12) are what keep a lake from rotting.

The storage foundation — columnar files (Lesson 4), on cheap object storage, organised by a table format — is what makes everything above it (transforming, querying, serving) both possible and affordable. Next we make these files fast to query not just by format but by layout: partitioning.

Pitfalls to Avoid

1. Running a bare lake with no guard rails. A pile of files with no transactions, no schema enforcement and no catalog becomes a data swamp — half-written files, duplicates, nothing trustworthy. Add a table format (Iceberg/Delta) and a catalog (Lesson 12).
2. Assuming a lake supports updates/deletes. Plain Parquet on S3 has no clean UPDATE/DELETE — needed for CDC, corrections and GDPR. That’s what table formats add; don’t try to hand-roll it with file juggling.
3. Vendor lock-in by default. Loading everything into a proprietary warehouse makes it hard to point other engines at your data. Open table formats on object storage keep the data portable — weigh this for a new platform.
4. Copying data between a lake and a warehouse endlessly. The old “lake for storage, warehouse for SQL, ETL between them” doubles storage and adds sync lag. A lakehouse lets one copy serve both — prefer it over maintaining two.
5. Over-engineering for a small shop. If your data is modest and a managed warehouse covers it, you don’t need a full Iceberg-on-S3 platform. Match the architecture to your scale; the lakehouse shines at scale and with multiple engines.
6. Forgetting the metadata/catalog layer. Table formats need a catalog to track tables (e.g. a REST catalog, Glue, Unity). Skipping it leaves you back at “which files are this table?” — set up the catalog, don’t just drop Parquet and hope.

Quick Recap

• Data lake: cheap object storage (S3/GCS), any format, schema-on-read — flexible and cheap but no transactions/updates/guard rails, degrades into a swamp if ungoverned.
• Data warehouse: managed columnar SQL (Snowflake/BigQuery/Redshift) — transactional, fast, structured but pricier, more locked-in, less flexible for raw/ML.
• Lakehouse: table formats (Iceberg, Delta, Hudi) add a metadata layer on top of Parquet files that brings ACID transactions, updates/deletes, schema evolution, and time travel — so data stays in open files on cheap storage, any engine reads it, with warehouse-grade correctness. The industry is converging here.
• Modern default: object storage + Parquet + a table format (+ a catalog); managed warehouses remain a fine lower-ops choice and increasingly read the same open formats. Never run an ungoverned bare lake.

Next lesson: making those files fast to query by layout — partitioning and file layout, where we measure partition pruning (8×) and the skew that wrecks distributed jobs (5.9×).