Netflix ingests five petabytes of event logs every day. The architecture that makes this tractable is not a single database or a monolithic pipeline — it’s a tiered system where ClickHouse handles the hot tier, processing 10.6 million events per second for sub-second queries on recent data, while Apache Iceberg governs cold storage, providing cost-efficient, query-capable archival that any compute engine in the organisation can read. This is not a Netflix-specific curiosity. It is the template for modern data architecture at scale, and the components that make it work have reached production maturity.
Why the Data Warehouse vs Data Lake War Is Over
For a decade, organisations chose between data warehouses (structured, fast, expensive, proprietary) and data lakes (unstructured, cheap, slow, chaotic). The warehouse camp offered ACID transactions and reliable query performance at the cost of schema rigidity and vendor lock-in. The lake camp offered flexibility and storage economics at the cost of governance, consistency, and query latency that made interactive analytics impractical.
The lakehouse architecture resolves the conflict by bringing ACID transaction semantics to object storage. The critical insight is that the consistency guarantees don’t need to live in the storage layer — they can live in the table format layer sitting above it. Open table formats like Apache Iceberg implement optimistic concurrency control, snapshot isolation, and atomic commits on top of plain object storage (S3, GCS, Azure Data Lake). The result is ACID transactions without proprietary infrastructure, time travel queries that let you query any historical snapshot of the data, schema evolution without rewrites, and multi-engine access where Spark, Flink, ClickHouse, Trino, and DuckDB all read and write the same tables without translation layers.
The Four Open Table Formats — and When to Use Each
Four table formats have emerged as the dominant options, each with a distinct design philosophy and strength profile:
- Apache Iceberg was designed from the ground up for cross-engine interoperability and scalability. Its specification is engine-agnostic — any engine that implements the Iceberg spec can read and write Iceberg tables. Hidden partitioning, partition evolution without data rewrites, and a file-level manifest structure that scales to billions of files without metadata bottlenecks make it the leading choice for multi-engine environments. Iceberg is winning the interoperability battle and is increasingly the default choice for new lakehouse deployments.
- Delta Lake originated within the Databricks ecosystem and remains the simplest choice for organisations already running on Spark. Its change data feed feature and tight integration with Databricks MLflow make it attractive for ML-heavy workloads. The trade-off is weaker cross-engine portability — non-Databricks engines treat Delta tables as a second-class citizen relative to Iceberg.
- Apache Hudi was built for real-time upserts and incremental data processing, use cases where Iceberg’s batch-oriented design shows its seams. Hudi’s merge-on-read and copy-on-write storage types let you tune the read/write performance trade-off. Incremental views — queries that efficiently return only records changed since a given timestamp — are a Hudi native feature that other formats implement awkwardly. Choose Hudi when your primary requirement is high-frequency upserts from CDC sources.
- Apache Paimon is the newest of the four and was designed specifically for streaming-first workloads with changelog semantics. Where Iceberg and Delta think in terms of snapshots, Paimon thinks in terms of streams of change events. It integrates natively with Flink and is the right choice when the lakehouse is primarily a materialised view over a streaming data source.
Hot Tier / Cold Tier: The Architecture Pattern You Need
The single most impactful architectural decision in a modern data platform is how you split hot and cold data — and what engine you put in front of each tier.
The cold tier is Iceberg on object storage. Data older than your hot window (typically 7–90 days depending on query patterns) lives here. Storage costs are minimal — S3 Standard is $0.023/GB/month, S3 Intelligent-Tiering can cut that further for infrequently accessed data. Iceberg makes this cold data fully queryable by any engine without a schema registry or serialisation layer. Time travel queries on cold data are a configuration parameter, not an engineering project.
The hot tier is ClickHouse. ClickHouse’s columnar MergeTree engine is engineered for analytical queries at sub-second latency over hundreds of billions of rows. It achieves this through sparse indexing, compressed columnar storage, vectorised execution, and aggressive use of SIMD instructions. The 25.7 through 25.9 release series added full Iceberg read/write parity, including ALTER UPDATE support — ClickHouse can now act as both a consumer and a producer of Iceberg tables. Catalog support spans the full enterprise landscape: REST catalog, AWS Glue, Databricks Unity Catalog, Hive Metastore, and Microsoft OneLake. This means ClickHouse slots into an existing lakehouse catalog without requiring a separate metadata infrastructure.
The pattern is clean: write all data to both ClickHouse (for hot analytics) and Iceberg (for durable cold storage). After the hot window expires, data remains queryable in Iceberg but is evicted from ClickHouse. Query routing — hot queries to ClickHouse, historical queries to Iceberg — can be transparent to consumers through a federated query layer.
Streaming Into the Lakehouse
The dominant streaming ingestion pattern is Kafka for durable, partitioned event delivery and Apache Flink for stateful stream processing. Flink’s Iceberg sink connector writes directly to Iceberg tables with configurable commit intervals, enabling near-real-time data availability in the cold tier with latencies in the seconds-to-minutes range.
The gap between “streaming” and “lakehouse” has historically been 5–15 minutes of latency — the time required to accumulate enough records to write a meaningful Iceberg file without producing millions of tiny files. Fluss, an open-source streaming storage system developed by the Apache Flink community, addresses this by providing sub-second write latency with columnar storage and primary-key semantics. Fluss sits between Kafka and Iceberg in the ingestion path: events land in Fluss at streaming speed, and Fluss continuously compacts and writes to Iceberg in the background. The result is a system that can answer “what is happening right now?” at sub-second latency and “what happened over the last three years?” against Iceberg without running two separate pipelines.
The Five Architectural Layers
A production lakehouse decomposes into five distinct layers, each with a well-defined responsibility:
- Ingestion: Kafka for event streaming, Apache Flink for stream processing and enrichment, Debezium for change data capture from operational databases. This layer is responsible for getting data into the platform reliably and with low latency.
- Storage: Iceberg or Delta Lake tables on S3 or GCS. All durable data lives here. The storage layer has no compute — it is purely responsible for organising and persisting data in an open, engine-agnostic format.
- Compute: Spark for large-scale batch transformations and historical backfills, Flink for stateful streaming computation, ClickHouse for interactive analytics on hot data. Multiple compute engines coexist without conflict because they all speak Iceberg.
- Catalog: Apache Polaris (open-source REST catalog), Databricks Unity Catalog, or AWS Glue. The catalog maintains the authoritative mapping between table names and data file locations, enforces access control, and tracks schema history. Without a catalog, multi-engine access degrades into a coordination problem.
- Serving: BI tools (Metabase, Superset, Tableau) connect via ClickHouse or Trino, ML feature stores read from Iceberg, REST APIs materialise query results for application consumption. The serving layer translates governed, versioned data into the format each consumer needs.
Schema Evolution Without Breaking Everything
Schema evolution is where lakehouses earn their keep on long-running production systems. Operational schemas change — columns are added, renamed, widened; partitioning strategies change as data volumes grow; field semantics shift as the business evolves. In a traditional data warehouse, schema changes are expensive migrations with downtime windows. In an Iceberg-based lakehouse, they are metadata operations.
Iceberg’s hidden partitioning decouples the physical partitioning of data from the logical schema exposed to queries. When your partitioning strategy needs to change — say, from daily partitions to hourly as query patterns shift — Iceberg performs partition evolution without rewriting existing data files. New data is written with the new partition scheme; old data is read using the old scheme. Queries are automatically routed to the correct partition layout based on the snapshot they’re reading.
Column renaming and type widening (e.g., promoting an integer column to a long) are handled at the metadata level. Existing Parquet files retain their original column names; the Iceberg manifest translates between the current schema and the physical layout. Producers and consumers can evolve their schemas independently as long as changes are backward-compatible — which Iceberg enforces at write time.
The goal of the modern data lakehouse is a single governed body of data that serves batch pipelines, streaming applications, interactive analytics, and AI/ML workloads without duplication, without conversion, and without the coordination overhead of maintaining separate stores for each access pattern. The architecture exists today. The components are production-ready. The primary barrier to adoption is organisational, not technical.