Apache Iceberg

Databricks implemented the data Lakehouse architecture through its open-source project Delta Lake.

Lakebase is a new type of database form proposed within the Databricks ecosystem, whose core concept is to build Postgres-compatible transactional database capabilities on top of a data lakehouse architecture.

Apache Iceberg and Apache Hudi, together with Delta Lake, are collectively known as the three major open formats of the lakehouse era.

Lakehouse resolves the contradiction between the lack of ACID transactions in data lakes and vendor lock-in by data warehouse vendors by building a transactional metadata layer (such as Delta Lake, Apache Iceberg, and Apache Hudi) on top of object storage.

Teams using the native support still need to pay attention to: the streaming write frequency and index update strategy must match the business freshness requirements; the refresh cycle of materialized views affects the real-time nature of query results; and Iceberg table metadata management may bring additional overhead as the number of tables grows.

Iceberg manages table state through a metadata layer, enabling multiple compute engines (such as Spark, Trino, Flink and Presto) to concurrently and safely read and write the same table.

Delta Lake, Iceberg and Hudi are the three mainstream open table format implementations today.

Delta Lake uses a transaction log (_delta_log), Apache Iceberg uses manifest files and a metadata tree, and Apache Hudi uses a timeline (Timeline) as the mechanism for implementing the metadata layer.

Iceberg provides index formats such as Puffin to supplement Bloom filters or finer-grained statistics, in order to improve query pruning effectiveness.

Iceberg's manifest files are Avro files, and each entry contains the file path, the partition it belongs to, record count, file size, and statistics such as the upper and lower bounds and null counts for each column; query planning uses these statistics and partition information to prune data files.

Each Iceberg write produces a new snapshot, and commit atomicity is achieved by renaming metadata files or through conditional updates to an external catalog, simplifying concurrency control from page-level locks to metadata-level optimistic concurrency.

Object storage such as S3 only guarantees the atomicity of a single-object PUT, with no cross-object rename or transactions, so Iceberg's atomic commits on object storage rely on mechanisms such as rename, Hive Metastore database transactions, or the CAS of a REST catalog.

Iceberg strips partition information out of the physical directory, with partitions existing as transformation expressions in metadata, so schema evolution does not require rewriting historical data.

InnoDB excels at high-frequency small transactions and low-latency point lookups, while Iceberg excels at massive batch writes, analytical scans and schema evolution.

InnoDB's complexity comes from the premise of 'in-place updates', while Iceberg's simplicity comes from the premise of 'immutable files'; when choosing a storage solution, one should first determine the write pattern and storage medium before deciding the metadata form.

Migrating online business from InnoDB to Iceberg is not just a matter of changing the connection string, but requires redefining the trade-offs among consistency, latency and cost.

Iceberg is an open table format for data lakes, and its metadata hierarchy is: table metadata JSON → manifest list → manifest file → data file, where each manifest file records data file paths, partitions, column-level statistics and file-level statistics.

Iceberg was initiated by Netflix and contributed to the Apache Foundation, with an emphasis on multi-engine compatibility (accessible by Spark, Trino, Flink, and Hive).

The lakehouse architecture is based on open storage formats (such as Parquet) and open table formats (such as Delta Lake, Iceberg, and Hudi), providing capabilities such as transactions, version management, and schema evolution on top of low-cost object storage.

All three—Delta Lake, Iceberg, and Hudi—support ACID transactions, time travel (historical version queries), schema evolution, and partition pruning.