Apache Iceberg

Apache Iceberg is an open-source technology that lets you handle huge volumes of data as if it were a neatly organized table. If raw materials are just piled up in a warehouse, it's hard to tell what arrived when — similarly, when large-scale data is stored only as raw files, changing its structure later or rolling it back to an earlier state becomes very cumbersome. Iceberg adds a management layer on top of that pile of files, much like a precise inventory ledger.

Specifically, it makes it safe to change a table's column structure, to let multiple programs read and write the same data at the same time without conflicts, and to look back at exactly what the data looked like at a certain point in the past. In effect, it brings the reliability of a bank's ledger to what would otherwise be just a storage warehouse for data.

Originally created at Netflix and later donated to the Apache Software Foundation, the project serves as a kind of common standard, allowing different data-processing tools to read and write the same data in a shared format. Companies building platforms for handling massive datasets, such as Databricks, also support compatibility with this format.