What makes a catalog open? Unity Catalog Advances the Open Lakehouse

What makes a catalog open? Unity Catalog Advances the Open Lakehouse | Databricks Blog Skip to main content

Summary

Customer-owned storage: Unlike other platforms with native tables in managed storage, Unity Catalog managed tables in customer-owned storage allow portability without data copy.

Unity Catalog is the most open catalog: As the first catalog to support the Apache Iceberg REST catalog spec for REGISTER and UNREGISTER, Unity Catalog allows you to move tables between catalogs without data duplication.

Best features available on tables you own: Open catalogs like Unity Catalog deliver all advanced table features on tables in your storage, not only on tables a provider stores for you.

While almost every platform now supports open formats, what makes a catalog truly open? An open catalog keeps your data in storage you own , gives you the best features there, and supports data portability : the ability to move your tables to another catalog while keeping the data in place . Portability is a core tenet of an open lakehouse, and it’s where most catalogs fall short. As the first catalog to support both REGISTER and UNREGISTER from the Apache Iceberg™ REST catalog spec, Unity Catalog in Databricks' Open Lakehouse expands what an open catalog means: your tables can move existing Iceberg tables in or out without any data movement. Databricks contributed UNREGISTER to the Iceberg REST catalog spec, making this an open standard. Read Unlocking Data Portability . This post walks through the three requirements of an open catalog, how Unity Catalog meets each one, and compares what happens to the same Iceberg table on different platforms. Beyond open formats: The three requirements of an open catalog How open a lakehouse is comes down to the catalog. Many providers still use a proprietary format for the primary table type in their catalog, so formats are important to pay attention to. Beyond this however, an open catalog must also meet three requirements:

Open catalog requirement What it means What happens without it 1. Storage you own Tables live in customer-owned storage rather than managed storage (i.e. “provider-controlled storage”) The bucket sits in the provider's cloud account, not yours. Storage is billed through the provider, so you pay storage “rent”, plus the provider will charge you for moving data out. 2. Best features Included Automatic optimizations and all advanced table features work on tables in customer-owned storage, not only those in provider-controlled storage Teams move tables into provider-controlled storage to get features the customer-owned option lacks. From then on, the provider's roadmap decides what you can do with your data.

3. Free to Move without Copy Tables can enter and leave the catalog using REGISTER and UNREGISTER, with data staying in place Changing catalogs means exporting and rewriting every table, to move it out of provider-controlled storage.

[1] Customer-owned storage for Unity Catalog managed tables and every table type Most data platforms support open table formats like Apache Iceberg. However, when those tables live in buckets that belong to the provider, your tables are no longer open. That makes the format open in name only: you can't see the files in your own cloud account, storage is billed through the provider, and leaving means copying every table out first. Unity Catalog takes the opposite approach. In Databricks, "managed" describes who maintains a table, not who owns its storage. The catalog handles optimization, maintenance, and upgrades, while the files stay in a cloud storage account you own, not in the proprietary storage other platforms often call "managed storage." In Databricks, 99% of UC managed tables are in customer-owned storage. On platforms built around provider-controlled storage, it’s the opposite: nearly all of their tables live in storage that the provider controls and are locked in a proprietary format by default. The difference shows up the day you decide to leave a catalog. [2] The best features should work on the tables you own Owning your storage only matters if the catalog treats those tables as first-class. Some platforms offer a customer-owned storage option, but reserve their best capabilities for tables in managed storage they control, creating an illusion of openness. The open path exists, but it comes with fewer features, so most tables would end up in provider managed storage anyway. With Unity Catalog managed tables, the best features are supported for customer-owned storage (“bring your own storage”) path, whereas other platforms have limitations: Fail-safe data protection. Snowflake and other platforms reserve fail-safe recovery for tables in their own storage. By contrast, Unity Catalog managed tables in customer-owned storage support data recovery: you can query earlier versions or restore dropped tables for up to 30 days. Automatic optimizations. In Snowflake , data compaction is costlier in provider-controlled storage. For other platforms, table maintenance and V-order apply to tables stored in customer-owned storage, rather than data reached through shortcuts. Databricks’ Predictive Optimization Runs on Unity Catalog managed tables in both Delta and Iceberg formats in customer-owned storage. Storage costs. Snowflake doesn't delete orphaned files from Iceberg tables in your own bucket, such as files left behind by failed writes, which means you keep paying for files no table uses. On Unity Catalog managed tables on the other hand, Predictive Optimization removes unreferenced files from your bucket automatically, saving you money.

Unity Catalog managed tables' automatic optimizations saved us over $1 million annually in storage costs while eliminating the need for tedious manual effort on a daily basis. —Abhinav Raghuvanshi, Associate Director of Data Engineering, Zepto [3] Open catalogs provides data portability: Unity Catalog does it without data copy The most important test for whether a catalog is open is portability: whether you can leave the catalog easily at any time by moving tables between catalogs without having to copy out data. Until recently, the ecosystem didn’t have a standard way to release a table from a catalog's management. Therefore, Databricks contributed UNREGISTER to the Apache Iceberg REST catalog specification which gives customers the freedom to leave without having to copy, move or rewrite any data files. In Unity Catalog,...