Transactions are Generally Available, advancing warehousing on the Lakehouse

Transactions are Generally Available | Databricks Blog Skip to main content

Multi-Statement Transactions are  Generally Available , enabling organizations to run complex, multi-step workflows with atomic guarantees without needing separate warehouse systems. Data warehouses power some of an enterprise’s most important workloads, including data ingestion, transformation, business intelligence, applications, and operational analytics. Many of these workloads require changes across multiple tables to succeed or fail together. Historically, organizations relied on separate warehouse systems for these transactional guarantees, adding expensive data movement, duplicate copies, fragmented governance, and operational complexity. Databricks is the first lakehouse architecture to support multi-statement transactions natively across both Delta Lake and Apache Iceberg tables. Alongside stored procedures, temporary tables, and SQL scripting, multi-statement transactions bring traditional data warehousing workloads to open, governed data on the lakehouse. Problems with transactions on traditional and cloud warehouses Traditional and cloud warehouses provide transactional guarantees, but often at the cost of a fragmented data architecture: Additional copies and data movement : Moving data into and out of a separate warehouse adds latency, cost, and inconsistency risk. Fragmented governance : Teams must duplicate and synchronize access policies, auditing, and operational controls across systems. Complex consistency workarounds : Developers need custom orchestration, reconciliation, and retry logic to keep related changes consistent.

Multi-statement transactions bring these guarantees directly to the open, governed lakehouse, reducing duplicate systems and the operational work required to keep them aligned. Transactions on the lakehouse deliver a unified foundation for warehousing Transactions on the lakehouse provide a foundation for running demanding warehouse workloads on open enterprise data. They combine support for open table formats with unified governance and high-concurrency performance. They also integrate with familiar warehouse development tools, data pipelines, and production observability capabilities. Open across Delta Lake and Apache Iceberg Multi-statement transactions support both Delta Lake and Apache Iceberg tables. Teams can preserve open table formats while gaining atomic guarantees across multiple statements and tables. They do not have to move governed data into a proprietary format or maintain a transactional copy in another system. Fast at high concurrency The lakehouse architecture supports row-level concurrency, allowing concurrent transactions to write to the same table as long as they do not update the same row. Unlike traditional data warehouses, which use locks to serialize writes, row-level concurrency enables more transactions to run in parallel. As a result, Databricks delivers significantly stronger performance than traditional cloud and legacy warehouses, allowing organizations to run demanding warehouse workloads directly on the lakehouse. Integrated with warehouse development and data pipelines Transactions are composable with other warehousing capabilities, enabling teams to build traditional data warehousing workloads on the lakehouse. For example, developers can stage intermediate results in a  temporary table , package multi-step business logic in a  stored procedure , and commit the entire workflow atomically. Transactions can also write to  streaming tables , bringing atomic guarantees to workflows that continuously ingest, transform, and publish data. Now, downstream pipelines can receive consistent updates, preventing them from processing incomplete data. Governed transactions on sensitive data Transactions respect Unity Catalog attribute-based access controls, including row-level security and column masking, on both reads and writes. This is especially important for workflows that stage, validate, and publish sensitive data. Observability in in production Production transactions need to be understandable as well as correct. Developers can use commands such as  DESCRIBE TABLE ,  SHOW COLUMNS , and  GET DIAGNOSTICS to inspect the tables involved in a transaction and capture information about transactional execution. These capabilities make transactional workflows easier to develop, operate, and troubleshoot. This fully integrated experience enables Fivetran to build their connectors on top of lakehouse transactions. "Fivetran History Mode preserves each version of a record using slowly changing dimensions (SCD) Type 2. Each source change is two destination writes: closing out the prior version and inserting the new active one - both must succeed together. Previously, with transactional guarantees scoped to a single statement, we ran these as separate operations and defended against partial failure with reconciliation logic and consistency checks on subsequent syncs. With multi-statement, multi-table transactions, we now commit the close and the insert as one atomic unit so either the full change is applied or none of it is. This let Fivetran remove an entire class of consistency-repair code and gave our customers a stronger correctness guarantee on historical data." —Egidio Terra, Staff Product Manager, Fivetran Likewise, Accolade uses transactions to process thousands of claims and liabilities records every day. "At Accolade, we process roughly 1,700 transactions in production every day to handle incoming claims and liabilities. Each of those updates has to complete reliably and consistently. Multi-statement, multi-table transactions let us run these multi-step operations with full transactional guarantees, directly on our governed lakehouse. That reliability is central to our broader migration of critical workloads to the open lakehouse, without standing up a separate warehouse." —Akshay Ajgaonkar, Engineering Manager, Accolade Together, open formats, centralized governance, row-level concurrency, familiar SQL features, pipeline integration, and observability create a complete transactional experience on the lakehouse. Key use cases for multi-statement transactions Multi-statement, multi-table transactions support several common patterns for modernizing ETL, migrating warehouse workloads, and publishing trusted data. Give ETL multi-table transactional guarantees An ETL job can update related tables as one unit. In this example, the order status...