The 40-year-old database rule agents just broke: How LTAP unifies OLTP and OLAP workloads

The 40-year-old database rule agents just broke: How LTAP unifies OLTP and OLAP workloads | Databricks Blog Skip to main content

Summary

For decades, operational (OLTP) and analytical (OLAP) systems have lived apart because of a physical storage tradeoff: rows for fast transactions, columns for broad analysis.

AI agents break that arrangement. They need to read and act on live operational data in near real time, and neither traditional pipelines nor HTAP systems can do that affordably or fast enough.

LTAP (Lake Transactional/Analytical Processing) unifies transactional and analytical workloads at the storage layer, not the engine layer, which is why it succeeds where HTAP has historically stalled.

Jonathan Katz has spent his career on both sides of a line that most of the data industry has simply learned to live with. As a longtime Postgres contributor and now a Senior Staff Product Manager at Databricks, he has watched operational and analytical systems operate as two separate worlds, bridged only by pipelines, copies, and compromise. In this conversation, Jonathan walks through why that separation existed in the first place, why AI agents are the thing finally breaking it, and how LTAP closes the gap by rethinking where the two worlds meet: not in a single engine, but in a unified storage layer. The assumption AI agents broke The industry has lived with a strict separation between operational and analytical database systems for decades. Why is the rise of autonomous AI agents breaking that apart? Jonathan Katz: There are two worlds of data. Operational data is what you touch when you're processing a credit card transaction or looking for fraud: very short, fast queries, looking at data line by line. Analytical data is what you've been accumulating for weeks, months, or years, and when you query it, you're looking across the entire dataset. Both are trying to return an answer as quickly as possible, but they go about it in completely different ways. That difference isn't arbitrary. It comes down to physics. If you store your data in rows, that's how you return a single fast answer as quickly as possible. If you store it in columns, that's how you scan and aggregate across everything as quickly as possible. Also, by design, an analytical query can consume the entire resources of a massively parallelized system to get an answer over a very large dataset, while an operational query is designed to consume as few resources as possible while still returning an answer quickly. These lead to two very different ways to think about how you design and manage your data system, and require different kinds of optimizations. Postgres , one of the most widely deployed databases on the planet, is built around rows because it's optimized for operational workloads. Analytical engines are built around columns for the opposite reason. Because of that fundamental physical difference, the two systems have always had to sit apart, and any time you wanted to analyze operational data, you had to ship it somewhere else. The rise of AI agents has changed what we need from databases. It’s part of why Lakebase’s architecture exists in the first place: databases built to keep up with how agents actually work. For example, agents are tasked with looking for fraud or anomalies, and those events are happening in hundreds of milliseconds. But that system is also handling a huge volume of writes and short reads at the same time. A single agent might be smart enough to know what query it needs to run. A herd of agents can easily overwhelm an operational system if there aren’t guardrails in place. What LTAP actually is What is LTAP, and how does it actually work under the hood? Jonathan Katz : LTAP, Lake Transactional/Analytical Processing, lets you run analytical queries directly against live operational data without moving that data anywhere and without putting load on the system serving your transactions. It does this by unifying transactional and analytical data in a single logical storage layer instead of forcing them through separate systems connected by a pipeline. Under the hood, LTAP is only possible because of how Lakebase itself is architected: stateless, ephemeral compute that's fully decoupled from storage in the lake. That separation, inherited from Neon, means the durable storage layer can handle high-throughput writes and periodically flush them down to object storage for permanence, independent of whatever compute happens to be running against it at any given moment. Since that data was already being optimized for cloud storage, why not represent it in the same columnar format the Lakehouse already uses, so engines like Apache Spark and SQL can read it directly and get high-performance analytical reads without a second copy? The hardest part was making sure nothing got lost in translation. Postgres has its own data types and encodings, and open formats like Iceberg and Delta have their own. We had to write the data out in a way that preserved the exact physical representation of the original Postgres data, without changing a single bit, and get it into a Parquet file. That's the piece that let us merge the operational and analytical representations of the same data into one. In practice, the storage layer runs in two tiers: a hotter tier that keeps data in row format for fast operational access, and a cooler tier that holds it in columnar format for analytical reads, so either side can get what it needs efficiently. Why LTAP succeeds where HTAP stalled HTAP tried to solve real-time analytics years ago and stalled out. Why does doing this at the lakehouse storage layer succeed where traditional HTAP failed? Jonathan Katz : You can make HTAP systems work, but they're expensive. They're clunky, hard to run, and generally not open. What's different about the LTAP model is that you get serverless operational compute and serverless analytical compute as two separate things. You can tailor exactly how much compute you're using for each workload independently, rather than paying for one system trying to do both jobs at once. Storage is the cheap part of any data system. Compute is the expensive part. That's the whole argument for unifying the storage layer instead of the engine: you get to keep the specialized, efficient engine for each job, and you only pay for compute where you actually need it, instead of running one expensive...