Autoscaling Lakebase Postgres

Autoscaling Lakebase Postgres | Databricks Blog Skip to main content

Summary

Autoscaling architectural requirement

When to adjust capacity up and down

How to adjust capacity without stopping PostgreSQL

Choosing a database instance size before you know the workload is an old building pattern. The process is generally wonky and feels very wasteful of compute, especially now that compute is becoming a luxury . Lakebase Postgres omits the sizing experience altogether thanks to autoscaling. Autoscaling responsiveness comes from in-place VM resizing and an algorithm that tracks CPU, memory, and the database’s working set. How autoscaling looks like for an arbitrary sample of Lakebase Postgres databases. Note how this is only one hour. The architectural requirement Traditional Postgres runs as a stateful process tied to a machine and its disks; replacing or resizing that machine is a database operation because the machine owns both execution and durable state. But the Lakebase Postgres architecture separates those responsibilities: The compute layer runs Postgres and executes queries. It uses RAM and local NVMe for low-latency access, and owns no durable state. The storage layer owns durability and history. WAL is replicated by safekeepers running on SSDs, pageservers (also SSDs) reconstruct page versions, and object storage keeps the long-term immutable record. (This blog post focuses on compute, but we wrote a deep dive on the storage piece if you are also interested.)

A compute node can therefore start, stop, move, or change size without moving the database underneath it. This is an essential foundation. Now, when it comes to implementing autoscaling, there are two parts to the story: first, one has to determine when to adjust capacity up and down, and second, how to do it without stopping Postgres. Let's cover both in order. Part I: The algorithm The three autoscaling signals To deduce when to resize, the Lakebase Postgres autoscaling algorithm tracks three signals, with each signal producing its own target compute size: CPU load: cpuGoalCU Memory use: memGoalCU Compute-cache working set size: lfcGoalCU

The final scaling target is the largest of the three, constrained to the minimum and maximum compute sizes that the user has configured for that database (the autoscaling limits) :

CPU (cpuGoalCU) CPU is the most straightforward of the three signals. The algorithm keeps a close watch on how hard the processor is working: Every five seconds , the autoscaler-agent reads the VM’s one-minute CPU load average. The CPU goal aims to keep that load at or below 90% of available CPU capacity. When the load rises above that target, cpuGoalCU increases. When sustained load falls, the goal falls with it.

Using a one-minute average filters very short fluctuations while still responding to meaningful changes in demand. The five-second polling interval lets the system update the target as that average moves. CPU alone, however, is not enough to autoscale Postgres properly. A query waiting for data to arrive over the network can show low CPU use while performing poorly. The algorithm also needs to account for memory and cache pressure. Memory (memGoalCU) Memory has a different failure mode from CPU. If demand briefly exceeds the available CPU, queries become slower; but if Postgres allocates more memory than the VM has, the kernel can terminate processes. The autoscaler therefore needs a much faster signal than CPU for memory exhaustion. So the system watches memory at two frequencies: Every five seconds , the autoscaler-agent reads overall memory metrics from the VM. Every 100 milliseconds , the vm-monitor checks memory used by Postgres .

The memory goal keeps use below 75% of allocated RAM. That headroom gives the system space to respond to new allocations and leaves memory for the guest operating system and other processes. The vm-monitor also checks every proposed downscale. Memory cannot be removed if doing so would leave the running processes without enough space. A bit of history: This polling approach replaced an earlier design based on the cgroup memory.high event. Crossing memory.high caused Linux to reclaim memory and throttle the processes inside the cgroup. Polling proved more predictable and stable while still giving the system a 100-millisecond view of Postgres memory. The compute cache (lfcGoalCU) The third signal measures whether the workload’s active data fits close to Postgres. The high level story is this: Lakebase Postgres separates storage and compute; when a page is not available locally, the compute requests it from the pageserver; the returned page is cached for subsequent reads. The compute cache, which we originally called the Local File Cache or (LFC), is a disk-backed cache sized to fit in the kernel page cache. It acts as a resizable extension of Postgres shared buffers. When a compute grows, the vm-monitor expands the cache to use part of the added memory. For many OLTP workloads, performance changes sharply once the working set fits in local memory. This exposes a blind spot in CPU-only autoscaling: cache misses leave queries waiting on network requests, which reduces CPU use. The system may therefore see low CPU pressure at the exact moment when a larger cache would improve performance. So in Lakebase Postgres, there’s a third autoscaling signal that estimates the Postgres working set directly. This is the most interesting part of the algorithm, so let’s look at how that estimate works. Zooming in: how we estimate the Postgres working set A workload’s working set is the set of database and index pages it accesses repeatedly over a given period. To exactly count every page for the purpose of autoscaling would require too much memory, so the classic way to solve for this is to rely on HyperLogLog , a probabilistic cardinality estimator that can estimate the number of distinct items in a set using a small, fixed amount of state. For each Postgres page access, a standard HyperLogLog implementation, Hashes the page identifier. Uses the first bits of the hash to select a register. Counts the leading zeroes in the remaining bits. Updates the selected register if this observation exceeds its previous value.

The distribution of those register values would provide an estimate of how many distinct pages have been observed. However, there’s an issue with simply using HyerLogLog for autoscaling: a standard HyperLogLog only grows. Once a register has observed a...