The AI Bill is coming

The AI Bill is coming | Databricks Blog Skip to main content

Summary

The cost curve is structural. AI tools are consumption-priced, so usage grows exponentially while revenue doesn’t. Unlimited tokens and hard spend caps both fail, and Databricks has moved to visibility plus progressive friction instead.

Four levers cut unit costs by as much as 90%. Shift defaults to more efficient models, route each task to the cheapest model that can handle it, give developers visibility into their spend, and cut token overhead. Token price alone is a poor predictor of what a model actually costs.

Every engineering org needs a control layer. An AI gateway sits between developers and models to handle access, routing, budgets, and tracing, making AI spend observable, steerable, and predictable.

In this conversation, Catherine Brown talks with Patrick about what happened when Databricks went all in on AI coding agents: how costs initially spiraled with growing usage , what actually brought unit costs down by as much as 90%, and why every engineering organization needs a control layer between its developers and its models. That control layer started as an internal fix for Databricks' own AI bill, and it became the inspiration for Unity Gateway .

Engineering output is the bottleneck

Catherine: You run Databricks' AI products, and you've also been the internal champion for AI adoption. Why has this become such a personal priority for you?

Patrick: Because engineering output is the main bottleneck in our business. Every company needs its engineering team to move faster, and if you can give them something that meaningfully improves their productivity, that's worth a lot. The early chat products were somewhat useful. But the ROI really changed maybe a year ago, when the first really good coding models showed up.

Catherine: How do you know whether you are getting productivity wins from AI tools?

Patrick: We track a bunch of output metrics: pull requests, features shipped and lines of code. None of them is perfect on its own. I think of R&D as a big machine. You put resources in, you get features and code out, and you can measure how much more is coming out. In aggregate, we're close to doubling output with the same team. The teams that rebuilt their processes around AI are going a lot faster than that.

From Tokenmaxxing to ValueMaxxing

Catherine: When Databricks first rolled out AI coding tools, what did you get wrong?

Patrick: It's kind of funny because at the start, we had the opposite problem. Nobody wanted to try the new stuff. I was going around bugging people. So we did what everyone did, which was remove every barrier: the most expensive model, as many tokens as you want. I think that came from a good place. You're trying to get people over the hump.

Then people actually started using it, and we saw this exponential cost curve. These tools are all consumption-priced now, not per seat, so a single user can, in principle, spend an unbounded amount of money. They can run a little loop on the most expensive model and go to lunch. Maybe that loop is doing something really valuable, but maybe it’s just burning tokens. If your costs grow exponentially and your revenue doesn't, at some point, you have a problem. We got to the point where the costs were threatening to reverse the efficiency gains, which were the whole reason we did this.

That's when I got very involved personally. I compared notes with leaders at other technology companies who I knew were adopting AI at scale (Coinbase, Uber, Stripe, and others). Collectively, many of us were pursuing similar techniques. Today, at Databricks, our token spend per head is roughly flat now, even as total usage and value keep going up.

Stop putting hard caps on your best developers

Catherine: Most companies' first instinct is a hard spend cap. You tried that. What happened?

Patrick: It's the obvious first move, and you do want the mechanism. But a hard stop is a bad experience. You cut someone off mid-task, and they lose their flow, maybe hours of work. And the people hitting the cap are often your best users. Cut them off, and some of them just go use a personal account, and now you can't see anything.

So we moved to friction that ramps up, and Unity Gateway handles all of it. First, a warning, then an approval, then we downshift you to a cheaper model so you can keep working. Actually cutting someone off is a last resort. We also send everyone a regular breakdown of their own spending with suggestions on where to save. That alone is worth about 10%. One company I know of puts the model and the cost right in every pull request. So it's basically public shaming if you use an expensive model for a small thing. I love it.

The best cost optimization is one developers never notice

Catherine: You've written about four levers for bringing these costs down. Which one has had the biggest impact?

Patrick: Our favorite one is kind of boring: better, cheaper models keep coming out, and you move traffic to them. There are probably five models released every week between the proprietary and open source vendors. Not every one is a step forward on cost, but maybe one every week or two is. It's a rising tide. You get the same output for less money, and nobody has to change anything. No fancy routing, nothing.

To give you a sense of how fast this moves: one of the recent cheap models roughly matches what a frontier model could do about nine months ago, at around 1% of the cost per task. That's a 100x drop in under a year. That's still preliminary, but it's the direction.

The catch is you have to actually move. If a new model comes out and nobody in your company is using it, it's a tree falling in the woods. So we got good at evaluating new models fast and shifting workloads over when one is good enough. Because everything goes through Unity Gateway, we can make that switch in one place, and developers don't have to change anything. That's been the most impactful thing we've done, 50% or more on its own. Add routing, spend visibility, and cutting token overhead, and in some cases, our unit costs are down as much as 90%.

A model's token price is a bad predictor of what it costs

Catherine: Your team benchmarks coding agents against Databricks' own multi-million-line codebase . Why not just trust the public benchmarks?

Patrick:...