Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend
Captured source
source ↗Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend | DigitalOcean
© 2026 DigitalOcean, LLC. Sitemap .
Product updates
Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend
By DigitalOcean
Updated: October 23, 2026 4 min read
<- Back to blog home
Prompt caching is now generally available on DigitalOcean Serverless Inference, across supported models in our open model catalog, with no changes to your request code required. Cached input is billed at up to 90% off and time to first token drops by up to 80%.*
An agent never starts over. Each new call adds the result of its last tool call to the end of a context that already holds the system prompt, the tool definitions, every file it has read, and every earlier action and result. On DigitalOcean Serverless Inference, more than 90% of the input tokens that agents send are tokens the model has already processed. Prompt caching cuts the cost of those tokens by up to 90% depending on the model, but only when the request reaches the replica holding the cache. Without cache-aware routing, a repeat request lands wherever the load balancer sends it, and on the wrong replica you pay full price for tokens you already sent.
DigitalOcean cache affinity routes each request to the replica that already holds most of its prompt. There’s no session ID to manage: a conversation’s second message already contains its first, so the router finds the match on its own. On our flagship open models, cache hit rates measured in our October 2026 testing ran between 85 and 95%.
How cache affinity makes hit rates reliable
Your cached context lives on one specific replica. If your next request lands somewhere else, that replica doesn’t have your cache, and you pay full price even though your prompt repeats. Serverless traffic spreads across many replicas, so without something steering repeat requests back to the right one, hit rate is mostly a coin flip.
During public preview, caching was on for everyone and the discount was real, but whether you hit the cache was mostly chance. Cache affinity fixes that. For every request, the router checks which replicas already hold most of a request’s prompt, narrows to that warm set, and only then balances load across whichever of those is least busy. If nothing is warm yet, it falls back to balancing across the whole pool. There’s no session ID to manage: a conversation’s second message already contains its first, so the router recognizes the match and sends it home on its own.
In our internal testing, across our flagship models, cache affinity has lifted hit rates from the mid-30% range into the 90%+ range. GLM-5.3 went from mid-single digits to 88–95%; Kimi K3’s benchmark prefix reuse runs from 84% on short prompts to 92%+ on long ones.**
Capabilities at a glance
GA adds two new developer controls, raises context windows, and ships a dashboard to see the results.
prompt_cache_key : Keep traffic separate inside your account
Caches on DigitalOcean are isolated between accounts. prompt_cache_key lets you separate traffic within your own account, so your legal team’s cached context is kept in a separate cache from your marketing team’s, or one end customer’s with another’s. Requests with different keys do not share cache, and each group still caches normally. It’s optional, and it doesn’t pin requests to a replica.
POST https : //inference.do-ai.run/v1/chat/completions
{
"model" : "kimi-k3" ,
"messages" : [ "... your system prompt, tools, and conversation ..." ] ,
"prompt_cache_key" : "tenant-4821-support-agent"
}
Responses API: Discount still applies
In addition to Chat Completions, prompt caching is now also supported on the Responses API. Every response tells you exactly how many tokens were cached, so you can verify the savings request by request.
Larger context windows: 256K tokens and above
Caching only helps the part of your prompt that fits in the window. GA raises context windows to 256K tokens and above, so more of a long codebase, document, or running conversation is eligible to be cached and reused as a single prefix.
Hit-rate and savings dashboard: See what’s working
Every response already reports cached and uncached token counts. The new prompt caching dashboard in the control panel shows what caching is doing for your own traffic: cache hit rate over time, a per-model breakdown, and an estimate of dollars saved, so you can find the prompts that aren’t benefiting and fix them.
Get started
Prompt caching applies automatically to every eligible request, so there’s nothing to turn on. prompt_cache_key is optional, and only needed if you want to group or separate traffic yourself.
Open Serverless Inference in the DigitalOcean Control Panel and generate a model access key if you don’t already have one.
Send requests to https://inference.do-ai.run/v1 the same way you already do.
Add prompt_cache_key to a request only when you want to pin a session or deliberately isolate traffic.
Cached tokens are billed at up to 90% off varying by model; anything that isn’t cached is billed at the standard input rate, and every response tells you exactly how many of each you used. Caches stay isolated per account by default, and cache isolation keys let you separate groups inside your own account too. Available today across DigitalOcean’s open model catalog, with coverage expanding over time.
- Actual discount and latency improvement vary by model and prompt.
** Figures reflect DigitalOcean benchmark testing in October 2026. Your results will vary.
About the author
DigitalOcean Author
See author profile
See author profile
Share
Product Updates
Start building today From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications. Sign up
Related Articles
Engineering Building a computer that can pause and resume in milliseconds: Inside the MicroVM architecture behind Managed Agents
Braden Bassingthwaite
October 15, 2026 20 min read
Read more
Product updates DigitalOcean Inference Router Enhancements: Performance Tuned Routing, No Rules to Write
DigitalOcean
October 13, 2026 4 min read
Read more
Product updates Introducing DigitalOcean Agent Data Services:...
Excerpt shown — open the source for the full document.
Notability
notability 3.0/10Routine cloud feature GA, minor industry relevance.