How a global fintech scaled coding agent traffic with Dedicated Model Inference

How a global fintech scaled coding agent traffic with Dedicated Model Inference Webflow Analyze/Optimize tracking bridge -->

The Open Source AI Stack: models, inference, routers, harnesses & tools →

📈 GLM-5.3 vs. GLM 5.3 Flash on DeepSWE →

🚀 DeepSeek V4 Pro 0813 vs. GPT-5.6 Sol on DeepSWE →

⚡ On-demand B200s now available on Together GPU Clusters →

All blog posts

Inference

Published 9/18/2026

How a global fintech scaled coding agent traffic with Dedicated Model Inference

Authors

Ankit Goyal

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Summary

A global fintech runs its coding assistant on GLM 5.2 through Together's Dedicated Model Inference, handling spiky, engineering-hours traffic that static capacity planning couldn't keep up with.

With DMI, the customer's engineers scale endpoints, roll out models, and test changes themselves, no tickets, no waiting on Together. The result: infrastructure that moves as fast as the teams adopting it.

The workload This customer ships financial products to millions of users across dozens of markets, and growth shows no sign of slowing. Sustaining that pace is an engineering problem before anything else, and the company's engineers lean on AI coding agents to do it. That puts inference on the critical path of how fast the company ships, rather than inside any single customer-facing feature. The workload runs on GLM-5.2, the mixture-of-experts model built for long-horizon coding and agentic work, served on Together. Traffic follows the working day: spiky, concentrated in engineering hours, and it climbs every time another team adopts agents into its workflow. The constraint: capacity planning couldn't keep up with adoption Operational control The customer came to Together after running coding workloads with other inference providers, and first consolidated onto our earlier dedicated offering. That offering worked, but wasn't built for how this workload actually behaves. The coding-assistant traffic isn't steady; it's peak-load and relatively low-TPS, concentrated in engineering hours, with sharp bursts in concurrency and prompt size as more teams put agents into their daily workflow. That shape is precisely why concurrency, not raw throughput, was the design priority when the workload moved to GLM-5.2. Under the earlier model, absorbing that kind of burst meant someone had to see it coming. Teams ready to move agents into their daily workflow often waited on capacity rather than provisioning it, and the customer's platform team absorbed the coordination for every one of them, filing requests and sizing clusters. The team worked to plan ahead, but planning stopped working once adoption became unpredictable in both timing and size. You can't forecast a burst that's driven by a hundred different engineering teams independently deciding to lean on their coding agent harder this week. When capacity is provisioned to yesterday's forecast and traffic is genuinely spiky, prefill capacity and KV cache headroom get exhausted exactly during the burst, which is when it matters most, producing exactly the kind of multi-minute request queuing that agentic coding workflows can't tolerate. Fixing that after the fact, versus giving the customer's own teams the ability to see load and scale ahead of it, is the difference between a coordination problem and an infrastructure one. What the customer required: self-service, observability, concurrency The customer set requirements for the Together team around autonomy, in addition to raw performance, and the workload's own shape makes clear why. The coding-assistant traffic runs at ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, a peak-load, relatively-low-TPS pattern where bursts in concurrency matter far more than raw tokens-per-second. That's the backdrop for what the customer asked of Together: Self-service provisioning : An engineering team should be able to stand up its own endpoint and put traffic on it without filing a request or waiting on the platform group, a shift from the earlier model, where every new team's capacity request went through Together and the customer's platform team in turn. Observability its own teams could act on : Usage and performance data available programmatically, so capacity decisions could sit with the teams making them, not get routed through a support queue when something like cache hit rate degrades. Throughput and fast scaling under concentrated load : Sustained performance during working-hours peaks, not benchmark conditions, running dozens of B200s across a multi-replica configuration at 256K context, sized specifically to hold concurrency headroom. Model fluidity : Room to swap models as the frontier advances, without renegotiation, demonstrated in practice by the move from GLM 5.1 to GLM 5.2 on the same account, plus a live tuning pass on cache and load-balancing parameters done as a config update, not a redeployment.

What shipped: full endpoint control, a metrics API, and model fluidity Endpoint configuration through the API, UI, or CLI Dedicated Model Inference exposes the full endpoint lifecycle: creation, sizing, scaling policy, and configuration changes. The customer's infrastructure team used exactly this when a migration reshaped their GLM 5.2 endpoint, shifting toward fewer, larger replicas, same total footprint, different ratio of replica count to chips per replica. When that re-shape hit near-100% prefill capacity a few days later, with requests queuing one to three minutes and decode throughput collapsing to roughly 5 tokens per second, the fix wasn't a new deployment or a ticket back to Together. It was a live configuration change: restoring the tuned cache-session-aware routing policy in place of DMI's default cache-aware-by-hash policy, and widening the max-inflight-per-worker threshold. All of it was pushed same day with zero downtime. Metrics API Programmatic access to endpoint usage and performance data is how the root cause was found. Together API Support traced a single 192-second slow request end-to-end through the metrics data and found it wasn't compute-bound, and had spent almost the entire span queued behind a 2.3M-token pending-prefill backlog from other requests, not its own 250K-token prompt. That's the...