A/B test models in production

A/B test models in production Webflow Analyze/Optimize tracking bridge -->

💰 Announcing our Series C. Intelligence should be abundant, not expensive →

🤝 Together AI & Y Combinator announce partnership to deliver the first dedicated YC GPU cluster →

⚡ On-demand B200s now available on Together GPU Clusters →

🚀 Now serving MiniMax-M3 for efficient inference →

All blog posts

Inference

Published 8/17/2026

A/B test models in production

A guide to implementing A/B testing

Authors

Zain Hasan, Zarni Phyo, Nikitha Suryadevara, Ted Cui

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Summary

A/B experiments allow you to split an endpoint's live traffic into fixed cohorts of one control and up to 20 variants, each with a percentage split of the traffic. This allows you to measure how a candidate model performs with real users at an exposure level you choose. Ramping up a variant can also be done with one call where you can promote the winner using a blue-green rollout. Deleting the experiment returns 100% of traffic to the control without the need to make any client-side or routing logic changes to unwind afterwards. Below we'll run an experiment on a live endpoint where we create at 95%/5%, ramp to 80%/20% and 50%/50%, then delete and check the observed traffic shares at every stage.

Implementing A/B testing for LLMs in production Sooner or later every team wants to answer the same question: is the new model actually better for our users compared to the current model? Not better on a benchmark but rather better on retention, thumbs-up rate, task completion, whatever your product actually measures. Shadow traffic can't answer that question. Shadowing tells you the candidate is operationally sound with respect to latency, errors, throughput, but its responses are discarded; no user ever acts on them. Quality questions need real exposure to end users where a cut of your users get model B, and you compare what happens. Typically teams build this themselves in the application layer using some combination of: A feature flag or a hash-mod-100 on user ID in the client code. Two endpoints (or two hardcoded model strings) the client switches between. A spreadsheet somewhere explaining what group A vs B means.

It works, but it entangles your experiment with your infrastructure in ways that hurt later: the routing logic ships with your application, the cohort split can drift as clients cache decisions, and even after the experiment "ends" the branching code lives on long afterward because nobody's sure it's safe to remove. The Together AI platform allows you to run A/B experiment logic at the endpoint level. How it works An A/B experiment attaches to an endpoint and declares members with exactly one control and one or more variants , each pointing at a deployment, each with a percent setting, that must sum to 100, controlling traffic routing.

How the endpoint router works is that whenever the base traffic sends a request to the control the experiment re-samples it among the arms and redistributes such that 95% stays on the control, 5% goes to the variant.

To be precise about the mechanism: the experiment subdivides the control's share of the base traffic split. Routing first resolves a request through the weight split; when the winner is the control of an A/B experiment, the request is re-sampled among the experiment's arms by their percents. With the control as the only entrypoint in the split member percents therefore are absolute traffic shares. Also worth noting is that a control whose split weight is zero gives the experiment nothing to subdivide and as a result the whole experiment receives no traffic.

Importantly variant deployments must not be in the endpoint's traffic split, the platform requires variants to carry zero weight; only the control lives in the base split. The experiment will own traffic routed to the variant entirely; its percentage is its traffic share. If a variant could also draw capacity-weighted traffic from the split, your measurements would be quietly wrong. One way to think about it is that you should set up the variant like a shadow deployment: created, READY , weight zero and then let the experiment percentage setting route to it. Another important point here is that A/B percents are true fixed traffic shares summing to 100% and are independent of replica counts. We made this deliberately different from traffic-split weights (which are per-ready-replica and follow capacity). An experiment is a measurement instrument ; you want the split to be constant while you measure and not drift with autoscaling. Creating a 95/5 experiment:

tg beta endpoints ab my-org/candidate-model --control $CONTROL_DEPLOYMENT --percent 5

Your clients won’t notice this experiment because on the surface the same endpoint name, API and keys persist. On the backend 5% of requests will now be answered by the variant candidate. Under the hood: ramping, measuring, ending Ramping is resending the member set There's no separate "ramp" API, an update will replace the full member list , which keeps the mental model simple (the experiment is always exactly what its members say) and makes every ramp an explicit reviewable change:

Week 2: candidate looks good at 5% —> go to 20%

client.beta.endpoints.ab_experiments.update( id=experiment_id, endpoint_id=endpoint_id, update_mask="members", etag=experiment.etag, # a teammate's concurrent ramp gets rejected, not overwritten members=[ {"deployment_id": control_dep, "percent": 80, "role": "AB_EXPERIMENT_MEMBER_ROLE_CONTROL"}, {"deployment_id": variant_dep, "percent": 20, "role": "AB_EXPERIMENT_MEMBER_ROLE_VARIANT"}, ], )

Updates are guarded by an etag because if a teammate ramped the experiment while you were composing your update, yours will be rejected instead of silently overwriting theirs.

With this API design you still need to make the common exposure choice of how much traffic to route to group B:

Split Signal speed Risk Use when

95/5 Slow (needs volume/time) Minimal New model, first real exposure

80/20 Moderate Contained Candidate survived 5%; you want a more significant readout

50/50 Fastest Half your users Late-stage confirmation between two known-good options

With up to 20 variant members you can also run multi-way tests, lets say for example you want to try out a full-precision endpoint along with...