Powering Inference For The Continual Learning Era

Powering Inference for the Continual Learning Era Announcing our Series F . Learn more

Community

Powering Inference for the Continual Learning Era

How Trajectory went from research to production at scale with Baseten.

Authors

Bola Malek

Michael Elabd

Arjun Karanam

Marylise Tauzia

Last updated May 27, 2026

Share

What is continual learning? Models are more powerful than ever. But the way we use them is static. You use a product, and the model behind it stays the same until the next quarterly release. Every time you run into an issue and find a workaround, you have to wait weeks or months for the model itself to hopefully get better. Benchmarks go up. Products don't. A new paradigm is on the horizon: continual learning. Models that get better the longer you use them. A legal product sharpens on legal work the longer lawyers use it. A support agent internalizes the shape of the tickets it actually sees. A coding product learns the codebase it lives inside. The model and the product stop being separate things. Getting there means solving many problems at once. At the research layer, how do you learn from production traces in a way that turns every interaction into a training signal? At the product layer, how do you let builders shape their model and the harness around it as a single object, so the way the product behaves and the way the model learns evolve together. At the infrastructure layer, how do you serve a model that is no longer a fixed artifact but a moving target? Trajectory is the team pioneering this paradigm. They take production traces, identify failure modes, and continuously retrain models to fix them. Over the last five months, we have been working with them on the infrastructure question, because the inference stack is what determines whether the loop can close fast enough to matter. The rest of this post is about what we have built together, and the road ahead. Why inference is different in this setting Inference is built around a model that holds still. Quantization, speculative decoding, KV cache reuse, prefix caching, every meaningful optimization in modern serving assumes that the weights are stable. The model is the constant; the request is the variable. Continual learning relaxes that assumption in two directions at once. The deploy cadence compresses drastically. Checkpoints arrive from training runs hourly. The pipeline from a freshly-trained model to a live endpoint has to be measured in minutes; anything longer breaks the feedback loop that the training side depends on. The number of distinct models multiplies. A single product can run a different model per customer, per environment, per experiment. The serving fleet routes thousands of subtly different variants rather than a handful of base models, and each variant needs the same first-class treatment: auth, routing, observability, versioning. What we've built with Trajectory Continual learning compresses every step between training and serving. A new adapter coming off a training run has minutes, not hours, before the signal that prompted the retrain goes stale. The pipeline we built with Trajectory takes a LoRA adapter from a finished training run and puts it behind a live OpenAI-compatible endpoint at model-{id}.api.trajectory.ai . Each stage looks superficially like normal model deployment, and each one is different under the hood because the assumptions have changed. Checkpoint arrives. A training run finishes and writes a LoRA adapter to training storage. This is the moment the clock starts.

Merge. The adapter is pulled and merged into the base model. A LoRA merge looks like a single matrix multiplication on paper. In modern model families it isn't always. There can be fused projections (where Q, K, V projections share one underlying matrix), MXFP4 quantization (where the merge has to respect block-wise scaling factors), and expert broadcasting in MoE models (where each expert's adapter has to be merged independently) where each imposes different correctness constraints that a naive merge does not respect. In a static deployment, you do this once and ship. Under continual learning, you do it every hour, on every customer's adapter, against a moving target of base model versions. We run architecture-aware correctness checks on every merged checkpoint, because shipping a silently broken model into a training loop poisons the next round of training too.

Packaging. The merged model is packaged with Truss, Baseten's framework for turning a model into a deployable artifact. Truss handles dependencies, runtime spec, and serving configuration. Without it, every new checkpoint would require hand-assembling the same boilerplate; with it, the merged model is one command from being deployable.

Validation against the serving runtime. Baseten's custom serving runtime applies the optimizations that make inference fast in production: quantization narrowed to the layers that survive it, RoPE scaling for extended context, speculative decoding with a matched draft. Each has correctness constraints that not every checkpoint satisfies. Under continual learning, a checkpoint that fails one of these is not a once-a-quarter release blocker; it is an hourly occurrence that has to be caught automatically before deployment, not after a user hits it.

Deployment and A/B test routing. The validated artifact is deployed onto Baseten's distributed network and routed at model-{id}.api.trajectory.ai . Routing is where one of the deeper inversions of continual learning shows up: every new checkpoint is treated as an experimental variant, not a promotion. In static-model serving, a new version replaces the old one. Under continual learning, the new version is a hypothesis, and the old version is its control. Our routing layer splits traffic between the two, partitions telemetry so the training pipeline can compare them on live production load, and only promotes when the data says to. The same routing layer also handles multi-tenancy: every model, route, and API key is scoped to a customer, with isolation inherited from the data model rather than bolted onto the serving layer.

Live with provenance. Once the endpoint is live, every request it serves is logged with full provenance: which model version answered it, deployed when, from which training run. This is the part of the pipeline that closes the loop. Inference is not the end state of a continually-trained model; it is the...