Ai Training Vs Inference

AI training vs. inference: what's the difference? Announcing our Series F . Learn more

Model performance

AI training vs. inference: what's the difference?

AI training teaches models to learn from data. Inference is what runs in production. Learn the key differences in hardware, cost, and optimization.

Authors

Chloe Florit

Last updated June 25, 2026

Share

TL;DR Training is when a model learns from data until it can do tasks like write code, answer questions, or generate images. Inference is what comes after: every time a user sends a request, the trained model generates a response. The two phases have different hardware needs, optimization techniques, and costs. This post walks through the model lifecycle from pretraining to serving, the four metrics that define inference performance (TTFT, TPOT, throughput, and latency), and the key technical differences between training and inference.

AI training is how a model learns: adjusting its weights on massive datasets until it can write code, answer questions, or generate images. AI inference is what comes after: the trained model generates outputs on new data it's never seen. Imagine hiring a scholar to explain Shakespeare to you. First, they spend years in university reading plays, discussing themes, and debating different interpretations. That's training. Once they've learned, you can ask them questions any time and get instant answers. That's inference. Most teams will spend far more time on inference than they ever will on training: training happens a limited number of times, and inference happens every time someone uses the model. This post breaks down the difference between the two. Where does AI inference show up? If you've used an AI product, you've triggered inference: When GPT OSS answers a question you asked

When you ask Cursor to write code

When AI flags anomalies in medical conversations and records (e.g., Abridge)

When Notion AI summarizes a meeting or drafts a document

In each situation, a trained model generates an output from a new input. No learning is happening at that moment; the model's weights are frozen. That’s inference. Note: If GPT OSS is deployed behind an API, you can ask it a question by calling that API (via curl in terminal, for example). From training to inference: the model lifecycle A model passes through multiple stages on its way to production. ✕ AI training to inference lifecycle 1. Pre-training This is where the model sees enormous amounts of data and learns patterns and relationships between inputs and outputs. Specifically, it runs a forward pass to generate predictions, computes loss to determine how wrong it was, then uses backpropagation to calculate how to update the weights to improve its responses. This process is repeated until the model captures broad knowledge about language, code, the world, or whatever data it's being trained on. 2. Post-training (fine-tuning) Post-training takes a pre-trained model and adjusts its weights so it can perform a specific task, using a specialized dataset. Imagine Baseten wants a customer support bot to quickly and accurately answer support tickets. A powerful LLM might not know the details of the products or specific terminology. So we would fine-tune the model on past support tickets and ideal responses. The results would be a model that knows how to respond in Baseten's voice, understands product-specific terminology, and can address common customer issues. Post-training works well in use cases where domain expertise is required. For example, Baseten Research recently partnered with Harvey on legal AI and post-trained Qwen3.5-27B against Harvey's Legal Agent Benchmark (LAB) , which covers real legal tasks graded by expert-written rubrics. As a result, a 27B open-weight model became competitive with frontier closed-source models at a fraction of the cost. 3. Optimization Once training is done, the model gets transformed for the target hardware through quantization and compilation: model weights are converted into an optimized format for a specific GPU or accelerator. This is where raw model artifacts become something that can run fast in production. 4. Deployment Deployment means setting up the infrastructure: allocating GPUs, setting up an API endpoint, and configuring autoscaling so the system adds GPUs when requests spike and scales down when traffic drops. GPUs are designed to run many calculations in parallel, which makes them great for the math that powers AI models. Once the model is loaded onto that hardware, the API endpoint provides a URL that applications can call to send inputs and receive model outputs. 5. Serving Serving is where live requests are handled in production, meeting speed and uptime commitments. This involves receiving incoming prompts, running them through the model, and streaming or returning the generated output. Optimizations like batching requests and caching common outputs help maximize throughput (how many requests the system can handle at once) and minimize latency (how long it takes to get a response for a single request). This is the phase users actually experience. How do you measure the success of inference? When you're running models in production, accuracy isn't enough. Users care about how fast the model responds, and you care about how scalable the system is: can your infrastructure handle many requests without slowing down response speed for individual users? Four metrics tell the inference story: Time to first token (TTFT) measures how quickly users see something after sending a request. High TTFT makes an app feel frozen or unresponsive. Even if the full answer arrives quickly afterward, that initial pause is what users remember. Time per output token (TPOT) measures the gap between each subsequent token. This is what makes streaming feel smooth or choppy. High TPOT means text trickles out in stutters instead of flowing. Throughput measures the number of tokens the system generates per second across all requests. It's a measure of system-level capacity, not individual response speed. Low throughput means the system can't scale to serve more users. Latency measures the full request-to-response time for a single request. This is the top-line SLA metric: does your app meet its speed requirements? On Baseten, latency is logged for every request. Dedicated deployments track TPOT (time per output token, or inter-token...