basetenlabs/truss

basetenlabs/truss

Description: The simplest way to serve AI/ML models in production

Language: Python

License: MIT

Stars: 1162

Forks: 109

Open issues: 75

Created: 2022-07-06T05:39:37Z

Pushed: 2026-06-10T23:22:44Z

Default branch: main

Fork: no

Archived: no

README:

Truss

The simplest way to serve AI/ML models in production

 

Truss is the CLI for deploying and serving ML models on Baseten. Package your model's serving logic in Python, launch training jobs, and deploy to production—Truss handles containerization, dependency management, and GPU configuration.

Truss lets you serve models with the Baseten Inference Stack as well as deploy models from any open-source framework: vLLM, SGLang, TensorRT-LLM, transformers, diffusers, PyTorch, TensorFlow, and more.

[Get started](https://docs.baseten.co/examples/deploy-your-first-model) | 100+ examples | Documentation

Why Truss?

• Write once, run anywhere: Package model code, weights, and dependencies with a model server that behaves the same in development and production.
• Fast developer loop: Iterate with live reload, skip Docker and Kubernetes configuration, and use a batteries-included serving environment.
• Support for all Python frameworks: From transformers and diffusers to PyTorch and TensorFlow to vLLM, SGLang, and TensorRT-LLM, Truss supports models created and served with any framework.
• Production-ready: Built-in support for GPUs, secrets, caching, and autoscaling when deployed to Baseten or your own infrastructure.

Installation

Install Truss with:

pip install --upgrade truss

Quickstart

Deploying a model to Baseten via Truss turns a Hugging Face model into a production-ready API endpoint. You write a config.yaml that specifies the model, the hardware, and the engine, then uvx truss push builds a TensorRT-optimized container and deploys it. No Python code, no Dockerfile, no container management.

This guide walks through deploying Qwen 2.5 3B Instruct, a small but capable LLM, from a config file to a production API. You'll set up Truss, write a config, deploy to Baseten, call the model's OpenAI-compatible endpoint, and promote to production.

Set up your environment

Before you begin:

• Sign up or sign in to Baseten.
• Install uv, a fast Python package manager. This guide uses uvx to run Truss commands without a separate install step.

Authenticate with Baseten

Generate an API key from Settings > API keys, then log in:

uvx truss login

Paste your API key when prompted:

💻 Let's add a Baseten remote!
🤫 Quietly paste your API_KEY:

You can skip the interactive prompt by setting BASETEN_API_KEY as an environment variable:

export BASETEN_API_KEY="paste-your-api-key-here"

Create a Truss project

Scaffold a new project:

uvx truss init qwen-2.5-3b && cd qwen-2.5-3b

When prompted, name the model Qwen 2.5 3B.

? 📦 Name this model: Qwen 2.5 3B
Truss Qwen 2.5 3B was created in ~/qwen-2.5-3b

This creates a directory with a config.yaml, a model/ directory, and supporting files. For engine-based deployments like this one, you only need config.yaml. The model/ directory is for [custom Python code](/examples/customize-a-model) when you need custom preprocessing, postprocessing, or unsupported model architectures.

Write the config

Replace the contents of config.yaml with:

model_name: Qwen-2.5-3B
resources:
accelerator: L4
use_gpu: true
trt_llm:
build:
base_model: decoder
checkpoint_repository:
source: HF
repo: "Qwen/Qwen2.5-3B-Instruct"
max_seq_len: 8192
quantization_type: fp8
tensor_parallel_count: 1

That's the entire deployment specification.

• model_name identifies the model in your Baseten dashboard.
• resources selects an L4 GPU (24 GB VRAM), which is plenty for a 3B parameter model.
• trt_llm tells Baseten to use [Engine-Builder-LLM](/engines/engine-builder-llm/overview), which compiles the model with TensorRT-LLM for optimized inference.
• checkpoint_repository points to the model weights on Hugging Face. Qwen 2.5 3B Instruct is ungated, so no access token is needed.
• quantization_type: fp8 compresses weights to 8-bit floating point, cutting memory usage roughly in half with negligible quality loss.
• max_seq_len: 8192 sets the maximum context length for requests.

---

Deploy

Push the model to Baseten:

We'll start by deploying in development mode so we can iterate quickly:

uvx truss push --watch

You should see:

✨ Model Qwen 2.5 3B was successfully pushed ✨

Model ID: abc1d2ef
Deployment ID: xyz123
Endpoint: https://model-abc1d2ef.api.baseten.co
Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123

👀 Watching for changes to truss...

You'll need the model ID to call the model's API. You can also find it in your Baseten dashboard.

Baseten now downloads the model weights from Hugging Face, compiles them with TensorRT-LLM, and deploys the resulting container to an L4 GPU. You can watch progress in the logs linked above.

Call the model

Engine-based deployments serve an OpenAI-compatible API. Once the deployment shows "Active" in the dashboard, call it using the OpenAI SDK or cURL. Replace {model_id} with your model ID from the deployment output.

Install the OpenAI SDK if you don't have it:

uv pip install openai

Create a chat completion:

import os
from openai import OpenAI

client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/development/sync/v1",
)

response = client.chat.completions.create(
model="Qwen-2.5-3B",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)

print(response.choices[0].message.content)…