novitalabs/sglang

novitalabs/sglang

Description: SGLang is a fast serving framework for large language models and vision language models.

Language: Python

License: Apache-2.0

Stars: 6

Forks: 1

Open issues: 0

Created: 2025-10-27T07:20:36Z

Pushed: 2026-05-22T06:35:23Z

Default branch: main

Fork: yes

Parent repository: sgl-project/sglang

Archived: no

README:

--------------------------------------------------------------------------------

Blog | Documentation | Roadmap | Join Slack | Weekly Dev Meeting | Slides

News

• [2026/01] 🔥 SGLang Diffusion accelerates video and image generation (blog).
• [2025/12] SGLang provides day-0 support for latest open models (MiMo-V2-Flash, Nemotron 3 Nano, Mistral Large 3, LLaDA 2.0 Diffusion LLM, MiniMax M2).
• [2025/10] 🔥 SGLang now runs natively on TPU with the SGLang-Jax backend (blog).
• [2025/09] Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput (blog).
• [2025/09] SGLang Day 0 Support for DeepSeek-V3.2 with Sparse Attention (blog).
• [2025/08] SGLang x AMD SF Meetup on 8/22: Hands-on GPU workshop, tech talks by AMD/xAI/SGLang, and networking (Roadmap, Large-scale EP, Highlights, AITER/MoRI, Wave).

More

• [2025/11] SGLang Diffusion accelerates video and image generation (blog).
• [2025/10] PyTorch Conference 2025 SGLang Talk (slide).
• [2025/10] SGLang x Nvidia SF Meetup on 10/2 (recap).
• [2025/08] SGLang provides day-0 support for OpenAI gpt-oss model (instructions)
• [2025/06] SGLang, the high-performance serving infrastructure powering trillions of tokens daily, has been awarded the third batch of the Open Source AI Grant by a16z (a16z blog).
• [2025/05] Deploying DeepSeek with PD Disaggregation and Large-scale Expert Parallelism on 96 H100 GPUs (blog).
• [2025/06] Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput (blog).
• [2025/03] Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X (AMD blog)
• [2025/03] SGLang Joins PyTorch Ecosystem: Efficient LLM Serving Engine (PyTorch blog)
• [2025/02] Unlock DeepSeek-R1 Inference Performance on AMD Instinct™ MI300X GPU (AMD blog)
• [2025/01] SGLang provides day one support for DeepSeek V3/R1 models on NVIDIA and AMD GPUs with DeepSeek-specific optimizations. (instructions, AMD blog, 10+ other companies)
• [2024/12] v0.4 Release: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs (blog).
• [2024/10] The First SGLang Online Meetup (slides).
• [2024/09] v0.3 Release: 7x Faster DeepSeek MLA, 1.5x Faster torch.compile, Multi-Image/Video LLaVA-OneVision (blog).
• [2024/07] v0.2 Release: Faster Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) (blog).
• [2024/02] SGLang enables 3x faster JSON decoding with compressed finite state machine (blog).
• [2024/01] SGLang provides up to 5x faster inference with RadixAttention (blog).
• [2024/01] SGLang powers the serving of the official LLaVA v1.6 release demo (usage).

About

SGLang is a high-performance serving framework for large language models and multimodal models. It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters. Its core features include:

• Fast Runtime: Provides efficient serving with RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization (FP4/FP8/INT4/AWQ/GPTQ), and multi-LoRA batching.
• Broad Model Support: Supports a wide range of language models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), reward models (Skywork), and diffusion models (WAN, Qwen-Image), with easy extensibility for adding new models. Compatible with most Hugging Face models and OpenAI APIs.
• Extensive Hardware Support: Runs on NVIDIA GPUs (GB200/B300/H100/A100/Spark), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more.

-…