NVIDIA/TensorRT-Edge-LLM

NVIDIA/TensorRT-Edge-LLM

Description: High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI

Language: Python

License: Apache-2.0

Stars: 528

Forks: 120

Open issues: 80

Created: 2025-10-02T17:40:17Z

Pushed: 2026-08-18T08:10:11Z

Default branch: main

Fork: no

Archived: no

README:

TensorRT Edge-LLM

High-Performance Large Language Model Inference Framework for NVIDIA Edge Platforms

Overview | Support Matrix | Quick Start | Performance | Documentation | Roadmap

---

Latest News

• [2026/08] TensorRT Edge-LLM 0.10.0 adds Day-0 support for **Qwen3.8-27B**.
• [2026/08] Release 0.10.0 adds support for **NVIDIA Nemotron-3.5 Lightning** with MTP and DFlash, **Cosmos3-Edge**, **DiffusionGemma**, **Nemotron-3.5-ASR**, and [DSpark](docs/source/user_guide/examples/speculative-decoding.md#dspark) speculative decoding, alongside an experimental [direct TensorRT engine builder](docs/source/user_guide/getting_started/direct-engine-builder.md) without ONNX export, multi-turn KV-cache reuse, and video input for the experimental OpenAI-compatible server.
• [2026/07] Support for the full Gemma 4 family (E2B / E4B / 12B / 26B-A4B / 31B — multimodal text + image + audio, with MTP), Qwen3-Omni and Nemotron-3 NVFP4, and DFlash speculative decoding (with DDTree for Qwen3 / Qwen3.5) landed across releases 0.9.0 and 0.9.1.

---

Overview

TensorRT Edge-LLM is NVIDIA's C++ inference runtime for text, vision, audio, speech, and action models on NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark. The supported frontend exports Hugging Face checkpoints to ONNX for C++ engine building; an experimental direct frontend builds engines from checkpoints without ONNX. Both paths use the same C++ deployment runtimes.

---

Getting Started

Check the **Official Support Matrix**, then follow the **Quick Start Guide**. Checkpoint IDs are listed in **Supported Models**.

---

Documentation

Introduction

• [Overview](https://nvidia.github.io/TensorRT-Edge-LLM/latest/overview.html) - What is TensorRT Edge-LLM and key features
• [Official Support Matrix](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/support-matrix.html) - Platform, JetPack, DriveOS, CUDA, TensorRT, and TensorRT Edge-LLM compatibility
• [Supported Models](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/supported-models.html) - Complete model compatibility matrix
• [Checkpoint Exporter](https://nvidia.github.io/TensorRT-Edge-LLM/latest/developer_guide/software-design/checkpoint-export.html) - Recommended ONNX export pipeline
• [Experimental Direct Engine Builder](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/direct-engine-builder.html) - Build all model components directly from a checkpoint

User Guide

• [Installation](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/installation.html) - Set up quantization, tensorrt_edgellm, and the C++ runtime
• [Quick Start Guide](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/quick-start-guide.html) - Run your first inference in ~15 minutes
• [Examples](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/examples/index.html) - End-to-end workflows
• [Quantization](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/features/quantization.html) - Create quantized checkpoints for tensorrt_edgellm
• [Experimental High-Level Python API and Server](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/examples/experimental-server.html) - vLLM-style API and OpenAI-compatible server
• [Input Format Guide](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/format/input-format.html) - Request format and specifications
• [Chat Template Format](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/format/chat-template-format.html) - Chat template configuration

Developer Guide

Software Design

• [Quantization Package Design](https://nvidia.github.io/TensorRT-Edge-LLM/latest/developer_guide/software-design/quantization-design.html) - Quantization package architecture
• [Engine Builder](https://nvidia.github.io/TensorRT-Edge-LLM/latest/developer_guide/software-design/engine-builder.html) - Building TensorRT engines
• [C++ Runtime Overview](https://nvidia.github.io/TensorRT-Edge-LLM/latest/developer_guide/software-design/cpp-runtime-overview.html) - Runtime system architecture
• LLM Inference Runtime

Advanced Topics

• [Customization Guide](https://nvidia.github.io/TensorRT-Edge-LLM/latest/developer_guide/customization/customization-guide.html) - Customizing TensorRT Edge-LLM for your needs
• [TensorRT Plugins](https://nvidia.github.io/TensorRT-Edge-LLM/latest/developer_guide/customization/tensorrt-plugins.html) - Custom plugin development
• [Tests](tests/) - Comprehensive test suite for contributors

---

Performance

See the **Performance Benchmarks** page for released benchmark results covering LLM and...