NVIDIA/TensorRT-LLM v1.0.0

v1.0.0

Repository: NVIDIA/TensorRT-LLM

Tag: v1.0.0

Published: 2025-09-24T12:53:50Z

Prerelease: no

Release notes:

TensorRT LLM Release 1.0

TensorRT LLM 1.0 brings 2 major changes: the PyTorch-based architecture is now stable and the default experience, and the LLM API is now stable. For more details on new developments in 1.0, please see below.

Key Features and Enhancements

• Model Support
• Add Mistral3.1 VLM model support
• Add TensorRT-Engine Qwen3 (dense) model support
• Add phi-4-multimodal model support
• Add EXAONE 4.0 model support
• Add Qwen3 MoE support to TensorRT backend

• Features
• Add support for sm121
• Add LoRA support for Gemma3
• Support PyTorch LoRA adapter eviction
• Add LoRA support for PyTorch backend in trtllm-serve
• Add support of scheduling attention dp request
• Remove padding of FusedMoE in attention DP
• Support torch compile for attention dp
• Add KV events support for sliding window attention
• Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE
• Add Piecewise CUDA Graph support for MLA
• Support mutliCtasKvMode for high-throughput MLA kernels
• Enable kvcache to be reused during request generation
• Add ADP schedule balance optimization
• Add chunked prefill support for MLA (Blackwell)
• Enable Multi-block mode for Hopper spec dec XQA kernel
• Add vLLM KV Pool support for XQA kernel
• Allow sending more than 2GiB through MPI by using mpi4py.util.pkl5
• Add support for fused gate_up_proj scales for FP8 blockwise
• Support FP8 row-wise dense GEMM in torch flow
• Enable fp8 SwiGLU to minimize host overhead
• Add Deepseek R1 FP8 Support on Blackwell
• Add support for MXFP8xMXFP4 in pytorch
• Support nvfp4 model and fp8 kv cache for MLA chunked prefill (Blackwell)
• Opensource MOE MXFP8-MXFP4 implementation
• Add support for Modelopt fp8_pb_wo quantization scheme
• Support deepEP fp4 post quant all2all dispatch
• Fuse w4a8 moe pre-quant scale on Hopper
• Support Weight-Only-Quantization in PyTorch Workflow
• Add support for per expert activation scaling factors
• Add ReDrafter support for Qwen
• Enable CUDA Graph for Nemotron-H
• Add support for YARN in NemotronNAS models
• Switch to internal version of MMProjector in Gemma3
• Disable add special tokens for Llama3.3 70B
• Auto-enable ngram with concurrency 1 (#6447)
• Fix wide EP when using DeepEP with online EPLB (#6429)
• Fix bugs caused by None attention_bias during Qwen3 model convert engine (#6344)
• Fix PD + MTP + overlap scheduler accuracy issue (#6136)
• Fix bug of Qwen3 when using fp4 on sm120 (#6065)
• Fix TMA error with GEMM+AR on TP=2 (#6075)
• Fix scaffolding aime test in test_e2e (#6140)
• Fix KV Cache overrides in trtllm-bench (#6103)
• Fix MOE benchmark to rotate buffers to prevent L2 cache reuse (#4135)
• Fix eagle3 two model disaggregated serving test (#6014)
• Fix chunked prefill + overlap scheduling (#5761)
• Fix mgmn postprocess error (#5835)
• Fallback to cubins for fp8 fmha kernels on Ada (#5779)
• Fix disagg + speculative decoding (#5558)
• Fix test_generate_with_seed CI failure. (#5772)
• Fix prompt adapter TP2 case (#5782)
• Fix disaggregate serving with attention DP (#4993)
• Fix a quote error introduced in #5534 (#5816)
• Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. (#5801)
• Fix lost requests for disaggregated serving (#5815)
• Update unit tests: skip all_close assert for dropout in attention, increase tolerance for rope op test (#5855)
• Fix GEMM+AR fusion on blackwell (#5563)
• Fix llama4 multimodal support (#5809)
• Fix Llama4 Scout FP4 crash issue (#5925)
• Fix max batch size and max tokens in kv cache estimations for Nemotron-H (#5371)
• Fix moe regression for sm120 (#5823)
• Fix Qwen2.5VL FP8 support (#5029)
• Fix the illegal memory access issue in moe gemm on SM120 (#5636)
• Fix tileN cannot % 16==0 & support sm89 deepgemm bmm (#5531)
• Fix incremental detokenization (#5825)
• Fix MoE workspace info by storing Torch tensor itself instead of data_ptr (#5900)
• Fix mistral unit tests due to transformers upgrade (#5904)
• Fix the Llama3.1 405B hanging issue. (#5698) (#5925)
• Fix Gemma3 unit tests due to transformers upgrade (#5921)
• Fix alltoall for llama4 (apply_router_weight_on_input=True) (#5902)
• Remove SpecConfig and fix thread leak issues (#5931)
• Fast redux detection in trtllm gen routing kernel (#5941)
• Fix cancel request logic (#5800)
• Fix errors in wide-ep scripts (#5992)
• Fix error in post-merge-tests (#5949)
• Fix missing arg to alltoall_prepare_maybe_dispatch (#5669)
• Fix attention DP doesn't work with embedding TP (#5642)
• Fix broken cyclic reference detect (#5417)
• Fix permission for local user issues in NGC docker container. (#5373)
• Fix mtp vanilla draft inputs (#5568)
• Fix mPtrExpertCounts allocation in MoE TRT-LLM backend (nvfp4) (#5519)
• Fix block scale fp8 support for deepseek v3 on Blackwell. (#5514)
• Fix the issue MoE autotune fallback failed to query default heuristic (#5520)
• Fix the unexpected keyword argument 'streaming' (#5436)

Known Issues

• When using disaggregated serving with pipeline parallelism and KV cache reuse, a hang can occur. This will be fixed in a future release. In the meantime, disabling KV cache reuse will fix this issue.
• Running multi-node cases where each node has just a single GPU is known to fail. This will be addressed in a future release.
• For the Llama 3.x and Llama 4 models, there is an issue with pipeline parallelism when using FP8 and NVFP4 weights. As a workaround, you can set the environment variable export TRTLLM_LLAMA_EAGER_FUSION_DISABLED=1.

What's Changed

• Qwen3: Fix eagle hidden states by @IzzyPutterman in https://github.com/NVIDIA/TensorRT-LLM/pull/6199
• [None][fix] Upgrade dependencies version to avoid security vulnerability by @yibinl-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/6506
• [None][chore] update readme for perf release test by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/6664
• [None][test] remove trt backend cases in release perf test and move NIM cases to llm_perf_nim.yml by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/6662
• [None][fix] Explicitly add tiktoken as required by kimi k2 by @pengbowang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/6663
• [None][doc]: remove the outdated features which marked as Experimental by @nv-guomingz in https://github.com/NVIDIA/TensorRT-LLM/pull/5995
• [https://nvbugs/5375966][chore] Unwaive...