NVIDIA/TensorRT-LLM v1.2.0rc5

v1.2.0rc5

Repository: NVIDIA/TensorRT-LLM

Tag: v1.2.0rc5

Published: 2025-12-01T06:35:45Z

Prerelease: yes

Release notes:

Announcement Highlights

• Vulnerability
• Two security vulnerabilities have been identified in the urllib3 package versions >= 1.24 and < 2.6.0. These issues will be addressed in the next release. For detailed information on the vulnerabilities, refer to the following advisories:

GHSA-gm62-xv2j-4w53 GHSA-2xpw-w6gg-jr37 To mitigate the issues immediately, users are advised to upgrade urllib3 to version 2.6.0 or later.

• Model Support
• Slimmed down implementation of Nemotron H (#9235)
• Add support Starcoder2 PyTorch backend (#8923)
• Add support MLA chunked prefill for DeepSeek V3.2 model (#9376)
• Add support AutoDeploy Nemotron-Flash (#9504)
• AutoDeploy: Add support Llama4 MoE handling (#9556)
• Add support for nano-v3 and super-v3 with PyTorch backend (#9261)
• AutoDeploy: Add support for nano v3 to custom implementation (#9465)

• API
• Add revision option to trtllm commands (#9498)
• Add support to override env vars in LLM API (#9104)
• Support Response API for general purpose (#9392)

• Feature
• Add support for KVCache reuse for DeepSeek V3.2 (#9383)
• Support Yarn on QwQ-32B model (#9059)
• Update DeepGEMM to include optimizations for DeepSeek-v3.2 (#9380)
• Cold L2 cache when doing autotune benchmarking (#8779)
• Improve TRTLLM MoE throughput for small hidden size (#9377)
• Add parser to layer-wise benchmarks (#9440)
• Support custom chat template for tool calling (#9297)
• Add draft token tree runtime on CDL (#8586)
• Top-p optimization by removing redundant softmax (#9411)
• Use FlashInfer's top_k_sampling_from_probs (#9457)
• Overlap context chunks in pipeline parallel mode (#9308)
• Improve all-to-all perf for large CP size in Helix (#9494)
• Support more accurate AR calculation (#9323)
• Support custom config of sharding (#9143)
• Integrate helix parallelism (#9342)
• Optimize RocketKV algorithm (#9333)
• Extend cute_dsl_nvfp4_gemm to sm103 (#9543)
• Add chat template kwargs support to longbench-v2 (#9544)
• Add Beam Search to TorchSampler (#8509)
• Unify nvfp4 gemm backend (#8963)
• Use FlashInfer.sampling by default (#9545)
• Add RocketKV usage doc and e2e accuracy test on LongBenchV2 (#9572)
• Alias to comply to LlmArgs (#9586)
• Update trtllm-gen nvfp4 kernels with better performance (#9510)
• Enable CuteDSL MoE with Large EP (#9592)
• Convert cuteDSL GEMM to opt-in feature (#9682)
• Optimize the load_weights method to include mapping parameter (#9583)
• Support torch compile for pipeline parallel Llama and DeepSeekV3 (#7838)
• Check if executor is shutdown in /health entrypoint (#9057)
• Add NIXL-LIBFABRIC support (#9225)
• Decouple disagg service from FastAPI (#8714)
• AutoDeploy: Add NVFP4 Cutlass MoE kernels (#9551)
• AutoDeploy: Draft Target Speculative Decoding (#9275)
• AutoDeploy: Support TRTLLM Sampler (#9641)
• AutoDeploy: Perf optimization for Attention and rmsnorm (#9719)
• AutoDeploy: Use router gemm op for Nemotron MOE (#9500)
• AutoDeploy: Remove redundant copies in mamba layers (#9461)
• AutoDeploy: Add A_log fusion for Mamba layers (#9422)
• AutoDeploy: Update dist ops (if not already) (#9301)
• AutoDeploy: Perf optimization entries (if not already in Feature) (#9719)

• Fix
• Modify qwen3-next sampling stop_tokens (#9331)
• Fix mismatched nvfp4 gemm sf shape (#9336)
• Enhance warning in cacheTransBuffer (#9390)
• Fix top-k outIndices with vectorized_process (#9404)
• Let KV cache manager block initialization respect dry run (#9093)
• Avoid cudaFree overlap with cuda graph (#9438)
• Fix TP support for DeepSeek-V3.2 on Hopper (#9484)
• Fix Qwen3-235B ATP accuracy issue with PDL (#9530)
• Correct virtual memory allocation alignment (#9491)
• Fix view operation on uncontiguous tensor (#9576)
• Extract GPU count from single-node stage names (#9599)
• Refine Piecewise Cuda Graph condition for DP (#9393)
• Enhance RPC robustness (#8711)
• Fix synchronization bugs in KvCacheTransferManager preventing corrupted blocks (#9056)
• Fix dist-serving performance by clearing CPU affinity (#9549)
• Fix wide ep MoE error (#9642)
• Fix LoRa enablement for GPT OSS Torch (#8253)
• Recover TRTLLM MoE performance for DEP (#9562)
• Fix error when processing batches containing both text and multimodal data (#8381)
• Fix deepseek_fp8_block_scales using 2D x_sf in TRTLLMGEN-MoE (#9658)
• Enable hmac in RPC (#9745)
• Start disagg workers and servers on free ports (#9694)
• Fix bug: deepseek_fp8_block_scales uses 2D x_sf instead of 1D (#9658)
• AutoDeploy: fix nano sharding config (#9668)
• AutoDeploy: Remove auto-tuner from nvfp4_gemm forward (#9497)

• Documentation
• Fix math formula rendering issues (#9481)
• Qwen3 deployment guide (#9488)
• KV Connector Docs (#9325)
• Deployment Guide for Kimi K2 Thinking on TensorRT LLM - Blackwell (#9711)
• Add feature docs for helix parallelism (#9684)
• Add examples showcasing OpenAI compatible APIs (#9520)
• Update Linux installation guide (#9485)
• Refine the slurm examples (#9548)
• Link to modelopt checkpoints in quick start guide (#9571)

• Test & Infra
• Rename AlltoAll backend names (#9329)
• Move build config from BaseLlmArgs to TrtLlmArgs (#9249)
• Reduce nested nvtx ranges (#9347)
• Add disagg and wideep multi-node multi-gpu test cases (#9356)
• Upgrade CuteDSL to 4.3.0 (#9444)
• Use flexcache for gh200 nodes (#9405)
• Evaluate helix parallelism with DSV3 Lite (#9597)
• AutoDeploy update cuda stream manager for multi-device (#9575)
• Add container notices and documentation (#9185)
• Increase warmup times in multi-gpu testing (#9578)

What's Changed

• [#9316][feat] AutoDeploy: Add the accuracy test for Nemotron MOE models by @nvchenghaoz in https://github.com/NVIDIA/TensorRT-LLM/pull/9317
• [#9096][feature] Auto Deploy: configurable fused MoE backend by @nzmora-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/9194
• [None][fix] Use fp32 for indexer weight_proj GEMM by @chang-l in https://github.com/NVIDIA/TensorRT-LLM/pull/9243
• [None][fix] Multimodal InputProcessor dummy builder fix by @yechank-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/8916
• [None][ci] waive test_disagg_server_restart by @QiJune in https://github.com/NVIDIA/TensorRT-LLM/pull/9326
• [None][chore] Revise the description of enable_autotuner. by @hyukn in https://github.com/NVIDIA/TensorRT-LLM/pull/9320
• [TRTLLM-9295][fix] use greedy decoding in test_openai_compatible_json_schema by @ixlmar in...