NVIDIA/TensorRT-LLM v1.2.0rc6

v1.2.0rc6

Repository: NVIDIA/TensorRT-LLM

Tag: v1.2.0rc6

Published: 2025-12-23T06:48:47Z

Prerelease: yes

Release notes:

Highlights

• Model Support
• Add DeepSeek-V3.2 to the supported models (#9893)
• Add llama4 scaling (#9771)

Support Mistral Large3 LLM part (#9820)

• API
• Add docs and examples for Responses API (#9946)
• Add eos_token_id in generation_config to sampling params (#9514)
• Update nspect version for api change (#9899)

• Feature
• 2D parallel EP TP support (#9459)
• Fused kernels (qknormrope + moe routing) and two-model MTP support for glm4moe (#9852)
• Add gather fc1 kernel by cuteDSL (#9618)
• Add GB300 support since it does not support segment (#9731)
• Add helixPostProcessNative kernel for cp_dim=2 (#9924)
• Added symetric memory AllReduce strategy (#8919)
• ConfigurableMoE support (#9772, #9858)
• Enable multistream for Linear Attention in Qwen3 (#9696)
• Enable PDL for indexer topK (#9843)
• Implement distributed tuning system (#9621)
• Implement sampling on 1-model EAGLE3 (#9885)
• Move D->H copies to a worker thread (#8463)
• Optimize the host overhead of _sample_async (#9935)
• Port fp4 quantization kernel optimization from FlashInfer (#9854)
• Support larger topK for NVLinkOneSided AlltoAll. (#9816)

• Fix
• Fix CUDA stream sync issue in ModelRunnerCPP (#6426)
• Fix accuracy issue in TRTLLM MoE (#9999)
• Fix PDL in TRTLLM MOE for dsv3 (#9799)
• Fix unterminated process issue for RemoteOpenAIServer (#9490)
• Fix PDL bugs with trtllm-gen fmha kernels (#9863)
• Use first PP rank's schedule result in other PP ranks to fix PP hang (#9659)

• Documentation
• Add config db and docs (#9420)
• Update doc for NVFP4 KV cache (#9475)
• Update documents for GB300 NVL72 (#9987)
• Update wide EP documents (#9724)

• Test & Infra
• Add Test Cases (#9506, #9833, #9881, #9415, #8800, #9511, #8716, #9886, #9645)
• Load balance decode token KV cache with helix parallelism (#9757)
• Update disagg benchmarking scripts to support context parallelism (#9720)
• Upgrade UCX to 1.20 (#9786, #9977)
• Upgrade NIXL to v0.8.0 (#9707)

What's Changed

• [https://nvbugs/5703953][fix] Preserving ip:port for trtllm-serve before initializing llm by @JunyiXu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9646
• [None][infra] Waive failed cases for main branch on 12/07 by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/9769
• [None][fix] Several minor fixes to CI setting by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/9765
• [OMNIML-3036][doc] Re-branding TensorRT-Model-Optimizer as Nvidia Model-Optimizer by @cjluo-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9679
• [None][feat] Enable NCCL_SYMMETRIC as default fallback for AllReduce by @nv-lschneider in https://github.com/NVIDIA/TensorRT-LLM/pull/9314
• [TRTLLM-9000][feat] Add multi-node Perf Tests into CI by @chenfeiz0326 in https://github.com/NVIDIA/TensorRT-LLM/pull/8800
• [None][test] add ntp tolerance in time metrics verification by @zhengd-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9741
• [TRTLLM-9603][feat] Enable ConfigurableMoE test in the CI by @xxi-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9645
• [https://nvbugs/5422621][test] Add GB 200 WIDEEP test case for RCCA 5422621 by @fredricz-20070104 in https://github.com/NVIDIA/TensorRT-LLM/pull/9506
• [None][fix] Fix two tuning cache miss issues. by @hyukn in https://github.com/NVIDIA/TensorRT-LLM/pull/9743
• [TRTLLM-9706] [doc] Update wide EP documents by @kaiyux in https://github.com/NVIDIA/TensorRT-LLM/pull/9724
• [https://nvbugs/5666804][test] only adding sampler config for limited models by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/9512
• [None][infra] Waive failed cases for main on 12/08 by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/9773
• [None][chore] Move the rocketkv e2e test to post-merge by @lfr-0531 in https://github.com/NVIDIA/TensorRT-LLM/pull/9768
• [None][chore] Enable tvm_ffi for cute dsl nvfp4_gemm to reduce host overhead. by @limin2021 in https://github.com/NVIDIA/TensorRT-LLM/pull/9690
• [TRTLLM-9431][perf] Enable multistream for Linear Attention in Qwen3-… by @nv-guomingz in https://github.com/NVIDIA/TensorRT-LLM/pull/9696
• [None][chore] Remove closed bugs by @xinhe-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9770
• [None][infra] update mooncake in docker images by @zhengd-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9584
• [None][test] Add Kimi k2 WIDEEP perf and accuracy cases by @fredricz-20070104 in https://github.com/NVIDIA/TensorRT-LLM/pull/9686
• [https://nvbugs/5527655][test] Add test case for RCCA 5527655 by @fredricz-20070104 in https://github.com/NVIDIA/TensorRT-LLM/pull/9511
• [http://nvbugs/5649010][fix] fix test_auto_scaling.py::test_worker_restart timeout by @reasonsolo in https://github.com/NVIDIA/TensorRT-LLM/pull/9775
• [None][fix] Switch AutoDeploy's default allreduce strategy to NCCL by @MrGeva in https://github.com/NVIDIA/TensorRT-LLM/pull/9666
• [TRTLLM-9506][fix] Fix AR for DeepSeek-R1 2 model path by @sunnyqgg in https://github.com/NVIDIA/TensorRT-LLM/pull/9661
• [TRTLLM-9089][chore] Port prepare_dataset into trtllm-bench by @FrankD412 in https://github.com/NVIDIA/TensorRT-LLM/pull/9250
• [https://nvbugs/5567586][feat] Ampere xqa swa specdec for GPT-OSS Eagle3-one-model by @jhaotingc in https://github.com/NVIDIA/TensorRT-LLM/pull/8383
• [TRTLLM-7967][chore] Add more tests by @yibinl-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/9415
• [https://nvbugs/5508267][fix] Proper handling of inactive canceled requests by @thorjohnsen in https://github.com/NVIDIA/TensorRT-LLM/pull/9280
• [#8921][feat] Added symetric memory AllReduce strategy by @MrGeva in https://github.com/NVIDIA/TensorRT-LLM/pull/8919
• [None][fix] Fix #8383 introduced TRTLLM backend python error by @jhaotingc in https://github.com/NVIDIA/TensorRT-LLM/pull/9804
• [#9753][feat] AutoDeploy: Implement add rms_norm fusion by @nvchenghaoz in https://github.com/NVIDIA/TensorRT-LLM/pull/9754
• [None][infra] Correct the waived test names due to a merge conflict by @yuanjingx87 in https://github.com/NVIDIA/TensorRT-LLM/pull/9803
• [None][fix] Fix PDL in TRTLLM MOE for dsv3 by @dmtri35 in https://github.com/NVIDIA/TensorRT-LLM/pull/9799
• [None][feat] Add llama4 scaling by @byshiue in https://github.com/NVIDIA/TensorRT-LLM/pull/9771
• [https://nvbugs/5677746][fix] Use first PP rank's schedule result in other PP ranks to fix PP hang by @jiaganc in...