NVIDIA/TensorRT-LLM v1.2.0rc3

Release v1.2.0rc3

Repository: NVIDIA/TensorRT-LLM

Tag: v1.2.0rc3

Published: 2025-11-21T08:35:56Z

Prerelease: yes

Release notes:

Announcement Highlights

• Model Support
• Enable Nemotron H MoE sharding (#8744)
• Support Latent MOE for Nemotron (#8955)
• Add TP support for DeepSeek-V3.2 (#8943)
• Support Glm4MoeForCausalLM (#8256)
• Add support for disagg in DSv3.2 (#8735)
• Add tool call parsing fixes and Qwen3 coder parser (#8817)

• API
• Add trtllm_ prefix for exposed metrics (#8845)
• Return logprobs incrementally in torch backend (#8785)
• Enable n > 1 in OpenAI API with PyTorch backend (#8951)
• Support json_schema in response_format (#8934)
• Add TRTLLM_NIXL_KVCACHE_BACKEND environment variable for NIXL backend selection (#9075)
• Prevent negative max_tokens passed into tllm request (#9037)

• Feature
• Fuse QK down_proj with indexer K + weight_proj for FP4 ckpt (#8771)
• Add swapsMmaAb sparseMla kernels (#8913)
• Implement Deep Research with scaffolding (#8452)
• Add rope and uk-bgemm overlap for MLA generation (#8495)
• Add NUMA-aware CPU affinity autoconfig (#8805)
• Add custom indexer k cache scatter op (#8960)
• Allow env variable to specify spawn process IPC address (#8922)
• Implement sampling using FlashInfer.sampling (#8581)
• Enhance the overlap scheduler for two-model spec decoding (#8706)
• Update TRTLLM Cutlass MoE kernels with ReLU2 (#9011)
• Unify MPI & Ray's req/response handling with RPC Client/Server (#8765)
• Use triton kernels for RocketKV prediction module (#8682)
• Support accuracy test and install from wheel (#9038)
• Add tree attention support for blackwell arch (#8975)
• Add simple optimizations for MTP 2-model (#9176)
• Enable early exit with overlap scheduler (#8587)
• Add dynamic draft length in spec decode (stage 1) (#8194)
• Add bias for FP4 TRT-LLM Gen MoE (#9220)
• Integrate CuteDSL NVFP4 grouped GEMM (#8880)
• Add ability to cancel disagg request if KV cache resources are exhausted (#9155)
• Make factory sharding the default (#9144)
• Enable simple sharding for latent experts (#9099)
• Update the indexer topK (#9255)
• Add fp8 dense for sm120 (#9174)
• Add specdec to nemotron nas (#8985)
• Use CUDAGraph to improve the tuning accuracy for AutoTuner (#9089)
• Add ReLU2 to TRTLLM Cutlass MoE BF16 kernels (#9191)
• Add pp_partition to customize each rank's layer number (#9003)
• Enable EPLB for trtllm-gen and cutlass backend (#8886)
• Add optimized trtllm-gen attention kernels on sm103 (#9081)
• Add MTP>1 support for DS-v3.2 (#9045)

• Benchmark
• Add Qwen3-Next to layer-wise benchmarks (#9065)
• Refactor benchmark infrastructure (#9207)
• Print device info in trtllm-bench report (#8584)
• Use torch.compile to fuse copy + layernorm within the LayerNorm module (#9052)
• Add torch.compile + multi-stream support for k-cache scatter and weight scaling (#8988)
• Adjust select_alltoall_method_type (#8950)

• Documentation
• Replace the relative links with absolute links in README.md (#8995)
• Update llama and llama4 example doc (#9048)
• Update doc/tests/chat_template for nano-v2-vlm (#8840)
• Add Mixed Precision Context and Generation section to Disagg (#8769)
• Add DeepSeek-V3.2-Exp document (#9141)
• Update docs for EPLB (#9166)
• Update the Flux autodeploy example (#8434)
• Update DS-R1 example doc (#9231)
• Update license (#8807)

• Fix & Infra
• Fix the logger once key issue and further compress log in AutoTuner (#8873)
• Fix disagg GPT-OSS test (#8870)
• Remove PyTorchConfig completely (#8856)
• Fix boost issue (#8996)
• Lock onnx version 1 in OpenAI API with PyTorch backend by @ixlmar in https://github.com/NVIDIA/TensorRT-LLM/pull/8951
• [TRTLLM-8999][infra] Reduce gb200 multi-node test stages by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/8778
• [None][infra] Waive failed tests for main 11/07 by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/9008
• [https://nvbugs/5637037][fix] Update unwaive list. by @bobboli in https://github.com/NVIDIA/TensorRT-LLM/pull/9001
• [None][chore] Lock onnx version 1 support for DS-v3.2 by @lfr-0531 in https://github.com/NVIDIA/TensorRT-LLM/pull/9045
• [None][chore] Remove is_disaggregated param in executor request queue by @pcastonguay in https://github.com/NVIDIA/TensorRT-LLM/pull/9049
• [https://nvbugs/5636912][fix] AutoDeploy: Unwaive the test by @nvchenghaoz in https://github.com/NVIDIA/TensorRT-LLM/pull/9018
• [None][feat] Enable EPLB for trtllm-gen and cutlass backend by @dongxuy04 in https://github.com/NVIDIA/TensorRT-LLM/pull/8886
• [None][fix] AutoDeploy: Use tmp folder for the load_moe_align by @nvchenghaoz in https://github.com/NVIDIA/TensorRT-LLM/pull/9101
• [None][ci] waive test_disaggregated_serving.py::TestQwen3_8B::test_chunked_prefill by @QiJune in https://github.com/NVIDIA/TensorRT-LLM/pull/9111
• [TRTLLM-9179][feat] add pp_partition to customize each rank's layer number by @dc3671 in https://github.com/NVIDIA/TensorRT-LLM/pull/9003
• [TRTLLM-9212][chore] move MoeLoadBalancerConfig to llm_args.py by @QiJune in https://github.com/NVIDIA/TensorRT-LLM/pull/9002
• [None][chore] Waive test_llm_rpc_streaming by @Superjomn in https://github.com/NVIDIA/TensorRT-LLM/pull/9113
• [None] [infra] Update CODEOWNERS for pre-commit-config.yaml by @venkywonka in https://github.com/NVIDIA/TensorRT-LLM/pull/9108
• [TRTLLM-9209][infra] Upgrade precommit-hooks to v6.0.0 by @cheshirekow in https://github.com/NVIDIA/TensorRT-LLM/pull/9097
• [None][ci] Waive test_llm_rpc and test_llm_rpc_streaming by @Superjomn in https://github.com/NVIDIA/TensorRT-LLM/pull/9118
• [#6507][fix] Fix precision issue due to KV layout mismatch for split/concat kernels by @ZhangGe6 in https://github.com/NVIDIA/TensorRT-LLM/pull/6917
• [TRTLLM-8816][feat] add optimized trtllm-gen attention kernels on sm103 by @PerkzZheng in https://github.com/NVIDIA/TensorRT-LLM/pull/9081
• [https://nvbugs/5640873][fix] Move thop tests to pre-merge by @HuiGao-NV in https://github.com/NVIDIA/TensorRT-LLM/pull/9094
• [None][fix] Clear indexer k cache reference before release cuda memory by @chang-l in https://github.com/NVIDIA/TensorRT-LLM/pull/9110
• [None][test] add deepseek and qwen cases for rtx series by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/8839
• [None][chore] Remove closed bugs by @xinhe-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9114
• [None][fix] waive failed tests by @xinhe-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/9090
• [None][infra] Waive failed...