NVIDIA/TensorRT-LLM v1.2.0rc2

v1.2.0rc2

Repository: NVIDIA/TensorRT-LLM

Tag: v1.2.0rc2

Published: 2025-11-07T07:10:24Z

Prerelease: yes

Release notes:

Announcement Highlights

• Model Support
• Optimize the routing kernel for DeepSeek V3; add MoE TRTLLM backend support for KimiK2 and Qwen-next (#7761)
• Support DeepSeek V3.2 with FP8/BF16 KV cache and NVFP4/BF16 KV cache (#8405)
• Add EVS support for nano-v2-vlm (#8024)
• Support Qwen3 reasoning and tool parsers (#8000, #8216)
• Add Nemotron MOE support in AutoDeploy, including FP8 MOE (#8469, #8737, #8599)

• API
• Support ignored prompt length via new sampling parameter (#8127)
• Replace unified attention before export (#8303)
• Add max_total_draft_tokens (#8366)
• Pass KvCacheRetentionConfig to torch LlmRequest (#8634)

• Feature
• Add cuBLASLt NVFP4 GEMM backend (#7943)
• Add FP8 rowwise GEMMs for B200 (#8332)
• Enable low-precision alltoall for CUTLASS/TRTLLMGen (#8675)
• Integrate MNNVL Throughput and refactor allreduce kernel for TRTLLM MoE (#8728, #8018)
• Enable RMS norm fusion for Nemotron MOE (#8563)
• Add base64 video input support (#8458)

• Fix & Infra
• Upgrade to DLFW 25.10, PyTorch 2.9.0, and Triton 3.5.0 (#8838)
• Fix FP8 blockwise GEMM performance with attention DP (#8501)
• Fix pipeline-parallel bubbles (#8687)
• Cache the AllReduce wrapper to avoid re-allocation hangs (#8803)
• Stabilize tests/CI with waives and slurm/CI updates (#8524, #8573, #8749, #8775, #8808, #8896, #8897)

• Benchmark
• Add Server-Client Perf Test in pytest for B200/B300 (#7985)
• Add layer-wise benchmarks and detailed KV cache transfer time breakdown (#8777, #8521)
• Add longbench v2 for long-context evaluation (#8604)
• Add benchmark to DeepConf and upload perf results to database (#8776, #8653)

• Documentation
• Add LLM-API change principles (#8350)
• Add docs for torch.compile and piecewise CUDA graph (#8527)
• Update public docs and add etcd auto-scaling tests (#8602)
• Clarify perf best practices and supported hardware for GPT-OSS (#8665)

• Known issue
• For this pre-release version, install using the specific version identifier: pip3 install tensorrt-llm==1.2.0rc2. Installing with pip3 install tensorrt-llm --pre will result in a broken dependency on onnx==1.20.0rc1. This issue will be resolved in the next release.

What's Changed

• [None][chore] update test duration by @xinhe-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8377
• [None][fix] Avoid overwrite of kv_cache_config.max_tokens for VSWA scheme for the KVCacheManager by @eopXD in https://github.com/NVIDIA/TensorRT-LLM/pull/8219
• [TRTLLM-4866] [test] Support waiving unit tests by waives.txt by @VALLIS-NERIA in https://github.com/NVIDIA/TensorRT-LLM/pull/8359
• [TRTLLM-8637][feat] Optimize the routing kernel for DeepseekV3 (MoE CUTLASS backend); Add support for 384 experts (MoE TRTLLM backend) by @ChristinaZ in https://github.com/NVIDIA/TensorRT-LLM/pull/7761
• [https://nvbugs/5542862][fix] Upgrade fmha_v2. by @yuxianq in https://github.com/NVIDIA/TensorRT-LLM/pull/8364
• [TRTLLM-8669][infra] Use artifactory mirror for install python by @ZhanruiSunCh in https://github.com/NVIDIA/TensorRT-LLM/pull/8394
• [TRTLLM-7255][feat] Add iteration log parser script for benchmark log by @yizhang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/6942
• [None][ci] move some test cases from H100 to A10 by @QiJune in https://github.com/NVIDIA/TensorRT-LLM/pull/8449
• [TRTLLM-8436][feat] batched sampling and top-k logprobs improvements by @ixlmar in https://github.com/NVIDIA/TensorRT-LLM/pull/8398
• [None][feat] Update devcontainer configuration to include additional extensions by @Funatiq in https://github.com/NVIDIA/TensorRT-LLM/pull/8369
• [https://nvbugs/5540752][fix] Support quantized Phi4 MM models by @pamelap-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/8190
• [https://nvbugs/5492250][fix] Remove isolated cases and unwaive cases by @HuiGao-NV in https://github.com/NVIDIA/TensorRT-LLM/pull/8492
• [TRTLLM-6055][infra] Slurm Test refactor by @yuanjingx87 in https://github.com/NVIDIA/TensorRT-LLM/pull/7176
• [https://nvbugs/5568676][fix] Remove test waive by @dongfengy in https://github.com/NVIDIA/TensorRT-LLM/pull/8437
• [#8461][feat] AutoDeploy: trtllm-serve bug fix + unit test by @lucaslie in https://github.com/NVIDIA/TensorRT-LLM/pull/8462
• [None] [chore] Add architecture-specific ATTRIBUTIONS files by @venkywonka in https://github.com/NVIDIA/TensorRT-LLM/pull/8468
• [#8272][feat] Enable chunked prefill for SSMs in AutoDeploy by @suyoggupta in https://github.com/NVIDIA/TensorRT-LLM/pull/8477
• [None][feat] Update 3rdparty/DeepGEMM to latest commit by @ruoqianguo in https://github.com/NVIDIA/TensorRT-LLM/pull/8488
• [None][feat] Support kv_cahce_reuse for HyperCLOVAX-Vision model by @yechank-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/7789
• [TRTLLM-8436][fix] restore list[list[list[int]]] in add_token by @ixlmar in https://github.com/NVIDIA/TensorRT-LLM/pull/8502
• [None][chore] Move submit.sh to python and use yaml configuration by @zerollzeng in https://github.com/NVIDIA/TensorRT-LLM/pull/8003
• [TRTLLM-7287][test] add multimodal chunked_prefill cases by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/8011
• [None][feat] Add alltoall to trtllm-gen MoE backend. by @bobboli in https://github.com/NVIDIA/TensorRT-LLM/pull/8481
• [TRTLLM-8638][fix] Add failed cases into waives.txt by @xinhe-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8486
• [None][ci] rebalance H100 stages by @QiJune in https://github.com/NVIDIA/TensorRT-LLM/pull/8491
• [None][feat] Support Qwen3 reasoning parser by @LinPoly in https://github.com/NVIDIA/TensorRT-LLM/pull/8000
• [None][infra] Add split algorithm for slurm by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/8516
• [TRTLLM-8638][fix] Remove closed bugs by @xinhe-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8478
• [None][chore] Update feature combination matrix for SWA kv cache reuse by @eopXD in https://github.com/NVIDIA/TensorRT-LLM/pull/8529
• [None][fix] the api_stability unify default values of None and inspect._empty by @Superjomn in https://github.com/NVIDIA/TensorRT-LLM/pull/8496
• [None][infra] Waive failed tests for main 10/21 by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/8524
• [None][doc] Facilitates the integration of the transfer agent by @Shixiaowei02 in https://github.com/NVIDIA/TensorRT-LLM/pull/7867
• [TRTLLM-8160][feat] Add max_total_draft_tokens by @yweng0828 in...