NVIDIA/TensorRT-LLM v1.2.0rc1

v1.2.0rc1

Repository: NVIDIA/TensorRT-LLM

Tag: v1.2.0rc1

Published: 2025-10-22T06:56:53Z

Prerelease: yes

Release notes:

Announcement Highlights

• Model Support
• Add GPT-OSS Sm120/Sm121 support (#7937)
• Fix: Disable DeepGEMM for Qwen3 MoE Attention layers (#8087)
• Fix: Update is_post_quant_all2all_supported for MNNVL (#8355)
• Support quantized model for nano-v2-vlm (#8304)
• Fix: Address illegal access when scale is not provided in Llama3/4 (#7960)
• Fix: Correct Qwen2.5-VL device_path error (#8057)
• Add post-merge test for Seed-OSS-36B-Instruct (#8321)
• Fix: Correct get_num_tokens_per_image for nano-v2-vlm (#8425)
• Add Kimi multi-nodes case (#8025)

• API
• Refine sampling strategy selection (BREAKING CHANGE) (#8132)
• Add cache_salt in LLM.generate (#8317)
• Add input tensor pre-hook function API for tuning (#6924)
• Add additional model outputs (#7206)
• Clean create_py_executor API (#8412)

• Benchmark
• Add request timing breakdown option in benchmark_serving (#8128)
• Fix bench_serving import error (#8296)
• Update disagg benchmark configs (#8289)
• Add multimodal data to dummy requests during memory profiling (#7539)
• Save runtime report periodically (#8312)
• Resolve sampling defaults in OpenAI API backend (#8121)

• Feature
• Add new orchestrator type: Ray (#7520)
• Implement HTTP disagg-cluster management (#7869)
• Add PDL support for more kernels (#7977)
• Enable rejection sampling for CDL (#7731)
• Add torch.compile support for CUDA Core GEMM op (#8261)
• Support block-sparse attention in trtllm gen FMHA kernels (#8301)
• Support SWA KV cache reuse OOW block detach (#7922)
• Add factory TP sharding of quantized models (#8123)
• Turn off speculative decode based on acceptance-length threshold (#7283)
• Enable VLM subgraphs and CUDA graph/compile in AutoDeploy (#8203)
• Add sparse attention framework and RocketKV support (#8086)
• Implement etcd storage for disagg cluster (#8210)
• Export scale factor properly for W4A8/NVFP4/FP8 (#8180)
• Reuse CUDA graph memory pool in normal forward flow (#8095)
• Revise TileN-related routing calculation in MoE backend (#8148)
• Develop DeepConf (#8362)
• Support per-expert pre-quant scale factor for W4A8 AWQ MoE (PyTorch) (#7286)
• Support cached tokens for OpenAI server (#7637)
• Add fmha_v2 kernel for head_dim=80 and SM100 to support VLM (#8392)
• Add topological graph helpers (#8457)
• Enable CUDA graph support for KvConnectorWorker API (#8275)
• Add chunked prefill support in AutoDeploy (#8158)
• Set nixl as default cache transceiver backend (#7926)
• Enable FP8 ContextMLA on GB300 (#8080)
• Skip unnecessary CUDA graph capture (#8050)
• Use device tensor index for MTP (#8062)

• Documentation
• Publish blog: Scaling Expert Parallelism in TensorRT LLM (Part 3) (#8323)
• Refine deployment guide by renaming TRT-LLM to TensorRT LLM (#8214)
• Document the role of d2t (#8174)
• Add Qwen3-next doc and L0 test case (#8288)
• Update AutoDeploy README: expert section on YAML configuration (#8370)
• Update TPOT/ITL docs (#8378)
• Add Ray orchestrator initial doc (#8373)
• Add documentation for CUDA 12.9 (#8411)
• Combine feature combination matrix documents (#8442)
• Add ATTRIBUTIONS-{CPP,Python}.md and update wheels setup (#8438)

What's Changed

• [None][feat] AutoDeploy: Nemotron-H accuracy test by @lucaslie in https://github.com/NVIDIA/TensorRT-LLM/pull/8133
• [None][feat] AutoDeploy: graph/module inputs with kwargs instead of args by @lucaslie in https://github.com/NVIDIA/TensorRT-LLM/pull/8137
• [TRTLLM-7349][feat] Adding new orchestrator type -- ray by @joyang-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/7520
• [None][autodeploy] small refactors on attention matching by @Fridah-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8079
• [#5255][autodeploy] Update FuseAllreduceResidualRMSNorm to use pattern matcher utility; remove fuse_collective by @Fridah-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/7545
• [TRTLLM-8189][chore] enhance GenerationExecutor with RPC (part1) by @Superjomn in https://github.com/NVIDIA/TensorRT-LLM/pull/5543
• [https://nvbugs/5521949][fix] Re-enable test_bielik_11b_v2_2_instruct_multi_lora, fix its API use with pytorch flow LoRA by @amitz-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8146
• [None][fix] Adding docker folder to Dockerfile by @pcastonguay in https://github.com/NVIDIA/TensorRT-LLM/pull/8138
• [None][chore] fix llmargs conflict by @Superjomn in https://github.com/NVIDIA/TensorRT-LLM/pull/8152
• [TRTLLM-8413][chore] resolve sampling defaults in OpenAI API backend by @ixlmar in https://github.com/NVIDIA/TensorRT-LLM/pull/8121
• [None][chore] AutoDeploy: clean up accuracy test configs by @lucaslie in https://github.com/NVIDIA/TensorRT-LLM/pull/8134
• [None][fix] Eagle: Attention DP by @IzzyPutterman in https://github.com/NVIDIA/TensorRT-LLM/pull/7939
• [None][feat] GPT-OSS Sm120/Sm121 Support by @farazkh80 in https://github.com/NVIDIA/TensorRT-LLM/pull/7937
• [None][chore] Increase operations-per-run to 1000 for stale action by @karljang in https://github.com/NVIDIA/TensorRT-LLM/pull/8162
• [None] [test] Add B300 cases to CI by @VALLIS-NERIA in https://github.com/NVIDIA/TensorRT-LLM/pull/8056
• [None][infra] Skip failed cases for main by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/8176
• [None][fix] Fix MTP illegal memory access by @mikeiovine in https://github.com/NVIDIA/TensorRT-LLM/pull/8161
• [https://nvbugs/5488576][fix] Propagate disable_finalize_fusion config flag in WIDEEP MoE backend by @sklevtsov-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/8141
• [None][test] add test-model-suites option in integration conftest.py by @ruodil in https://github.com/NVIDIA/TensorRT-LLM/pull/8016
• [https://nvbugs/5455140][fix] unwaive tests related to GB200 OOM by @lancelly in https://github.com/NVIDIA/TensorRT-LLM/pull/8159
• [https://nvbugs/5550283][fix] update test case to the latest MoE API by @xxi-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8165
• [TRTLLM-8414][chore] BREAKING CHANGE: refine sampling strategy selection by @ixlmar in https://github.com/NVIDIA/TensorRT-LLM/pull/8132
• [None][chore] Waive some tests failing on main post merge by @brb-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/8186
• [https://nvbugs/5541545][fix] Remove test_llama4 by @mikeiovine in https://github.com/NVIDIA/TensorRT-LLM/pull/8031
• [https://nvbugs/5522746][fix] unwaive tests caused by node issues after rebooting by @lancelly in...