NVIDIA/TensorRT-LLM v1.2.0rc0

v1.2.0rc0

Repository: NVIDIA/TensorRT-LLM

Tag: v1.2.0rc0

Published: 2025-09-30T07:55:55Z

Prerelease: yes

Release notes:

Announcement Highlights

• Model Support
• Support nano_v2_vlm in pytorch backend (#7207)
• Add Tencent HunYuanDenseV1 model support (#7081)
• Support Seed-OSS model in pytorch backend (#7496)
• GPT-OSS MXFP4 support (#7451)
• API
• Support new structural tag API (upgrade XGrammar to 0.1.25) (#7893)
• Enable regex and EBNF grammar in trtllm-serve (#7925)
• Optionally disable server GC and worker GC (#7995)
• Add serialization/deserialization options for AutoTuner profiling cache (#7738)
• Cherry-pick from (#7598) Make low_precision_combine as a llm arg (#7898)
• Benchmark
• Add gpt-oss serve benchmark tests (#7638)
• Exit as early as possible and propagate exit status correctly for multi-node testing (#7739)
• Add gpt oss model for trtllm perf test (#7328)
• Add generation logits case for llama3 (#7759)
• Feature fix model issue for disagg serving (#7785)
• Add deepseek r1/v3 model with chunked prefill cases (#7124)
• Add accuracy benchmark in stress test (#7561)
• Add NoSmem epilogue schedule and dynamic cluster shape for sm10x group gemm (#7757)
• Rename llm_perf_full to llm_perf_core and add missing cases (#7899)
• Update benchmark script (#7860)
• Add multi-nodes test for disagg-serving (#7470)
• Update llm_models_root to improve path handling on BareMetal environment (#7876)
• Add DS-R1/Qwen3 test cases for RTX 6000 (#7662)
• Add NIM perf test cases (#7924)
• Fix the tactic sorting in TrtllmGenBatchedGemmRunner::getValidConfigIndices (#7419)
• Improve the failure message for accuracy test suite (#7994)
• Update get_sysinfo.py to avoid UnboundLocalError (#7982)
• Update disagg gen-only benchmark. (#7917)
• Feature
• Phi4-mm image modality inference optimization (#7918)
• Add NVFP4 x FP8 moe kernels (#7821)
• Enable KV cache reuse and chunked prefill for mistral3.1 (#7628)
• Enable two-model spec dec for MTP Eagle (#7001)
• Support EPLB in Qwen3 MoE (#7443)
• Eagle3 cuda graph support for the first draft model inference (#7363)
• Enable run_post_quant_allgather for MoE TRTLLM backend (#6794)
• Enable gpt oss on DGX H100. (#6775)
• Add gpt-oss chunked prefill tests (#7779)
• Eagle, use last hidden post norm (#7546)
• Optimize Qwen2/2.5-VL performance (#7250)
• Support kvcache reuse and chunk prefill for phi4mm (#7723)
• Support attention dp for qwen3 dense model (#7618)
• AutoDeploy Fix memory leak in fuse_moe (#7844)
• Enable overlap scheduler for two-model spec decoding (#7651)
• Add support of CUDA13 and sm103 devices (#7568)
• Add Cute DSL nvfp4 linear op (#7632)
• Enable LM tp for MTP, under attention dp case (cherry-pick #7128) (#7571)
• Add an example of KV cache host offloading (#7767)
• Helix: make softmax stats pointer available to attention gen (#6865)
• AutoDeploy: graph-less transformers mode for HF (#7635)
• Cherry-pick DeepGEMM related commits from release/1.1.0rc2 (#7716)
• Add swapab, tileN64, cga sync support for cute dsl nvfp4 gemm (#7764)
• FP8 Context MLA integration (Cherry-pick https://github.com/NVIDIA/TensorRT-LLM/pull/6059 from release/1.1.0rc2) (#7610)
• Update CUTLASS to 4.2 and enable SM103 group gemm (#7832)
• Cherry-pick fix to reuse pytorch memory segments occupied by cudagraph (#7747)
• Helix: add custom position ids to MLA kernels (#6904)
• Support for partial sharding from factory (#7393)
• KV cache transmission in disagg with CP on gen side (#7624)
• Cherry-pick from #7423 Support fp8 block wide ep cherry pick (#7712)
• E-PD Disagg Support via llmapi (3/N) (#7577)
• Add batch waiting when scheduling (#7416)
• Use list instead of torch tensor for new tokens in update requests (#7730)
• Support multi-threaded tokenizers for trtllm-serve (cherry-pick) (#7776)
• Support JIT mha.cu for SPEC_DEC in runtime (#6078)
• Batched sampling by strategy (supersedes enable_mixed_sampler, cf. TRTLLM-7156) (#7294)
• Enable prompt_logprobs in pytorch backend (#7580)
• Support SWA KV cache reuse (#6768)
• Return topk logprobs in torch backend (#7756)
• CapturedGraph to support max_batch_size > max(cuda_graph_batch_sizes) (#7888)
• Revert " Return topk logprobs in torch backend (#7756)" (#7969)
• DeepEP LL fp8 dispatch/combine (#7927)
• Helix: add alltoall op (#6815)
• Optimize kv cache transfer TEP (#7613)
• Add environment variable to adjust block pool allocation ration under kv cache manager (#7923)
• Add a standalone buffer cache class and reuse buffers between cduagraph and no-graph flow (#7669)
• Add static tree sampling and verification (#7161)
• Add support for KVCache transfer from KVCache reuse path (#6348)
• Added AutoDeploy backend support to test_perf.py (#7588)
• Speed up concat k and copy k_nope in context phase using torch.compile (#8044)
• Documentation
• Fix the link in the doc (#7713)
• Clean the doc folder and move the outdated docs into lega… (#7729)
• Add doc for KV cache salting support (#7772)
• Fix section header of llm_kv_cache_offloading example (#7795)
• Update Documentation link to point to docs instead of docs source code (#6495)
• Cherry-pick deployment guide update from 1.1.0rc2 branch to main branch (#7774)
• Tech blog: Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly (#7864)
• Update tech blog12 (#7884)
• Add known issues to llmapi doc (#7560)
• Add blackwell information into support matrix (#6740)
• Fix a invalid link and a typo. (#7634)
• Use hash id for external link (#7641)
• Add labels description note into llm api section (#7696)
• Enhance api reference doc by labeling stable APIs (#7751)
• Add 1.0 release notes (#7605)
• Scaffolding tech blog part one (#7835)
• Update docker cmd in quick start guide and trtllm-serve … (#7787)
• Replace the main in the examples' link with commit id. (#7837)
• Rename TensorRT-LLM to TensorRT LLM for homepage and the … (#7850)
• Add a guide for modifying APIs (#7866)
• Update Perf-Overview.md for release/1.0 (#7848)
• Add stable label to all the un-labelled arguments in LLM class (#7863)
• Fix invalid links in perf benchmarking. (#7933)
• Add Llama PP known issue to release note (#7959)
• Add acknowledgements in scaffolding tech blog (#7983)
• Add scaffolding tech blog to cover (#8021)
• Refine perf overview.md and correct the error link in per… (#8035)
• Scaffolding tech blog fix a typo (#8042)
• Document hang issue caused by UnpicklingError (#8049)

What's Changed

• [None][feat] Eagle, use last hidden post norm...