NVIDIA/TensorRT-LLM v1.1.0rc4

v1.1.0rc4

Repository: NVIDIA/TensorRT-LLM

Tag: v1.1.0rc4

Published: 2025-09-10T07:32:18Z

Prerelease: yes

Release notes:

Announcement Highlights:

• Model Support
• Support phi-4 model in pytorch backend (#7371)
• Support Aggregate mode for phi4-mm (#7521)
• API
• Implement basic functionalities for Responses API (#7341)
• Support multiple postprocess workers for chat completions API (#7508)
• Report failing requests (#7060)
• Benchmark
• Test trtllm-serve with --extra_llm_api_options (#7492)
• Feature
• Add MOE support for dynamic cluster shapes and custom epilogue schedules (#6126)
• Autotune TRT-LLM Gen MoE when using CUDA graphs (#7285)
• Enable guided decoding with speculative decoding (part 2: one-model engine) (#6948)
• Separate run_shape_prop as another graph utility (#7313)
• MultiLayer Eagle (#7234)
• Fuse d2t to logitsBitmaskKernel and fix a race condition in one-model spec (#7481)
• Add NVFP4 x FP8 (#6809)
• Support hashing and KV cache reuse for videos (#7360)
• Add MCTS and TOT tree-based inference controllers to Scaffolding (#7490)
• Introduce QKNormRoPEAttention module (#6830)
• AutoDeploy: flexible args for sequence interface + AD multi-modal input processor + llama4 VLM example (#7221)
• Support KV cache salting for secure KV cache reuse (#7106)
• trtllm-gen kernels support sm103 (#7570)
• Move stop_criteria to sample_async (#7041)
• KV cache transfer for uneven pp (#7117)
• Update multimodal utility get_num_tokens_per_image for better generalization (#7544)
• AutoDeploy: set torch recompile_limit based on cuda_graph_batch_sizes and refactored (#7219)
• Add Request specific exception (#6931)
• Add DeepSeek-v3-0324 e2e torch test (#7413)
• Add 8-GPU test cases for RTX6000 (#7083)
• add gptoss 20g tests (#7361)
• Nixl support for GDS (#5488)
• CMake option to link statically with cublas/curand (#7178)
• Extend VLM factory and add Mistral3 factory (#7583)
• Documentation
• fix example in docstring (#7410)
• Fix formatting error in Gemma3 readme (#7352)
• Add note about trtllm-serve to the devel container (#7483)
• add GPT OSS Eagle3 blog (#7140)
• 1.0 Documentation. (#6696)
• Update kvcache part (#7549)
• Rename TensorRT-LLM to TensorRT LLM. (#7554)
• refine docs for accuracy evaluation of gpt-oss models (#7252)

What's Changed

• [https://nvbugs/5485430][fix] Copy the nanobind file when using precompiled package by @jiaganc in https://github.com/NVIDIA/TensorRT-LLM/pull/7334
• [None][infra] Using local variables in rerun function by @yiqingy0 in https://github.com/NVIDIA/TensorRT-LLM/pull/7198
• [None][ci] Correct docker args for GPU devices and remove some stale CI codes by @chzblych in https://github.com/NVIDIA/TensorRT-LLM/pull/7417
• [https://nvbugs/5476580][fix] unwaive test_nvfp4_4gpus by @Superjomn in https://github.com/NVIDIA/TensorRT-LLM/pull/7454
• [None][test] auto reuse torch empty cache on qa test by @crazydemo in https://github.com/NVIDIA/TensorRT-LLM/pull/7421
• [None][doc] fix example in docstring by @tomeras91 in https://github.com/NVIDIA/TensorRT-LLM/pull/7410
• [TRTLLM-6643][feat] Add DeepSeek-v3-0324 e2e torch test by @aalanwyr in https://github.com/NVIDIA/TensorRT-LLM/pull/7413
• [None][infra] waive test case failed on post-merge by @HuiGao-NV in https://github.com/NVIDIA/TensorRT-LLM/pull/7471
• [TRTLLM-7208][feat] Implement basic functionalities for Responses API by @JunyiXu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/7341
• [https://nvbugs/5453992][unwaive] Unwaive llama quickstart test by @peaceh-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/7242
• [None][infra] Waive failed tests on main branch 0902 by @EmmaQiaoCh in https://github.com/NVIDIA/TensorRT-LLM/pull/7482
• [None][chore] Fix formatting error in Gemma3 readme by @karljang in https://github.com/NVIDIA/TensorRT-LLM/pull/7352
• [https://nvbugs/5470782][fix] Add specific test names for test_deepseek.py by @SimengLiu-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/7318
• [https://nvbugs/5458798][fix] Disabled test_trtllm_bench_backend_comparison due to timeout by @MrGeva in https://github.com/NVIDIA/TensorRT-LLM/pull/7397
• [None][chore] Add note about trtllm-serve to the devel container by @MartinMarciniszyn in https://github.com/NVIDIA/TensorRT-LLM/pull/7483
• [None][chore] rm executor config in kv cache connector by @leslie-fang25 in https://github.com/NVIDIA/TensorRT-LLM/pull/7372
• [None][perf] Add MOE support for dynamic cluster shapes and custom epilogue … by @djns99 in https://github.com/NVIDIA/TensorRT-LLM/pull/6126
• [None][perf] Autotune TRT-LLM Gen MoE when using CUDA graphs by @jinyangyuan-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pull/7285
• [TRTLLM-7261][feat] Support phi-4 model in pytorch backend by @Wanli-Jiang in https://github.com/NVIDIA/TensorRT-LLM/pull/7371
• [https://nvbugs/5480289][fix] release slot manager in mtp MTPHiddenStatesManager by @yweng0828 in https://github.com/NVIDIA/TensorRT-LLM/pull/7340
• [https://nvbugs/5488141][fix] Unwaive llama3 test_eagle3 by @mikeiovine in https://github.com/NVIDIA/TensorRT-LLM/pull/7486
• [https://nvbugs/5472947][fix] wait on isend handles before reusing buffers by @amukkara in https://github.com/NVIDIA/TensorRT-LLM/pull/7462
• [TRTLLM-7363][test] Add 8-GPU test cases for RTX6000 by @StanleySun639 in https://github.com/NVIDIA/TensorRT-LLM/pull/7083
• [https://nvbugs/5485593][fix] improve accuracy/test_disaggregated_serving.py by @reasonsolo in https://github.com/NVIDIA/TensorRT-LLM/pull/7366
• [None][doc] add GPT OSS Eagle3 blog by @IzzyPutterman in https://github.com/NVIDIA/TensorRT-LLM/pull/7140
• [None][fix] Fix KV cache recompute in draft_target spec decode by @mikeiovine in https://github.com/NVIDIA/TensorRT-LLM/pull/7348
• [TRTLLM-7028][feat] Enable guided decoding with speculative decoding (part 2: one-model engine) by @syuoni in https://github.com/NVIDIA/TensorRT-LLM/pull/6948
• [None][chore] Remove two unused parameters in create_py_executor by @leslie-fang25 in https://github.com/NVIDIA/TensorRT-LLM/pull/7458
• [#7222][autodeploy] Separate run_shape_prop as another graph utility by @Fridah-nv in https://github.com/NVIDIA/TensorRT-LLM/pull/7313
• [None][fix] Fix a numerical stability issue for XQA with spec dec by @lowsfer in https://github.com/NVIDIA/TensorRT-LLM/pull/7114
• [https://nvbugs/5470769][fix] fix disagg-serving accuracy test case by @reasonsolo in https://github.com/NVIDIA/TensorRT-LLM/pull/7338
• [TRTLLM-7876][test] Test trtllm-serve with --extra_llm_api_options by...