togethercomputer/xorl-wheels sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_cb2e4a7

FlashInfer 0.6.18 SM100a+SM103a MoE-LoRA AOT — SwiGLU LoRA up-delta units, EP scale lookups

Repository: togethercomputer/xorl-wheels

Tag: sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_cb2e4a7

Published: 2026-10-06T15:40:00Z

Prerelease: yes

Release notes: This prerelease contains the AOT sgl_fused_moe_trtllm_sm100 for FlashInfer 0.6.18, built for sm100a (B200) and sm103a (B300). It replaces sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_1b699bf.

It fixes a correctness bug in NVFP4 SwiGLU MoE LoRA that predates 1b699bf, and an out-of-bounds read on expert-parallel ranks. Upgrade if you serve NVFP4 MoE LoRA.

Changes since 1b699bf

The kernel source was built from tgl PR #1246 (togethercomputer/tgl, commit f46fab8). #1246 was reverted (#1249) and re-landed as TGL private overrides in #1250 (b82330d1f5 on qingyang/multilora). The overlay moved to python/sglang/private/kernels/ops/moe/trtllm_lora_temp/data with identical contents (same git tree cb2e4a7), so this wheel is unchanged and still valid.

• The up half of the LoRA delta is now in true units. The unfused GEMM1 applies the gate alpha to both halves, so the up half stays in gate units until it is multiplied by linScale. In per-token-activation mode, linScale = g1_alphas_up / g1_alphas.
• Old behaviour: the SwiGLU kernels added the true-units LoRA delta first and rescaled afterwards. This scaled the up-half LoRA delta by up/gate, and left activation_lora_input (the down-LoRA input) off by gate/up.
• Affected checkpoints: any whose gate and up halves carry different NVFP4 global scales. For example, Qwen3-30B-A3B NVFP4 has up/gate ratios of 0.52–1.22.
• Effect: base (no-adapter) output is unchanged. LoRA output went from 0.961 cosine to the reference to the NvFP4 floor.
• Coverage: all three activation kernels are fixed (scalar, vectorized, fused act-quant).
• Expert-parallel lookups are rank-local. The packed routing ids are global expert ids, but the per-expert scales and SiTU betas are sized num_local_experts. Every EP rank except rank 0 read them past the end.
• The kernels now rebase by local_expert_offset and bounds-check against local_num_experts.
• activation::run fails if routing ids arrive without the local range.
• `sgl_trtllm_bf16_routed_moe_lora` accepts only SwiGLU. Its activation kernel only computes SwiGLU.

Install

pip install --no-deps --require-hashes -r <(echo "https://github.com/togethercomputer/xorl-wheels/releases/download/sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_cb2e4a7/xorl_sgl_trtllm_aot-0.6.18+csrccb2e4a7.cu131-py3-none-linux_x86_64.whl --hash=sha256:d9ca306749eef26af0a7ba11d0321bc1374a87b89b9a5c450c3be12c82b16ece")

FlashInfer's `is_aot` only checks that the `.so` exists. It does not check the GPU architecture or the source tree, so the wheel must match both the GPUs and the tgl overlay tree you run.

Build

• Source commit: f46fab876f6da35fd935410ea3904fe4dccb38a3
• Kernel source tree: cb2e4a740c86c27a8ce9e68891efe9d1b3bb152c
• Tree path: python/sglang/private/kernels/ops/moe/trtllm_lora_temp/data (since tgl #1250; built from python/sglang/kernels/ops/moe/trtllm_lora_temp/data at f46fab8, same tree)
• FlashInfer Python and cubin: 0.6.18
• FlashInfer JIT cache: 0.6.18+cu130
• Torch: 2.13.0+cu130
• CUDA compiler: 13.1.115
• Host compiler: gcc 11.4 (Ubuntu 22.04)
• TVM FFI: 0.1.11
• GPU architecture: sm100a + sm103a (10 + 10 cubins)
• Minimum host libraries: GLIBC 2.34, GLIBCXX 3.4.29

Rebuild when the kernel source tree or the FlashInfer version changes.

Validation

Op-level, B200 and B300, FLASHINFER_DISABLE_JIT=1

test/registered/lora/test_trtllm_fp4_situ_moe_lora.py (now test/tgl/registered/lora/): 6 passed + 2 subtests on each GPU. The table compares against the 1b699bf binary.

| Case | 1b699bf | cb2e4a7 | | --- | --- | --- | | SwiGLU + LoRA, distinct gate/up scales | 0.961 | pass | | EP rank (experts 4–7 of 8), SiTU | NaN | pass | | EP rank, SwiGLU (scalar / vectorized / fused act-quant) | 0.960 / fail / fail | pass | | SiTU with and without LoRA (Kimi-K3 dims) | pass | pass |

The AOT artifact loaded with is_aot=True in about 0.2 s on both GPUs.

NVFP4 multi-LoRA harness, B300

• Model: NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4, which has distinct gate/up scales
• Hardware: four B300 GPUs, TP=4
• Adapters: eight rank-16 shared-outer adapters
• Backends: experimental_sgl_trtllm, virtual experts
• Harness: test/manual/lora/test_moe_lora_trtllm_correctness.py

The B200 host was fully occupied by another job, so the harness ran on B300. The B200 binary is covered by the op test above.

| Run | Single | Batched | Mixed | Total | | --- | --- | --- | --- | --- | | cb2e4a7, run 1 | 8/8 | 7/8 | 12/12 | 27/28 | | cb2e4a7, run 2 | 8/8 | 8/8 | 12/12 | 28/28 | | cb2e4a7, run 3 | 8/8 | 8/8 | 12/12 | 28/28 | | 1b699bf (previous), run 1 | 8/8 | 8/8 | 12/12 | 28/28 | | 1b699bf (previous), run 2 | 8/8 | 8/8 | 12/12 | 28/28 |

About the run 1 miss: adapter_6 produced CARBON-33922-WOLF instead of CARBON-3392-WOLF, in the batched phase. It did not reproduce in two more runs with the same binary and code. It looks consistent with a near-tie greedy flip caused by batch-composition numerics. The harness checks passwords, not magnitudes, so it is not sensitive to the LoRA-delta scaling this release fixes; the op-level test above covers that.

Each run used a fresh FlashInfer workspace, and no sgl_fused_moe_trtllm_sm100 JIT build appeared.

B300: Kimi-K3 end to end

• Model: nvidia/Kimi-K3-NVFP4
• Hardware: eight B300 GPUs, TP=8
• LoRA targets: q_proj k_proj v_proj q_b_proj kv_b_proj gate_up_proj down_proj
• Code: the same tgl source, JIT-built

| Check | Result | | --- | --- | | KDA q/k/v adapter rejected at load (HTTP 400) | pass | | MLA, routed-expert and shared-expert ×20 adapters load and serve, solo and in mixed batches | pass | | Each strong adapter's batched output is nearest its own solo output | pass | | Zero-B adapter equals base | pass |

Performance has not been benchmarked.