NVIDIA/Megatron-LM core_v0.15.0

NVIDIA Megatron Core 0.15.0

Repository: NVIDIA/Megatron-LM

Tag: core_v0.15.0

Published: 2025-12-17T23:08:29Z

Prerelease: no

Release notes:

• Features
• Performance
• Fused QKV preprocessing with precomputed RoPE caches (3x preprocessing speedup, 10-14% E2E) (MR \!3912)
• Use new TE interface for user buffers (MR \!3886)
• Add CPU activation offloading via TE (MR \!4286)
• Add setting to support Adam or AdamW optimizer (MR \!3866)
• MoE
• Add DTensor support for EP and DSv3 modules (MR \!3955)
• Add HybridEP backend to Flex Dispatcher (PR \!2176)
• Implement NVFP4 Zero Padding for MoE (PR \!1985)
• Compute shared experts before router (MR \!4068)
• Enable bias in expert MLP (MR \!3858)
• Model support
• Add YaRN support for GPT-OSS (MR \!4044)
• Add FP8 init for MTP (MR \!3958)
• Add fp8\_dpa option for FP8 scaling (MR \!4053)
• FSDP
• Enable joint training of parallel modules (MR \!3850)
• Inference
• Add CUDA Graph runner lookup table cache (up to 2x E2E speedup) (MR \!4082)
• Add MoE dropping and padding router for CUDA Graph \+ decode (MR \!3816)
• Integrate unified memory for dynamic inference context (MR \!3985)
• Post-training
• Add GPT-OSS ModelOpt support with quantization, import/export (MR \!4169)
• Enable KD support with hybrid training loop (MR \!4021)
• Add ModelOpt pruning example (MR \!4022)
• RL
• Add importance sampling and partial rollouts to Megatron RL (MR \!4000)
• Add sequence packing for RL (MR \!4191)
• Ease of use
• Handle CUDA absence during import (MR \!4120)
• Enable SWA mixing with attention (MR \!3855)
• Bug fixes
• Fix convergence bug in MXFP8 parameter gradient buffer reuse (MR \!3999)
• Fix loss mask cloning to prevent incorrect updates (MR \!4164)
• Fix metadata loss in checkpoints (MR \!4182)
• Fix FSDP grad accum fusion support (MR \!4018)
• Fix non-TE optimizer checkpoint issue (MR \!3931)
• Fix BERT virtual pipeline parallelism (MR \!3993)
• Fix gc.freeze() slowdown by adding gc.collect() on last layer (MR \!4003)
• Fix full iteration CUDA graph non-tensor handling (MR \!4019)
• Fix model\_auto\_sync mis-set and add gradient assertion (MR \!4062)
• Fix HF import dtype and checkpoint loading issues (MR \!4095)
• Fix missing initialization in ProcessGroupCollection (MR \!4159)
• Fix sink attention TP (MR \!4173)
• Fix 1f1b overlap unit tests for MTP standalone (MR \!4210)
• Fix stale state dict handling (MR \!4226)
• Known issues
• New Contributors
• @marksverdhei made their first contribution in #1980
• @Skylion007 made their first contribution in #2047
• @azzhipa made their first contribution in 5db6704
• @vicoooo26 made their first contribution in 5db6704
• @A-transformer made their first contribution in e002b5c
• @chaitanyadwivedii made their first contribution in 20b3954

We'd like to thank all our external contributors whose work was merged in this release:

• External Contributor Acknowledgements
• Fix ImportError and NameError in examples/run_simple_mcore_train_loop.py by @marksverdhei in #1980
• Optimizer refactor: clean up public get_megatron_optimizer interface by @Skylion007 in #2047
• Typo fixes from community with co-authors @vicoooo26, @azzhipa, @A-transformer in 5db6704 and e002b5c
• Fix router input jitter dtype by @chaitanyadwivedii in 20b3954

Note: Some...