basetenlabs/Megatron-LM

basetenlabs/Megatron-LM

Description: Ongoing research training transformer models at scale

Language: Python

License: NOASSERTION

Stars: 1

Forks: 0

Open issues: 0

Created: 2025-12-15T22:04:30Z

Pushed: 2026-06-05T21:25:55Z

Default branch: main

Fork: yes

Parent repository: NVIDIA/Megatron-LM

Archived: no

README:

Megatron-LM & Megatron Core ===========================

GPU-optimized library for training transformer models at scale

⚡ Quick Start

# 1. Install Megatron Core with required dependencies
pip install --no-build-isolation megatron-core[mlm,dev]

# 2. Clone repository for examples
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
pip install --no-build-isolation .[mlm,dev]

→ [Complete Installation Guide](#installation) - Docker, pip variants (dev,lts,etc.), and system requirements

Latest News

• 📣 NEW! [Megatron Dev Branch](https://github.com/NVIDIA/Megatron-LM/tree/dev) - early access branch with experimental features.
• 🔄 [Megatron Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge) - Bidirectional converter for interoperability between Hugging Face and Megatron checkpoints, featuring production-ready recipes for popular models.
• [2025/08] [MoE Q3-Q4 2025 Roadmap](https://github.com/NVIDIA/Megatron-LM/issues/1729) - Comprehensive roadmap for MoE features including DeepSeek-V3, Qwen3, advanced parallelism strategies, FP8 optimizations, and Blackwell performance enhancements.
• [2025/08] [GPT-OSS Model](https://github.com/NVIDIA/Megatron-LM/issues/1739) - Advanced features including YaRN RoPE scaling, attention sinks, and custom activation functions are being integrated into Megatron Core.
• [2025/06] [Megatron MoE Model Zoo](https://github.com/yanring/Megatron-MoE-ModelZoo) - Best practices and optimized configurations for training DeepSeek-V3, Mixtral, and Qwen3 MoE models with performance benchmarking and checkpoint conversion tools.
• [2025/05] Megatron Core v0.11.0 brings new capabilities for multi-data center LLM training (blog).

Previous News

• [2024/07] Megatron Core v0.7 improves scalability and training resiliency and adds support for multimodal training (blog).
• [2024/06] Megatron Core added supports for Mamba-based models. Check out our paper An Empirical Study of Mamba-based Language Models and code example.
• [2024/01 Announcement] NVIDIA has released the core capabilities in Megatron-LM into **Megatron Core** in this repository. Megatron Core expands upon Megatron-LM's GPU-optimized techniques with more cutting-edge innovations on system-level optimizations, featuring composable and modular APIs. Explore the [Megatron Core intro](#Megatron Core) for more details.

Table of Contents

Getting Started

• [Quick Start](#-quick-start)
• [Latest News](#latest-news)
• [Megatron Overview](#megatron-overview)
• [Project Structure](#project-structure)
• [Megatron-LM: Reference Implementation](#megatron-lm-reference-implementation)
• [Megatron Core: Production Library](#megatron-core-production-library)
• [Installation](#installation)
• [Docker (Recommended)](#-docker-recommended)
• [Pip Installation](#pip-installation)
• [System Requirements](#system-requirements)

Core Features

• [Performance Benchmarking](#performance-benchmarking)
• [Weak Scaling Results](#weak-scaling-results)
• [Strong Scaling Results](#strong-scaling-results)
• [Ecosystem Libraries](#ecosystem-libraries)

Training

• [Training](#training)
• [Getting Started](#getting-started)
• [Data Preparation](#data-preparation)
• [Parallelism Strategies](#parallelism-strategies)
• [Data Parallelism (DP)](#data-parallelism-dp)
• [Tensor Parallelism (TP)](#tensor-parallelism-tp)
• [Pipeline Parallelism (PP)](#pipeline-parallelism-pp)
• [Context Parallelism (CP)](#context-parallelism-cp)
• [Expert Parallelism (EP)](#expert-parallelism-ep)
• [Parallelism Selection Guide](#parallelism-selection-guide)
• [Performance Optimizations](#performance-optimizations)

Resources

• [Examples](./examples/) - Training scripts and tutorials
• Documentation - Official docs
• [Roadmaps](#roadmaps) - Development roadmaps and feature tracking
• [Community & Support](#community--support) - Get help and contribute
• [Getting Help](#getting-help)
• [Contributing](#contributing)
• [Citation](#citation)

Megatron Overview

Project Structure

Megatron-LM/
├── megatron/
│ ├── core/ # Megatron Core (kernels, parallelism, building blocks)
│ │ ├── models/ # Transformer models
│ │ ├── transformer/ # Transformer building blocks
│ │ ├── tensor_parallel/ # Tensor parallelism
│ │ ├── pipeline_parallel/ # Pipeline parallelism
│ │ ├── distributed/ # Distributed training (FSDP, DDP)
│ │ ├── optimizer/ # Optimizers
│ │ ├── datasets/ # Dataset loaders
│ │ ├── inference/ # Inference engines
│ │ └── export/ # Model export (e.g. TensorRT-LLM)
│ ├── training/ # Training scripts
│ ├── inference/ # Inference server
│ ├── legacy/ # Legacy components
│ └── post_training/ # Post-training (RLHF, etc.)
├── examples/ # Ready-to-use training examples
├── tools/ # Utility tools
├── tests/ # Comprehensive test suite
└── docs/ # Documentation

Megatron-LM: Reference Implementation

Reference implementation that includes Megatron Core plus everything needed to train models.

Best for:

• Training state-of-the-art foundation models at scale with cutting-edge performance on latest NVIDIA hardware
• Research teams exploring new architectures and training techniques
• Learning distributed training concepts and best practices
• Quick experimentation with proven model configurations

What you get:

• Pre-configured training scripts for GPT, LLama, DeepSeek, Qwen, and more.
• End-to-end examples from data prep to evaluation
• Research-focused tools and utilities

Megatron Core: Composable Library

Composable library with GPU-optimized building blocks for custom training frameworks.

Best for:

• Framework developers…