Open Source Llm Training Is A Mess Here Is How It All Works

Open-source LLM training is a mess. Here is how it all works. Announcing our Series F . Learn more

Foundations

Open-source LLM training is a mess. Here is how it all works.

There are too many libraries in the open-source LLM training ecosystem, and nobody tells you which ones actually matter, or how they relate.

Authors

Paras Stefanopoulos

Last updated March 18, 2026

Share

There are too many libraries in the open-source LLM training ecosystem, and nobody tells you which ones actually matter, or how they relate. A little about myself: I joined Parsed as the CTO/founding engineer, where we trained custom models for customers. We have since been acquired by Baseten, where we continue to expand. It has been my job to ensure our researchers can reliably train what they want, whenever they want. All that to say, I’ve trudged through the pain of entering this field from the outside. Initially, I felt flooded with a billion different libraries. It felt like everyone just had this pre-requisite knowledge because they matured technically alongside the development of the repos. However, for people coming into the space, it’s very unclear what the responsibilities of the different repos are and how they fit together. The point of this post is to provide the information I wish I had when starting. This post maps the components of the modern open-source LLM training: what each one does, what it depends on, and when you'd reach for one over another. The stack has four layers: Systems: GPU runtime, communication, kernel authoring

Core runtime: PyTorch, checkpoint formats

Training: model definitions, performance primitives, scaling frameworks, adaptation

Inference: serving runtimes, deployment optimization

✕

I assume we’re using NVIDIA GPUs. The ecosystem just hasn't been focused on alternatives; there are flickers of light (i.e., tinygrad), but they have not yet filtered through to the mainstream market.I will also preface that all of these libraries are constantly evolving. This overview is a snapshot in time, and between writing and posting, I may be out of date with the evolution of many packages. If I've gotten something wrong (likely), please let me know 💚 Systems layer These are the components below PyTorch. You rarely interact with them directly, but they are the lowest level, leading to machine code running on the hardware. ✕

Compute Unified Device Architecture (CUDA) The GPU runtime + an extension of C++ which allows you to control all things GPU.

All tensor computation on NVIDIA hardware goes through CUDA.

PyTorch, cuBLAS, NCCL all sit on top of CUDA

cuBLAS (CUDA Basic Linear Algebra Subprograms) NVIDIA's dense linear algebra library

A library of common operations implemented in CUDA, i.e. General Matrix Multiply (GEMM)

Transformers are dominated by GEMM operations

Alternatives exist like Nvidia’s CUTLASS, these are not mentioned in this post for brevity.

NVIDIA Collective Communications Library (NCCL) Library used to coordinate distributed GPU workloads

Implements AllReduce, AllGather, ReduceScatter — the collectives that underly every distributed training strategy

These are also just highly optimized kernels written in CUDA

DeepSpeed, PyTorch FSDP2, Megatron, all use NCCL to coordinate the transfer of data between GPUs

NVLink and InfiniBand NVLink: high-bandwidth interconnect between GPUs within a node (900 GB/s bidirectional on Hopper)

InfiniBand: high-bandwidth interconnect between nodes

The ratio of compute speed to interconnect bandwidth determines whether a parallelism strategy is communication-bound

Triton Python Domain Specific Language (DSL) for writing GPU kernels. Your triton code ends up as PTX, which is GPU-level assembly code (CUDA compiles to this too)

People use Triton to write kernels in an easier-way than straight CUDA or CUTLASS.

flash-linear-attention is a collection of kernels for optimized linear attention, written in Triton.

Core runtime PyTorch Nearly every package in this post builds on, wraps, patches, or exports into PyTorch

Tensors, autograd, optimizers, CUDA integration, distributed primitives

PyTorch lets you describe the calculations required for your model, it can then execute these operations efficiently on a GPU (or supported accelerator)

Two PyTorch-native features worth calling out: FSDP / FSDP2 : PyTorch's built-in Fully Sharded Data Parallel (FSDP) implementation — the native equivalent of DeepSpeed ZeRO. Shards parameters, gradients, and optimizer states across GPUs. FSDP2 adds per-parameter sharding control and composes with tensor parallelism and torch.compile.

DTensor and DeviceMesh : Abstractions for representing distributed tensors and GPU topologies. FSDP2 and PyTorch's tensor parallelism are built on these. They're what give TorchTitan its "pure PyTorch" distributed training path.

FSDP2 is awesome and makes multi-gpu distributed training easy. We speak more about FSDP2 and other parallelism options later. Safetensors Just a file format, designed by HuggingFace. It has pretty much become the de facto standard for storing model weights.

Used by Transformers, vLLM, SGLang, TensorRT-LLM, Megatron-Bridge

Training stack WTF are Transformers, they're everywhere! Imagine if every research group maintained its own git repo to describe its unique model architectures. There would be a slew of custom implementations, config formats, and checkpoint layouts. At a high level, Transformers gives the ecosystem one place to put and one way to represent: Architecture specs (the PyTorch modules for a model family)

Configuration (given an architecture, how many layers are in the 4B version, what is the hidden dimension?)

Tokenizers / processors

✕

When a new transformer drops, the research lab (or a third party) will commit the specification (architecture + tokenizers), written in PyTorch, in the transformers repo. If you go to the config.json within a model's files, you will find the model_type key. This will map to the architecture implementations in transformers. For example, for Qwen3.5-9B we can find the spec and its variants here . Other libraries can then either use these implementations directly, or try to wrap them/hook into them to patch in changes. For some, they can map from these reference implementations to optimized implementations (Megatron-Bridge converts to and from Megatron). Transformers also has a simple training loop, which can be used to run SFT and...