Padding Minimization Efficiency

Model-agnostic padding minimization for LLM training

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

TL;DR

Padding is a significant source of wasted compute when training LLMs. This remains a problem for hybrid Transformer-SSM models, where methods like sequence packing are not easily applicable. We managed to eliminate ~90% of padding-related overhead by applying a model-agnostic approach that utilizes micro-batch-level truncation and padding-aware micro-batching. This approach provides a simple and broadly applicable alternative across existing and future model architectures that dramatically improves training efficiency.

Addressing inefficiencies in online-RL training

Training efficiency is a key concern in large-scale model development, affecting both compute usage and wall-clock time, and ultimately cost. These tradeoffs become especially pronounced in online-RL training, where training runs are long-lived and compute-intensive, making inefficiencies compound quickly.

Over the past few months, our Labs in Front series has explored several directions for improving the efficiency of our online-RL training pipelines. This includes work from our colleagues on improving data efficiency through dynamic data snoozing , which primarily optimizes the rollout generation phase, as well as efforts to scale multi-node deployments of judge LLMs efficiently , which target the reward computation phase. Together, these efforts reflect a broader push to reduce waste and improve utilization across different stages of the training process.

In this post, we focus on the policy update phase of the training process, which encompasses the reference forward pass and the training forward–backward pass. Our focus on this phase complements prior work in this series on inference and reward computation. We identify padding-related waste within the policy update phase as a significant source of inefficiency, and show how to mitigate it in a model-agnostic way.

Training systems built for transformers meet other architectures

Much of the modern LLM training stack is built around transformer models, and many high-impact efficiency optimizations are implemented specifically for transformer workloads. At the same time, non-transformer and hybrid architectures are increasingly common in practice, often motivated by the need to address known inefficiencies and limitations of standard transformers. Examples include: Qwen3-next (DeltaNet) , Nemotron (Mamba2) , Granite (Mamba2) , and our own Jamba models (Mamba) . As these architectures gain adoption, gaps emerge where transformer-specific optimizations no longer apply and previously hidden sources of inefficiency become visible.

We illustrate this setting by training Jamba2-3B , our hybrid model architecture that interleaves attention layers with Mamba state-space layers, using VeRL , a popular online-RL training framework for LLMs. VeRL inherits many of these transformer-centric design choices, which surface clearly when training hybrid architectures like Jamba.

One of the most prominent missing optimizations in this setting is efficient handling of padding, the addition of dummy tokens so variable-length sequences fit a fixed tensor shape, which can become a major source of wasted compute.

Handling padding waste

Training batches typically contain sequences with highly variable lengths, which is handled by padding all sequences in a batch to the same length through the addition of dummy tokens. While padding makes batching possible, these tokens carry no information and are still processed by the model, consuming compute and memory without contributing useful work. Prior work analyzing common NLP datasets shows that under standard fixed-length batching, padding can account for up to about 50% of all processed tokens, and in some realistic cases even more ( Krell et al., 2021 ).

In RL training, this inefficiency is further amplified by the variability in model-generated response lengths, which can differ substantially even for similar prompts. As a result, padding-related waste affects multiple phases of the policy update step, most notably the reference forward pass and the training forward–backward pass. Since these phases account for a significant fraction of overall step time, excessive padding can become a bottleneck for training efficiency.

Padding waste in a fixed-length micro-batch. A micro-batch padded to a fixed maximum sequence length (e.g., 8 tokens, as shown here) contains real tokens followed by padding tokens. In this example, 16 out of 40 tokens (40%) are padding, meaning a substantial fraction of the computation and memory is spent on tokens that carry no information.

Eliminating padding in transformer training: sequence packing

For transformer training, the canonical fix is sequence packing, where multiple variable-length sequences are concatenated into a single longer sequence, completely eliminating the need for padding. Transformers interleave token-wise layers, such as MLPs and normalization layers, with attention layers. Token-wise layers operate independently on each token and therefore handle packed sequences without any special treatment.

Attention layers, however, mix information across positions in the sequence. In many pretraining and SFT pipelines, naive packing without explicitly enforcing sequence boundaries often works in practice, as models learn to treat special sequence start and end tokens as soft delimiters. Online-RL is more sensitive to inference–training mismatch: a difference in rollout-time and training-time conditioning can introduce instability. For this reason, it is crucial to enforce sequence boundaries so that tokens from one sequence do not attend to tokens from another.

Modern attention implementations support this by respecting sequence boundaries and avoiding attention across sequences. Crucially, this does not use naive masking, which would still incur quadratic computation over the entire concatenated sequence. Instead, efficient implementations take sequence boundaries into account directly to restrict attention computation so that tokens only attend within their original sequence.

Sequence packing eliminates padding by concatenation. Multiple variable-length sequences are concatenated into a single dense sequence, avoiding padding entirely. Efficient attention implementations respect sequence boundaries so that tokens attend only within their...