Reap

Cerebras Skip to main content

Cerebras Announces First Quarter 2026 Results >>

Oct 16 2025 REAP: One-Shot Pruning for Trillion-Parameter Mixture-of-Experts Models Mike Lasby Ivan Lazarevich Nish Sinnadurai Sean Lie Yani Ioannou Vithursan Thangarasa

TL;DR: We introduce REAP (Router-weighted Expert Activation Pruning), a new one-shot method for compressing Mixture-of-Experts (MoE) language models. Our key finding is that for generative tasks like code generation pruning low-impact experts is fundamentally better than merging them. REAP removes up to 50% of experts from models as large as 1 trillion parameters while largely maintaining baseline model quality. For instance, with the Qwen3-480B-Coder-FP8 model, REAP at 50% pruning retains 97.6% of its baseline non-agentic coding ability and 96.7% on the agentic SWE-Bench benchmark. We are open-sourcing the complete codebase and pruned model checkpoints on HuggingFace to encourage further research. Leveraging Expert Redundancy for MoE Compression Sparsely-activated Mixture-of-Experts (SMoE) models achieve their high quality by  decoupling their total parameter count from their computational cost [1]. This allows them to leverage a much larger parameter budget for greater model capacity, while only activating a small, computationally efficient subset of “expert” networks for any given input. However, this efficiency comes with a steep memory cost. Models such as Qwen3-480B-Coder and Kimi-K2 contain hundreds of billions of parameters, even though only a fraction are active at any time. This memory footprint is a significant barrier to deployment and research. Prior works show a critical insight: large MoE models contain significant  expert redundancy [2, 3]. Not all experts contribute equally to the tasks we care about, and expert usage is often highly imbalanced. Some experts are rarely chosen by the model's router, and their impact on the final output can be minimal even when they are selected. This redundancy creates an opportunity for model compression. By identifying and removing these low-impact experts, we can achieve a significant reduction in memory footprint without degrading model quality. This leads to a fundamental question: is it better to remove these experts entirely with pruning or to combine them with others with merging? While merging seems intuitively appealing, our work reveals a critical flaw in that approach. What Goes Wrong with Expert Merging Recent work suggests that merging experts is superior to pruning them [4, 5]. These results seem intuitive; instead of discarding an expert entirely, why not average its weights with a similar one to preserve some of its learned information? Early studies confirm this, showing that merging outperforms pruning on discriminative tasks like multiple-choice question answering. However, our work shows this does not hold true for generative tasks . In our experiments, pruning proves to be the superior strategy, consistently achieving higher model quality than merging across all generative benchmarks.These are tasks like code generation, tool calling, mathematical reasoning, or creative writing, which require the model to produce diverse or structured output instead of just picking an answer from a list. This finding highlights that compression strategies optimized for discriminative tasks may not directly translate to generative settings. Expert Merging Challenges: Functional Subspace Collapse Our work shows that merging experts introduces a fundamental and  irreducible error . To understand why, we first need to look at how a healthy MoE layer works. The router's main job is to perform  input-dependent mixing . For example, it might combine 70% of Expert A with 30% of Expert B for one token, then adapt to a 40/60 mix for the next. This dynamic ability is crucial for generating high-quality, nuanced output. Current merging techniques inhibit this capability as they combine the router’s gate-values for the merged experts by summation. When you merge experts A and B, you replace them with a single, static average. The router loses its freedom to choose; it is now forced to use that one fixed average for  all  inputs. This loss of dynamic control is what causes the irreducible error. We call this outcome  functional subspace collapse , because the range of possible outputs the model can produce dramatically shrinks. You simply cannot recreate the flexibility of dynamic mixing from a static average. The Core Intuition - The irreducible error from merging is proportional to: How much the router varies its mixing strategy (policy variability) How different the two experts are (expert gap) The magnitude of their gate-values (router scale)

Pruning avoids this error by maintaining the router's operational freedom. When an expert is removed, the router's control over all  surviving  experts remains completely independent. As a result, the router can continue to modulate each one dynamically, fully preserving the model's ability to adapt its mixing strategy based on the input. We visualized this collapse by projecting expert activations onto their first two principal components across different layers of a model. For example, in early layers (Layer 0 shown in Figure 1), the original 128 experts of Qwen3-30B-A3B form a compact distribution along the diagonal. After pruning 50% of the experts, the 64 surviving experts maintain the same geometric structure, overlaying the original distribution faithfully. Merging, however, contracts the distribution of expert activations toward the center, a visible but modest compression. The contrast becomes more evident in late layers (Layer 47 shown in Figure 2), where experts have specialized for distinct computational roles. The original experts span a wide range from PC1 coordinates of approximately -100 to 200. Pruning preserves this full breadth with 64 experts distributed across the same space. But merging induces catastrophic collapse: all merged experts compress into a tight cluster near the center, representing nearly a 100x reduction in functional diversity. This dramatic difference validates our theoretical analysis that irreducible error is proportional to policy variability. Early layers exhibit lower policy variability and thus modest collapse, while late layers with high policy variability suffer severe functional collapse when specialized experts are merged. REAP: Pruning by Impact, Not Just Frequency If...