Moe Guide Calculator

Cerebras Skip to main content

Cerebras Announces First Quarter 2026 Results >>

Oct 14 2025 MoE Math Demystified: What Does 8x7B Actually Mean? Daria Soboleva Etienne Goffinet

MoE Fundamentals | Router Wars | Debugging Dead MoE Models | MoE at Scale | MoE Math Demystified

This video breaks down MoE inference arithmetic and deployment bottlenecks across different hardware setups. If you can’t open the video displayed above, please use this link to open it on YouTube: https://youtu.be/gHpDBoyCOrE What does 8x7B actually mean? You probably thought it meant 8 experts with 7B active parameters per token. We did too. Turns out it is actually 13B active parameters. But wait, where does 13B come from? This is exactly the kind of confusion this post clears up ( skip to the answer ). We'll explain what those numbers actually mean for inference by answering how much memory you need, how many GPUs, and what the commonly hit bottlenecks are in production deployment. We'll show that single-GPU deployment is memory-bound, multi-GPU setups are communication-bound, and specialized hardware like Cerebras WSE is compute-bound. Originally, we set out to write a simple post on MoE arithmetic. Then we kept digging. And digging. What started as basic math turned into a full explanation of inference bottlenecks, hardware architectures, and deployment strategies. Welcome to MoE inference 101. The title stayed, but the scope didn't. Up to this point, the Mixture-of-Experts (MoE) series has focused on training aspects. In part 3, you trained your own MoE model and scaled it in part 4 to production size. Now what? We shift to inference. During inference, model weights are frozen, no gradients or optimizer states. This sounds much simpler compared to training mode. But! Despite doing less work, MoE inference has its own challenges. Fun fact: people who train models rarely think about inference costs, and vice versa. Your authors are no exception, but we are trying to be better. So, if you're deploying MoE models, or you trained an MoE model and want to know what your design choices have led to, or you simply want to understand how to run MoE inference efficiently, keep scrolling. How much memory do I need? Want to avoid hitting an OOM error during inference deployment? Let’s examine two components that dominate memory space: model weights and kv-cache. To simplify our calculations, we are going to use standard modern transformer setup. We use RoPE positional embeddings (Su et al., 2023), SwiGLU nonlinearity (Shazeer, 2020; Dauphin et al., 2016), layer norms, multi-head attention, untied embeddings, and industry standard learned routing (Soboleva, 2025a). Model weights Let's walk through the math. First, we'll calculate a single decoder block memory requirement, then account for all layers , and finally add the remaining network parameters not included in the decoder blocks (bottom to top). Bias terms are omitted from the following equations as they are negligible (regardless of the model size). Our MoE model consists of an embedding layer, followed by decoder blocks, and an unembedding layer. Each decoder block contains two layer norms, an attention layer, a router, and an MoE layer with expert networks (Figure 1). Figure 1: Visual breakdown of MoE model decoder architecture. The MoE model consists of an embedding layer, followed by decoder blocks, and an unembedding layer. Each decoder block includes two layer norms, an attention layer, a router, and an MoE layer with expert networks. We start with embedding weights: The embedding layer consists of an input embedding matrix of size . Following the embedding layer, next comes the layer norm: You can see that we have a multiplier here. This is because we have two layer norms per decoder block, and each layer norm stores gains and biases of the same size . Next, let's do the math for the attention layer:

In attention, we account for four weight matrices (query, key, value, and output) each with size . Perfect, now comes the first MoE component, the router: In the case of learned routing, we need to store a weight matrix of size with learnable router weights. The last thing we need to account for is the MoE layer weights: Each expert network uses SwiGLU nonlinearity and thus requires three linear transformations (gated linear unit, up-projection, and down projection). With dynamic routing, we can’t predict which experts will be activated, so you must provision enough device memory to store all experts. Thus, MoE models require more memory capacity than dense networks with equivalent active parameters. Finally, we use untied embeddings, so we need to account for an additional memory of an unembedding matrix of size : Combining all of the weights together in the decoder block, we get:

Accounting for decoder blocks in the network, and adding embedding and unembedding weights, we get the total amount of memory we need in bytes to hold model weights: Next, let's answer the question of how much compute we need to run inference. How much compute do I need? Just as we estimated the amount of memory required to run inference with our MoE model, here we will focus on estimating the number of FLOPs (Floating Point Operations). Later, we will convert our FLOPs into throughput and latency metrics. These metrics will help us answer how fast our MoE model will run. There are two important stages in inference. The first one is called prefill. During prefill, the model processes the entire prompt all at once. During the second stage, we start generating our answer. With autoregressive transformers, generation happens one token at a time. Each new token depends on the tokens previously generated. We call the second stage decode. Let's focus on calculating the FLOPs we spend during the prefill stage and then do a separate analysis for decode. Most of the operations in the transformer consist of matrix-vector, matrix-matrix, and Hadamard computations. So let's establish how many FLOPs are needed for these base operations. Prefill Cool, now we are ready to dive into the actual computations. Note that we will ignore an embedding layer. It is usually implemented as lookup table and doesn't require arithmetic operations. We start with layer norms: We need 7 FLOPs per element (for sequence with tokens and hidden dimension ) to calculate layer norm FLOPs. This comes from computing the mean and variance across the embedding...