Deepseek Model Architecture

DeepSeek v3 and R1 Model Architecture: Why it's powerful and economical

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Deepseek Model Architecture DeepSeek v3 and R1 Model Architecture: Why it's powerful and economical

PUBLISHED 2/7/2025

Table of Contents TLDR; Architecture Overview More Aggressive Mixture-of-Expert What changed from Deepseek v2 to v3?

Structural change

Local Balanced Routing to Experts

First ever FP8 Precision OSS LLM pre-training

Side Note on fp8 quantization How is it done?

Fine granularity quantization More Mantissa bits of fp8

Side Note on quantization errors: Better Accumulation Precision:

Online Quantization More customized data format and scaling factor Conclusion

Table of Contents

TLDR;

• This article introduces the architecture of DeepSeek v3 and DeepSeek R1, which shares exactly the same architecture. • Explain DeepSeek MoE (Mixture of Experts) and FP8 pre-training in depth.

Architecture Overview

DeepSeek v3 and R1 continue to use the traditional Transformer block, incorporating SwiGLU, RoPE, and RMSNorm. It also inherits Multi-head Latent Attention (MLA) and radical Mixture-of-Experts (MoE) introduced by DeepSeek v2. But what makes DeepSeek v3 so remarkable? Despite compute limitations, it leverages the scaling law by adopting a more aggressive MoE and utilizing FP8 precision for training .

More Aggressive Mixture-of-Expert

As we all know, linear layers of Feed-Forward Network are low-rank in nature (That’s why LoRA performs exceptionally well), that most parameters in the FFN are not as important. That leaves optimization opportunities: how to only activate the useful parameters for each incoming prompt? The result is a sparsely-activated model, more famously known as Mixture of Experts (MoE). (MoE does not seem like the most appropriate name, since the MoE under LLM context emphasizes more on sparsity than expertise. There aren’t any real “experts” involved here, and adding actual expertise could potentially have negative effects). Experiments from Mixtral have demonstrated that sparse large language models employing 8 experts, where only 2 are activated during inference, can achieve quality benchmarks comparable to similar-sized dense models. This opens up the possibility of achieving the same quality with the same amount of parameters, but at a much lower inference-time computation cost. Let’s recap the scaling law : As the number of parameters increases, larger models tend to achieve lower loss values by the end of pre-training. Therefore conversely, with the same inference cost, we can achieve higher quality benchmarks by increasing sparsity and boosting model size. This also raises an intriguing question: what if we move beyond the traditional paradigm of 8 experts with 2 activated? By dramatically increasing the number of experts—perhaps to over 100 or even 200—while maintaining a reasonable number of activated experts, we could potentially construct an ultra-large model. Model Name Activated / Total Parameters Number of Activated / Total Experts Mixtral 8x22B 39B out of 141B 2 out of 8 Grok-1 by xAI 86B out of 314B 2 out of 8 DBRX by Databricks 36B out of 132B 4 out of 16 Deepseek v2 21B out of 236B 8 out of 162 (2 shared) DeepSeek v3 37B out of 671B 9 out of 257 (1 shared)

What changed from Deepseek v2 to v3?

Structural change

The number of layers in DeepSeek v2 and v3 are nearly identical, with 60 and 61 layers respectively. However, the number of routed experts per layer increased by 60%, from 160 to 256. Doubling the FFN size means significantly more capacity for knowledge and memory. v3 also inherits the concept of the “shared expert”, i.e. an always-activated expert. Each FFN layer has 1 shared expert. Also for better representation of the input data, v3 increases the all-experts-activated layer from 1 to 3. So given 29.36M parameters per expert, and (61-3)_9 + 3 _ 257 = 1354 activated experts, we have 37.96B activated FFN parameters in total. Local Balanced Routing to Experts

How is a token assigned to an expert? The assignment is based on the token-to-expert affinity in embedding space. However, if all tokens repeatedly get routed to the same expert, this leads to an issue known as routing collapse. Routing collapse negatively impacts model quality during pre-training: even when the inputs are diverse, the model consistently selects only a few experts, saturating these parameters, while hindering sufficient training on other experts. DeepSeek v2 introduced three auxiliary losses—expert-level, device-level, and communication-level—to avoid routing collapse. However, these auxiliary losses can negatively impact model quality if they overshadow the token-to-expert affinity: this token is better suited for this expert, but routed to other experts for the sake of “balance”. Thus, v3 eliminates these auxiliary losses entirely and instead introduces a bias term to the gating value. This bias term is only used for routing purposes instead of being included in the overall loss, and only gets manually adjusted when its corresponding expert is overloaded/underloaded. Therefore the load balancing objective doesn't compete with the quality optimization objective. While DeepSeek v3 suffers from significantly worse load balancing, it ultimately results in better overall model performance.

First ever FP8 Precision OSS LLM pre-training

FP8 has been widely adopted as a quantization format during LLM inference, but using fp8 during training is a novel and innovative approach. Advantages of Using fp8 for Training: Compute Efficiency : Nvidia’s Tensor Core FP8 FLOPS are exactly double that of FP16. FP8 enables faster matrix multiplications and improves overall training speed. Memory Savings : Compared with bf16, fp8 reduces the memory in half, which allows larger and deeper models to fit within the same hardware constraints. Efficient Communication : fp8 lowers data transfer bandwidth requirements in distributed training, reducing communication overhead and improving synchronization efficiency across multiple GPUs. However, FP8 also introduces additional challenges: lower precision means lower numerical stability, leading to higher error rates per computation. The DeepSeek team invested countless engineering efforts to minimize quantization and computation errors. And they achieved a miracle: eventually, compared with the bf16 baseline, the relative...