deepseek-ai/DeepEP-Ascend

deepseek-ai/DeepEP-Ascend

Description: A high-performance communication library for machine learning training and inference on Huawei Ascend NPUs.

Language: C++

Stars: 123

Forks: 7

Open issues: 0

Created: 2026-09-30T00:44:59Z

Pushed: 2026-09-30T00:53:14Z

Default branch: main

Fork: no

Archived: no

README:

DeepEP-Ascend

（中文介绍）DeepEP-Ascend 是面向华为昇腾 NPU 的高性能机器学习训练与推理通信库，提供 MoE dispatch/combine 的专家并行（EP）all-to-all 操作，支持 FP8 dispatch 和延迟 epilogue。此外还提供流水线并行（PP）、面向上下文并行和数据并行（CP/DP）的 Bucket 集合通信，以及 Engram 远端内存访问等通信原语（开发中）。其公开 buffer API 与 NVIDIA 版 DeepEP 对齐。Ascend C 内核使用 HCCL/HCOMM、UBMEM 和 URMA 完成通信，并通过 DeepJIT 在运行时编译。

(English introduction) DeepEP-Ascend is a high-performance communication library for machine learning training and inference on Huawei Ascend NPUs. It provides expert-parallel (EP) all-to-all operations for MoE dispatch and combine, including FP8 dispatch and deferred epilogues. It also offers communication primitives (work in progress) for pipeline parallelism (PP), context and data parallelism through Bucket collectives (CP/DP), and remote memory access through Engram. Its public buffer APIs are aligned with the NVIDIA version of DeepEP. Ascend C kernels use HCCL/HCOMM, UBMEM and URMA, and are compiled at runtime through DeepJIT.

Performance

Measured on Ascend 950DT NPUs with CANN 9.2.0 and the manually configured PoC HDK described under [Recommended HDK and firmware](#recommended-hdk-and-firmware). All measurements use netlayer 1, the supernode's external Clos network.

The following runs use 16,384 tokens/rank capacity, hidden size 7168, top-6 routing over 256 experts, 32 AI cores / 64 AIVs, expert alignment 128 and no bias. Dispatch uses FP8 with row-major scales; combine uses BF16. Each rank uses 10 warmups and 50 samples with cache flushing.

| EP size | Dispatch (GB/s) | Combine (GB/s) | | --- | ---: | ---: | | EP8 | 373–375 | 345–347 | | EP16 | 348–352 | 338–341 | | EP32 | 335–340 | 320–324 | | EP64 | 323–327 | 294–298 | | EP128 | 313–320 | 272–278 |

Bandwidth ranges cover all ranks, using timings that include issue and drain but exclude final epilogues. Sustained dispatch reaches roughly 90–95% of the physical payload bandwidth limit for EP sizes up to 32. Larger EP sizes and combine remain under optimization; combine has additional local reduction overhead and HBM contention with URMA. Huawei firmware improvements are also planned to reduce this contention.

Quick start

Requirements

• Linux on an Ascend host with supported NPU drivers and firmware.
• Ascend 950 (A5) with UBMEM connectivity and UBC_CTP/URMA channels between the participating ranks.
• An even number of ranks for multi-rank communication.
• CANN with Ascend C, the Bisheng compiler, HCCL and HCOMM headers/libraries.
• Python 3.10 or newer, a matching PyTorch/torch_npu installation, and NumPy for the tests.
• A C++20 compiler and standard library supporting std::format.

The validated stack is Ascend 950DT, CANN 9.2.0, Python 3.12, PyTorch 2.13.0+cpu and torch_npu 2.13.0rc1, with the PoC HDK configuration described below. Kernel support on other Ascend generations or CANN versions has not been established by these measurements.

Installation builds the host extension; device kernels are compiled on first use. Keep the CANN toolkit and Bisheng available at runtime. Set ASCEND_HOME_PATH to the active CANN installation, or use ASCEND_TOOLKIT_HOME.

Recommended HDK and firmware

Use Huawei's Q3 commercial HDK release for Atlas 850E when it becomes publicly available. Huawei has advised that this release includes the configuration required for full-bandwidth operation. Public availability is currently planned for mid-October 2026, around October 15, through the Huawei Atlas 850E software download page, where users can apply for access when the release becomes available. The date is a vendor release plan; availability is subject to Huawei's publication schedule.

The performance results in this README were collected using a PoC HDK supplied to DeepSeek, with additional manual configuration. That configuration is not a publicly distributed release. Users on earlier PoC versions may observe lower bandwidth and should first check their HDK version and configuration. Use the Q3 commercial release, once available, as the recommended public deployment baseline; these measurements are not results from that unreleased commercial version.

Installation

Activate the CANN environment and install the matching PyTorch/torch_npu packages first, then:

git clone https://github.com/deepseek-ai/DeepEP-Ascend.git
cd DeepEP-Ascend

# Fetch the required DeepJIT submodule over HTTPS.
git submodule update --init --recursive third-party/deep_jit

python -m pip install --no-build-isolation .

For development, use bash develop.sh to build and link the extension into the source tree. The clangd-ascend submodule and CMake configuration are for IDE/debugging support.

Features and API compatibility

Training, inference prefill and decoding share the same EPBuffer interface and the deep_ep Python package name. The implementation follows DeepEP's EPBuffer-based V2.5 APIs; supported modes and stream behavior are specific to Ascend.

| Interface | Ascend support | | --- | --- | | EPBuffer | Expanded BF16/FP8 dispatch, BF16 combine, cached handles, deterministic layouts, expert padding and deferred epilogues | | PPBuffer | Send/receive between adjacent pipeline ranks | | EngramBuffer | NPU-backed BF16/FP8 tables and per-layer fetch hooks | | BucketBuffer | Batched all-gather, registered storage, multiple groups and sessions for ordinary tensors | | BufferAllocator, BufferBase, deep_ep.comm | Allocation planning, buffer lifecycle, communicator queries, barriers and stream access |

PP, Engram and Bucket are experimental; remaining work is listed under [Ongoing](#ongoing). EP requires do_expand=True and allow_multiple_reduction=True; do_handle_copy must remain False.

Routing indices use deep_ep.topk_idx_t, currently fixed to torch.int64. Explicit RDMA service levels and QP counts...