NousResearch/DisTrO

NousResearch/DisTrO

Description: Distributed Training Over-The-Internet

Stars: 1034

Forks: 56

Open issues: 1

Created: 2024-08-26T15:30:54Z

Pushed: 2025-10-14T15:37:16Z

Default branch: main

Fork: no

Archived: no

README:

DisTrO

This is the repository for DisTrO (Distributed Training Over-The-Internet), a family of low latency distributed optimizers that reduce inter-GPU communication requirements by three to four orders of magnitude.

• [x] Aug. 26th, 2024: DisTrO (Preliminary Report)
• [x] Dec. 2nd, 2024: DeMo Optimization (Paper) (Code), original seed research/idea for DisTrO
• [x] Dec. 2nd, 2024: Nous trains a 15b model using DisTrO
• [x] May 14th, 2025: Psyche Network
• [x] May 14th, 2025: Nous Consilience 40b LLM, Huggingface
• [x] Oct. 14th, 2025: DeMo Optimization (Paper v2), (Production Code)

Join us on Discord if you're interested in helping research and build the future of distributed training.