Drop-in import replacement for HuggingFace Transformers that adds Expert Parallelism, DeepEP dispatch, and TransformerEngine kernels to MoE training without API changes.
Summary
Cuts MoE fine-tuning time by 3.4–3.7x and GPU memory by 29–32% while preserving the from_pretrained() API, letting you scale from single-node 30B models to multi-node 550B training without rewriting training code.
Why it matters
Cuts MoE fine-tuning time by 3.4–3.7x and GPU memory by 29–32% while preserving the from_pretrained() API, letting you scale from single-node 30B models to multi-node 550B training without rewriting training code.
Implementation verdict
Replaces HuggingFace Transformers v5 for MoE workloads. Requires PyTorch 2.5+, CUDA 12.1+, and a distributed setup config for multi-GPU; hand-tuned for Qwen3, Nemotron, GPT-OSS, DeepSeek V3, falls back to vanilla HF otherwise. Ready now for both single-node and distributed MoE fine-tuning.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.