Speeding Up Small Batch Decoding for Mixture of Experts
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 6 concepts
Key Takeaways
- Small-batch decoding with Mixture of Experts models is bottlenecked by repeated expert weight loading from memory.
- DeaMoE groups experts into several departments where experts within the same department share common backbone gate, up, and down projection matrices.
- DeaMoE incorporates small private sub-matrices within each expert for unique specialization.
- DeaMoE reduces per-step loaded weights by up to 50.9 percent compared to vanilla Mixture of Experts.
Summary & Methodology Analysis
Mixture of Experts, a neural network design pattern where only a subset of network parameters is activated per token, suffers from a severe bottleneck in real-time interactive applications. During small-batch decoding, inference is constrained by repeated expert weight loading from memory, which increases latency and limits throughput. To solve this, the paper introduces DeaMoE, which alters the standard Mixture of Experts component structure. Standard systems use a router function assigning tokens to top-k experts and compute using individual gate, up, and down projection matrices. DeaMoE groups all experts into several departments, meaning experts within the same department share common backbone gate, up, and down projection matrices, while incorporating small private sub-matrices within each expert for unique specialization.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main problem addressed by the paper?
Mixture of Experts inference in small-batch decoding is bottlenecked by repeated expert weight loading from memory, which significantly increases latency and limits throughput in real-time interactive applications.
Q2. What is the core contribution of DeaMoE?
DeaMoE groups all experts into several departments with shared backbone matrices, incorporates small private sub-matrices for unique specialization, and implements a two-stage routing strategy.
Q3. How much does DeaMoE reduce per-step loaded weights?
DeaMoE reduces per-step loaded weights by up to 50.9 percent compared to vanilla Mixture of Experts.
Q4. How does the two-stage routing strategy work in DeaMoE?
Tokens are first routed to departments, and departments process collected tokens uniformly before distributing them to specific experts.
Q5. What are the standard Mixture of Experts components mentioned in the paper?
Standard components include a router function assigning tokens to top-k experts, and expert computation using individual gate, up, and down projection matrices.
Q6. What are the limitations of the speedup gains in DeaMoE?
Speedup gains are smaller for smaller expert models on high-bandwidth hardware like the H100 because the bottleneck shifts away from weight loading when weights fit into L2 cache.
Q7. What happens if routing constraints are too strict?
Enforcing strict department coverage during routing degrades model quality compared to soft group-limited constraints.
Q8. Which models and datasets are listed in the extracted facts?
The models or datasets include DeaMoE, DeepSeek-V3, DeepSeek-V2, Qwen3-235B-A22B, Kimi-K2, Baseline-7B, RedPajama-v1, BoolQ, PIQA, SIQA, HSwag, WinoG, RaceH, AnliR1, AnliR2, AnliR3, OBQA, PTB, WikiText-103, and WikiText-2.
Q9. Does the paper specify exact training costs for DeaMoE?
The paper does not specify this.