Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Abstract summary

Introduces a sparsely-gated mixture-of-experts (MoE) layer for deep neural networks that achieves over 1,000x improvement in model capacity with minimal computational overhead. A trainable gating network routes each input to a small subset of expert sub-networks, enabling models with billions of parameters while keeping inference compute tractable. The architecture achieved state-of-the-art results on language modeling and machine translation benchmarks.

Our summary in our own words — see the canonical source links below for the original abstract.

Why we cite this paper

Mixture of Experts is the architectural foundation behind the most efficient frontier AI models (Mixtral 8x7B, DeepSeek MoE, GShard). For venture analysis, MoE adoption signals a startup's engineering sophistication: teams deploying MoE architectures achieve better cost-per-token economics, which directly impacts gross margin and unit economics. We track MoE adoption patterns in open-source AI startups as a momentum signal.

Key findings

• 1Sparsely-gated MoE layers achieve 1,000x+ capacity increase with only 6-7% compute overhead by activating only a subset of experts per input.
• 2A trainable gating network learns to route inputs to the most relevant experts, balancing load across the network automatically.
• 3Models with up to 137 billion parameters were trained efficiently, establishing the viability of trillion-parameter architectures.
• 4MoE enables conditional computation: model capacity scales with parameters but inference cost scales only with active experts.
• 5The batch-domain penalty for load balancing ensures no single expert becomes a bottleneck, a technique still used in modern MoE implementations.

Canonical sources

Frequently Asked Questions

Other research papers