Mixture of experts (MoE)

Mixture of experts, or MoE, is an architecture where each layer holds several parallel sub-networks called experts. A router selects a small subset, often 2 of 8, to process each token, and the rest of the layer sits idle. The model carries the memory footprint of its full parameter count while doing the compute work of a much smaller one.

Mistral's Mixtral 8x7B popularized the approach, running 8 expert blocks per layer with 2 active per token. That works out to about 47 billion total parameters and roughly 13 billion active on any forward pass. DeepSeek-V3 pushed the idea much further, with 256 routed experts per layer and 671 billion total parameters, of which around 37 billion activate per token.

Plan GPU memory around the total parameter count, since every expert has to sit resident even though only a few run per token. Compute cost tracks the active count instead. A 671 billion parameter MoE will not fit a single card no matter how few experts fire per token.

Sources

Last verified August 29, 2026.