Mixture-of-Experts Explained: Routing, Load Balancing, and Why Frontier Models Use Them

A dense transformer spends the same compute on every token: every parameter in every feed-forward layer participates in every forward pass. A mixture-of-experts (MoE) model breaks that link. It holds many parallel feed-forward blocks, called experts, and a small learned router sends each token to only a few of them. Total parameter count grows with the number of experts; per-token compute grows only with the number of experts each token actually visits. That decoupling is why so many open-weight and efficiency-focused models released since 2024 are sparse rather than dense, and why understanding routing has become basic literacy for anyone training or serving large models.

The core mechanism

In a standard transformer block, attention is followed by a feed-forward network (FFN). In an MoE layer, that single FFN is replaced by N expert FFNs plus a router. For each token, the router computes a score per expert, typically a linear projection of the token’s hidden state followed by a softmax, selects the top-k experts, and returns a weighted sum of their outputs. Attention layers usually stay dense; sparsity lives in the FFNs, which hold most of a transformer’s parameters.

The modern form traces to Shazeer et al.’s sparsely-gated mixture-of-experts layer (2017), which used noisy top-k gating to train LSTM-based models with up to 137 billion parameters. Google’s GShard and then the Switch Transformer (Fedus, Zoph and Shazeer, 2021) brought the idea to transformers at scale, with Switch simplifying routing to a single expert per token.

Total versus active parameters

The headline numbers for MoE models always come in pairs:

Active parameters approximate per-token FLOPs. Total parameters approximate memory footprint. The gap between the two columns is both the attraction and the operational cost of the architecture.

Routing choices

Routing design has moved through several distinct patterns, each a trade-off between quality, stability, and hardware efficiency.

Top-k token choice

Each token picks its k highest-scoring experts. Top-1 (Switch) minimizes compute and communication; top-2 (Mixtral) gives each token a blend of two specialists and tends to be more robust. The weakness is that nothing stops many tokens from picking the same expert.

Expert choice

Zhou et al. (2022) inverted the selection: each expert picks its top tokens from the batch, guaranteeing perfectly even load. The cost is that some tokens may be picked by no expert and others by several, and the scheme depends on seeing the whole batch, which complicates autoregressive decoding.

Fine-grained and shared experts

DeepSeek’s MoE design splits capacity into many small experts and adds a shared expert that every token passes through. The shared expert absorbs common knowledge, so routed experts can specialize more narrowly; many small experts give the router more combinations to work with. DeepSeek-V3 routes each token to 8 of 256 routed experts plus the shared one.

Load balancing: the hard part

If the router is left alone, it collapses. Early in training a few experts get slightly better, receive more tokens, improve faster, and attract still more traffic, while the rest starve. Collapse wastes parameters and, in distributed training, overloads the devices that host popular experts.

Capacity factors and token dropping

Expert-parallel systems allocate each expert a fixed buffer per batch, set by a capacity factor (for example 1.25 times the perfectly even share). Tokens routed to a full expert are dropped from that layer and pass through on the residual connection only. A higher capacity factor drops fewer tokens but wastes memory and compute on padding.

Auxiliary balancing losses

The Switch Transformer added an auxiliary loss that penalizes uneven distribution of tokens and router probability across experts. It works, but it adds a second objective that competes with the language-modelling loss; set its weight too high and quality suffers, too low and imbalance returns. The ST-MoE work added a router z-loss to keep router logits small and training numerically stable.

Auxiliary-loss-free balancing

DeepSeek-V3 replaced the main balancing loss with a per-expert bias term added to routing scores only for top-k selection. After each step, the bias of an overloaded expert is decreased and that of an underused expert increased. Because the bias does not enter the gating weights or the gradient of the main objective, the report argues it balances load with less damage to model quality, while retaining a very small sequence-level auxiliary loss as a safeguard.

Why frontier and efficient models use MoE

The case is economic. For a fixed training compute budget, a sparse model can hold far more parameters than a dense one. The Switch Transformer paper reported up to 7x faster pre-training than a dense T5 baseline with the same computational resources, and Mixtral’s paper reports that it outperforms or matches Llama 2 70B across the benchmarks it evaluated while using roughly 13B active parameters per token.

At inference the advantage is per-token compute: generation cost scales with active parameters, which helps throughput and latency at a given quality. That is a large part of why MoE has spread from frontier-scale systems to mid-sized open-weight releases intended for single-node deployment.

The costs that do not show up in the headline

• Memory, not FLOPs, becomes the binding constraint. All experts must be resident somewhere, so a 671B-total model needs the memory of a 671B model even though each token touches 37B. This shapes serving architecture more than any other property.
• Communication overhead. Expert parallelism places experts on different devices, and each MoE layer requires all-to-all exchanges of tokens. Interconnect bandwidth often limits throughput.
• Small-batch inefficiency. At low batch sizes, each expert sees few tokens, so hardware utilization drops. MoE pays off best under high-concurrency serving.
• Fine-tuning instability. Sparse models have historically been more prone to overfitting and router drift during fine-tuning than dense models of similar active size, so fine-tuning recipes often freeze or regularize the router.
• Specialization is not semantic. Analyses, including Mixtral’s own, find that experts tend to specialize by syntax and token patterns more than by human-readable topics. Treating an expert as “the maths expert” is usually a misreading.

Practical guidance

When comparing MoE models, read both parameter counts and plan hardware around the total, not the active figure. Check whether a model’s inference cost estimates assume the batch sizes you will actually run. For training, choose a balancing strategy before tuning anything else, monitor per-expert token share as a first-class metric, and treat dropped-token rates as a quality signal rather than an implementation detail.