Mixture of Experts: Routers, Experts and Load Balancing

Learn how Mixture of Experts models run only part of the network for each token: the router, top-k expert selection, load-balancing loss, capacity limits, and the difference between total and active parameters.

28 minAdvancedCode Examples

The Core Idea: A Mixture of Experts replaces the feed-forward block with many experts and a router that sends each token to only the best few. The model can know a lot (large total size) while computing little per token (small active size).

  1. Beginner: A Team of Specialists

In a normal (dense) LLM, every token goes through every parameter. Making the model smarter means making it bigger, and a bigger model costs more for every single token.

A Mixture of Experts (MoE) model works like a hospital instead of one doctor. There are many specialists (experts) and a receptionist (the router) who sends each patient to the two or three who fit best. The hospital as a whole knows a huge amount, but each patient only uses a small part of it.

Dense modelEveryone works
Per token: Every parameter is used
Cost: Grows with total size
MoE modelOnly some work
Per token: The router picks the top few experts
Cost: Grows with the active size

Big idea: MoE separates how much the model knows (total parameters) from how much it computes per token (active parameters).

  1. Beginner: Where the Experts Live

In a Transformer block, the attention part is usually kept as is. The feed-forward part, which holds most of the parameters, is replaced by several parallel feed-forward networks, the experts, plus a small router network in front of them.

1
Attention: Runs normally for every token.
2
Router: A tiny linear layer scores every expert for this token.
3
Select top-k: Keep the best kk experts (often 1 or 2).
4
Run and mix: The chosen experts process the token, and their outputs are blended using the router weights.

Note: Experts do not usually specialize in neat human topics like math or French. They tend to specialize in subtler patterns, such as kinds of tokens or syntax.

  1. Medium: Routing in Detail

For each token the router computes a score for every expert and turns the scores into probabilities with a softmax. It keeps the top kk experts and renormalizes their weights so they add up to 1. The output is the weighted sum of those experts:

y=∑i∈top-kgi⋅Ei(x)y = \sum_{i \in \text{top-}k} g_i \cdot E_i(x)

SymbolMeaning
Ei(x)E_i(x)The output of expert number i for token x
gig_iThe router weight for that expert (the chosen weights add up to 1)
top-kOnly the k best experts run, usually k = 1 or 2

Why it saves compute: If the model has 8 experts and picks 2, only a quarter of the feed-forward parameters do any work for each token.

  1. Medium: The Load-Balancing Problem

A router that is left alone tends to collapse: it sends almost everything to a few favorite experts, which get better at their job and attract even more tokens, while the others stay untrained. That wastes capacity and makes the busy experts a bottleneck.

Training therefore adds an auxiliary load-balancing loss. It multiplies, for every expert, the fraction of tokens it received by the average probability the router gave it, then adds them up. The value is smallest when the load is spread evenly.

FixIdea
Auxiliary balance lossPenalize uneven token counts across experts
Capacity limitCap how many tokens one expert may take per batch. Extra tokens skip it or are dropped
Noise in the routerSmall random noise encourages exploring other experts
Shared expertsSome newer designs keep one or two experts that every token uses, plus many routed ones

Rule of thumb: If an MoE model trains badly, check the expert load first. A few overloaded experts are the classic sign.

  1. Advanced: Total vs. Active Parameters and Serving

MoE models are described with two numbers. Total parameters decide how much memory you need, because all experts must be loaded. Active parameters decide how much computing each token costs.

Model typeMemory needCompute per tokenStrength
Dense, 13BAbout 13B parametersAbout 13BSimple to run
MoE, about 47B total, about 13B active (for example Mixtral-style 8 experts, 2 per token)About 47B parametersAbout 13BKnowledge of a large model at the speed of a smaller one
Serving challenges
Experts are often spread across several GPUs (expert parallelism), so tokens must be sent between GPUs and back. That communication, plus keeping every expert loaded, is the main cost of MoE.

When to choose MoE: MoE shines when you can afford the memory and want high quality at lower per-token compute, typically in large-scale serving.

  1. Visual Explanation: The Router and the Experts

The figure follows one token through an MoE layer and then compares the three numbers you need to think about with MoE models.

Part 1 · One Token Through an MoE LayerThe router picks 2 of 8 experts
RoutingRouter
Token vector arrives after attention
↓
Router scores all 8 experts
↓
Keep the top 2, weights add up to 1
↓
Chosen: expert 0 (0.69) and expert 3 (0.31)
ComputingExperts
Expert 0 processes the token
↓
Expert 3 processes the token
↓
Outputs are blended by the weights
↓
Layer output (6 experts stayed idle)
Part 2 · Dense vs. MoESame quality goal, different cost shape
Total parameters
Memory
All experts must be loaded.
Active parameters
Compute
Only the chosen experts run.
Load balance
Health
Work should spread evenly across experts.
Click to see what a balance loss looks likeCompare ▼
The balance loss equals the number of experts times the sum over experts of (fraction of tokens sent to it) × (average router probability for it). It is about 1.0 when work is perfectly even and grows when a few experts dominate.

Part 1 shows the route of a single token: score, choose, compute, blend. Part 2 separates memory, compute and balance. Open the bar for the balance-loss formula in words.

  1. Python Implementation: A Top-2 Router and a Balance Check

This script builds a router from scratch for 1,000 tokens and 8 experts: it picks the top 2 experts per token, counts how many tokens each expert receives, computes a load-balancing loss, and works out total versus active parameters for an example model.

moe_router.pyPython 3.10+ · NumPy only
import numpy as np rng = np.random.default_rng(0)tokens, d_model, n_experts, top_k = 1000, 16, 8, 2 x = rng.normal(size=(tokens, d_model))router_w = rng.normal(size=(d_model, n_experts)) * 0.5 def softmax(z):    z = z - z.max(axis=1, keepdims=True)    e = np.exp(z)    return e / e.sum(axis=1, keepdims=True) # 1. The router scores every expert for every token, then keeps the top-kprobs = softmax(x @ router_w)chosen = np.argsort(probs, axis=1)[:, ::-1][:, :top_k]            # best 2 experts per tokengates = np.take_along_axis(probs, chosen, axis=1)gates = gates / gates.sum(axis=1, keepdims=True)                  # weights of the 2 experts add up to 1print("Token 0 goes to experts", chosen[0], "with weights", np.round(gates[0], 2)) # 2. How evenly is the work spread?load = np.bincount(chosen.ravel(), minlength=n_experts)print("Tokens handled by each expert:", load.tolist())print("Busiest / average expert load:", round(float(load.max() / load.mean()), 2)) # 3. Load-balancing loss: small when the router spreads work evenlyfraction_tokens = load / load.sum()                              # share of assignments per expertfraction_prob = probs.mean(axis=0)                               # average router probability per expertaux_loss = n_experts * np.sum(fraction_tokens * fraction_prob)print("Balance loss (1.0 means perfectly even):", round(float(aux_loss), 3)) # 4. Total vs active parametersdef moe_params(shared, expert, n_experts, top_k):    return shared + n_experts * expert, shared + top_k * experttotal, active = moe_params(shared=1.6e9, expert=5.6e9, n_experts=8, top_k=2)print("Example MoE: total", round(total / 1e9, 1), "B parameters, active per token", round(active / 1e9, 1), "B")
EXPECTED OUTPUT (your numbers may differ slightly)
Token 0 goes to experts [0 3] with weights [0.69 0.31]
Tokens handled by each expert: [241, 270, 240, 223, 283, 267, 273, 203]
Busiest / average expert load: 1.13
Balance loss (1.0 means perfectly even): 1.009
Example MoE: total 46.4 B parameters, active per token 12.8 B

The tokens spread fairly evenly across the 8 experts (the busiest has only about 13% more than average), and the balance loss is close to 1.0. The last line shows why MoE is attractive: about 46 billion parameters of knowledge, but only about 13 billion used per token. The sizes there are example numbers.

Interactive Knowledge Check

Question 1: What does the router in an MoE layer do?
Question 2: Why is a load-balancing loss used?

Key Points

✓MoE replaces the feed-forward block with several experts plus a router. Attention stays the same.
✓The router scores every expert, keeps the top k and blends their outputs with renormalized weights.
✓Only the chosen experts run, so compute follows active parameters, not total parameters.
✓All experts must still be stored, so memory follows total parameters.
✓Routers can collapse onto a few experts, so training adds a load-balancing loss and sometimes capacity limits.
✓Serving MoE needs expert parallelism and fast communication between GPUs.

The Big Picture

MoE in One Line

Many experts + a router + top-k selection = big knowledge at small per-token cost

MoE is one of the main ways modern models grow without every token paying for every parameter.

Up Next: Having a big model is only half the job: you also have to serve it to many users. In Efficient LLM Serving you will learn batching, paged memory and speculative decoding.