The Core Idea: A Mixture of Experts replaces the feed-forward block with many experts and a router that sends each token to only the best few. The model can know a lot (large total size) while computing little per token (small active size).
- Beginner: A Team of Specialists
In a normal (dense) LLM, every token goes through every parameter. Making the model smarter means making it bigger, and a bigger model costs more for every single token.
A Mixture of Experts (MoE) model works like a hospital instead of one doctor. There are many specialists (experts) and a receptionist (the router) who sends each patient to the two or three who fit best. The hospital as a whole knows a huge amount, but each patient only uses a small part of it.
Big idea: MoE separates how much the model knows (total parameters) from how much it computes per token (active parameters).
- Beginner: Where the Experts Live
In a Transformer block, the attention part is usually kept as is. The feed-forward part, which holds most of the parameters, is replaced by several parallel feed-forward networks, the experts, plus a small router network in front of them.
Note: Experts do not usually specialize in neat human topics like math or French. They tend to specialize in subtler patterns, such as kinds of tokens or syntax.
- Medium: Routing in Detail
For each token the router computes a score for every expert and turns the scores into probabilities with a softmax. It keeps the top experts and renormalizes their weights so they add up to 1. The output is the weighted sum of those experts:
| Symbol | Meaning |
|---|---|
| The output of expert number i for token x | |
| The router weight for that expert (the chosen weights add up to 1) | |
| top-k | Only the k best experts run, usually k = 1 or 2 |
Why it saves compute: If the model has 8 experts and picks 2, only a quarter of the feed-forward parameters do any work for each token.
- Medium: The Load-Balancing Problem
A router that is left alone tends to collapse: it sends almost everything to a few favorite experts, which get better at their job and attract even more tokens, while the others stay untrained. That wastes capacity and makes the busy experts a bottleneck.
Training therefore adds an auxiliary load-balancing loss. It multiplies, for every expert, the fraction of tokens it received by the average probability the router gave it, then adds them up. The value is smallest when the load is spread evenly.
| Fix | Idea |
|---|---|
| Auxiliary balance loss | Penalize uneven token counts across experts |
| Capacity limit | Cap how many tokens one expert may take per batch. Extra tokens skip it or are dropped |
| Noise in the router | Small random noise encourages exploring other experts |
| Shared experts | Some newer designs keep one or two experts that every token uses, plus many routed ones |
Rule of thumb: If an MoE model trains badly, check the expert load first. A few overloaded experts are the classic sign.
- Advanced: Total vs. Active Parameters and Serving
MoE models are described with two numbers. Total parameters decide how much memory you need, because all experts must be loaded. Active parameters decide how much computing each token costs.
| Model type | Memory need | Compute per token | Strength |
|---|---|---|---|
| Dense, 13B | About 13B parameters | About 13B | Simple to run |
| MoE, about 47B total, about 13B active (for example Mixtral-style 8 experts, 2 per token) | About 47B parameters | About 13B | Knowledge of a large model at the speed of a smaller one |
When to choose MoE: MoE shines when you can afford the memory and want high quality at lower per-token compute, typically in large-scale serving.
- Visual Explanation: The Router and the Experts
The figure follows one token through an MoE layer and then compares the three numbers you need to think about with MoE models.
Click to see what a balance loss looks likeCompare ▼
Part 1 shows the route of a single token: score, choose, compute, blend. Part 2 separates memory, compute and balance. Open the bar for the balance-loss formula in words.
- Python Implementation: A Top-2 Router and a Balance Check
This script builds a router from scratch for 1,000 tokens and 8 experts: it picks the top 2 experts per token, counts how many tokens each expert receives, computes a load-balancing loss, and works out total versus active parameters for an example model.
import numpy as np rng = np.random.default_rng(0)tokens, d_model, n_experts, top_k = 1000, 16, 8, 2 x = rng.normal(size=(tokens, d_model))router_w = rng.normal(size=(d_model, n_experts)) * 0.5 def softmax(z): z = z - z.max(axis=1, keepdims=True) e = np.exp(z) return e / e.sum(axis=1, keepdims=True) # 1. The router scores every expert for every token, then keeps the top-kprobs = softmax(x @ router_w)chosen = np.argsort(probs, axis=1)[:, ::-1][:, :top_k] # best 2 experts per tokengates = np.take_along_axis(probs, chosen, axis=1)gates = gates / gates.sum(axis=1, keepdims=True) # weights of the 2 experts add up to 1print("Token 0 goes to experts", chosen[0], "with weights", np.round(gates[0], 2)) # 2. How evenly is the work spread?load = np.bincount(chosen.ravel(), minlength=n_experts)print("Tokens handled by each expert:", load.tolist())print("Busiest / average expert load:", round(float(load.max() / load.mean()), 2)) # 3. Load-balancing loss: small when the router spreads work evenlyfraction_tokens = load / load.sum() # share of assignments per expertfraction_prob = probs.mean(axis=0) # average router probability per expertaux_loss = n_experts * np.sum(fraction_tokens * fraction_prob)print("Balance loss (1.0 means perfectly even):", round(float(aux_loss), 3)) # 4. Total vs active parametersdef moe_params(shared, expert, n_experts, top_k): return shared + n_experts * expert, shared + top_k * experttotal, active = moe_params(shared=1.6e9, expert=5.6e9, n_experts=8, top_k=2)print("Example MoE: total", round(total / 1e9, 1), "B parameters, active per token", round(active / 1e9, 1), "B")The tokens spread fairly evenly across the 8 experts (the busiest has only about 13% more than average), and the balance loss is close to 1.0. The last line shows why MoE is attractive: about 46 billion parameters of knowledge, but only about 13 billion used per token. The sizes there are example numbers.
Interactive Knowledge Check
Key Points
The Big Picture
Many experts + a router + top-k selection = big knowledge at small per-token cost
MoE is one of the main ways modern models grow without every token paying for every parameter.
Up Next: Having a big model is only half the job: you also have to serve it to many users. In Efficient LLM Serving you will learn batching, paged memory and speculative decoding.