Writing

Today's frontier LLMs are almost exclusively sparse MoEs, but it took the field many years to arrive at the architecture and training recipe they now mostly share. I revisit the papers and model releases that got us there and try to quantify what each idea added by measuring the uplift of MoEs over dense models.

A single paper showing that method X beats the alternatives almost never settles the question. I go through the reasons why the claims in a paper tend to be broader than the evidence behind them, illustrate them with examples from the MoE literature and describe what a reader can do about it.

Part 1 built a testbed that makes expert specialization measurable. In this second part I show how we used it on token balancing, and what the choice of method, the strength it is applied with and the scope it is computed over do to the routing an MoE actually learns.

Training a good MoE means training all experts sufficiently while having them specialize in non-redundant ways. Utilization is easy to measure, specialization is not. I walk through a paper we recently published proposing a routing testbed that makes specialization directly measurable, by constructing a data mix that comes with a strong, empirically validated prior for how tokens should be routed.

Mixture of expert models promise better model quality at the FLOPs of a dense model, and while the quality is real, the reality is more complex, involving memory, parallelism, communication and very careful implementation.