Writing — Tobias Falke

MoEs Over Time: What Changed And How Much Does it Matter?

Thu, 10 Sep 2026 00:00:00 +0000

After having spent the majority of my work from 2023 to 2026 on pretraining MoE models, I’ve always wanted to look back and write down how the field evolved over that period and which changes ended up actually making a difference.¹ That’s what this post is about. Today’s LLMs, especially the largest frontier models, are almost exclusively sparse MoE models. And while the idea of MoEs has been around for quite some time, it took a while for the field to arrive at this state. I will revisit the seminal papers and model releases that created that shift from dense to sparse.

But more importantly, I will also try to quantify the benefit that MoEs brought. The first post of this series discussed the central promise of sparse MoE models, getting the quality of a larger model for the training and inference cost of a smaller model (and the fine print that comes with that promise). I will refer to that here as the uplift: Given an MoE with $N_{a}$ active and $N_{t}$ total parameters, the model will show quality comparable to a dense model sized somewhere between those two; we’ll refer to that size as the effective size $N_{eff}$. The uplift factor is then defined as $$uplift = \frac{ N_{eff} }{ N_{a} }$$ expressing how much better the MoE is than a dense model with matched active parameters. To illustrate that metric: An uplift factor of 2.5 at sparsity 4% means an MoE with 10B active parameters (and 250B total) would be as good as a dense model with 25B parameters.

That uplift of course depends on the sparsity of the MoE. The sparser it is, the further the quality can increase beyond the active parameters, at the price of having to handle the total parameters. In this post, I will denote sparsity as activation ratio $N_{a} / N_{t}$, so lower means sparser. If we keep the sparsity fixed, uplift becomes a function of how well an MoE is trained to make the most out of the total parameters, which comes down to the MoE architecture and the training recipe. And how these developed over time is what this post is about.

A brief history of MoEs

A selection of influential MoE papers and representative MoE model releases from 2020 to today.

The research era (2017-2022)

Sparse MoEs as we know them today were first introduced by Shazeer et al. (2017), who tested the idea in LSTM-based models on language modeling and machine translation tasks, slightly predating the transformer architecture (June 2017) that would later dominate language modeling. GShard (Lepikhin et al., 2020) adopted the idea into transformers and tested top-2 routing at various degrees of sparsity. Switch-Transformers (Fedus et al., 2021), which became one of the most influential MoE papers of this era, simplified the setup to top-1 routing, which was previously believed to be unstable. GLaM (Du et al., 2021) later scaled up the GShard top-2 setting and ST-MoE (Zoph et al., 2022) discussed training stability challenges and proposed using z-loss for routers as one mitigation. Notably, all of that early fundamental work came out of Google. Concurrently, Meta (then Facebook) also contributed to the topic, introducing the linear assignment approach to token balancing in BASE layers (Lewis et al., 2021), hash routing (Roller et al., 2021) and thorough comparisons to dense models (Artetxe et al., Dec 2021). Expert choice (Zhou et al., 2022) challenged the dominant top-k routing paradigm, but it never caught on due to its difficulties with auto-regressive decoding. Clark et al. (2022), which I tend to cite heavily across my blog, combined much of that early work in a comprehensive scaling study of MoEs and dense models that precisely quantified the effect of sparsity. Also in this post, I will rely on it for measuring uplift.

The work in these years has introduced, beyond the general concept of how to integrate sparse experts and routing into transformers, many of the core ideas and design decisions that MoEs have relied on since then: routing paradigms (token choice with top-k, expert choice, hash routing), expert parallelism, capacity factors and token dropping, MoE layer placement (e.g. every nth layer), token balancing approaches (auxiliary load balancing loss, BASE-style linear assignment, Sinkhorn approximation) and stability measures such as router z-loss. Unlike today, all of this was done purely as published research. The rise of LLMs and the regular release of such models had yet to start. Which is why I call it the research era.

The ChatGPT and Llama era (2023-2024)

In November 2022, ChatGPT was released, and language models transitioned from being a research problem to powering real-world products. Various companies started training their own LLMs and releasing them as either closed models behind APIs or as public checkpoints. And while that development somewhat reduced the willingness of companies to publish research as openly as they did before, the open-weights part of the LLM space filled that gap by providing (more or less) detailed tech reports. Meta’s Llama was the first such example. After the first open models were exclusively dense, Mixtral (Jiang et al., 2024) in late 2023 was the first open MoE of this era, using top-2 routing over 8 experts with a Llama-inspired backbone. Right after, DeepSeek published DeepSeek-MoE (Dai et al., 2024), which turned out to be the most influential MoE paper of this era. It introduced the key new ideas of this time, namely expert granularity and shared experts. Expert granularity, also known as fine-grained experts, replaces experts sized as FFNs in dense models, which previous work relied on, with experts smaller by a factor $G$, but at the same time increases the total number of experts as well as the number of active ones by the same factor $G$. That keeps both total and active parameters unchanged, but gives the router a much larger space to choose from, e.g. 4 out of 32 instead of 1 out of 8. Shared experts, on the other hand, are FFNs that are always selected in addition to the routed experts, hence a dense path in parallel to the MoE.

Gemini 1.5 (Gemini Team, 2024) was the first closed model publicly confirmed to be an MoE (while GPT-4 was always just rumoured to be so). Notable open MoEs in 2024, among many others, included Grok-1 (xAI, 2024), still following Mixtral’s architecture closely, DBRX (Databricks, 2024), which was slightly more granular (2x), Snowflake Arctic (Snowflake AI Research, 2024), featuring a parallel attention architecture variant, and OLMoE (Muennighoff et al., 2024), which adopted DeepSeek’s ideas (but decided to not use shared experts) and was accompanied by a particularly detailed tech report. Another trend at the time, to produce MoEs quickly given their rising popularity, was to upcycle them from existing dense models, such as Qwen’s first MoE (Qwen Team, 2024). While that is obviously cheaper, it also leaves a significant part of the MoE’s potential untapped, resulting in lower uplift factors.

Compared to the research era, this era did not introduce as many new ideas (though with granularity one of the most influential ones), but is rather characterized by the slow yet steady adoption of the MoE paradigm in the LLM landscape. Many released models still remained dense transformers though. The last missing piece, as it turned out, was a successful blueprint for delivering the MoE promise at scale considering both modeling and systems aspects for efficient training and inference (see this post). While DeepSeek-V2 (DeepSeek-AI, 2024) already introduced some of these aspects, e.g. device-limited routing, only DeepSeek-V3 was that full holistic picture and marks the beginning of the next era.

The DeepSeek V3 era (2025-2026)

DeepSeek-V3 was released in December 2024 (DeepSeek-AI, 2024). On the modeling side, it kept DeepSeek-MoE’s ideas of fine-grained experts and shared experts, and added loss-free token balancing based on expert biases (and switched to sigmoid activations). But more importantly, it came with detailed discussions of the necessary engineering and systems work to train and inference an MoE of that size (670B parameters). Several other models followed that blueprint and modified it only slightly, such as Kimi K2 (Kimi Team, 2025) which increased sparsity and trained with Muon, or GLM 4.5 (GLM-4.5 Team, 2025) which varied the width-to-depth ratio, and Ant Group’s Ling/Ring family of models (Ling Team, 2025). This continued in 2026, when several US open model releases also followed the DSV3 architecture closely (Arcee Trinity, Laguna, Inkling).

Qwen 3 (Yang et al., 2025), the first from-scratch MoE of the Qwen series (as opposed to being upcycled), differed more by using no shared experts and relying on the established auxiliary loss for balancing.² In mid 2025, OpenAI released their first ever open-weights MoE, gpt-oss (OpenAI, 2025), also featuring fine-grained but no shared experts. And 2025 had the last release of the Llama series, Llama 4 (Meta, 2025), but since its impact in the field remained far behind its predecessors, I would not want to put too much weight on its MoE architecture (it was the first (and only) MoE in the Llama series and made a few unorthodox choices). LatentMoE (NVIDIA, 2026) was one of the rare new ideas of this era that found adoption, e.g. in MAI Thinking 1.

In these (almost) two years following the DSV3 launch, MoE clearly became the dominant approach for at least the largest LLMs. Many new companies entered the field and contributed additional closed or open models. The models also grew in size, up to 2.8T total parameters in the latest Kimi K3, and the activation ratio settled at 3-5%. But in the architecture and training recipe itself, not much changed: All models use fine-grained experts, some add shared experts, some not, some have initial dense layers, others not, most use sigmoid or softmax for the router, and expert biases or the load balancing loss (globally in either case) for token balancing. Of course, a lot of other changes went into these models, such as various ideas for more efficient attention architectures, or increasingly lower precision training, but I want to focus here on the MoE architecture and training. On that side, even in the most recent models as of writing this in late September 2026, such as Qwen 3.8 Next Flash, GLM 5.3 Flash, DeepSeek V4.1 Flash and MiMo V2.6 Flash, nothing new enters the picture.

Measuring uplift

Having reviewed the MoE models of the previous years and what went into them, I want to now turn to measuring the MoE uplift over dense models, $N_{eff} / N_{a}$, as introduced at the beginning of the post.

Controlled scaling studies

The best data source to measure MoE uplift is a controlled scaling study that compares MoE models with dense models. Luckily, several such studies have been done over the years. Since they train dense and MoE models under otherwise identical conditions, such as same training data, same backbone architecture (attention type, FFN type, width, depth, etc.), the loss difference between matched dense and MoE models can be fully attributed to the MoE architecture and training recipe and hence reflects what I call the uplift. And in addition, scaling studies typically sweep dimensions such as model size, data volume and sparsity, allowing us to measure uplift along these dimensions.

As I already mentioned, Clark et al. (2022) is a great example of such work, with the only caveat that they trained models of all sizes and sparsities on a single fixed amount of data. But since they released loss values for all models and all intermediate checkpoints, we can slice out a portion that holds tokens-per-parameter (tpp) constant as in Chinchilla-style scaling laws (at 315 tpp in this case). This yields the following plot of active parameters vs. validation loss at different sparsities:

Validation loss as a function of active parameters and sparsity (as activation ratio), in log-log-space, based on Clark et al. 2022. The dotted line illustrates how MoEs, here at 57M active parameters and 4.5% activation ratio, are mapped to their dense equivalent size by matching at validation loss.

To compute uplift, we can fit the dense model loss curve with a linear function in log-log-space that maps parameters to loss, invert it, take the loss of an MoE at a specific sparsity and parameters $N_{a}$ and compute the dense parameters showing that same loss, which gives us $N_{eff}$ and thereby the uplift factor $N_{eff} / N_{a}$. The following table shows the resulting uplift factors for Clark’s models that have between 15M and 370M active parameters:

Act. Ratio	Uplift Range
50.0%	1.22x - 1.38x
16.7%	1.62x - 1.88x
4.5%	2.15x - 2.54x
1.2%	2.64x - 3.12x

As expected, the sparser the model (the lower the activation ratio), the higher the uplift is. For the nowadays typical sparsity range of 4%-8% activation ratio, the uplift is around 1.9 to 2.5. This uplift factor characterizes the effectiveness of the specific MoE setup used in that study: MoE in every second layer, top-1 routing, full-sized experts following the FFN sizing in dense models (expansion factor 4.0) and Sinkhorn balancing. That is very close to the Switch-Transformers recipe which was the most popular approach in the research era.³

Three years later, Ludziejewski et al. (2025) ran a similar scaling study. The MoE setup was again Switch-style, in this case even closer⁴, so it gives us a second data point for the effectiveness of that recipe. The table below compares the uplift factors, but also reveals that data volume (and hence tpp) is an important factor for uplift:

Act. Ratio	Clark	Ludz (A)	Ludz (B)
30.0%	1.39x - 1.60x	1.17x - 1.21x	1.37x - 1.43x
16.7%	1.62x - 1.88x	1.30x - 1.37x	1.72x - 1.83x
8.8%	1.90x - 2.22x	1.45x - 1.55x	2.24x - 2.41x
4.5%	2.15x - 2.54x	1.62x - 1.77x	2.98x - 3.19x
tpp	315	20	23 - 64
$N_{act}$	16.5M - 368M	321M - 1.1B	321M - 1.1B

In setting A, the tpp for the active parameters is held constant, as we did for Clark, but here at the Chinchilla-derived value of 20 (the data does not allow using the same tpp for both studies). The uplift factors are lower, which I assume is primarily due to this undertraining aspect. But Ludziejewski’s data also lets us do something more interesting: Keeping the data volume fixed per active parameter underestimates the MoE’s potential since it scales data only with the lower active size, but not the true effective size, putting it at a disadvantage compared to the dense model. A more honest comparison would be a constant tpp against $N_{eff}$, which is what setting B shows (resulting in tpp w.r.t. $N_{a}$ of 23-64), leading to significantly higher MoE uplift of 2.2 - 3.2 at the previously mentioned sparsity range.

As the third relevant data point, Tian et al. (2025) published an MoE vs. dense scaling study in the same year. Their MoE setup is almost identical to Ludziejewski’s, but they add the two new ingredients DeepSeek proposed in 2024, fine-grained experts and shared experts, making it a representative of the second era. And, luckily, they model the influence of granularity directly in their study, which allows us to measure uplift at $G=1$ (full experts, resulting in Switch-style except with a shared expert), $G=4$ (DeepSeek-MoE’s granularity) and $G=24$ (the optimum in their sweep). The overtraining regime is comparable to setting B of Ludziejewski, scaling data with effective size.⁵ The following figure shows the results next to the other two studies:

MoE uplift at different activation ratios measured for Switch-style and DeepSeek-style MoE architectures at different tokens-per-parameter regimes scaled with either active (tpp-act) or effective (tpp-eff) model size.

That picture illustrates what those three studies can tell us about MoE uplift in the first two eras of my MoE history above. Using the 9% activation ratio point as a reference:

The Switch-style architecture and training recipe provides about 1.5x uplift at Chinchilla-optimal tpp=20 training and about 2.0x with stronger overtraining at tpp=315. Scaling data volume with $N_{eff}$ instead of $N_{a}$ puts it at 2.2x - 2.4x even in the Chinchilla regime.
The DeepSeek-style architecture with fine-grained and shared experts improves upon that, landing at 2.7x - 3.1x uplift with $N_{eff}$-based compute-optimal data scaling.

So clearly, from the first to the second era, the MoE benefit as measured by uplift increased and we can attribute it to DeepSeek’s architecture innovations. But beyond that, the existing data unfortunately cannot tell us much more, since there are no controlled studies that ablate every single decision (e.g. granularity without shared expert, Switch but no z-loss). As far as I know, those three are the only studies usable for our purpose here.⁶

Released models

A second data source for uplift is released models with their benchmark numbers. But, we can only use that data to measure uplift if a model family comes with both dense and MoE models, and with enough of them on the dense side to have a scaling law to map the MoE quality against. Such a comparison is somewhat clean, as models of the same family usually share a common backbone architecture and use the same training data, so we can attribute the difference to MoE’s sparsity and routing, but it remains less clean than the previously discussed scaling studies, especially because released models often don’t even disclose all relevant details.

I use the following methodology: Given an MoE and multiple dense models from a model family, fit a function from $log_{10}(N)$ to score across the dense points, invert it at the MoE’s score and use the resulting $N_{eff}$ to compute uplift. I say score, because these models typically don’t report perplexities but accuracies on benchmarks. That’s why I do it per benchmark reported for both dense and MoE and then take the median out of these uplift factors. The MoE needs to score in between the dense points (to not land in the extrapolation region of the dense fit) and I skip benchmarks that are close to saturated (dense fit not improving at least 8 points across 10x parameters).⁷

In fact, although plenty of models were released over the past years, many as open models (see the history section above), not many satisfy the requirement of having both MoEs and comparable dense models in a family. I found these to be usable:

GLaM (12/2021): GLaM was trained by Google in 2021, and since this was the research era, it is strictly speaking not a released model of course, as only the paper was published. It comes with a 4-point dense ladder from 130M to 137B parameters, matching MoEs at 1.7B and 8B active and using top-2 routing over 64 experts every other layer with full-sized experts. A typical Switch-style recipe, though with top-2 routing instead of top-1, similar to the earlier GShard. The uplift factor computed for it is 1.80-1.92, with the caveat that the paper only reports already aggregated benchmark scores rather than individual ones.
Artetxe et al. (12/2021): This one is also a paper of the research era, similarly featuring four dense models with one suitable MoE that we can match against them. Different from the rest, it reports perplexities, so we can handle it akin to the scaling studies. The MoE is similar to GLaM, top-2, every other layer, and particularly sparse with 512 full-sized experts. The resulting uplift factor is 1.76.
DeepSeekMoE (01/2024): This data point is the first DeepSeek-style one, from the very paper that introduced it. Fine-grained experts with granularity 4, 64 routed experts in total, 6 active, plus 2 shared experts, in every layer but the first. The limitation is that the dense side has only two points, 7B and 67B, between which the MoE with 2.8B active lands. That means any uplift we can measure here is at least 2.5 and benchmarks with lower factors cannot be captured. The uplift, under these caveats, lands at 3.86 as the median of 5 benchmarks.
Qwen 3 (04/2025): This is the cleanest data point, the dense ladder has six rungs, the 30B-A3B MoE lands within that range, data is the same, hyperparameters were tuned on either side. The architecture is as already mentioned earlier in the post: 128 experts, top-8, no shared expert, global batch load balancing via auxiliary loss, every layer MoE. The uplift of that recipe ends up being 5.3 as the median of 12 benchmarks. All benchmarks measured on the base models (pretraining only).
Qwen 3.5 (02/2026): Qwen 3.5 is similar to 3, except that I could only find results for the posttrained version of the models. Again, only the smallest MoE can be used because only that falls into the range covered by the dense ladder, it uses 256 experts, now also a shared one, and otherwise the same setup as Qwen 3 (when it comes to MoE). Here the resulting uplift factor is 6.47 across 18 benchmarks.
Gemma 4 (04/2026): Gemma is similar to the Qwen 3.5 case: we also need to rely on posttraining numbers, and in addition, the dense side has only two data points. And worse, one of them even differs more by using a different multi-modal stack. The MoE architecture is similar to the other two, 128 experts, top-8, wide shared expert, unknown balancing. The uplift is 5.63 across 10 benchmarks.

Despite the limited number of suitable models, we have good coverage of the eras: Two examples of Switch-style MoEs from the research era (GLaM and Artetxe), one example of DeepSeek-style models from the ChatGPT/Llama-era (DeepSeek-MoE itself) and three examples from the most recent era (Qwen 3, 3.5 and Gemma). This is how it looks when placing the six uplift factors at their sparsity points in (a slightly simplified version of) the previous plot:

The six released MoEs placed at their activation ratio and measured uplift next to (a subset of) the uplift factors derived from scaling studies. From Switch-style to DeepSeek-style and with increasing granularity, the uplift increases significantly.

What can we see? The two era 1 data points, GLaM and Artetxe, fall into the range that the scaling studies already established for Switch-style MoEs. DeepSeek-MoE from era 2 lands above that, as predicted by the scaling studies for fine-grained models, and shows even higher uplift at G=4 than Tian’s study, which most likely is due to higher overtraining. The most recent era 3 models bring uplift to new heights, with 6.5 for Qwen 3.5, and I attribute that to both overtraining and increased granularity. The MoE architecture and training recipe itself, apart from the granularity, is not different from the original DeepSeek after all. And that’s probably the most interesting finding of this data: While the last two years brought almost no new architecture or training innovations to the MoE setup, uplift still improved, indicating that the labs at least managed to squeeze more out of that established setup.

That being said, these data points are also only somewhat reliable, for one because they rely on a particular selection of benchmarks that differs from model to model (but is consistent within the dense-MoE-family of course), second because we compare pretrained and posttrained models in some cases, and third we compare uplifts computed against perplexities with uplifts against benchmark accuracies. What is not a confound here though is the fact that models got better by other means (e.g. better data, attention, optimization, posttraining) since that tends to benefit both dense models and MoEs equally and therefore largely cancels out in the uplift.

Which changes matter?

While uplift factors are a nice concept and we could collect them from quite a few sources, they still leave many questions unanswered. They simply don’t exist for all small single-factor ablations that we would want to have to judge all the MoE architectures and training recipes proposed over the years. Given those gaps in the data, here’s a more opinionated review, based on the existing data, adoption across the model landscape and personal experience:

What is here to stay: Expert granularity is undeniably the concept with the strongest positive evidence that was also adopted across the board. For shared experts, it’s a bit weaker, and the interaction with the MoE layer placement, as also pointed out in the MAI Thinking report, might have contributed to that lack of clarity. For all layers (but the first) MoE models, shared experts are adopted by the majority.

What disappeared: Using MoEs in every nth layer, instead of all of them, was popular in the research era but has mostly disappeared lately. I’m not sure though if there are strong reasons against it; after all, it makes shared experts unnecessary and it provides more design flexibility by decoupling depth from sparsity. What definitely disappeared, probably due to the effectiveness of fine-grained experts, is increasing top-k independent of that, such as the top-2 with full-sized experts as in GShard, GLaM and ST-MoE. Token dropping is typically avoided as much as possible nowadays. And expert choice, given its shortcomings, never really got adopted in language modeling.

What’s less clear: Token balancing remains crucial, but no single method dominates. While BASE’s linear assignment and Clark’s Sinkhorn mostly disappeared, the auxiliary loss as well as DeepSeek’s expert biases are both actively used. Both, or even all four, can be effective if used at the right token scope, which is the more important factor. I discuss that particular topic in-depth in this post. What’s less clear is if initial dense layers, or fixed routed MoE layers as in the recent DeepSeek, are needed, and if so, how many of them. Models still vary in this regard. And similarly, both softmax and sigmoid, and even other activation functions, are still used for the router.

What’s next?

Given all that, the history of models and methods, the changes in uplift that we could measure, and my verdict on the methods above, one might ask what’s next for MoEs. One possible answer, supported by the most recent developments, would be “nothing at all”. While there is still a plethora of academic papers being published on MoE architecture and training, very little has found adoption in recent LLMs, and as I argued above, most details have not changed since DeepSeek-MoE almost three years ago. I’m very much looking forward to the paper that shows us how to obtain 10x or 20x uplift, but maybe that simply never comes and with what we have right now the potential of MoEs is fully leveraged already. Given the increasing focus on other aspects in recent years, such as efficient long-context attention methods or increased depth-wise information flow (e.g. mHC, attention residuals), this might turn out to be true.

That being said, there are two related threads worth watching since they both bring in forms of sparsity in other parts of the model: First, extra embedding tables. Since an embedding is merely a lookup operation, it is sparse in the sense of every token just engaging with its own parameter row, and in terms of FLOPs even better than MoEs since a lookup requires no FLOPs at all. Gemma 3n, later Gemma 4, and also STEM added a table per layer. DeepSeek’s Engram extended the idea from unigrams to n-grams, which DeepSeek V4.1 adopted. And second, looping. Varying the depth of a network per token is an old idea already explored with RNNs, e.g. by Graves (2016), and later integrated into transformers as Universal Transformers. At the time, this was typically framed as adaptive compute or early exit to save compute on simpler inputs. But, the very same idea can also be used to increase compute dynamically. While MoEs add extra parameters at constant FLOPs, looping adds extra FLOPs at constant parameters. Two recent papers that look at this are Lee et al. (2026) and Wang et al. (2026). Looping becomes particularly powerful when combined with MoEs: While a looped dense transformer can only repeatedly apply a fixed learned function, a looped MoE, since it routes based on input representations, can apply a different computation at each iteration of the loop even though the parameters are reused.

As I argued in a previous post, this is often only fully clear in retrospect. ↩︎
But, crucially, at global batch scope, as I discussed here. ↩︎
The main difference from Switch is using Sinkhorn instead of the balancing auxiliary loss, but as I show in this post, that makes only a minor difference when controlling for scope. Clark et al. also test hash routing and RL-based routing, but those diverge more from typical settings of that era. ↩︎
It used the auxiliary loss for balancing as Switch-Transformers did instead of Clark’s Sinkhorn. But, it also differed by using MoEs in every layer and adding router z-loss. ↩︎
More precisely, each dense and MoE model is trained for the compute-optimal data volume given its active parameters, as defined by the scaling law. That is close to a Chinchilla-style constant tokens-per-parameter ratio against the effective size, but not equivalent. ↩︎
Similar studies were done by Krajewski et al. (2024) and Abnar et al. (2025) but they did not release their raw data as Clark did nor can their fitted scaling functions be inverted to map from dense to MoEs as they are not well-conditioned for this use case, leading to vastly changing uplift factors with even tiny input changes. ↩︎
With that filter, using a linear fit is reasonable, otherwise, a sigmoidal fit is typically used when trying to capture the full score range. ↩︎

Why I don't trust any single paper's claims

Wed, 12 Aug 2026 00:00:00 +0000

This recent post on X reminded me of an article I always wanted to write about how to read and interpret published papers and their claims appropriately:

While the fraud part probably (hopefully) applies to just a small minority of papers, the overclaiming issue is indeed a large problem. The topic often comes up when discussing papers with people newer to the field, in particular interns, so I always wanted to write down my thinking about this issue and how I approach it.

The core problem is that a lot of papers published at ML conferences, at least in my field, express their findings in much more general terms than the actual experimental evidence warrants, making the naive reader, such as the inexperienced intern, who takes those claims literally, overestimate their importance and validity. I had cases where interns literally made statements like “why would anyone not use method X given that paper Y showed it is the best”. In this post, I’ll argue why it is almost never the case that a single paper showing that some X is superior to alternatives does so in a way that this question would be fully settled for the future.

As a disclaimer, my experience is primarily from the LLM training literature. Other fields, even closely related topics within ML/AI, might not have this issue to the same degree, since it depends on the publishing/reviewing customs and the nature of the experiments.

The quantitative gap

Let me start by illustrating the problem quantitatively. Over the last few years, I worked primarily on mixture of experts (MoE) architectures and training recipes. If you query arXiv for papers mentioning “mixture of experts” in the abstract, there are 2959 in the last three years, and 975 that have both “mixture of experts” and “routing” in the abstract (which makes it more likely they discuss new architectures and training recipes, as opposed to being just applications). All papers that propose something new, a slightly different architecture, new routing method, new auxiliary loss, etc., come with experiments that show their proposed method outperforms the established methods. If that evidence didn’t exist, the paper wouldn’t be published at all.

And yet, while those proposed methods are presumably all beneficial, the changes that really get adopted in the field are orders of magnitude fewer. Comparing MoEs from three years ago, e.g. Mixtral, with the most recent ones, such as DeepSeek V4, Kimi K3 or GLM 5.3, the number of MoE-related differences is probably something like 10 or 20. Even if it is a bit more (certainly depends on what counts as a change), and even if the 1000 to 3000 papers above overestimate the number of proposed improvements (a few hundred might be more accurate), there is undeniably a large gap that remains. The vast majority of proposed innovations are simply forgotten.

Why does the gap exist?

Because in practice, a lot of proposed innovations turn out to not actually make a difference, or at least not a positive one. So the interesting question is, why do reality and the paper disagree?

There are several reasons I can think of:

Faked Experiments: Some papers might truly be fabricated, with experimental results being completely fake. I hope that is rare, but it probably does exist.
Buggy Experiments: Some papers show positive results that are caused by issues in the experimental setup, such as bugs in the code, data issues, etc. that benefit the proposed method in a way that it looks more promising than it would in a non-buggy setup. That makes the experimental result meaningless, as the experiment doesn’t measure what it is supposed to measure, and the result is similar to case 1, just not done intentionally and usually unknown to the authors until a future reproduction attempt reveals it. This might sound like a rare case, but everyone who has ever found a bug in their own experimental setup whose fix completely changed the results can tell you that this is quite common even when setting things up very carefully. Variations of this are cases that are not implementation mistakes, but flaws in the experimental design that bias the outcome without the authors noticing.
Undisclosed Limitations: In this case, the benefit of a method is real, but it comes at a cost that is not fully disclosed. Reviewers are good at spotting some of these and typically push back, for instance when a paper shows improved accuracy without accounting for the increased cost, so fair experimental setups are usually the case. Another more subtle version of this though is a careful selection of the experimental setup, in which alternative methods are compared fairly, but it is a setup that hides a problem that the method would have in other settings. In other words, it does not generalize beyond the settings that were tested in the paper, but the claims might be worded so broadly that the reader assumes it does.
Experimental Breadth: This last one is similar to 3, where the issue is also generalization, though not at the conceptual level – where a method cannot be applied in another setting at all – but at the empirical level, meaning one can use it, but it simply doesn’t work as well. I think this case is the most relevant one in practice. The main reason is that training LLMs is a very complex process, models can have different sizes, different architectures, different training hyperparameters, use different training data – the list is very long – and there are lots of interactions between all of them. An experiment that shows method A is better than B or C does so in the context of all these other factors set to some value, aka the experimental setup, and no paper manages to cover a wide variety of such settings since the space is simply too large to explore. That leaves many generalization questions open: You can’t say if A-better-than-B really generalizes to all possible settings or if there are interactions with other factors that change the picture.

Ordered that way, the four categories form a spectrum of how far the claim is from the evidence behind it: The evidence is fake, it is wrong without the authors knowing, it is real but its price is not fully disclosed, or it is real and honestly reported but covers only a slice of the settings one might care about. The further down that list, the less anyone did anything wrong but the more common the case is.

While the reasons above explain the limitations of the experimental evidence a paper has, the true issue is of course something else: How the paper communicates the results. A statement saying “A beats B in the case of model C across sizes D, E and F on data G and …” would be honest but doesn’t read very well. The experiment section of a paper usually gives this honest statement because it specifies the full setting. But in the abstract, the introduction and the conclusions, the claims are often shortened, dropping the important qualifiers that make the non-careful reader expect more than the evidence really backs up. Clearly, these overclaiming habits are not just sloppy writing by the authors, but also incentivized by the existing norms and reviewing practices in the field, which demand bold claims to get papers accepted. But no matter what the exact causes are, what the careful reader needs to pay attention to doesn’t change.

Examples

I want to try illustrating the issue categories with a few examples, but that is trickier than it seems: Most of the gap between the number of published and actually adopted methods is just silently forgotten, no follow-up paper ever reports what the specific issue was. Maybe category 1, maybe 2, maybe 3, or maybe nothing at all and it just got overlooked. So naturally, the specific examples I will go over are in fact bad examples because they are the exception. I hope they are useful nonetheless to clarify my points.

In my first post, I mentioned the scaling study of Clark et al. (2022), which was the first broad study that showed how the benefit of sparse MoE models over their dense counterparts scales with increasing sparsity. While it did show that very well and provided useful data points for everyone working on MoEs, the experimental setup had one subtle flaw that makes it an example of category 2: All models, no matter their size in either active or total parameters, were trained on the exact same token budget, rather than co-scaling the token budget with the model size as later scaling studies did. That introduced an artifact into the data suggesting that the MoE benefit diminishes with scale, which disappears when instead keeping the tokens-per-parameter constant.¹

As an example of category 3, I want to discuss the Autonomy of Experts paper from ICML 2025. It’s a nice idea to get rid of routers in MoEs by instead using the activation norms of experts themselves as the routing signal. Since one cannot compute those activations for all experts – then we would lose the whole sparsity – the authors take one weight of each expert, the gate projection of SwiGLU-style FFNs, replace that part with a low-rank factorization and compute only the down-projection step for all experts. That way, the part evaluated for all experts remains small and the remainder of the expert computation is still done only for the selected experts. The conceptual advantage is that with no router, just experts, there is only one thing to train and no risk of misalignment between router and experts. In the paper’s experiments, the model improves by one point across a variety of benchmarks.

What one might miss at a first read is that the idea does not transfer from the paper’s 8 experts top-2 setting to more contemporary models with larger numbers of fine-grained experts. The factorization idea essentially replaces the $(d,E)$ router with a $(d,E,r)$ “router”, where $r$ is the low rank, and this is a dense part of the model since it is computed for every token. That factor $r$ growth in dense parameters implies that the sparse part of the model – the actual experts – needs to shrink to keep the overall active parameters constant, and that introduces a problem: Either $r$ is low, shifting few parameters from sparse to dense but creating a bottleneck due to the low rank, or it is increased, resolving the bottleneck but reducing the expert capacity. At 8 experts, top-2, 768 hidden dim, 3072 FFN dim (the paper’s setting), this effect is almost invisible. For a DeepSeek V3-style architecture with 256 experts, top-8, 7168 hidden dim, that’s not the case: A low rank dimension of 193, which is already fairly low for a model that size, would shift so many parameters to the dense region that in fact no parameters are left for the sparse experts at all.² Only with ranks lower than 193 can the idea be applied at all, and it introduces bottlenecks that hurt the model quality badly. Because of that, it is a great example of the category of not even conceptually generalizing beyond the experimental setup of the paper.

Category 4, which I claimed is the most frequent one, is about empirical generalization beyond the paper’s setting. As I argued, this happens typically because there are simply too many factors involved to account for all of them in ablations. One example is shared experts: DeepSeek-MoE introduced them in early 2024 and most MoEs adopted them since. The paper even has an ablation, showing that shared experts alone, independent of the also proposed fine-grained experts, improve their models. But one additional factor that is different from earlier work is the layer placement: While earlier work like GShard and Switch Transformer combined MoE layers with interleaved dense layers, DeepSeek-MoE used them in all layers. The usefulness of shared experts strongly depends on that context. This has been largely ignored even by subsequent work that adopted shared experts. I only know of MAI-Thinking-1 explicitly calling this out more than two years later, reporting that they see no value of shared experts in interleaved architectures. Similarly, other factors of the MoE setup can influence experiments, for example, a routing scheme that looks promising under softmax routing activations might not work with sigmoid, or vice versa. And as I showed in the previous post, scope is an important variable in token balancing that can change comparisons (assuming they are even done at the same scope in the first place).

What can readers do?

As a reader, this implies you should be skeptical of the claims a paper makes, especially in the abstract and introduction, and instead look at the exact experiments reported and make your own judgment on what conclusions the results support. That will catch some overclaiming, mainly of categories 2 and 3, but only those that a careful reviewer should already have caught (assuming the paper went through a reviewing process). Other issues, and in particular category 4, how far a proposed method generalizes empirically beyond the necessarily limited experiments of its paper, cannot be solved by careful reading.

I see two main options to address that case:

Wait for adoption: Adoption by the field is the strongest signal that something is really useful across a variety of settings. That is also why I titled this whole post saying that a single paper is not enough. A method that only looks good because the paper overclaims will simply be ignored by subsequent work. A paper that is cited a lot seems to have more value. But more important than citations, which can mean many things, is whether a proposed technique gets adopted in later work. If a new architecture becomes the default choice in future work’s baselines or in released LLMs, then it must truly be good. But obviously, this solution has a price: The evidence is only clear with a long delay and only by the time everyone has already adopted it, so it does not work if you want to be an early adopter.
Try yourself: Test everything that looks promising on your own against your own baselines. That gives you a second set of experiments that complement the ones in the paper, and importantly, rather than answering the broad question of what all the settings are that the findings generalize to, it answers it for that one specific target setting you really care about: your own baseline. This is the relevant question in particular if your goal for following the literature is to train your own production LLMs for which you want to adopt potential improvements. And obviously, what it trades off compared to the first approach is time against costs.

Summary

The gap I started this post with does not mean that most of these papers are wrong. In most cases they are not: They report results that were really obtained in the setup they describe, and it is mostly the abstract and the introduction, not the experiment section, that make those results sound like general findings. That is also why I don’t think one should stop reading papers, as the tweet I started with suggests, but rather stop reading their claims as settled answers, and instead confirm them yourself or wait for the field as a whole to settle it.

Krajewski et al. (2024) point this out. My own plot in the first post also demonstrates that even with Clark et al.’s data itself the diminishing returns disappear when selecting only the subset with constant tokens-per-parameter ratios. ↩︎
A normal MoE activates $N_{act}^{MoE} = d \cdot E + k \cdot 3 \cdot d \cdot d_{ffn}$ parameters per token. AoE changes that to $N_{act}^{AoE} = d \cdot E \cdot r + k \cdot (r \cdot d_{wide} + 2 \cdot d \cdot d_{wide})$, with $d_{wide}$ set to keep the total (but not active) parameters constant. If $d \cdot E \cdot r$ alone already matches $N_{act}^{MoE}$, no parameters are left for the actual experts. For DSV3, that happens at $r = (E + k \cdot 3 \cdot d_{ffn}) / E = 193$. Any feasible $r$ is hence significantly lower than $d_{ffn} = 2048$. For DSV4 Pro, it is 145 vs. 3072. And for the paper’s smaller top-2 model, it is 2305 compared to 3072. ↩︎

A Routing Testbed for MoEs, Part 2: Visualizing Balancing Methods

Mon, 06 Jul 2026 00:00:00 +0000

In Part 1 I argued that two properties of the learned routing decide how much of an MoE’s total capacity is actually used. The load should be balanced across the experts, so that no expert stays undertrained and wastes the capacity it holds, and the experts should specialize, so that the additional parameters learn distinct things instead of duplicating each other. Load balance can be read off the routing statistics of any training run, while specialization cannot, which is what motivated our MoE Routing Testbed: A data mix of Wikipedia in eight languages with distinct scripts provides a strong prior for how tokens should be routed, a reference routing derived from that prior which we verified to be better than what our models learn on their own and the two metrics of expert utilization and routing purity that quantify the two goals separately.

With that setup, the routing behavior of a training run becomes directly visible instead of having to be inferred from a validation loss. This second part of the post uses it on the mechanism that shapes routing most directly, namely the token balancing that essentially every MoE training run applies in one form or another. We trained hundreds of configurations covering four balancing methods and the ways in which they can be applied. Via the expert utilization and routing purity metrics that our testbed can measure, the differences between these balancing approaches can be nicely visualized, as I show in this post.

The token balancing methods

We compared four approaches from the literature that have been used in MoE work over the past years:

The load balancing loss (LBL), introduced with GShard and popularized by the Switch Transformer, is the oldest of them and still the most widely used one. For each MoE layer it computes two quantities per expert: The share $f_i$ of the token-expert assignments in the batch that went to expert $i$, and the average routing probability $P_i$ that the router assigned to expert $i$ over the same tokens. The auxiliary loss added to the language modeling loss is then $\alpha E \sum_{i=1}^{E} f_i P_i$. While $f$ is the distribution we want to be uniform, the surrogate $P$ is used additionally to make the loss differentiable, which is not the case for $f$ due to top-$k$ selection. $\alpha$ is a hyperparameter that weighs this auxiliary loss against the main LM loss.

Balanced assignment (BA), introduced with BASE layers, decouples balancing from the router instead of penalizing it. Given the score $s_{ti}$ that the router assigns to each token $t$ in a batch for each expert $i$, it looks for the binary assignment that maximizes the total score of the selected token-expert pairs, subject to every token receiving $k$ experts and every expert receiving the same number of tokens. This is a linear assignment problem and can be solved with an auction algorithm. BASE layers itself assigns a single expert per token, the $k>1$ case used here is the straightforward generalization of that. Balance is perfect by construction rather than encouraged, and there is no strength left to tune, but as a consequence also no flexibility in that regard.

Sinkhorn routing (SH), as used in the scaling study of Clark et al. (2022), keeps the same assignment view as BA but relaxes it. Rather than solving the assignment exactly, it adds an entropy term to the objective and the solution of that relaxed problem is found by the Sinkhorn algorithm. A handful of iterations are sufficient and they are cheap matrix operations, so the cost stays well below an exact solver, at the price of balance that is only approximate. Like BA, it acts on the routing itself rather than on the loss modifying the router.

The expert bias (EB) method, introduced by Wang et al. (2024) and popularized by DeepSeek V3, often called auxiliary-loss-free balancing, takes a third route and manipulates the selection. Each expert gets a scalar bias $b_i$ that enters the top-$k$ selection only, so the selected experts are those with the highest $s_{ti} + b_i$, while the gating weights that scale their outputs are still computed from the unbiased $s_{ti}$. The bias hence steers which experts are chosen without distorting how their outputs are combined. After every step, the biases are updated by a fixed increment $\gamma$ based on the observed load, as $b_i \leftarrow b_i + \gamma \operatorname{sign}(\bar{c} - c_i)$, with $c_i$ the number of tokens expert $i$ received in that step and $\bar{c}$ the average across experts, so the bias decreases for experts above their share and increases for those below it. Balance is therefore approached over the course of many steps instead of being enforced within a single one, and the appeal of the method is that no additional gradient competes with the LM objective.

Three dimensions, not one

These four methods are usually compared as alternatives, i.e. the question is posed as which balancing method to use. But the method is only one part of the configuration. Applying it also requires deciding how hard the balance is enforced, and, before that, over which set of tokens it is applied. We therefore treat balancing as three separate dimensions: The method (how balance is encouraged), the strength (how hard it is enforced) and the scope (over which tokens it is computed). Only LBL and EB come with a strength dial at all, and for EB it turns out to be a rather poor one.¹ The scope, on the other hand, applies to all four methods equally, and it is the dimension that is almost never stated, since it follows from how the balancing statistics happen to be aggregated in a given implementation.

Separating the three makes the comparison a lot more informative, because a difference attributed to a method may well come from the scope at which it was applied.

The role of scope

The effect of the scope is easiest to see in the extreme case of computing the balance over a single training sequence. Sequences are typically packed from a single data source, so such a sequence can be for instance entirely code, and a balancing mechanism applied at this scope pushes the router to spread code tokens uniformly over all experts. That is the opposite of what specialization requires, namely code tokens concentrating on the experts that handle code. At sequence scope, utilization and specialization are directly in conflict by construction.

The other end of the spectrum, and arguably the ideal case, is a scope large enough to have a representative sample of the full training distribution. If the domains are represented at comparable rates, in the testbed exactly so, with eight domains at 1/8 each, then a router that sends each domain to its own group of experts is perfectly specialized and perfectly balanced at the same time. The conflict is hence not inherent to balancing, it is created by computing the balance over token sets that are too small and too homogeneous to reflect the training distribution and hence the distribution that experts should specialize over.

What makes this practically relevant is where real training frameworks sit on that spectrum. As Qiu et al. (2025) pointed out, most public MoE implementations compute the balancing statistics over the tokens that happen to be present on one device. In large-scale training with gradient accumulation or pipeline parallelism, such a per-device micro-batch often contains just one or two sequences, so that training frequently operates at the bad end of that spectrum without anyone having made that decision.

Experimental setup

The testbed lets us measure all of this cheaply. We swept the three dimensions systematically: Four methods (LBL, EB, SH, BA), five scopes (balancing over 1, 2, 8, 16 or all 64 sequences of the global batch, controlled via micro-batch size and cross-device aggregation) and for LBL and EB a range of strengths, resulting in 98 configurations in total. The model is a small single-layer MoE with 50M active parameters (270M total), 32 fine-grained experts under top-4 routing, trained for 20k steps (about 10.5B tokens) at a global batch size of 64 x 8192 tokens. No token dropping is applied in the default setup.

Both our metrics, expert utilization and routing purity, are computed over the full global batch – independent of the scope at which the balancing acts – which is the detail I flagged in Part 1 and which matters here: The token load can be balanced over experts globally without being balanced locally, but not the other way round.

Result 1: Scope decides the trade-off

The following plot shows what I’ve been referring to as the visualization power of the routing testbed. When plotting the expert utilization vs. routing purity trade-offs for different configurations, we can clearly see the effect that scope has:

Expert utilization vs. routing purity for the four LBL strength sweeps and the single-point results of BA, SH and EB. Color encodes the balancing scope, shape the method and the balancing strength is swept along each LBL line (strong balancing bottom right, weak top left). At a scope of one sequence, balancing trades utilization one for one against purity, while at larger scopes both become achievable at once. Without any balancing (gray diamond), specialization emerges on its own, at the cost of significant utilization.

At a scope of just one sequence, the strength sweep is an almost straight line, from high utilization at low purity for the strongest weight down to low utilization at high purity for the weakest one. Every bit of balance is paid for with purity here, which is exactly the conceptual conflict described above and hence the intuitive result to expect. The run without any balancing, included as the gray diamond, confirms that specialization is what the router does on its own: It lands among the highest purity values of the whole sweep without any help, but its utilization is low. With growing scope, the lines bend towards the top-right corner. At a scope of 16 sequences, the strongest balancing already starts at higher purity and the best configurations reach around 0.9 utilization and 0.6 purity at the same time, resolving the conflict.

The validation loss on the testbed mix follows that trade-off closely, with the lowest loss of every run in the plot coming from the large-scope LBL configurations, while the strongly balanced sequence-scope configurations produce the highest. Coloring the same landscape by that loss makes the correspondence easier to see:

The same runs and the same axes as above, colored by the validation loss on the testbed mix instead of by scope. The lowest losses sit in the upper right, where utilization and purity are high at the same time.

Result 2: Methods are equally good

In the two plots above, the alternative balancing methods were drawn as squares (BA), triangles (SH) and crosses (EB), which unlike LBL have no lines, because there is no strength to vary. BA, true to its auction formulation, pins utilization at exactly 1.0 in every configuration, so its four points sit on a vertical line at the right edge, ordered by scope. Sinkhorn behaves similarly at slightly lower utilization, and EB, applied at global scope as DeepSeek proposed it, reaches 0.90 utilization at 0.56 purity, the same neighborhood as LBL at its largest scopes. The differences between the methods are small compared to the effect of the scope and at a sufficiently large scope all four of them work well.

That is worth emphasizing, given how much effort the field spent on the method question. EB was introduced with the claim of being superior to the auxiliary loss, and Qiu et al. (2025) later showed that the two are on par once compared at the same scope, so the original comparison had confounded method with scope. Our experiments confirm this finding, but due to our testbed, we can do so by comprehensively searching the three-dimensional design space and accurately measuring and visualizing the effects, rather than just comparing the loss of two larger models. The same conclusion was recently also reported in Microsoft’s MAI-Thinking-1, which uses a global-batch balancing loss aggregated across data-parallel workers and micro-batches, and the report states that the aggregation strategy mattered much more than the type of loss according to their ablations, matching our observations here.

Result 3: The role of token dropping

Many training setups cap the number of tokens an expert can process, i.e. they use a capacity factor and drop the tokens beyond it, trading model quality for predictable compute and communication. We repeated the grid search experiments under such caps, which inverts the picture:

The landscape without dropping (left) and with a capacity factor 1.0 (right), for LBL at scopes of 1, 8 and 64 sequences plus EB, on a shared loss scale. Under the cap, the reachable purity falls from 0.71 to 0.47 and the best run becomes a strongly balanced sequence-scope configuration.

Two things change: The purity ceiling reduces substantially, so the specialized part of the landscape is no longer reachable at all. And the best run, ringed in both panels, moves from EB at global scope to LBL at a scope of one sequence, while the overall loss level rises. The reason is that the capacity constraint is enforced per device, so only strong local balancing avoids heavy dropping under it. Since the damage from the dropped tokens seems to outweigh what the specialization gains, the best strategy becomes to avoid dropping by strong local balancing. This is also in line with the experience at MAI, who observed that conclusions about balancing methodology can flip depending on the expert capacity and converged on a fully dropless implementation as in our baseline setting.

Result 4: Validation loss hides the effects

One could ask whether the testbed is needed for any of this, or whether a plain validation loss would have shown the same. We checked that by training the same configurations on Dolma3, the open training mix of the OLMo 3 models, and the following plot colors the trade-off landscape by the loss of both side by side:

The same runs and the same axes as above, colored by validation loss. On the testbed mix (left) the loss lines up with the position in the landscape, with the well-utilized and well-specialized runs in the upper right achieving the lowest loss. On Dolma3 (right) the loss varies over a comparable range, but with only a faint correspondence to the position left. Each panel has its own loss scale.

The two panels hold the same runs at the same coordinates and differ only in which loss the color encodes. On the testbed mix, the color follows the position: The upper right of the landscape is where the low losses sit. On Dolma3, the same runs scatter their colors across the landscape with no usable ordering left, even though the spread of the losses is nearly the same. Measured across all 98 configurations as a rank correlation between the loss and how far up and to the right a run sits, the testbed reaches -0.80 and Dolma3 only -0.28. At 50M active parameters, a loss on a normal pretraining mix simply cannot distinguish routing configurations, and metrics that look at the routing itself are needed instead.

What about expert choice?

Expert choice (EC) routing (Zhou et al., 2022) often comes up in discussions about token balancing and I want to say a few words on why we did not include it in the paper. EC inverts the assignment problem – experts pick their top-$k$ tokens instead of tokens picking experts – and thereby it achieves perfect balance by construction. The catch however is causality: Whether an expert picks a token depends on the other tokens in the batch, including future positions of the same sequence. For an autoregressive decoder this leaks future information during training, and when generating token by token, the assignment cannot be reproduced the way it was trained. Proposed fixes bolt on extra routers and training phases but have rarely been used by production MoE models.

Beyond the causality issue, EC also has another flaw: The balance is not just guaranteed by construction, it is even impossible to remove, which can be problematic at inference time where the token distribution cannot be controlled. If the model has to process a batch of tokens coming purely from one domain, the EC assignment will force them over all experts nonetheless, no matter if they are specialized in that domain or not. LBL, BA, SH and EB do not face this problem because for each of them the balancing can simply be turned off at inference time.

Expert threshold (ET) routing (Sun et al., 2026) is a recent attempt to repair the causality flaw of EC. Each expert maintains a threshold, tracked as an exponential moving average calibrated on the global token distribution, and a token is routed to an expert whenever its score clears that threshold, a decision that depends only on the token itself and hence restores causality. That does indeed solve the problem, but introduces new ones instead: Calibrating the threshold successfully can be challenging. The number of experts per token is no longer fixed and the dynamic allocation might be counterintuitive, as it is driven by the token’s routing score distribution over experts against per-expert (but cross token) thresholds. And the key quality of EC, balance by construction, is given up in this case.

Summary

This concludes my walk through the MoE Routing Testbed paper. Its special data mix with clearly distinguishable domains enables directly measuring expert specialization. Using it, we can clearly measure and visualize how token balancing choices affect the trade-offs between expert specialization and expert utilization. Scope, as noticed by previous work, is the most important factor. Only if it is large enough can the two goals be achieved at the same time instead of conflicting. The balancing method itself, i.e. using a load balancing auxiliary loss or expert biases, plays only a minor role.

For EB, the bias update rate is not really a knob for balancing strength but for how fast the biases adapt and hence reach an equilibrium at all. We observe that empirically: within a certain range balancing works but the degree of balance stays constant, and outside that range EB stops balancing altogether. ↩︎

A Routing Testbed for MoEs, Part 1: Measuring Routing Effectiveness

Sun, 05 Jul 2026 00:00:00 +0000

The first post of this series revisited the central promise of sparse MoE models: sparsity decouples model quality from cost and allows training high quality models for less compute. But one aspect that framing quietly assumes is that the total parameters are actually put to good use. In a recent paper that I co-authored with colleagues at Amazon, MoE Routing Testbed, we argue that whether an MoE can fully realize its potential depends heavily on two properties of the routing, load balance and expert specialization, and we propose a testbed to measure that. In this (two-part) post, I will walk you through that paper.

Balance and specialization determine the sparsity benefit

A sparse MoE with $A$ active parameters and $N$ total parameters comes with the promise of enabling better model quality than a dense model with $A$ parameters. In the worst case, which is the router collapsing to route every token to just a single expert (or fixed set of $k$ experts) each time, the quality in fact collapses to just that dense counterpart (or worse), making it the lower bound for the MoE quality, or at least a baseline it should certainly beat. A dense model with $N$ parameters is an upper reference, and presumably an ambitious one that the MoE will never reach given that it limits FLOPs per token to $A$. The interesting question of practical MoE training is where exactly between the dense quality of $A$ and $N$ the MoE will land.

Our paper builds on the idea that there are two key properties of the learned routing that determine where on that model quality spectrum the MoE will land:

Load Balance: The router should distribute tokens equally across the experts. This is the well-known thing about MoE training, usually discussed in the context of training stability (a collapsed router can throw off the whole training) or efficiency (the more balanced the load, the more efficient the EP setup). But beyond these, balance is also important for model quality more generally. An imbalanced load that over-utilizes some experts implies that other under-utilized experts receive fewer tokens, and hence less training data, leaving their parameters under-trained compared to the other experts. As a result, the full model capacity $N$ is not leveraged as much as in a perfectly balanced case.
Expert Specialization: The router should route tokens in a way that allows experts to specialize. This is a somewhat overloaded term in the MoE literature, referring to different concepts and even more proposed metrics to capture them. For our purposes here, we define good specialization as avoiding duplication: The additional parameters a sparse model has are maximally useful if they are used to learn distinct things. That happens mostly automatically in a dense model, but in a sparse MoE it depends also on the router. Only if it learns to separate the input space in a meaningful way, each expert can learn a distinct slice of the overall problem, while a less optimal routing can lead to duplicated learning across the experts. Such duplication wastes model capacity and lowers the effective capacity $N$ to some $N’<N$, reducing as a result the quality benefit over $A$ that the sparsity brings.

The first goal is what the well-known balancing techniques for MoE training address. The second is just as important, but much harder to even see clearly. And that difference in visibility is what motivated us to set up a special training setup for MoE models in which expert specialization can be directly quantified – the MoE routing testbed.

Specialization needs a reference

Specialization is a claim about what an expert has learned, but what we can observe easily is where the router sends tokens. The two are connected: if an expert receives tokens from all over the data distribution, it cannot specialize on any coherent subset; if it receives a coherent subset, it at least has the chance to. So routing behavior is the natural place to look for specialization and that’s also how most previous work approached it. The problem is that for real training data, we don’t know what coherent subsets exist at the token level, and therefore we cannot say whether any given routing pattern is good, bad or just different.

The solution is to control the data. If the training mix is composed of clearly separated domains, we have a strong prior for what an ideal router should do: send tokens from the same domain to the same group of experts, so that each group can specialize on its domain. That immediately allows us to measure specialization in a very intuitive way: Given the tokens that are being routed to an expert, we can simply calculate to what extent they come from a single domain – the higher that fraction, the better specialized the expert is.

The domains even provide a reference routing, which is a routing that would route all tokens from each domain cleanly to a dedicated (set of) experts. For that reference to be meaningful, it should in fact be a good routing in the sense of leading to a very well-trained MoE on that training data: ideally known to be optimal, and failing that, at least demonstrably better than what routers learn on their own. How much of that certainty we can get depends entirely on the data, which made the data mix the central design decision of the testbed.

Designing the data mix

We considered various options to build data mixes for this testbed that represent different trade-offs between two core goals: knowing what optimal routing looks like vs. being close to realistic LLM pretraining data. The following figure illustrates that:

The design space for the testbed data: the closer a data source is to the real LLM training data, the less certain we can be about what optimal routing looks like. Our testbed relies on Wikipedia in different languages, which is a balanced trade-off between the extremes of the spectrum.

Previous work such as Dikkala et al. (2023) sits at one end of the spectrum: With fully synthetic data sampled from mixtures of Gaussians, the optimal routing is known by construction and can be used to assess any learned routing. But this data is far from what MoEs work with in the real LLM pretraining case, namely natural language, and raises questions about whether findings from this setup transfer. Synthetic data such as sentences sampled from context-free grammars as used in the Physics of Language Models line of work get closer to real language, but also lose the property of having a known optimal routing. Text classification datasets bring natural text whose labels give a routing prior, but they are orders of magnitude too small to train language models on.

The other end of the spectrum is the actual target use case, a large and diverse pretraining mix as used to train general-purpose LLMs. That is the real data we care about, but we have no idea about the size, shape and number of slices of that data that different experts should specialize in to obtain the optimal language modeling performance. Taking distinct domains from such a corpus and mixing them in equal proportions introduces a stronger prior for how experts should specialize, but domains are a weak signal for the individual token: many tokens of a news article could equally appear in a novel.

We landed on a data mix that we think offers the most interesting trade-off, given that you can’t have fully realistic data with a known optimal routing anyway: Wikipedia in eight languages with different scripts: German, Korean, Japanese, Chinese, Hebrew, Thai, Hindi and Arabic. Language is a domain boundary that survives down to the token level, and using distinct scripts sharpens it further, since tokens from different scripts mostly don’t even share surface forms. At the same time, every sequence is real human text. This provides strong token-level priors for how tokens should be routed, as I show with experiments later in this post. We believe this data option is also the most interesting addition to the existing literature: While Dikkala et al. occupy the left end of the spectrum, and various other papers sit on the right end by trying to measure specialization in the real setup (see 2.3 in the paper), we are, to my knowledge, the first to build a testbed from the middle of that spectrum.

Two structural decisions complete the mix: Each domain gets exactly 1/8 of the mix, so that all domains are equally important and – crucially – a perfectly balanced expert load and a perfectly specialized routing are compatible with each other by construction. And the model is matched to the mix: we pair the 8-domain data with 32 experts under top-4 routing, so that the experts divide evenly into eight groups of four, one group per domain.

Not every token has a domain

The clean picture above has one flaw: not all tokens are domain-specific, even in this mix. Digits, punctuation, whitespace and many named entities appear across all eight languages. A reference that forces every token into its sequence’s domain group has to make arbitrary calls for these generic tokens – the digit “7” in a Thai article is not meaningfully Thai. Naively enforcing domain routing for all tokens builds that arbitrariness into the reference, and as we’ll see later, that measurably hurts.

We therefore split tokens into a domain-specific set and a generic set. A small feed-forward classifier is trained to predict the domain from the token embedding, reaching 79% test accuracy¹ on that dataset, and its confidence serves as a proxy for domain-specificity: tokens it classifies correctly with high confidence are treated as domain-specific, the rest as generic. The reference routing then prescribes domain-aligned routing only for the domain-specific tokens and leaves the generic ones to be routed freely.

Measuring against the reference

With domains attached to tokens, two metrics fall out naturally that capture the two goals I introduced initially:

Routing purity captures expert specialization: for each expert, take the fraction of its received tokens that come from its most common domain, and average across experts: $\frac{1}{E} \sum_{i=1}^{E} \max_{d} \frac{f_{id}}{f_i}$ where $E$ is the number of experts, $f_i$ the share of all token-expert assignments that go to expert $i$ and $f_{id}$ the share that goes to expert $i$ and comes from domain $d$. A purity of $1$ means every expert serves exactly one domain; with $D$ domains the lower end is $1/D$, which is where a routing that ignores domains entirely ends up. Note that purity does not check which expert serves a domain – tokens from the same domain should travel together, but which expert hosts them is irrelevant.
Expert utilization captures the load balancing goal: it is maximal when all experts receive equal token counts and decreases as load concentrates: $\sum_{i=1}^{E} \min\left(f_i, \frac{1}{E}\right)$ reflecting that an over-used expert implies under-utilized ones elsewhere and hence reduces the effective total capacity of the model due to under-training some experts. Since top-$k$ routing sends every token to $k$ distinct experts, the worst case is a collapse onto a fixed set of $k$ experts, which puts the lower end at $k/E$.

Expert utilization can be computed simply from the routing statistics of how many tokens go to each expert. It is also independent of our testbed. Purity, on the other hand, looks at how the tokens are routed, i.e. which specific tokens go where, not just pure statistics, and for that it relies on our testbed setup and the domain-induced reference routing. We compute it only over the domain-specific tokens, ignoring the generic ones. Note that matching the model to the mix, i.e. $D = E/k$, givens both metrics the same range as their lower ends coincide at $1/D = k/E$ so the common range is $[1/D, 1]$. Both metrics are computed over the full global batch, which is an important detail I’ll discuss in Part 2 of the post.

Verifying the reference routing

Having given up the certainty of the synthetic Gaussian world, we need to make the empirical case instead: a reference routing is only a meaningful comparison target if following it is actually better than anything the model learns on its own – otherwise “closeness to the reference” measures closeness to a mediocre policy.

We verify this empirically by training MoEs on the data mix while constraining the routing to follow the domain structure. Router logits are masked so that domain-specific tokens can only select experts from their domain’s assigned group, while generic tokens route freely. Sweeping the classifier-confidence threshold that splits the tokens into domain and generic moves the setup continuously from fully learned routing (no tokens constrained) to naive domain routing (all tokens constrained), changing the achieved validation loss:

Validation loss as a function of the fraction of tokens constrained to domain-based routing. Forcing all tokens into domain routing (ratio 1.0) is worse than learned routing on average; constraining roughly the most domain-specific half of tokens gives the lowest loss of all configurations, below every fully learned run (gray points).

Naive domain routing – forcing every token into its sequence’s domain group – lands at the highest validation loss of the sweep, and higher than many fully learned routings, confirming that arbitrary domain assignments for generic tokens damage the model. Learned routing itself spans a wide range depending on the balancing configuration that Part 2 will discuss. The reference with a roughly 50:50 split, constraining the half of tokens the classifier is most confident about, reaches the lowest loss, below every fully learned configuration we trained. That makes it a useful reference and we can meaningfully measure routing purity against that target.

What’s next

We now have a routing testbed: a data mix with clearly defined domains, an empirically validated reference routing and two metrics – routing purity and expert utilization – that make the router’s two competing goals separately visible instead of collapsed into a single loss number. In Part 2, I show how that can clearly visualize the effects of various popular token balancing methods on learned routing and that scope is what matters most, much more than the actual balancing approach.

Note that the fact that this classifier is not perfect is actually desirable: A classifier achieving 100% accuracy of mapping tokens to domains would not help us distinguishing domain-specific tokens from generic ones. ↩︎

Mixture of Expert Models: Why FLOPs Don't Tell the Whole Story

Wed, 24 Jun 2026 00:00:00 +0000

This post is the first in a (hopefully) longer series on LLM pretraining. I have spent the last three years on pretraining LLMs, and in particular, mixture of expert (MoE) models, closely watching the development of the field while experimenting with and training said models myself. While I can’t and won’t talk about the exact details of what I worked on at my employer (except for a few things that have been published), I will share some high-level observations, thoughts about the technology and learnings from the work, which still gives plenty of interesting topics to write about.

Since my focus has been on MoEs for the majority of the time, I want to start the series revisiting the central promise of sparse models such as the commonly used Transformer-based MoEs: By activating just a subset of a model’s parameters for each token, model quality and compute – the two key units whose relationship scaling laws describe – become decoupled. The parameters of a model break down into total parameters and active parameters, the former reflecting the model’s total capacity available for learning and hence the quality it can achieve, while the latter indicates the compute it requires to train and inference the model, typically expressed in FLOPs. Or put simpler: You can get better models without paying for it. At least not the price known from dense models.

That central promise is valid, but it comes with a few caveats that are discussed a lot less than the headline itself, namely memory and communication. The remainder of this post will focus on these two to paint the more honest picture of the MoE promise. As we’ll see, it requires a bit more effort to fully reap the benefits than just changing the architecture.

Quality

When I started working on LLM pretraining about three years ago, MoE architectures existed primarily in research papers, while the dominating open models – Llama 1 and 2 were released at that time – were exclusively dense. Only by the end of 2023, the first MoE models were publicly released, most notably Mixtral, and soon after, DeepSeek-MoE. Today, sparse MoE architectures are the standard, used by open frontier models (e.g. DeepSeek, Qwen, GLM) as well as closed ones¹.

Qwen3.5 is a more recent example that nicely illustrates the impact sparsity has on model quality because it comes in both dense and sparse versions that were trained under comparable settings. As shown in the plot, there is a clear gap between dense and sparse MoE models at matched active parameters, which is the MoE promise I keep referring to:

Model quality of dense vs. sparse Qwen3.5 models at matched active parameters. Quality is measured here as the average across the 24 benchmarks reported for all 6 models. Scores taken from the respective model cards.

A more controlled version of this comparison was first done by the scaling study of Clark et al. (2022), which trained dense models as well as sparse counterparts of increasing sparsity. As you can see in the following plot of their data, the MoE promise is not just a constant benefit, but of course increases with the degree of sparsity. The more experts they add to a dense model, the more quality improves (or here, validation loss decreases).

Model quality measured as validation loss at different levels of sparsity (top-1 out of N experts) at different active parameters. Based on data from Clark et al. (2022) (S-BASE routing, top-1, every 2nd layer).

Similar findings have been confirmed by later scaling studies, such as Krajewski et al. (2024) and Abnar et al. (2025), and the quality effect is uncontested.² But, at what cost does it come? Looking at just FLOPs, it seems to be free, since FLOPs are a function of only the active parameters and thus would predict the same cost for dense and sparse models at matched active parameters. And that’s exactly where the story isn’t fully complete.

Memory

The first issue with sparse MoEs is that the additional experts, even if inactive and therefore not contributing to compute, still need to be present in memory. The selection of experts, and their separation into active and inactive, is done per token by the learned router, and that selection can of course differ across the tokens in a batch. So while per token the compute is limited to the number of selected experts, across a larger pool of tokens all experts get eventually used. Compute is a function of active parameters, but the memory for storing the weights is still a function of the total model size.

There is an elegant solution to the memory problem and it has been around almost as long as MoEs themselves: Expert Parallelism (EP).

Expert parallelism (right) brings the memory footprint from the naive solution (middle) back to dense parity (left) but at the cost of introducing two new all-to-all collectives (during forward as well as backward passes).

As depicted in the figure above, the idea of EP is best explained by comparing to a dense model and the naive MoE setup that incurs the high memory cost we just discussed. While the dense setup, when combined with data parallelism, holds exactly the same copy of the model on each device, the same approach applied to MoEs results in the FFN memory footprint growing E-fold with E experts. In addition, each expert is heavily underutilized since just a subset of the local batch will be routed to it.

EP, as shown in the right-most column, fixes the memory issue by distributing the experts across devices, as the name already suggests. In the extreme case of having an expert parallel degree as high as the number of experts, the memory footprint reduces to the dense case and we get the sparsity benefit for free. Well, still not completely yet, because we have solved the memory issue, but there is now communication needed to send the tokens to the device where their assigned expert is located, and similarly send the expert outputs back to the original device, via all-to-all collectives.

Before we turn to communication, I want to point out how elegantly EP interacts with DP: Under a perfectly balanced load, i.e. when each expert is assigned the same number of tokens in a batch, the all-to-all communication becomes a token exchange, with each device sending as many tokens out as it receives and keeping the same-sized local batch before and after. This is only possible because EP does not act in a new orthogonal parallelization dimension but operates inside the existing data parallelism, which provides the tokens that can be exchanged with each other. And as a result, it also solves the underutilization problem: Each expert can now process a full-sized batch similar to the FFN in the dense model. This also demonstrates why token balancing is such a crucial topic in MoE training, which we’ll discuss in a dedicated upcoming post.

Communication

EP solves memory but adds communication, which creates a new problem: It now takes longer to run a pass through the model. And compared to other types of parallelism, like the gradient synchronization necessary in data parallel training, this communication even lies on the critical path of every forward and backward pass, meaning it can’t be as easily overlapped with computation. Nonetheless, overlapping compute and communication is the solution, it just takes more effort to achieve that in MoEs.

DeepSeek’s DualPipe or Megatron’s interleaved 1F1B with A2A overlap are pipelining approaches that achieve significant overlap, but doing so requires very careful scheduling of micro batches across pipeline stages, decoupling forward and backward computations (and even finer-grained subsets of the backward) or overlapping subsets of experts. Even with that, what can be hidden depends on the time communication and computation portions of the model take, which depend on the exact architecture (defining the matmul sizes as well as the communication payloads) and the hardware (determining the compute and networking capabilities). Beyond the plain MoE setup, specific architectural variations like using a shared expert, or more drastically, a fully parallel transformer architecture (as in PaLM), can provide additional opportunity for overlap, but changes like the parallel layers come with consequences also for model quality. On the architecture side, fine-grained experts are another important factor, since besides their positive impact on model quality, they also remove the need for tensor parallelism for experts, which would complicate successfully overlapping the all-to-all even more. But more importantly, even for the conceptually best overlapping strategy, whether it delivers in practice further depends on very careful implementation: carefully managing CUDA streams, allocating sufficient SMs to compute and communication kernels, avoiding stragglers, etc. And finally, the hardware. While it determines the timing of communication and compute blocks and hence the potential for overlapping, it shapes the model in even more ways. Most importantly for MoEs, communication bandwidth changes dramatically when going from intra-node communication (via NVLink) to inter-node (via InfiniBand), on an H100 the bandwidth difference is about 18x. Overlapping the all-to-all with compute is a completely different challenge if that communication is across nodes or stays within one. And at what number of devices that boundary sits depends again on the hardware being used (e.g. a GB200 NVL72 rack has 72 GPUs in NVLink instead of just 8).

DeepSeek V3 was an excellent example to demonstrate what it takes to handle all these challenges: careful pipelining with DualPipe, custom communication kernels with DeepEP, limited inter-node communication via node-limited routing, fine-grained experts to avoid tensor parallelism, to name just a few ingredients. Compared to using EP to reduce the memory bottleneck, hiding the communication requires a lot more effort and tricks, especially at the implementation level, and it relies on a careful co-design of model architecture, implementation and hardware. Only with that, the matched compute compared to a dense model also translates to matched latency and training cost.

That being said, fully hiding all communication and reaching cost parity with dense models is not strictly necessary to make MoEs attractive. As discussed before, the appeal of sparse models is that they bring a sizable improvement in model quality, and depending on the size of that, even with some amount of communication overhead, and hence higher training cost than the dense counterpart, the sparse model can still offer a better cost/quality trade-off.

Inference

So far, the discussion assumed training when talking about memory, communication, latency and cost. But the same challenges also exist at inference time, and in fact, they become even more challenging. I don’t want to go deep into inference here, but I’d like to highlight two important differences that inference adds.

First, batching. We saw above that EP interacts nicely with DP and thereby hides the memory cost associated with the additional total parameters a sparse model brings. But that assumes large batches of tokens are processed. During inference, they don’t come naturally, but need to be constructed on the fly by combining independent incoming requests. This is typically already done in modern inference solutions to amortize the cost of reading model weights from HBM during decoding, but MoEs add another reason why inference should be run with batches as large as possible: distributing experts over devices fits them in the available memory, and distributing tokens across the same dimension keeps all those devices busy.

Second, the data distribution is no longer under control. As I briefly hinted at above, an important factor for the efficiency of EP is that the token load across the experts is even. During training, this is achieved by actively encouraging the router to assign tokens in a balanced fashion. But crucially, that balance is defined with respect to the training distribution, a diverse mix of data across all domains a model should learn. At inference time, we don’t control the data distribution, it is simply the sum of the incoming requests. If they happen to concentrate on a certain domain at some point in time, the expert usage of the resulting batch is necessarily skewed, even if the model was nicely balanced during training. The resulting all-to-all communication and expert matmuls mirror that imbalance and stragglers increase latency. For that reason, various MoE-specific inference solutions have been proposed that depart from the regular expert placement used in training and instead dynamically deploy experts redundantly or not at all based on demand.

Summary

The adoption of sparse MoE architectures has been one of the key ingredients for scaling LLMs over the past years to the capabilities they have today. Better quality at same FLOPs was the, somewhat simplified, promise they brought to the table. As we saw in this post, the reality is a little more complex, memory and communication need to be handled, and implementation is where that promise becomes reality or not. A good illustration of that is how much space the DeepSeek V3 tech report spends describing model architecture vs. system implementation. Despite those hurdles, sparsity has now fully conquered the landscape of LLMs and dense models have become rare, especially at the frontier.

Officially only Gemini was confirmed to be an MoE, GPT-4 was rumored to be one, Claude remains unknown, but in 2026 the probability is very high that all closed frontier models are sparse. ↩︎
One claim of Clark et al. (2022), namely that the sparsity benefit diminishes with scale, has been refuted by later work and found to be an artifact of their study using a fixed amount of tokens to train all models. My plot above accounts for that and uses their intermediate data points instead of final ones to keep a constant token-per-parameter ratio. Only then, the curves remain parallel. ↩︎