MoEs Over Time: What Changed And How Much Does it Matter?

After having spent the majority of my work from 2023 to 2026 on pretraining MoE models, I’ve always wanted to look back and write down how the field evolved over that period and which changes ended up actually making a difference.¹ That’s what this post is about. Today’s LLMs, especially the largest frontier models, are almost exclusively sparse MoE models. And while the idea of MoEs has been around for quite some time, it took a while for the field to arrive at this state. I will revisit the seminal papers and model releases that created that shift from dense to sparse.

But more importantly, I will also try to quantify the benefit that MoEs brought. The first post of this series discussed the central promise of sparse MoE models, getting the quality of a larger model for the training and inference cost of a smaller model (and the fine print that comes with that promise). I will refer to that here as the uplift: Given an MoE with $N_{a}$ active and $N_{t}$ total parameters, the model will show quality comparable to a dense model sized somewhere between those two; we’ll refer to that size as the effective size $N_{eff}$. The uplift factor is then defined as $$uplift = \frac{ N_{eff} }{ N_{a} }$$ expressing how much better the MoE is than a dense model with matched active parameters. To illustrate that metric: An uplift factor of 2.5 at sparsity 4% means an MoE with 10B active parameters (and 250B total) would be as good as a dense model with 25B parameters.

That uplift of course depends on the sparsity of the MoE. The sparser it is, the further the quality can increase beyond the active parameters, at the price of having to handle the total parameters. In this post, I will denote sparsity as activation ratio $N_{a} / N_{t}$, so lower means sparser. If we keep the sparsity fixed, uplift becomes a function of how well an MoE is trained to make the most out of the total parameters, which comes down to the MoE architecture and the training recipe. And how these developed over time is what this post is about.

A brief history of MoEs

The research era (2017-2022)

Sparse MoEs as we know them today were first introduced by Shazeer et al. (2017), who tested the idea in LSTM-based models on language modeling and machine translation tasks, slightly predating the transformer architecture (June 2017) that would later dominate language modeling. GShard (Lepikhin et al., 2020) adopted the idea into transformers and tested top-2 routing at various degrees of sparsity. Switch-Transformers (Fedus et al., 2021), which became one of the most influential MoE papers of this era, simplified the setup to top-1 routing, which was previously believed to be unstable. GLaM (Du et al., 2021) later scaled up the GShard top-2 setting and ST-MoE (Zoph et al., 2022) discussed training stability challenges and proposed using z-loss for routers as one mitigation. Notably, all of that early fundamental work came out of Google. Concurrently, Meta (then Facebook) also contributed to the topic, introducing the linear assignment approach to token balancing in BASE layers (Lewis et al., 2021), hash routing (Roller et al., 2021) and thorough comparisons to dense models (Artetxe et al., Dec 2021). Expert choice (Zhou et al., 2022) challenged the dominant top-k routing paradigm, but it never caught on due to its difficulties with auto-regressive decoding. Clark et al. (2022), which I tend to cite heavily across my blog, combined much of that early work in a comprehensive scaling study of MoEs and dense models that precisely quantified the effect of sparsity. Also in this post, I will rely on it for measuring uplift.

The work in these years has introduced, beyond the general concept of how to integrate sparse experts and routing into transformers, many of the core ideas and design decisions that MoEs have relied on since then: routing paradigms (token choice with top-k, expert choice, hash routing), expert parallelism, capacity factors and token dropping, MoE layer placement (e.g. every nth layer), token balancing approaches (auxiliary load balancing loss, BASE-style linear assignment, Sinkhorn approximation) and stability measures such as router z-loss. Unlike today, all of this was done purely as published research. The rise of LLMs and the regular release of such models had yet to start. Which is why I call it the research era.

The ChatGPT and Llama era (2023-2024)

In November 2022, ChatGPT was released, and language models transitioned from being a research problem to powering real-world products. Various companies started training their own LLMs and releasing them as either closed models behind APIs or as public checkpoints. And while that development somewhat reduced the willingness of companies to publish research as openly as they did before, the open-weights part of the LLM space filled that gap by providing (more or less) detailed tech reports. Meta’s Llama was the first such example. After the first open models were exclusively dense, Mixtral (Jiang et al., 2024) in late 2023 was the first open MoE of this era, using top-2 routing over 8 experts with a Llama-inspired backbone. Right after, DeepSeek published DeepSeek-MoE (Dai et al., 2024), which turned out to be the most influential MoE paper of this era. It introduced the key new ideas of this time, namely expert granularity and shared experts. Expert granularity, also known as fine-grained experts, replaces experts sized as FFNs in dense models, which previous work relied on, with experts smaller by a factor $G$, but at the same time increases the total number of experts as well as the number of active ones by the same factor $G$. That keeps both total and active parameters unchanged, but gives the router a much larger space to choose from, e.g. 4 out of 32 instead of 1 out of 8. Shared experts, on the other hand, are FFNs that are always selected in addition to the routed experts, hence a dense path in parallel to the MoE.

Gemini 1.5 (Gemini Team, 2024) was the first closed model publicly confirmed to be an MoE (while GPT-4 was always just rumoured to be so). Notable open MoEs in 2024, among many others, included Grok-1 (xAI, 2024), still following Mixtral’s architecture closely, DBRX (Databricks, 2024), which was slightly more granular (2x), Snowflake Arctic (Snowflake AI Research, 2024), featuring a parallel attention architecture variant, and OLMoE (Muennighoff et al., 2024), which adopted DeepSeek’s ideas (but decided to not use shared experts) and was accompanied by a particularly detailed tech report. Another trend at the time, to produce MoEs quickly given their rising popularity, was to upcycle them from existing dense models, such as Qwen’s first MoE (Qwen Team, 2024). While that is obviously cheaper, it also leaves a significant part of the MoE’s potential untapped, resulting in lower uplift factors.

Compared to the research era, this era did not introduce as many new ideas (though with granularity one of the most influential ones), but is rather characterized by the slow yet steady adoption of the MoE paradigm in the LLM landscape. Many released models still remained dense transformers though. The last missing piece, as it turned out, was a successful blueprint for delivering the MoE promise at scale considering both modeling and systems aspects for efficient training and inference (see this post). While DeepSeek-V2 (DeepSeek-AI, 2024) already introduced some of these aspects, e.g. device-limited routing, only DeepSeek-V3 was that full holistic picture and marks the beginning of the next era.

The DeepSeek V3 era (2025-2026)

DeepSeek-V3 was released in December 2024 (DeepSeek-AI, 2024). On the modeling side, it kept DeepSeek-MoE’s ideas of fine-grained experts and shared experts, and added loss-free token balancing based on expert biases (and switched to sigmoid activations). But more importantly, it came with detailed discussions of the necessary engineering and systems work to train and inference an MoE of that size (670B parameters). Several other models followed that blueprint and modified it only slightly, such as Kimi K2 (Kimi Team, 2025) which increased sparsity and trained with Muon, or GLM 4.5 (GLM-4.5 Team, 2025) which varied the width-to-depth ratio, and Ant Group’s Ling/Ring family of models (Ling Team, 2025). This continued in 2026, when several US open model releases also followed the DSV3 architecture closely (Arcee Trinity, Laguna, Inkling).

Qwen 3 (Yang et al., 2025), the first from-scratch MoE of the Qwen series (as opposed to being upcycled), differed more by using no shared experts and relying on the established auxiliary loss for balancing.² In mid 2025, OpenAI released their first ever open-weights MoE, gpt-oss (OpenAI, 2025), also featuring fine-grained but no shared experts. And 2025 had the last release of the Llama series, Llama 4 (Meta, 2025), but since its impact in the field remained far behind its predecessors, I would not want to put too much weight on its MoE architecture (it was the first (and only) MoE in the Llama series and made a few unorthodox choices). LatentMoE (NVIDIA, 2026) was one of the rare new ideas of this era that found adoption, e.g. in MAI Thinking 1.

In these (almost) two years following the DSV3 launch, MoE clearly became the dominant approach for at least the largest LLMs. Many new companies entered the field and contributed additional closed or open models. The models also grew in size, up to 2.8T total parameters in the latest Kimi K3, and the activation ratio settled at 3-5%. But in the architecture and training recipe itself, not much changed: All models use fine-grained experts, some add shared experts, some not, some have initial dense layers, others not, most use sigmoid or softmax for the router, and expert biases or the load balancing loss (globally in either case) for token balancing. Of course, a lot of other changes went into these models, such as various ideas for more efficient attention architectures, or increasingly lower precision training, but I want to focus here on the MoE architecture and training. On that side, even in the most recent models as of writing this in late September 2026, such as Qwen 3.8 Next Flash, GLM 5.3 Flash, DeepSeek V4.1 Flash and MiMo V2.6 Flash, nothing new enters the picture.

Measuring uplift

Having reviewed the MoE models of the previous years and what went into them, I want to now turn to measuring the MoE uplift over dense models, $N_{eff} / N_{a}$, as introduced at the beginning of the post.

Controlled scaling studies

The best data source to measure MoE uplift is a controlled scaling study that compares MoE models with dense models. Luckily, several such studies have been done over the years. Since they train dense and MoE models under otherwise identical conditions, such as same training data, same backbone architecture (attention type, FFN type, width, depth, etc.), the loss difference between matched dense and MoE models can be fully attributed to the MoE architecture and training recipe and hence reflects what I call the uplift. And in addition, scaling studies typically sweep dimensions such as model size, data volume and sparsity, allowing us to measure uplift along these dimensions.

As I already mentioned, Clark et al. (2022) is a great example of such work, with the only caveat that they trained models of all sizes and sparsities on a single fixed amount of data. But since they released loss values for all models and all intermediate checkpoints, we can slice out a portion that holds tokens-per-parameter (tpp) constant as in Chinchilla-style scaling laws (at 315 tpp in this case). This yields the following plot of active parameters vs. validation loss at different sparsities:

To compute uplift, we can fit the dense model loss curve with a linear function in log-log-space that maps parameters to loss, invert it, take the loss of an MoE at a specific sparsity and parameters $N_{a}$ and compute the dense parameters showing that same loss, which gives us $N_{eff}$ and thereby the uplift factor $N_{eff} / N_{a}$. The following table shows the resulting uplift factors for Clark’s models that have between 15M and 370M active parameters:

As expected, the sparser the model (the lower the activation ratio), the higher the uplift is. For the nowadays typical sparsity range of 4%-8% activation ratio, the uplift is around 1.9 to 2.5. This uplift factor characterizes the effectiveness of the specific MoE setup used in that study: MoE in every second layer, top-1 routing, full-sized experts following the FFN sizing in dense models (expansion factor 4.0) and Sinkhorn balancing. That is very close to the Switch-Transformers recipe which was the most popular approach in the research era.³

Three years later, Ludziejewski et al. (2025) ran a similar scaling study. The MoE setup was again Switch-style, in this case even closer⁴, so it gives us a second data point for the effectiveness of that recipe. The table below compares the uplift factors, but also reveals that data volume (and hence tpp) is an important factor for uplift:

In setting A, the tpp for the active parameters is held constant, as we did for Clark, but here at the Chinchilla-derived value of 20 (the data does not allow using the same tpp for both studies). The uplift factors are lower, which I assume is primarily due to this undertraining aspect. But Ludziejewski’s data also lets us do something more interesting: Keeping the data volume fixed per active parameter underestimates the MoE’s potential since it scales data only with the lower active size, but not the true effective size, putting it at a disadvantage compared to the dense model. A more honest comparison would be a constant tpp against $N_{eff}$, which is what setting B shows (resulting in tpp w.r.t. $N_{a}$ of 23-64), leading to significantly higher MoE uplift of 2.2 - 3.2 at the previously mentioned sparsity range.

As the third relevant data point, Tian et al. (2025) published an MoE vs. dense scaling study in the same year. Their MoE setup is almost identical to Ludziejewski’s, but they add the two new ingredients DeepSeek proposed in 2024, fine-grained experts and shared experts, making it a representative of the second era. And, luckily, they model the influence of granularity directly in their study, which allows us to measure uplift at $G=1$ (full experts, resulting in Switch-style except with a shared expert), $G=4$ (DeepSeek-MoE’s granularity) and $G=24$ (the optimum in their sweep). The overtraining regime is comparable to setting B of Ludziejewski, scaling data with effective size.⁵ The following figure shows the results next to the other two studies:

That picture illustrates what those three studies can tell us about MoE uplift in the first two eras of my MoE history above. Using the 9% activation ratio point as a reference:

• The Switch-style architecture and training recipe provides about 1.5x uplift at Chinchilla-optimal tpp=20 training and about 2.0x with stronger overtraining at tpp=315. Scaling data volume with $N_{eff}$ instead of $N_{a}$ puts it at 2.2x - 2.4x even in the Chinchilla regime.
• The DeepSeek-style architecture with fine-grained and shared experts improves upon that, landing at 2.7x - 3.1x uplift with $N_{eff}$-based compute-optimal data scaling.

So clearly, from the first to the second era, the MoE benefit as measured by uplift increased and we can attribute it to DeepSeek’s architecture innovations. But beyond that, the existing data unfortunately cannot tell us much more, since there are no controlled studies that ablate every single decision (e.g. granularity without shared expert, Switch but no z-loss). As far as I know, those three are the only studies usable for our purpose here.⁶

Released models

A second data source for uplift is released models with their benchmark numbers. But, we can only use that data to measure uplift if a model family comes with both dense and MoE models, and with enough of them on the dense side to have a scaling law to map the MoE quality against. Such a comparison is somewhat clean, as models of the same family usually share a common backbone architecture and use the same training data, so we can attribute the difference to MoE’s sparsity and routing, but it remains less clean than the previously discussed scaling studies, especially because released models often don’t even disclose all relevant details.

I use the following methodology: Given an MoE and multiple dense models from a model family, fit a function from $log_{10}(N)$ to score across the dense points, invert it at the MoE’s score and use the resulting $N_{eff}$ to compute uplift. I say score, because these models typically don’t report perplexities but accuracies on benchmarks. That’s why I do it per benchmark reported for both dense and MoE and then take the median out of these uplift factors. The MoE needs to score in between the dense points (to not land in the extrapolation region of the dense fit) and I skip benchmarks that are close to saturated (dense fit not improving at least 8 points across 10x parameters).⁷

In fact, although plenty of models were released over the past years, many as open models (see the history section above), not many satisfy the requirement of having both MoEs and comparable dense models in a family. I found these to be usable:

• GLaM (12/2021): GLaM was trained by Google in 2021, and since this was the research era, it is strictly speaking not a released model of course, as only the paper was published. It comes with a 4-point dense ladder from 130M to 137B parameters, matching MoEs at 1.7B and 8B active and using top-2 routing over 64 experts every other layer with full-sized experts. A typical Switch-style recipe, though with top-2 routing instead of top-1, similar to the earlier GShard. The uplift factor computed for it is 1.80-1.92, with the caveat that the paper only reports already aggregated benchmark scores rather than individual ones.
• Artetxe et al. (12/2021): This one is also a paper of the research era, similarly featuring four dense models with one suitable MoE that we can match against them. Different from the rest, it reports perplexities, so we can handle it akin to the scaling studies. The MoE is similar to GLaM, top-2, every other layer, and particularly sparse with 512 full-sized experts. The resulting uplift factor is 1.76.
• DeepSeekMoE (01/2024): This data point is the first DeepSeek-style one, from the very paper that introduced it. Fine-grained experts with granularity 4, 64 routed experts in total, 6 active, plus 2 shared experts, in every layer but the first. The limitation is that the dense side has only two points, 7B and 67B, between which the MoE with 2.8B active lands. That means any uplift we can measure here is at least 2.5 and benchmarks with lower factors cannot be captured. The uplift, under these caveats, lands at 3.86 as the median of 5 benchmarks.
• Qwen 3 (04/2025): This is the cleanest data point, the dense ladder has six rungs, the 30B-A3B MoE lands within that range, data is the same, hyperparameters were tuned on either side. The architecture is as already mentioned earlier in the post: 128 experts, top-8, no shared expert, global batch load balancing via auxiliary loss, every layer MoE. The uplift of that recipe ends up being 5.3 as the median of 12 benchmarks. All benchmarks measured on the base models (pretraining only).
• Qwen 3.5 (02/2026): Qwen 3.5 is similar to 3, except that I could only find results for the posttrained version of the models. Again, only the smallest MoE can be used because only that falls into the range covered by the dense ladder, it uses 256 experts, now also a shared one, and otherwise the same setup as Qwen 3 (when it comes to MoE). Here the resulting uplift factor is 6.47 across 18 benchmarks.
• Gemma 4 (04/2026): Gemma is similar to the Qwen 3.5 case: we also need to rely on posttraining numbers, and in addition, the dense side has only two data points. And worse, one of them even differs more by using a different multi-modal stack. The MoE architecture is similar to the other two, 128 experts, top-8, wide shared expert, unknown balancing. The uplift is 5.63 across 10 benchmarks.

Despite the limited number of suitable models, we have good coverage of the eras: Two examples of Switch-style MoEs from the research era (GLaM and Artetxe), one example of DeepSeek-style models from the ChatGPT/Llama-era (DeepSeek-MoE itself) and three examples from the most recent era (Qwen 3, 3.5 and Gemma). This is how it looks when placing the six uplift factors at their sparsity points in (a slightly simplified version of) the previous plot:

What can we see? The two era 1 data points, GLaM and Artetxe, fall into the range that the scaling studies already established for Switch-style MoEs. DeepSeek-MoE from era 2 lands above that, as predicted by the scaling studies for fine-grained models, and shows even higher uplift at G=4 than Tian’s study, which most likely is due to higher overtraining. The most recent era 3 models bring uplift to new heights, with 6.5 for Qwen 3.5, and I attribute that to both overtraining and increased granularity. The MoE architecture and training recipe itself, apart from the granularity, is not different from the original DeepSeek after all. And that’s probably the most interesting finding of this data: While the last two years brought almost no new architecture or training innovations to the MoE setup, uplift still improved, indicating that the labs at least managed to squeeze more out of that established setup.

That being said, these data points are also only somewhat reliable, for one because they rely on a particular selection of benchmarks that differs from model to model (but is consistent within the dense-MoE-family of course), second because we compare pretrained and posttrained models in some cases, and third we compare uplifts computed against perplexities with uplifts against benchmark accuracies. What is not a confound here though is the fact that models got better by other means (e.g. better data, attention, optimization, posttraining) since that tends to benefit both dense models and MoEs equally and therefore largely cancels out in the uplift.

Which changes matter?

While uplift factors are a nice concept and we could collect them from quite a few sources, they still leave many questions unanswered. They simply don’t exist for all small single-factor ablations that we would want to have to judge all the MoE architectures and training recipes proposed over the years. Given those gaps in the data, here’s a more opinionated review, based on the existing data, adoption across the model landscape and personal experience:

What is here to stay: Expert granularity is undeniably the concept with the strongest positive evidence that was also adopted across the board. For shared experts, it’s a bit weaker, and the interaction with the MoE layer placement, as also pointed out in the MAI Thinking report, might have contributed to that lack of clarity. For all layers (but the first) MoE models, shared experts are adopted by the majority.

What disappeared: Using MoEs in every nth layer, instead of all of them, was popular in the research era but has mostly disappeared lately. I’m not sure though if there are strong reasons against it; after all, it makes shared experts unnecessary and it provides more design flexibility by decoupling depth from sparsity. What definitely disappeared, probably due to the effectiveness of fine-grained experts, is increasing top-k independent of that, such as the top-2 with full-sized experts as in GShard, GLaM and ST-MoE. Token dropping is typically avoided as much as possible nowadays. And expert choice, given its shortcomings, never really got adopted in language modeling.

What’s less clear: Token balancing remains crucial, but no single method dominates. While BASE’s linear assignment and Clark’s Sinkhorn mostly disappeared, the auxiliary loss as well as DeepSeek’s expert biases are both actively used. Both, or even all four, can be effective if used at the right token scope, which is the more important factor. I discuss that particular topic in-depth in this post. What’s less clear is if initial dense layers, or fixed routed MoE layers as in the recent DeepSeek, are needed, and if so, how many of them. Models still vary in this regard. And similarly, both softmax and sigmoid, and even other activation functions, are still used for the router.

What’s next?

Given all that, the history of models and methods, the changes in uplift that we could measure, and my verdict on the methods above, one might ask what’s next for MoEs. One possible answer, supported by the most recent developments, would be “nothing at all”. While there is still a plethora of academic papers being published on MoE architecture and training, very little has found adoption in recent LLMs, and as I argued above, most details have not changed since DeepSeek-MoE almost three years ago. I’m very much looking forward to the paper that shows us how to obtain 10x or 20x uplift, but maybe that simply never comes and with what we have right now the potential of MoEs is fully leveraged already. Given the increasing focus on other aspects in recent years, such as efficient long-context attention methods or increased depth-wise information flow (e.g. mHC, attention residuals), this might turn out to be true.

That being said, there are two related threads worth watching since they both bring in forms of sparsity in other parts of the model: First, extra embedding tables. Since an embedding is merely a lookup operation, it is sparse in the sense of every token just engaging with its own parameter row, and in terms of FLOPs even better than MoEs since a lookup requires no FLOPs at all. Gemma 3n, later Gemma 4, and also STEM added a table per layer. DeepSeek’s Engram extended the idea from unigrams to n-grams, which DeepSeek V4.1 adopted. And second, looping. Varying the depth of a network per token is an old idea already explored with RNNs, e.g. by Graves (2016), and later integrated into transformers as Universal Transformers. At the time, this was typically framed as adaptive compute or early exit to save compute on simpler inputs. But, the very same idea can also be used to increase compute dynamically. While MoEs add extra parameters at constant FLOPs, looping adds extra FLOPs at constant parameters. Two recent papers that look at this are Lee et al. (2026) and Wang et al. (2026). Looping becomes particularly powerful when combined with MoEs: While a looped dense transformer can only repeatedly apply a fixed learned function, a looped MoE, since it routes based on input representations, can apply a different computation at each iteration of the loop even though the parameters are reused.