Mixture of Expert Models: Why FLOPs Don't Tell the Whole Story

This post is the first in a (hopefully) longer series on LLM pretraining. I have spent the last three years on pretraining LLMs, and in particular, mixture of expert (MoE) models, closely watching the development of the field while experimenting with and training said models myself. While I can’t and won’t talk about the exact details of what I worked on at my employer (except for a few things that have been published), I will share some high-level observations, thoughts about the technology and learnings from the work, which still gives plenty of interesting topics to write about.

Since my focus has been on MoEs for the majority of the time, I want to start the series revisiting the central promise of sparse models such as the commonly used Transformer-based MoEs: By activating just a subset of a model’s parameters for each token, model quality and compute – the two key units whose relationship scaling laws describe – become decoupled. The parameters of a model break down into total parameters and active parameters, the former reflecting the model’s total capacity available for learning and hence the quality it can achieve, while the latter indicates the compute it requires to train and inference the model, typically expressed in FLOPs. Or put simpler: You can get better models without paying for it. At least not the price known from dense models.

That central promise is valid, but it comes with a few caveats that are discussed a lot less than the headline itself, namely memory and communication. The remainder of this post will focus on these two to paint the more honest picture of the MoE promise. As we’ll see, it requires a bit more effort to fully reap the benefits than just changing the architecture.

Quality

When I started working on LLM pretraining about three years ago, MoE architectures existed primarily in research papers, while the dominating open models – Llama 1 and 2 were released at that time – were exclusively dense. Only by the end of 2023, the first MoE models were publicly released, most notably Mixtral, and soon after, DeepSeek-MoE. Today, sparse MoE architectures are the standard, used by open frontier models (e.g. DeepSeek, Qwen, GLM) as well as closed ones¹.

Qwen3.5 is a more recent example that nicely illustrates the impact sparsity has on model quality because it comes in both dense and sparse versions that were trained under comparable settings. As shown in the plot, there is a clear gap between dense and sparse MoE models at matched active parameters, which is the MoE promise I keep referring to:

A more controlled version of this comparison was first done by the scaling study of Clark et al. (2022), which trained dense models as well as sparse counterparts of increasing sparsity. As you can see in the following plot of their data, the MoE promise is not just a constant benefit, but of course increases with the degree of sparsity. The more experts they add to a dense model, the more quality improves (or here, validation loss decreases).

Similar findings have been confirmed by later scaling studies, such as Krajewski et al. (2024) and Abnar et al. (2025), and the quality effect is uncontested.² But, at what cost does it come? Looking at just FLOPs, it seems to be free, since FLOPs are a function of only the active parameters and thus would predict the same cost for dense and sparse models at matched active parameters. And that’s exactly where the story isn’t fully complete.

Memory

The first issue with sparse MoEs is that the additional experts, even if inactive and therefore not contributing to compute, still need to be present in memory. The selection of experts, and their separation into active and inactive, is done per token by the learned router, and that selection can of course differ across the tokens in a batch. So while per token the compute is limited to the number of selected experts, across a larger pool of tokens all experts get eventually used. Compute is a function of active parameters, but the memory for storing the weights is still a function of the total model size.

There is an elegant solution to the memory problem and it has been around almost as long as MoEs themselves: Expert Parallelism (EP).

As depicted in the figure above, the idea of EP is best explained by comparing to a dense model and the naive MoE setup that incurs the high memory cost we just discussed. While the dense setup, when combined with data parallelism, holds exactly the same copy of the model on each device, the same approach applied to MoEs results in the FFN memory footprint growing E-fold with E experts. In addition, each expert is heavily underutilized since just a subset of the local batch will be routed to it.

EP, as shown in the right-most column, fixes the memory issue by distributing the experts across devices, as the name already suggests. In the extreme case of having an expert parallel degree as high as the number of experts, the memory footprint reduces to the dense case and we get the sparsity benefit for free. Well, still not completely yet, because we have solved the memory issue, but there is now communication needed to send the tokens to the device where their assigned expert is located, and similarly send the expert outputs back to the original device, via all-to-all collectives.

Before we turn to communication, I want to point out how elegantly EP interacts with DP: Under a perfectly balanced load, i.e. when each expert is assigned the same number of tokens in a batch, the all-to-all communication becomes a token exchange, with each device sending as many tokens out as it receives and keeping the same-sized local batch before and after. This is only possible because EP does not act in a new orthogonal parallelization dimension but operates inside the existing data parallelism, which provides the tokens that can be exchanged with each other. And as a result, it also solves the underutilization problem: Each expert can now process a full-sized batch similar to the FFN in the dense model. This also demonstrates why token balancing is such a crucial topic in MoE training, which we’ll discuss in a dedicated upcoming post.

Communication

EP solves memory but adds communication, which creates a new problem: It now takes longer to run a pass through the model. And compared to other types of parallelism, like the gradient synchronization necessary in data parallel training, this communication even lies on the critical path of every forward and backward pass, meaning it can’t be as easily overlapped with computation. Nonetheless, overlapping compute and communication is the solution, it just takes more effort to achieve that in MoEs.

DeepSeek’s DualPipe or Megatron’s interleaved 1F1B with A2A overlap are pipelining approaches that achieve significant overlap, but doing so requires very careful scheduling of micro batches across pipeline stages, decoupling forward and backward computations (and even finer-grained subsets of the backward) or overlapping subsets of experts. Even with that, what can be hidden depends on the time communication and computation portions of the model take, which depend on the exact architecture (defining the matmul sizes as well as the communication payloads) and the hardware (determining the compute and networking capabilities). Beyond the plain MoE setup, specific architectural variations like using a shared expert, or more drastically, a fully parallel transformer architecture (as in PaLM), can provide additional opportunity for overlap, but changes like the parallel layers come with consequences also for model quality. On the architecture side, fine-grained experts are another important factor, since besides their positive impact on model quality, they also remove the need for tensor parallelism for experts, which would complicate successfully overlapping the all-to-all even more. But more importantly, even for the conceptually best overlapping strategy, whether it delivers in practice further depends on very careful implementation: carefully managing CUDA streams, allocating sufficient SMs to compute and communication kernels, avoiding stragglers, etc. And finally, the hardware. While it determines the timing of communication and compute blocks and hence the potential for overlapping, it shapes the model in even more ways. Most importantly for MoEs, communication bandwidth changes dramatically when going from intra-node communication (via NVLink) to inter-node (via InfiniBand), on an H100 the bandwidth difference is about 18x. Overlapping the all-to-all with compute is a completely different challenge if that communication is across nodes or stays within one. And at what number of devices that boundary sits depends again on the hardware being used (e.g. a GB200 NVL72 rack has 72 GPUs in NVLink instead of just 8).

DeepSeek V3 was an excellent example to demonstrate what it takes to handle all these challenges: careful pipelining with DualPipe, custom communication kernels with DeepEP, limited inter-node communication via node-limited routing, fine-grained experts to avoid tensor parallelism, to name just a few ingredients. Compared to using EP to reduce the memory bottleneck, hiding the communication requires a lot more effort and tricks, especially at the implementation level, and it relies on a careful co-design of model architecture, implementation and hardware. Only with that, the matched compute compared to a dense model also translates to matched latency and training cost.

That being said, fully hiding all communication and reaching cost parity with dense models is not strictly necessary to make MoEs attractive. As discussed before, the appeal of sparse models is that they bring a sizable improvement in model quality, and depending on the size of that, even with some amount of communication overhead, and hence higher training cost than the dense counterpart, the sparse model can still offer a better cost/quality trade-off.

Inference

So far, the discussion assumed training when talking about memory, communication, latency and cost. But the same challenges also exist at inference time, and in fact, they become even more challenging. I don’t want to go deep into inference here, but I’d like to highlight two important differences that inference adds.

First, batching. We saw above that EP interacts nicely with DP and thereby hides the memory cost associated with the additional total parameters a sparse model brings. But that assumes large batches of tokens are processed. During inference, they don’t come naturally, but need to be constructed on the fly by combining independent incoming requests. This is typically already done in modern inference solutions to amortize the cost of reading model weights from HBM during decoding, but MoEs add another reason why inference should be run with batches as large as possible: distributing experts over devices fits them in the available memory, and distributing tokens across the same dimension keeps all those devices busy.

Second, the data distribution is no longer under control. As I briefly hinted at above, an important factor for the efficiency of EP is that the token load across the experts is even. During training, this is achieved by actively encouraging the router to assign tokens in a balanced fashion. But crucially, that balance is defined with respect to the training distribution, a diverse mix of data across all domains a model should learn. At inference time, we don’t control the data distribution, it is simply the sum of the incoming requests. If they happen to concentrate on a certain domain at some point in time, the expert usage of the resulting batch is necessarily skewed, even if the model was nicely balanced during training. The resulting all-to-all communication and expert matmuls mirror that imbalance and stragglers increase latency. For that reason, various MoE-specific inference solutions have been proposed that depart from the regular expert placement used in training and instead dynamically deploy experts redundantly or not at all based on demand.

Summary

The adoption of sparse MoE architectures has been one of the key ingredients for scaling LLMs over the past years to the capabilities they have today. Better quality at same FLOPs was the, somewhat simplified, promise they brought to the table. As we saw in this post, the reality is a little more complex, memory and communication need to be handled, and implementation is where that promise becomes reality or not. A good illustration of that is how much space the DeepSeek V3 tech report spends describing model architecture vs. system implementation. Despite those hurdles, sparsity has now fully conquered the landscape of LLMs and dense models have become rare, especially at the frontier.