The Experiment Funnel: How to Organize LLM Pretraining Experiments

Designing meaningful and informative experiments is a key skill of any scientist and should be a common task for anyone working with machine learning models. However, with the complexity of contemporary LLMs, it is sometimes difficult to find the “perfect” experimental setup, especially when collaborating in a larger team that works together on a single model. Moreover, in today’s automated world where coding agents can reliably take care of many of the previously time-consuming parts of running ML experiments – implementing new features, setting up code and configuration files, babysitting cluster jobs – designing the experiment itself becomes an even higher-leverage activity than it was in the past.

Despite that importance, not much is written about this topic in the typical tech reports that accompany released LLMs. A few notable exceptions are Hugging Face’s write-ups, such as The Smol Training Playbook, which has a section on how the ablation setup was defined, and the MAI-Thinking-1 tech report, which has a similar discussion in Section 2.2.

In this post, I will also write down a few thoughts on that topic, drawing from several years of LLM pretraining experience. As usual in this blog, I tend to focus primarily on pretraining, but parts of it will be equally applicable to posttraining and other kinds of LLM experimentation. The context I assume is that a team experiments in order to prepare a larger training run for which the right architecture and training recipe choices need to be made. That is different from my earlier post on experiments in published papers: Here we only care about making the best choices for that one model to be trained, not to show that a method or idea is beneficial across a variety of settings.

Goals & Challenges

A well-designed experiment should satisfy at least these objectives:

  • Correct Treatment: Obviously, it needs to test what we want to test, and only that; otherwise the result is meaningless.
  • Real Effect: Observed changes in metrics need to be distinguishable from noise and capture the relevant model capabilities.
  • Desirable Scaling: Since experiments are typically done with models smaller than the target size, the effect needs to persist when scaling up.
  • Cheap & Fast: The more I can test with a given budget, the more data points I have to make my final choices for the target run.

Correct Treatment

This sounds obvious, but in practice it still often goes wrong. The code needs to be free of bugs and the baseline and treatment models need to differ only by what’s supposed to be tested and nothing else. Changes need to be as small as possible for accurate accounting of which factors matter. Typical issues are a feature that isn’t actually running, a checkpoint that is silently corrupted before being evaluated or a systems change that interacts with the modeling change under treatment. For instance, if comparing a treatment against a baseline requires also changing the parallelism to stay in the memory budget, which normally shouldn’t affect the model quality, it’s good practice to have tests that actually verify the training codebase is invariant across parallelism settings. Full reproducibility of runs, and full documentation of their setup, is important to allow correct comparisons across time, such as reusing a baseline trained a while ago. And finally, a meaningful comparison also has to be fair, for instance by dedicating equal hyperparameter tuning effort to baseline and treatment.

Real Effect

Even if our experiment is apples to apples and compares the intended change and nothing else, what effect can be observed in evaluation metrics depends a lot on the experimental setup. Model size (and training horizon) is one important factor, in direct conflict with the cost goal: We want to use the smallest amount of compute that gives informative results. Which metrics to look at is another factor, and maybe the single most important one in good experimental design. It needs to consider what model capabilities matter for the experiment, how to weight or aggregate them if multiple, which datasets and benchmarks to use to measure them, whether perplexities are informative enough or downstream benchmark accuracies are needed, what the noise threshold of a specific benchmark is, and many more. Again, there is a trade-off between cost/speed and the quality of the observed signal, e.g. computing perplexities is faster than running downstream benchmarks. Most importantly, there is no single answer to all these questions: Each experiment, or at least type of experiment (e.g. data mix, architecture, long context), needs a dedicated setup that balances signal quality with execution cost and time.

Desirable Scaling

Given the cost goal, experiments are typically done at the smallest size that gives a reliable signal. But the effect of a treatment can also be a function of the model size, or more generally, the amount of compute (= model size + data volume). To ensure that a change is also beneficial at the target size, or more generally, to see how it trends with compute, an experiment should be run at different sizes. If the effect decreases with scale, it might only matter for small models and not be meaningful at scale. If it stays constant or even increases, it’s a strong candidate for the target model. Scaling laws are a useful tool to formalize this idea and MAI-Thinking-1 gives a few examples in Section 2.2 (as well as their later blog post on MAI-Base-2). I reproduced two examples here to illustrate my point about scaling behavior, showing improvements that increase with scale (left) and that are roughly constant (right)1:

Activation gradient clipping0%10%20%30%40%10221023More experts (top-8 MoE)10201021Training compute (FLOPs)Efficiency gain over baselineL18 - 2.0e21 FLOPs - efficiency gain +1.9%L24 - 1.0e22 FLOPs - efficiency gain +6.9%L30 - 2.9e22 FLOPs - efficiency gain +9.5%L36 - 7.0e22 FLOPs - efficiency gain +16.2%L42 - 1.9e23 FLOPs - efficiency gain +27.5%L18L24L30L36L42256 experts - 4.0e19 FLOPs - efficiency gain +0.1%256 experts - 2.3e20 FLOPs - efficiency gain -0.6%256 experts - 1.0e21 FLOPs - efficiency gain +1.2%256 experts - 3.6e21 FLOPs - efficiency gain -0.6%512 experts - 4.1e19 FLOPs - efficiency gain +15.3%512 experts - 2.3e20 FLOPs - efficiency gain +17.7%512 experts - 1.1e21 FLOPs - efficiency gain +21.6%512 experts - 3.6e21 FLOPs - efficiency gain +21.3%1024 experts - 4.1e19 FLOPs - efficiency gain +27.9%1024 experts - 2.3e20 FLOPs - efficiency gain +34.1%1024 experts - 1.1e21 FLOPs - efficiency gain +39.4%1024 experts - 3.6e21 FLOPs - efficiency gain +40.3%256 (baseline)512 experts1024 experts
Efficiency gain over a baseline ladder, i.e. how much more training compute the baseline needs to match the treatment’s loss. Left: activation gradient clipping in MAI-Base-2, whose gain grows from 2% to 28% across the ladder. Right: 512 or 1024 instead of 256 experts in a top-8 MoE in MAI-Thinking-1, whose gain levels off. Redrawn from Figure 8 of the MAI Base 2 post and Figure 3 (weighted average) of the MAI-Thinking-1 report.

Also in this case, a careful experimental setup is necessary to obtain clean results. If one just picks arbitrary models of different sizes, e.g. a 200M, 1B and 7B that happen to exist, the trends can be misleading. At each compute budget, we need a clean comparison of baseline and treatment models with only the intended difference, but also across the budgets the models need to form a consistent setting that is scaled in size but otherwise the same. This is known as a scaling ladder. That means, for example, that both width and depth need to be scaled jointly to keep the aspect ratio constant; you shouldn’t increase just the layers or just the width. Model size and data volume need to be scaled jointly to keep the overtraining factor constant. And which overtraining factor to use can be important as well and might change findings. Even the data mix itself needs to be carefully controlled. For instance, if some sources are used for multiple epochs at the target compute budget, they should be sliced to a smaller dataset that gets repeated for the same number of epochs in smaller settings instead of using a larger slice only once. That way all steps on the ladder face consistent amounts of data repetition during training.2 All steps of the ladder also need equally well-tuned hyperparameters to make their performance comparable.3

Conflicts

The goals discussed above build nicely on each other: correctness is the minimal requirement, getting reliable signals from the metrics the next level and seeing that consistently across scale the final one. The fourth one, making the experiments cheap and fast, is the one that tends to conflict. The bigger the model size, and the longer the training, the cleaner the signals. The larger the ladder, the cleaner the scaling trend. But all of it makes it slower and more costly to run. So one of the key goals of a good experimental setup is to find the right trade-off for this conflict. A second tension is between having dedicated settings for what’s under test vs. a single shared setting that makes all experiments comparable. While the latter is eventually needed to make decisions across the full list of potential changes, dedicated setups can often speed up the initial exploration of ideas and provide clearer insights via problem-specific metrics and diagnostics.

The Experiment Funnel

At this point it is probably clear that it is quite hopeless to try to find a single ablation setup that works for everything. Instead, I think a good way to resolve the conflicts discussed above and thereby achieve a good use of experimentation resources is to rely on a range of different experimental setups arranged in a funnel-like sequence, which I call the experiment funnel:

  • Level 1: Diagnostic toy settings for fast exploration. Very simple settings that allow fast iteration. Dummy data, very small models, all unrelated pieces removed. To achieve that, this setting is typically specific to a certain topic, e.g. one for testing MoE router balancing methods, one for long-context attention, etc. Data and metrics are more diagnostic, to stress-test the idea, not to measure general capabilities on standard data mixes. The MoE routing testbed I described in a previous post is an example of this level, though I think there is value in having it even simpler: Sometimes it can be just a notebook that runs in seconds or a few minutes to truly allow fast exploration.

  • Level 2: Canonical baseline per experiment type. This experimental setup gets closer to the intended target model, using typical data mixes, benchmarks and metrics, but at small compute scale. The exact setting, e.g. the compute budget, the allocation to model size and data volume, the set of metrics, can still differ based on the type of experiment. Testing a new data mix might be more informative in one setting while a new architecture idea is better tested in another. But per type, this is where different ideas first meet and can be compared to each other in a setup close to the real setting.

  • Level 3: Scaling behavior on joint ladder. This level now tests an idea at different scales and compares the scaling behavior against a common baseline. It is the scaling ladder setting that I described above and that MAI’s tech report describes in detail. It also unifies the setting and the baseline to beat for every potential change, so that it becomes the test that all ideas coming out of the lower levels, no matter if architecture, data, optimizer or whatever else, need to pass to make it into the target run. To keep costs bounded, the ladder is run at a low tokens-per-parameter ratio.

  • Level 4: Integration ladder with overtraining. As the final step towards the target run, the ladder from level 3 should be run with higher overtraining factors close to the target run, since that additional training horizon can change things. But since it is the most expensive type of experiment, it can only be run occasionally, for example as an integration run that bundles all promising changes that passed the previous levels to test the current best recipe.

Level 1Toy settingsper topic$manyLevel 2Per-type baselineper type$$fewerLevel 3Joint laddershared$$$fewLevel 4Integrationshared$$$$bundledSetupCost per runIdeasTargetrun
The experiment funnel. Each line is one idea, colored by experiment type. Ideas are first explored in parallel in topic-specific toy settings, then compared per experiment type and finally tested on one shared scaling ladder, with many of them dropped (crosses) at every level. The survivors are bundled into an integration run at the overtraining factor of the target run.

The exact funnel and its levels don’t have to look exactly as described above, but I hope the key ideas are clear. Throughout the funnel, the cost of each experiment increases, the diverse setups unify towards the target run and the metrics become more robust. The expectation is of course that also the number of ideas tested at each level decreases significantly along the way, with many initial ideas discarded, resulting in an efficient resource allocation that allows both fast and wide exploration of ideas in the cheap initial settings and reliable confirmation of the most promising ideas for the final run. If a large team collaborates on that final target run, levels 3 and 4 are where the work done in parallel flows together. And levels 1 and maybe 2 are where high degrees of automation with coding agents can be used to explore ideas in depth.

Summary

Good experiments are the foundation of every decision that goes into a large training run, but how to best design those experiments can be challenging. Clean signals and scaling trends require compute, while exploring many ideas requires experiments to be cheap and fast. And while ideas are best explored in setups dedicated to their type, decisions eventually have to be made in one shared setting. My proposed experiment funnel admits that this cannot all be done in a single setup and instead spreads the work across several: wide and cheap exploration at its wide end, careful and expensive confirmation at its narrow end. With coding agents taking over more and more of the execution, I expect getting such structures right to become one of the most valuable parts of the work.


  1. The plot about sparsity illustrates another interesting aspect of experiment design: Increasing sparsity by keeping active parameters fixed and adding more experts, as done by MAI in that case, is a fair comparison in the sense of comparing at fixed FLOPs and hence compute, but ignores the resulting difference in total parameters that need to be managed (see my first post). With no bound on total parameters, any extra sparsity will generally improve the model, even if returns diminish. It therefore doesn’t help for picking a sparsity. For practical decisions, more objectives need to be incorporated, such as training and inference efficiency. ↩︎

  2. MAI Base 2’s blog post discusses that aspect of robust scaling ladders. ↩︎

  3. A recent paper from Meta, Lourie et al. (2026), points out that very small model sizes in fact need the most hyperparameter tuning effort. ↩︎