Why I don't trust any single paper's claims

This recent post on X reminded me of an article I always wanted to write about how to read and interpret published papers and their claims appropriately:

While the fraud part probably (hopefully) applies to just a small minority of papers, the overclaiming issue is indeed a large problem. The topic often comes up when discussing papers with people newer to the field, in particular interns, so I always wanted to write down my thinking about this issue and how I approach it.

The core problem is that a lot of papers published at ML conferences, at least in my field, express their findings in much more general terms than the actual experimental evidence warrants, making the naive reader, such as the inexperienced intern, who takes those claims literally, overestimate their importance and validity. I had cases where interns literally made statements like “why would anyone not use method X given that paper Y showed it is the best”. In this post, I’ll argue why it is almost never the case that a single paper showing that some X is superior to alternatives does so in a way that this question would be fully settled for the future.

As a disclaimer, my experience is primarily from the LLM training literature. Other fields, even closely related topics within ML/AI, might not have this issue to the same degree, since it depends on the publishing/reviewing customs and the nature of the experiments.

The quantitative gap

Let me start by illustrating the problem quantitatively. Over the last few years, I worked primarily on mixture of experts (MoE) architectures and training recipes. If you query arXiv for papers mentioning “mixture of experts” in the abstract, there are 2959 in the last three years, and 975 that have both “mixture of experts” and “routing” in the abstract (which makes it more likely they discuss new architectures and training recipes, as opposed to being just applications). All papers that propose something new, a slightly different architecture, new routing method, new auxiliary loss, etc., come with experiments that show their proposed method outperforms the established methods. If that evidence didn’t exist, the paper wouldn’t be published at all.

And yet, while those proposed methods are presumably all beneficial, the changes that really get adopted in the field are orders of magnitude fewer. Comparing MoEs from three years ago, e.g. Mixtral, with the most recent ones, such as DeepSeek V4, Kimi K3 or GLM 5.3, the number of MoE-related differences is probably something like 10 or 20. Even if it is a bit more (certainly depends on what counts as a change), and even if the 1000 to 3000 papers above overestimate the number of proposed improvements (a few hundred might be more accurate), there is undeniably a large gap that remains. The vast majority of proposed innovations are simply forgotten.

Why does the gap exist?

Because in practice, a lot of proposed innovations turn out to not actually make a difference, or at least not a positive one. So the interesting question is, why do reality and the paper disagree?

There are several reasons I can think of:

1. Faked Experiments: Some papers might truly be fabricated, with experimental results being completely fake. I hope that is rare, but it probably does exist.
2. Buggy Experiments: Some papers show positive results that are caused by issues in the experimental setup, such as bugs in the code, data issues, etc. that benefit the proposed method in a way that it looks more promising than it would in a non-buggy setup. That makes the experimental result meaningless, as the experiment doesn’t measure what it is supposed to measure, and the result is similar to case 1, just not done intentionally and usually unknown to the authors until a future reproduction attempt reveals it. This might sound like a rare case, but everyone who has ever found a bug in their own experimental setup whose fix completely changed the results can tell you that this is quite common even when setting things up very carefully. Variations of this are cases that are not implementation mistakes, but flaws in the experimental design that bias the outcome without the authors noticing.
3. Undisclosed Limitations: In this case, the benefit of a method is real, but it comes at a cost that is not fully disclosed. Reviewers are good at spotting some of these and typically push back, for instance when a paper shows improved accuracy without accounting for the increased cost, so fair experimental setups are usually the case. Another more subtle version of this though is a careful selection of the experimental setup, in which alternative methods are compared fairly, but it is a setup that hides a problem that the method would have in other settings. In other words, it does not generalize beyond the settings that were tested in the paper, but the claims might be worded so broadly that the reader assumes it does.
4. Experimental Breadth: This last one is similar to 3, where the issue is also generalization, though not at the conceptual level – where a method cannot be applied in another setting at all – but at the empirical level, meaning one can use it, but it simply doesn’t work as well. I think this case is the most relevant one in practice. The main reason is that training LLMs is a very complex process, models can have different sizes, different architectures, different training hyperparameters, use different training data – the list is very long – and there are lots of interactions between all of them. An experiment that shows method A is better than B or C does so in the context of all these other factors set to some value, aka the experimental setup, and no paper manages to cover a wide variety of such settings since the space is simply too large to explore. That leaves many generalization questions open: You can’t say if A-better-than-B really generalizes to all possible settings or if there are interactions with other factors that change the picture.

Ordered that way, the four categories form a spectrum of how far the claim is from the evidence behind it: The evidence is fake, it is wrong without the authors knowing, it is real but its price is not fully disclosed, or it is real and honestly reported but covers only a slice of the settings one might care about. The further down that list, the less anyone did anything wrong but the more common the case is.

While the reasons above explain the limitations of the experimental evidence a paper has, the true issue is of course something else: How the paper communicates the results. A statement saying “A beats B in the case of model C across sizes D, E and F on data G and …” would be honest but doesn’t read very well. The experiment section of a paper usually gives this honest statement because it specifies the full setting. But in the abstract, the introduction and the conclusions, the claims are often shortened, dropping the important qualifiers that make the non-careful reader expect more than the evidence really backs up. Clearly, these overclaiming habits are not just sloppy writing by the authors, but also incentivized by the existing norms and reviewing practices in the field, which demand bold claims to get papers accepted. But no matter what the exact causes are, what the careful reader needs to pay attention to doesn’t change.

Examples

I want to try illustrating the issue categories with a few examples, but that is trickier than it seems: Most of the gap between the number of published and actually adopted methods is just silently forgotten, no follow-up paper ever reports what the specific issue was. Maybe category 1, maybe 2, maybe 3, or maybe nothing at all and it just got overlooked. So naturally, the specific examples I will go over are in fact bad examples because they are the exception. I hope they are useful nonetheless to clarify my points.

In my first post, I mentioned the scaling study of Clark et al. (2022), which was the first broad study that showed how the benefit of sparse MoE models over their dense counterparts scales with increasing sparsity. While it did show that very well and provided useful data points for everyone working on MoEs, the experimental setup had one subtle flaw that makes it an example of category 2: All models, no matter their size in either active or total parameters, were trained on the exact same token budget, rather than co-scaling the token budget with the model size as later scaling studies did. That introduced an artifact into the data suggesting that the MoE benefit diminishes with scale, which disappears when instead keeping the tokens-per-parameter constant.¹

As an example of category 3, I want to discuss the Autonomy of Experts paper from ICML 2025. It’s a nice idea to get rid of routers in MoEs by instead using the activation norms of experts themselves as the routing signal. Since one cannot compute those activations for all experts – then we would lose the whole sparsity – the authors take one weight of each expert, the gate projection of SwiGLU-style FFNs, replace that part with a low-rank factorization and compute only the down-projection step for all experts. That way, the part evaluated for all experts remains small and the remainder of the expert computation is still done only for the selected experts. The conceptual advantage is that with no router, just experts, there is only one thing to train and no risk of misalignment between router and experts. In the paper’s experiments, the model improves by one point across a variety of benchmarks.

What one might miss at a first read is that the idea does not transfer from the paper’s 8 experts top-2 setting to more contemporary models with larger numbers of fine-grained experts. The factorization idea essentially replaces the $(d,E)$ router with a $(d,E,r)$ “router”, where $r$ is the low rank, and this is a dense part of the model since it is computed for every token. That factor $r$ growth in dense parameters implies that the sparse part of the model – the actual experts – needs to shrink to keep the overall active parameters constant, and that introduces a problem: Either $r$ is low, shifting few parameters from sparse to dense but creating a bottleneck due to the low rank, or it is increased, resolving the bottleneck but reducing the expert capacity. At 8 experts, top-2, 768 hidden dim, 3072 FFN dim (the paper’s setting), this effect is almost invisible. For a DeepSeek V3-style architecture with 256 experts, top-8, 7168 hidden dim, that’s not the case: A low rank dimension of 193, which is already fairly low for a model that size, would shift so many parameters to the dense region that in fact no parameters are left for the sparse experts at all.² Only with ranks lower than 193 can the idea be applied at all, and it introduces bottlenecks that hurt the model quality badly. Because of that, it is a great example of the category of not even conceptually generalizing beyond the experimental setup of the paper.

Category 4, which I claimed is the most frequent one, is about empirical generalization beyond the paper’s setting. As I argued, this happens typically because there are simply too many factors involved to account for all of them in ablations. One example is shared experts: DeepSeek-MoE introduced them in early 2024 and most MoEs adopted them since. The paper even has an ablation, showing that shared experts alone, independent of the also proposed fine-grained experts, improve their models. But one additional factor that is different from earlier work is the layer placement: While earlier work like GShard and Switch Transformer combined MoE layers with interleaved dense layers, DeepSeek-MoE used them in all layers. The usefulness of shared experts strongly depends on that context. This has been largely ignored even by subsequent work that adopted shared experts. I only know of MAI-Thinking-1 explicitly calling this out more than two years later, reporting that they see no value of shared experts in interleaved architectures. Similarly, other factors of the MoE setup can influence experiments, for example, a routing scheme that looks promising under softmax routing activations might not work with sigmoid, or vice versa. And as I showed in the previous post, scope is an important variable in token balancing that can change comparisons (assuming they are even done at the same scope in the first place).

What can readers do?

As a reader, this implies you should be skeptical of the claims a paper makes, especially in the abstract and introduction, and instead look at the exact experiments reported and make your own judgment on what conclusions the results support. That will catch some overclaiming, mainly of categories 2 and 3, but only those that a careful reviewer should already have caught (assuming the paper went through a reviewing process). Other issues, and in particular category 4, how far a proposed method generalizes empirically beyond the necessarily limited experiments of its paper, cannot be solved by careful reading.

I see two main options to address that case:

• Wait for adoption: Adoption by the field is the strongest signal that something is really useful across a variety of settings. That is also why I titled this whole post saying that a single paper is not enough. A method that only looks good because the paper overclaims will simply be ignored by subsequent work. A paper that is cited a lot seems to have more value. But more important than citations, which can mean many things, is whether a proposed technique gets adopted in later work. If a new architecture becomes the default choice in future work’s baselines or in released LLMs, then it must truly be good. But obviously, this solution has a price: The evidence is only clear with a long delay and only by the time everyone has already adopted it, so it does not work if you want to be an early adopter.
• Try yourself: Test everything that looks promising on your own against your own baselines. That gives you a second set of experiments that complement the ones in the paper, and importantly, rather than answering the broad question of what all the settings are that the findings generalize to, it answers it for that one specific target setting you really care about: your own baseline. This is the relevant question in particular if your goal for following the literature is to train your own production LLMs for which you want to adopt potential improvements. And obviously, what it trades off compared to the first approach is time against costs.

Summary

The gap I started this post with does not mean that most of these papers are wrong. In most cases they are not: They report results that were really obtained in the setup they describe, and it is mostly the abstract and the introduction, not the experiment section, that make those results sound like general findings. That is also why I don’t think one should stop reading papers, as the tweet I started with suggests, but rather stop reading their claims as settled answers, and instead confirm them yourself or wait for the field as a whole to settle it.