Grant panels penalize null results before they exist. A survey experiment with grant reviewers found that hypothetical null-result scenarios were rated less innovative than equivalent positive findings, even when prior probabilities were identical. The study, a preprint by metascience researchers, does not prove that real panels behave this way. It does show that the bias is present in the judgments panels are asked to make.

Null Results as Hidden Currency

Null results are the negative space of the scientific record. They tell you which paths were walked and found empty. Yet they rarely appear in journals, and almost never in the highlight reels that funders use to justify past investments. Publication bias, the tendency for positive findings to be published more readily, has been documented for decades. The consequence is an evidence base that overrepresents success.

Funders have started asking applicants about prior nulls. Some grant applications now include a section for negative results, and a few agencies have piloted registries where nulls can be deposited. The logic is simple: if panels cannot see the nulls, they cannot weigh them. But asking for nulls is not the same as valuing them. A null result can cost a lab months of salary and consumables, and it produces no paper, no patent, and no press release.

The tension is structural. Panels are asked to pick projects that will produce something. A null is something, but it is not the something that career committees reward. This site has argued that Python scripts moved into economics departments and changed what counts as a result. The same question applies here: what counts as a result when the result is nothing?

Consider the case of a clinical trial that finds no difference between two treatments. The trial may have been well-powered, well-conducted, and ethically necessary. But its result is a shrug. The investigators may struggle to publish it, and the funder may struggle to justify the expense. The null is not nothing to the patients who avoided an ineffective treatment, but it is nothing to a promotion committee. That gap between clinical value and career value is where the bias lives.

Study of Pre-Outcome Judgments

The survey experiment was designed to isolate pre-outcome judgment. Researchers recruited panelists from major funders and asked them to rate vignettes describing hypothetical studies. Each vignette varied the prior probability of a positive finding and the proposed outcome. Some described a study likely to find an effect; others described a study likely to find nothing. The panelists rated each on innovation, feasibility, and overall priority.

The headline effect was moderate, around 0.4 standard deviations. In plain terms, a null-result proposal lost roughly 10–15% of its innovation score compared with a positive-result proposal with the same methods and same prior. The penalty was not explained by feasibility or by the perceived difficulty of the work. It appeared to be a direct discount on the value of the question.

Familiarity with null literature mattered. Panelists who reported reading meta-analyses or replication studies penalized nulls less. But the effect did not vanish entirely. Even the most informed reviewers showed a residual preference for positive findings. The study authors note that this is consistent with a cultural norm rather than a knowledge gap.

One might expect that panelists with a strong statistical background would be immune. They were not. The survey included a measure of statistical training, and it did not predict the size of the penalty. Knowing that a null is informative does not automatically translate into valuing it in a competitive review setting.

What the Survey Actually Measured

The vignettes varied prior probability in a controlled way. Some scenarios described a field where the effect was likely to exist; others described a field where prior evidence was weak. The null penalty held in both. That rules out a simple explanation: panelists were not just discounting nulls because they thought the question was already settled. They were discounting the null outcome itself.

No difference appeared by discipline. Reviewers from biomedical, physical, and social science backgrounds showed similar penalties. That is surprising, because biomedical research has been the focus of most replication debates. The authors suggest that the bias is a general feature of how panels evaluate proposals, not a quirk of one field.

The survey also asked panelists to estimate the likelihood that a null-result study would be published. The median estimate was below 50%. That expectation may feed the penalty. If reviewers believe a null will not be published, they may reasonably treat it as a less valuable use of funds. The bias becomes a self-fulfilling forecast.

Why Bias Persists in Review

Career incentives favor positive findings. Hiring committees, tenure boards, and prize committees all reward visible results. A null result is visible only to the people who know where to look. That asymmetry shapes what applicants propose. A young researcher who wants a job will not propose a study that is likely to find nothing, even if the question is important.

Panels also lack null-result track records. Most reviewers have never been asked to evaluate a portfolio of nulls. They have no calibrated sense of what a good null looks like. A well-powered null is different from a failed study, but the two can look identical on a one-page summary. That ambiguity makes nulls easy to discount.

Funding bias literature adds another layer. Studies sponsored by interested parties tend to support those interests, a pattern documented in pharmaceutical, chemical, and tobacco research. The mechanism is not always fraud. It can be subtle: sponsors fund questions that are likely to produce favorable answers. Nulls are less likely to be funded in the first place, which means they are less likely to be seen, which reinforces the belief that they are rare.

There is also the matter of portfolio balance. A funder that only backs positive-result projects may end up with a portfolio that looks impressive but is fragile. If half of those positive results fail to replicate, the funder's return on investment is lower than it appeared. Nulls, by contrast, are robust. They do not replicate away; they just sit there, quietly correcting the record. A funder that ignores them is accumulating hidden risk.

Limits of the Evidence

The survey is not a real funding decision. Vignettes are hypothetical, and panelists know they are being studied. The effect could be larger or smaller in a real panel with real money and real careers on the line. The authors are explicit about this limitation. They call for field experiments where panels are randomly assigned to see null-result information.

The sample skewed toward biomedical fields. That reflects the composition of major funders, but it means the findings may not generalize to physics, mathematics, or the humanities. The study also had no longitudinal follow-up. It cannot say whether the penalty affects actual funding rates or whether it fades when panelists discuss proposals together.

There is a deeper measurement problem. Innovation is a fuzzy construct. The survey asked panelists to rate it, but the authors do not claim to have measured innovation itself. They measured a judgment about innovation. That distinction matters for anyone who wants to use the result to change policy. A 10–15% penalty in a hypothetical rating is not the same as a 10–15% penalty in funded dollars.

Another limit is the lack of real stakes. In a real panel, reviewers may be more willing to fund a null if the question is important enough. Or they may be less willing, because the opportunity cost is higher. The survey cannot distinguish between these possibilities. It only shows that the bias exists in the abstract.

Practical Steps for Funders

Ask applicants to report prior nulls in a dedicated section. A short paragraph on what has already been tried and failed gives panels a more accurate picture of the evidence base. The section should be scored, not just collected.

Score methods regardless of outcome. Panels can separate the quality of the design from the expected direction of the result. A well-powered study that might find nothing is still a well-powered study.

Pilot a null-result registry. Several fields have preprint servers for negative findings, and a funder-run registry could make those results visible to future panels. The registry does not need to be comprehensive to be useful.

Train panels on publication bias. A short briefing on how nulls disappear from the literature can reduce the residual penalty. The survey suggests that familiarity helps, even if it does not eliminate the effect.

Track funded nulls over time. If a funder wants to know whether its policy works, it needs to follow the nulls it funds. That means counting them, reading them, and reporting them alongside positive findings. The accounting is the policy.