Priming talks, the kind where a speaker's warmth or confidence is rated from a short video clip, keep stalling in replication. The reason is procedural: the human raters doing the coding are not blind to the hypothesis, and their guesses drive the numbers. This piece breaks down the choices that produce that result and what to change.

The Raters Who Guess the Hypothesis

Most priming studies in social and behavioural science depend on human coders. A researcher records short clips of participants giving a talk, then asks raters to score each clip for warmth, competence, or anxiety. The raters are usually undergraduate research assistants, not naive observers, and they see the same clips the original authors saw.

Those raters infer the study's aim from context cues: the consent form, the lab's reputation, the order of clips. This is a demand characteristic, a well-documented tendency for research participants to behave in ways that confirm what they think the experimenter wants. When raters guess the hypothesis, their ratings shift toward the predicted direction.

The contested claim is simple: raters are not blind. A 2016 survey of social psychology coding practices found that fewer than half of published studies reported any blinding procedure for raters. That gap is not a minor oversight; it is a direct pathway for experimenter expectation to enter the data.

What the Coders Actually See

Raters typically watch 30-second to two-minute clips of a speaker. They see facial expressions, posture, hand gestures, and vocal pitch. They do not see the priming manipulation itself, which happened before the camera rolled. So they rely on residual cues: a slight smile, a forward lean, a pause before answering.

These cues are noisy. A speaker who is nervous about being recorded looks much the same as one who was primed to feel powerful. The rater's job is to map that noise onto a scale. Prior expectations fill the gaps. If the rater believes the study is about power priming, a firm handshake reads as confidence; if the rater believes it is about rejection, the same handshake reads as defensiveness.

Training sessions can reduce this inference, but they also teach raters what the hypothesis is. The standard fix, keeping raters unaware of condition, is rarely implemented because it requires stripping identifying details from clips and running separate coding teams for each condition. That is expensive and slow.

Evidence from Replication Attempts

Many classic priming effects have failed to replicate in preregistered trials. The most cited examples come from social priming, where subtle cues like holding a warm cup are said to change behaviour. Large-scale replications have found effect sizes near zero, with confidence intervals that exclude the original estimate.

Rater guessing predicts effect size. When coders in a replication are asked to guess the hypothesis, the studies with the highest guess rates show the largest attenuation. This pattern appears across several independent replication projects, though the exact proportion varies by subfield. Roughly half of published priming effects attenuate by more than 50 percent in preregistered trials.

A related piece on this site, Optogenetics Papers Depend on a Virus Batch Nobody Standardised, makes a parallel point about hidden procedural variation. The mechanism differs, but the lesson is the same: when a key step is unstandardised, replication becomes a coin flip.

Procedural Choices That Matter

Blinding raters to condition is the single most effective step. This means removing any clip features that reveal the manipulation: no lab logos, no condition-specific prompts, no metadata in the filename. Raters should also be unaware of the study's hypothesis, which is harder than it sounds because lab culture leaks.

Multiple raters per clip help. If three raters score the same clip and their scores diverge, that divergence is a signal. Inter-rater reliability, often measured with intraclass correlation, should be reported alongside the main effect. Low reliability means the effect is riding on rater noise.

Training sessions reduce inference but can also teach the hypothesis. A better approach is to train raters on a separate set of clips from a different study, then move them to the target clips without explanation. Preregistering coding rules and hypotheses locks in the analysis before data collection, which prevents post hoc tweaking.

Why Reasonable Scientists Disagree

Some researchers see priming as a robust phenomenon with decades of evidence. They point to meta-analyses that show small but reliable effects, and they argue that replication failures often come from subtle procedural differences, not from a fundamental flaw.

Others cite publication bias. The file drawer problem means null results rarely get published, so the literature overrepresents positive findings. When you correct for that bias, the average effect shrinks. The debate over ecological validity matters too: lab priming tasks are artificial, so a null result in the lab may not reflect real-world behaviour.

There is no consensus on rater awareness. Some labs report that raters rarely guess the hypothesis; others find guess rates above 60 percent. The disagreement is partly methodological, partly cultural. Labs that routinely measure guessing find more of it, which suggests the problem is underreported rather than absent.

Practical Steps for Cleaner Priming Studies

Run a rater-blind pilot before the main study. Strip all condition-identifying details from a small set of clips, have raters score them, and check whether their ratings differ by condition. If they do, your blinding is incomplete.

Check rater guess rates. After coding, ask each rater to guess the hypothesis. A guess rate above chance means your effect may be contaminated. Report that number in the paper, even if it is uncomfortable.

Report inter-rater reliability for every coded variable. Use intraclass correlation or Krippendorff's alpha, and include confidence intervals. If reliability is low, the effect size is not trustworthy.

Preregister coding rules and analysis. Specify how clips will be trimmed, how raters will be trained, and what counts as a guess. This site has argued in Reproducibility Checks Stall Where Code Repositories Stop Accepting Deposits that infrastructure gaps quietly kill replication. The same holds for coding protocols.

Share clips and codebooks. Even anonymised clips and the exact rating instructions let other labs attempt a direct replication. Without them, every new study is a fresh guess about what the original team actually did.

Designing a Rater-Blind Coding Protocol

A rater-blind protocol is not just about hiding condition labels. It means designing the entire coding workflow so that no rater can infer the hypothesis from any part of the procedure. Start with the clips themselves. Record all participants against a neutral background, with identical lighting and camera angle. Avoid clothing or objects that might correlate with condition. If the priming manipulation involves, say, a writing task, ensure that the task materials are not visible in the clip. After recording, assign each clip a random identifier that carries no information about condition. Store the key linking identifiers to condition in a separate file accessible only to the lead researcher.

When training raters, use a separate set of clips from a different study or from a pilot with no experimental manipulation. Train them on the rating scales and the concept definitions, but do not mention the hypothesis. If raters ask what the study is about, provide a cover story that is plausible but unrelated to the true aim. This is standard practice in survey research, but it is rare in behavioural coding.

During coding, present clips in a random order and ensure that raters cannot see which clips belong to the same participant. If multiple clips per participant are used, separate them across different coding sessions. Also, prevent raters from discussing the clips with each other. Independent coding is essential; if raters compare notes, they may converge on a shared guess about the hypothesis.

After coding, collect guess data. Ask each rater to write down what they think the study was testing. Code these guesses for accuracy. If the guess rate is high, you can statistically control for guessing by including it as a covariate, but that is a weak fix. Better to re-train or re-design.

The Trade-off Between Blinding and Ecological Validity

Blinding raters often requires stripping away contextual details that make the clips more natural. For example, if you remove all background cues, the speaker may appear in a sterile environment that feels artificial. This can reduce ecological validity—the extent to which the findings generalise to real-world settings. But the alternative, leaving cues in, invites bias. The trade-off is real, and there is no perfect solution. Researchers must decide which is more important for their specific question. If the goal is to test a theoretical mechanism, internal validity matters more; if the goal is to predict behaviour in a specific setting, some ecological validity may be worth the risk of bias.

One compromise is to run two versions of the study: a tightly blinded lab version and a more naturalistic version with additional controls. If the effects converge, confidence increases. If they diverge, the difference itself is informative. This approach is costly, but it is more honest than pretending that a single study can achieve both goals.

Reporting Standards and Transparency

Even the best protocol is useless if it is not reported clearly. Journals should require authors to state whether raters were blind to condition and hypothesis, how blinding was achieved, and what steps were taken to prevent inference. If blinding was not possible, authors should explain why and discuss the potential for bias. Similarly, inter-rater reliability should be reported for every coded variable, not just the primary outcome. Confidence intervals for reliability coefficients are also important, as they indicate the precision of the estimate.

Preregistration helps, but it is not a panacea. A preregistered study can still have unblinded raters. The key is to preregister the blinding procedure itself. Specify exactly how clips will be prepared, how raters will be recruited and trained, and how guess data will be collected. Then, when the study is published, readers can judge whether the protocol was adequate. Some journals now offer badges for open data and open materials, but badges for blinding are rare. That needs to change.

Finally, share the codebook. The codebook contains the definitions of each rating scale, examples of high and low scores, and decision rules for ambiguous cases. Without it, other labs cannot replicate the coding. Even with it, replication is hard, but at least the target is clear. The replication crisis in priming is not just about statistics; it is about the mundane, unglamorous details of how data are produced. Fixing those details will not make headlines, but it will make the science more trustworthy.