Most behavioral science findings that were published as robust in the 2000s do not reproduce at their original size. That is my thesis, and the large replication projects of the last decade support it. The more useful question is not whether the field has a problem, but which findings survived, which shrank, and what separates the two groups. I will argue that the answer is mostly about design and statistics, not about the topic of a study, and that a reader can apply the same tests the replication teams used.
The Paradigm Clash: Discovery Science Versus Confirmation Science
The field grew up on a model of discovery. A researcher finds a surprising effect in a small sample, publishes it, and other labs build on it. Within that model, a failed replication is treated as noise, a change in population, or a sign that the original was subtly different. Replicators work from the opposite assumption: a finding is only as strong as its ability to reappear under a close copy of the original procedure, run by people who did not invent it.
Neither camp is wrong about everything. Exploratory work is necessary. The problem starts when exploratory results are read as confirmed effects, and when journals reward the first clean result rather than the tenth. The replication crisis is the moment the community noticed that the second model had been missing for a long time.
The Numbers: What the Large Projects Found
The first systematic test was the Open Science Collaboration’s replication of 100 studies from three psychology journals. The result was blunt. Ninety-seven percent of the original studies reported a statistically significant effect, but only 36% of the replications did, and the average replication effect was about half the size of the original (Open Science Collaboration, 2015).
Economics went through the same exercise. Camerer and colleagues replicated 18 laboratory experiments published in Science and Nature and found that 11 of them, or 61%, replicated. The average effect in the replications was about 66% of the original (Camerer et al., 2016). Two years later, the same group looked at 21 social science experiments from the same two journals. Thirteen replicated, or 62%, and the effects were about half the original size (Camerer et al., 2018).
| Project | Field | Studies tested | Replicated | Effect relative to original |
|---|---|---|---|---|
| Open Science Collaboration (2015) | Psychology | 100 | 36% significant | About half |
| Camerer et al. (2016) | Experimental economics | 18 | 11 (61%) | About 66% |
| Camerer et al. (2018) | Social science (Nature/Science) | 21 | 13 (62%) | About 50% |
The pattern across these projects is consistent. Effects are rarely zero, but they are almost always smaller than the first report. That matters for behavioral science because many applied claims were built on the original magnitudes, not on what a replication would predict.
Case Study One: Ego Depletion
Ego depletion, the idea that self-control draws on a limited resource that runs down after an effortful task, was one of the field’s most cited ideas. I wrote about it in 2007, when it was still a live research program, in my early post on ego depletion and executive functioning. That post explained the theory as a plausible account of how executive control might work. Reading it now is a useful exercise in how a plausible mechanism can be presented with more confidence than its evidence supports.
A registered multi-lab test changed the picture. Hagger and colleagues ran a preregistered replication across 23 laboratories with 2,141 participants and found an effect of d = 0.04, with a confidence interval from -0.07 to 0.15 (Hagger et al., 2016). An effect that small is indistinguishable from zero at this sample size. The original researchers and commentators raised a fair objection: the depleting task may have been too weak to produce a depletion effect in the first place. I think that objection deserves weight. It also illustrates the problem. A theory that survives only when the manipulation is calibrated after the fact is hard to falsify, and a theory that is hard to falsify is not yet a useful basis for an intervention.
A later multi-lab project with 36 labs and 3,531 participants reported d = 0.06 and a Bayes factor of roughly 4 in favor of the null hypothesis (Vohs et al., 2021). A Bayes factor of 4 is weak evidence, but it points in the same direction as the other projects.
Case Study Two: Social Priming and the Retracted Signing Study
Social priming, the claim that subtle cues such as words or images change behavior outside awareness, is the second case. I covered one of the classic examples on this site in a 2010 post on unconscious goal priming. The studies were striking, which was part of their appeal. Striking results are also the ones most likely to be false positives under the conditions described in the next section.
The clearest documented case in behavioral science is the 2012 PNAS paper by Shu and colleagues, which reported that people who signed an honesty declaration at the top of a form reported more truthfully than people who signed at the bottom. The paper was retracted in 2021 (Shu et al., 2012; retraction notice, PNAS, 2021). I will describe only what the retraction notice states. The retraction matters for this discussion because the effect was widely taught, cited in policy discussions, and used as a template for later work. Any applied claim that depends on a single striking study with an unusually large effect should carry that history in mind.
Why It Happened: Researcher Degrees of Freedom
Simmons, Nelson and Simonsohn (2011) gave the field its most useful diagnosis. They showed that flexibility in data collection and analysis, such as deciding when to stop collecting data, which covariates to include, or which outcome to report, can push the false-positive rate far above the nominal 5%. Their paper, “False-Positive Psychology,” did not accuse anyone of fraud. It showed that honest researchers working in good faith could produce significant results from noise.
This is why the replication problem is not mainly about bad people. It is about incentives and about the statistical habits that those incentives reward. I would put the fix in three places:
- Preregistration: writing down the hypothesis, sample size and analysis before data collection.
- Power: designing studies to detect the effect sizes that matter for practice, not the effect sizes that happened to be significant in a pilot.
- Publishing the null: treating a well-designed null result as a contribution rather than a failure.
What Survived
A replication crisis narrative can slide into the claim that nothing in behavioral science is real. That is not what the evidence says, and it is a mistake I want to avoid.
Several findings have held up across many labs, samples and settings. Defaults shape choices in a way that is large enough to matter in practice; the effect of default enrollment on retirement plan participation is the best-known example, and I discuss the applied side in the nudge theory guide. The anchoring effect in judgments under uncertainty has a long replication record. Loss aversion, the tendency to weigh losses more than equivalent gains, has been reproduced many times, although its size varies a lot across contexts. I would not promise a consistent coefficient for it. The lesson I draw from the loss aversion guide is that the direction of an effect is often more reliable than its magnitude.
The surviving findings share features that the failed ones lacked. They have large samples, clear manipulations, outcomes that are measured directly rather than through a self-report that is easy to bend, and a theory that predicts when the effect should disappear. Behavioral science does better when it treats a finding as a claim about a boundary condition, not as a universal law.
How to Read a Behavioral Science Claim
The following checklist is what I use before I rely on a finding in client work. It is short on purpose.
| Question | What a strong answer looks like | Warning sign |
|---|---|---|
| Was the study preregistered? | Yes, with a public protocol | No protocol, or a protocol written after the data |
| How large was the sample? | Powered for the effect that matters | Small groups, large claimed effects |
| How big is the effect? | Reported with a confidence interval | Only “significant” is reported |
| Has it been replicated independently? | By labs that did not invent it | Only by the original group |
| Is the outcome measured directly? | Behavior or objective records | Self-report alone for a behavioral claim |
| What is the mechanism? | Stated and testable | Story without a falsifiable prediction |
A finding that passes most of these checks is worth testing in your own setting. A finding that fails several is a hypothesis, not a basis for a product decision.
In My View
I think the replication crisis is best understood as a correction, not a collapse. The field did what science does when its methods are tested against reality, and it found that some of its earlier claims were too large. In my view, the most important shift is not a list of banned findings but a change in what counts as evidence. A single striking study should now prompt a replication question, not a press release.
Where I disagree with some critics is on the value of the applied programs themselves. Choice architecture and default design rest on a smaller set of findings that have been tested repeatedly, and they can be evaluated with ordinary tools: a controlled trial, a clear outcome, and a stopping rule. That is a more modest claim than the one many 2010 books made, and it is a more defensible one.
The forward horizon is clear. Organizations that build on behavioral science need their own replication habits. Before rolling out a nudge at scale, a team should run a preregistered pilot with a pre-specified effect size, publish the result whether or not it works, and check whether the effect holds in a second population. That is more work than reading a summary of a famous study. It is also the only way to know what a finding will do in your organization.
Frequently Asked Questions
Which behavioral science findings replicate?
The most reliable results are those with large, preregistered replications and clear manipulations, such as default effects and anchoring. Many striking single-study results, including several in social priming and ego depletion, have replicated weakly or not at all. The exact proportion depends on the field and the project; the multi-lab studies above give the numbers.
Is behavioral economics in trouble because of the replication crisis?
Partly. The economics replications were better than the psychology ones, with about 61% of experiments replicating in the 2016 study, but effect sizes were still smaller than originally reported. The field is not in trouble in the sense of having no findings; it is in trouble if it treats each striking result as settled.
Does a retraction mean the whole topic is false?
No. A retraction tells you that a specific paper’s results cannot be relied upon as reported. It does not settle whether the broader idea is true. The honest reading is narrower: that particular evidence is gone, and any claim that depended on it needs other support.
How can a practitioner check a behavioral science claim quickly?
Look for preregistration, a confidence interval on the effect size, an independent replication, and a direct outcome measure. If a claim fails most of these, run a small controlled test before you commit budget to it.
References
- Camerer, C. F., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280). https://doi.org/10.1126/science.aaf0918
- Camerer, C. F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637-644. https://doi.org/10.1038/s41562-018-0399-z
- Hagger, M. S., et al. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4). https://doi.org/10.1177/1745691616652873
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716
- Shu, L. L., et al. (2012). Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end. PNAS, 109(38). https://doi.org/10.1073/pnas.1209746109 (retracted 2021; retraction notice: https://doi.org/10.1073/pnas.2115397118)
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11). https://doi.org/10.1177/0956797611417632
- Vohs, K. D., et al. (2021). A multisite preregistered paradigmatic test of the ego-depletion effect. Psychological Science, 32(10). https://doi.org/10.1177/0956797621989733