Reproducibility Project

The Reproducibility Project is a series of crowdsourced collaborations aiming to reproduce published scientific studies, finding high rates of results which could not be replicated. It has resulted in two major initiatives focusing on the fields of psychology[1] and cancer biology.[2] The project has brought attention to the replication crisis, and has contributed to shifts in scientific culture and publishing practices to address it.[3]

The project was led by the Center for Open Science and its co-founder, Brian Nosek, who started the project in November 2011.[4]

Results

Brian Nosek of University of Virginia and colleagues sought out to replicate 100 different studies, all published in 2008.[5] The project pulled these studies from three different journals, Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition, published in 2008 to see if they could get the same results as the initial findings. 97 of the original studies had significant effects, but of those 97, only 36% of the replications yielded significant findings (p value below 0.05),[6] and the effects were often smaller than those in the original papers. The authors emphasized that the findings reflect a problem that affects all of science and not just psychology, and that there is room to improve reproducibility in psychology.

In 2021, the project showed that of 193 experiments from 53 top papers about cancer published between 2010 and 2012, only 50 experiments from 23 papers could be replicated. Moreover, it showed that the effect sizes of that fraction were 85% smaller on average than the original findings. None of the papers had its experimental protocols fully described and 70% of experiments required asking for key reagents.[7][8]

Impact

The project, along with broader action in response to the replication crisis, has helped spur changes in scientific culture and publishing practices.[3][4] The results of the Reproducibility Project might also affect public trust in psychology.[9][10] Lay people who learned about the low replication rate found in the Reproducibility Project subsequently reported a lower trust in psychology, compared to people who were told that a high number of the studies had replicated.[11][9]

See also

• Invalid science
• John Ioannidis
• Meta-analysis
• Metascience
• Proteus phenomenon
• Publication bias
• Replication crisis
• Scientific method

External links

• Official website for the Reproducibility Project: Psychology
• Official website for the Reproducibility Project: Cancer Biology

References

1. Yong, Ed (27 August 2015). "How Reliable Are Psychology Studies?". The Atlantic. Retrieved 7 November 2023.
2. Nelson, Bryn; Wiles, Austin (15 September 2022). "A troubling lack of replicability for cancer biology studies: After an ambitious project struggled to replicate high-profile studies, researchers are calling for a new focus on protocol and data sharing as essential steps for building confidence in the field". Cancer Cytopathology. 130 (9): 656–657. doi:10.1002/cncy.22639. ISSN 1934-662X. PMID 36108146.
3. Loken, Eric (8 April 2019). "The replication crisis is good for science". The Conversation. Retrieved 7 November 2023.
4. Apple, Sam (22 January 2017). "The Young Billionaire Behind the War on Bad Science". Wired.
5. Weir, Kristen. "A reproducibility crisis?". American Psychological Association. Retrieved 24 November 2016.
6. Open Science Collaboration (August 2015). "PSYCHOLOGY. Estimating the reproducibility of psychological science". Science. 349 (6251): aac4716. doi:10.1126/science.aac4716. hdl:10722/230596. PMID 26315443. S2CID 218065162.
7. "Dozens of major cancer studies can't be replicated". Science News. 7 December 2021. Retrieved 19 January 2022.
8. "Reproducibility Project: Cancer Biology". www.cos.io. Center for Open Science. Retrieved 19 January 2022.
9. Wingen, Tobias; Berkessel, Jana B.; Englich, Birte (24 October 2019). "No Replication, No Trust? How Low Replicability Influences Trust in Psychology". Social Psychological and Personality Science. 11 (4): 454–463. doi:10.1177/1948550619877412. ISSN 1948-5506. S2CID 210383335.
10. Anvari, Farid; Lakens, Daniël (19 November 2019). "The replicability crisis and public trust in psychological science". Comprehensive Results in Social Psychology. 3 (3): 266–286. doi:10.1080/23743603.2019.1684822. ISSN 2374-3603.
11. "The Replication Crisis Lowers The Public's Trust In Psychology — But Can That Trust Be Built Back Up?". Research Digest. 31 October 2019. Retrieved 30 November 2019.