Skip to main content Accessibility help
×
Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-22T20:51:29.990Z Has data issue: false hasContentIssue false

13 - The Importance of Replication

from Part II - Important Methodological Considerations

Published online by Cambridge University Press:  12 December 2024

John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

This chapter offers a broad review of why replication is important to science by considering all aspects of the construct from definition to publication. The chapter introduces critical considerations about how to discuss replication given the complexity of its meaning as well as the challenges in conducting and interpreting it. Additionally, the chapter describes why replications are critical for any single construct as well as their contribution to generalizability across scientific disciplines. By conducting high-quality replications before and after effects are published, researchers can remain confident in their contributions to science.

Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, S. F. (2020). Misinterpreting p: The discrepancy between p values and the probability the null hypothesis is true, the influence of multiple testing, and implications for the replication crisis. Psychological Methods, 25(5), 596609. https://doi.org/10.1037/met0000248CrossRefGoogle ScholarPubMed
Anderson, S. F., & Maxwell, S. E. (2016). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods, 21(1), 112. https://doi.org/10.1037/met0000051CrossRefGoogle ScholarPubMed
Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452454. https://philpapers.org/rec/BAKSL-2CrossRefGoogle ScholarPubMed
Bakker, M., Van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543554. https://doi.org/10.1177/1745691612459060CrossRefGoogle ScholarPubMed
Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531533. https://doi.org/10.1038/483531aCrossRefGoogle ScholarPubMed
BITSS (Berkeley Initiative for Transparency in the Social Sciences). (2019, October 12). Resource Library. www.bitss.org/resource-library (retrieved December 16, 2022).Google Scholar
Bohannon, J. (2014). Psychology. Replication effort provokes praise – and “bullying” charges. Science, 344(6186), 788789.CrossRefGoogle ScholarPubMed
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., et al. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217224. https://doi.org/10.1016/j.jesp.2013.10.005CrossRefGoogle Scholar
Crocker, J., & Cooper, M. L. (2011). Addressing scientific fraud. Science, 334(6060), 1182. https://doi.org/10.1126/science.1216775CrossRefGoogle ScholarPubMed
Cuccolo, K., Irgens, M. S., Zlokovich, M. S., Grahe, J., & Edlund, J. E. (2021). What crowdsourcing can offer to cross-cultural psychological science. Cross-Cultural Research, 55(1), 328. https://doi.org/10.1177/1069397120950628CrossRefGoogle Scholar
Cumming, G. (2013). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. Routledge.CrossRefGoogle Scholar
Dafoe, A. (2014). Science deserves better: The imperative to share complete replication files. PS: Political Science & Politics, 47(1), 6066. https://doi.org/10.1017/S104909651300173XGoogle Scholar
Easley, R. W., Madden, C. S., & Gray, V. (2013). A tale of two cultures: Revisiting journal editors’ views of replication research. Journal of Business Research, 66(9), 14571459. https://doi.org/10.1016/j.jbusres.2012.05.013CrossRefGoogle Scholar
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., & Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 6882. https://doi.org/10.1016/j.jesp.2015.10.012CrossRefGoogle Scholar
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D. J., Buttrick, N. R., Chartier, C. R., et al. (2020). Many Labs 5: Testing pre-data-collection peer review as an intervention to increase replicability. Advances in Methods and Practices in Psychological Science, 3(3), 309331.CrossRefGoogle Scholar
Edlund, J. E., Cuccolo, K., Irgens, M. S., Wagge, J. R., & Zlokovich, M. S. (2022). Science through replication studies. Perspectives on Psychological Science, 17(1), 216225. https://doi.org/10.1177/1745691620984385CrossRefGoogle ScholarPubMed
Eitzel, M. V., Cappadonna, J. L., Santos-Lang, C., Duerr, R. E., Virapongse, A., West, S. E., et al. (2017). Citizen science terminology matters: Exploring key terms. Citizen Science: Theory and Practice, 2(1), 2. http://doi.org/10.5334/cstp.96Google Scholar
Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555561. https://doi.org/10.1177/1745691612459059CrossRefGoogle Scholar
Field, S. M., Hoekstra, R., Bringmann, L., & van Ravenzwaaij, D. (2019). When and why to replicate: As easy as 1, 2, 3? Collabra: Psychology, 5(1), 46. https://doi.org/10.1525/collabra.218CrossRefGoogle Scholar
Forstmeier, W., Wagenmakers, E.-J., & Parker, T. H. (2017). Detecting and avoiding likely false‐positive findings – a practical guide. Biological Reviews, 92(4), 19411968. https://doi.org/10.1111/brv.12315CrossRefGoogle ScholarPubMed
Gernsbacher, M. A. (2018). Writing empirical articles: Transparency, reproducibility, clarity, and memorability. Advances in Methods and Practices in Psychological Science, 1(3), 403414. https://doi.org/10.1177/2515245918754485CrossRefGoogle ScholarPubMed
Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7(6), 562571. https://doi.org/10.1177/1745691612457576CrossRefGoogle ScholarPubMed
Grahe, J. E., & Cuccolo, K. (2023, May 19). Replications of Willow in color. https://osf.io/deu8yGoogle Scholar
Grahe, J. E., Williams, K. D., & Hinsz, V. B. (2000). Teaching experimental methods while bringing smiles to your students’ faces. Teaching of Psychology, 27(2), 108111. https://doi.org/10.1207/S15328023TOP2702_06CrossRefGoogle Scholar
Hardwicke, T. E., Tessler, M. H., Peloquin, B. N., & Frank, M. C. (2018). A Bayesian decision-making framework for replication. Behavioral and Brain Sciences, 41, e132.CrossRefGoogle ScholarPubMed
Hedges, L. V. (2019). The statistics of replication. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 15(S1), 314. https://doi.org/10.1027/1614-2241/a000173CrossRefGoogle Scholar
Hedges, L. V., & Schauer, J. M. (2019). Statistical analyses for studying replication: Meta-analytic perspectives. Psychological Methods, 24(5), 557570. https://doi.org/10.1037/met0000189CrossRefGoogle ScholarPubMed
Hendriks, F., Kienhues, D., & Bromme, R. (2020). Replication crisis = trust crisis? The effect of successful vs failed replications on laypeople’s trust in researchers and research. Public Understanding of Science, 29(3), 270288. https://doi.org/10.1177/0963662520902383CrossRefGoogle ScholarPubMed
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466(7302), 29. https://doi.org/10.1038/466029aCrossRefGoogle Scholar
Hesse, B. W. (2018). Can psychology walk the walk of open science? American Psychologist, 73(2), 126. https://doi.org/10.1037/amp0000197CrossRefGoogle ScholarPubMed
Hinsz, V. B., & Tomhave, J. A. (1991). Smile and (half) the world smiles with you, frown and you frown alone. Personality and Social Psychology Bulletin, 17(5), 586592. https://doi.org/10.1177/0146167291175014CrossRefGoogle Scholar
Honey-Rosés, J., Anguelovski, I., Chireh, V. K., Daher, C., Konijnendijk van den Bosch, C., Litt, J. S., et al. (2021). The impact of COVID-19 on public space: An early review of the emerging questions – design, perceptions and inequities. Cities & Health, 5(S1), S263S279. https://doi.org/10.1080/23748834.2020.1780074CrossRefGoogle Scholar
Ioannidis, J. P. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124CrossRefGoogle ScholarPubMed
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018). How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation. Nature Biomedical Engineering, 2, 797809. https://doi.org/10.1038/s41551-018-0314-yCrossRefGoogle ScholarPubMed
Islam, M. R. (2018). Sample size and its role in Central Limit Theorem (CLT). International Journal of Physics and Mathematics, 1(1), 3747. https://doi.org/10.31295/ijpm.v1n1.42Google Scholar
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524532. https://doi.org/10.1177/0956797611430953CrossRefGoogle ScholarPubMed
Kass-Hout, T. A., Xu, Z., Mohebbi, M., Nelsen, H., Baker, A., Levine, J., et al. (2016). OpenFDA: An innovative platform providing access to a wealth of FDA’s publicly available data. Journal of the American Medical Informatics Association, 23(3), 596600. https://doi.org/10.1093/jamia/ocv153CrossRefGoogle ScholarPubMed
Kelly, A. E., & Cuccolo, K. (2022). Supporting college students during times of transition: Pedagogical recommendations based on pandemic learning data. College Teaching, 72(1), 1527. https://doi.org/10.1080/87567555.2022.2071825CrossRefGoogle Scholar
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196217. https://doi.org/10.1207/s15327957pspr0203_4CrossRefGoogle ScholarPubMed
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., et al. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456CrossRefGoogle ScholarPubMed
Klein, R. A., Cook, C. L., Ebersole, C. R., Vitiello, C., Nosek, B. A., Hilgard, J., et al. (2022). Many Labs 4: Failure to replicate mortality salience effect with and without original author involvement. Collabra: Psychology, 8(1), 35271. https://doi.org/10.1525/collabra.35271CrossRefGoogle Scholar
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R.B. Jr, Bahník, Š., Bernstein, M. J., et al. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142152. https://doi.org/10.1027/1864-9335/a000178CrossRefGoogle Scholar
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B. Jr, Alper, S., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443490. https://doi.org/10.1177/2515245918810225CrossRefGoogle Scholar
Kraus, M. W. (2013). Look everyone: A social priming finding with direct replications! Psych Your Mind [blog], November 6. https://psych-your-mind.blogspot.com/2013/11/look-everyone-social-priming-finding.htmlGoogle Scholar
Kraus, M. W. (2015). Americans still overestimate social class mobility: A pre-registered self-replication. Frontiers in Psychology, 6, 1709. https://doi.org/10.3389/fpsyg.2015.01709CrossRefGoogle ScholarPubMed
Kraus, M. W., & Tan, J. J. (2015). Americans overestimate social class mobility. Journal of Experimental Social Psychology, 58, 101111. https://doi.org/10.1016/j.jesp.2015.01.005CrossRefGoogle Scholar
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863CrossRefGoogle Scholar
Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259269. https://doi.org/10.1177/2515245918770963CrossRefGoogle Scholar
Lehrer, J. (2011). Trials and errors: Why science is failing us. Wired, December 16. www.wired.com/2011/12/ff-causationGoogle Scholar
Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584585. www.science.org/doi/full/10.1126/science.aal3618CrossRefGoogle ScholarPubMed
Moshontz, H., Campbell, L., Ebersole, C. R., IJzerman, H., Urry, H. L., Forscher, P. S., et al. (2018). The Psychological Science Accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science, 1(4), 501515. https://doi.org/10.1177/2515245918797607CrossRefGoogle ScholarPubMed
Napoli, P. M., & Karaganis, J. (2010). On making public policy with publicly available data: The case of US communications policymaking. Government Information Quarterly, 27(4), 384391. https://doi.org/10.1016/j.giq.2010.06.005CrossRefGoogle Scholar
Neuliep, J. W. (1990). Editorial bias against replication research. Journal of Social Behavior and Personality, 5(4), 8590.Google Scholar
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., et al. (2015). Scientific standards: Promoting an open research culture. Science, 348(6242), 14221425. https://doi.org/10.1126/science.aab2374CrossRefGoogle ScholarPubMed
Nosek, B. A., & Bar-Anan, Y. (2012). Scientific Utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217243. https://doi.org/10.1080/1047840X.2012.692215CrossRefGoogle Scholar
Nosek, B. A., & Errington, T. M. (2020). What is replication? PLOS Biology, 18(3), e3000691. https://doi.org/10.1371/journal.pbio.3000691CrossRefGoogle ScholarPubMed
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615631.CrossRefGoogle ScholarPubMed
Open Science Collaboration. (2015). Psychology. Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.CrossRefGoogle Scholar
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528530. https://doi.org/10.1177/1745691612465253CrossRefGoogle Scholar
Patterson, M. L. (2008). Back to social behavior: Mining the mundane. Basic and Applied Social Psychology, 30(2), 93101. https://doi.org/10.1080/01973530802208816CrossRefGoogle Scholar
Patterson, M. L., & Tubbs, M. E. (2005). Through a glass darkly: Effects of smiling and visibility on recognition and avoidance in passing encounters. Western Journal of Communication, 69(3), 219231. https://doi.org/10.1080/10570310500202389CrossRefGoogle Scholar
Popper, K. R. (1963). Science as falsification. Conjectures and Refutations, 1, 3339.Google Scholar
Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10 (712). https://doi.org/10.1038/nrd3439-c1CrossRefGoogle ScholarPubMed
Rahman, M. K., Gazi, M. A. I., Bhuiyan, M. A., & Rahaman, M. A. (2021). Effect of Covid-19 pandemic on tourist travel risk and management perceptions. PLOS ONE, 16(9), e0256486. https://doi.org/10.1371/journal.pone.0256486CrossRefGoogle ScholarPubMed
Rohlfing, I., & Zuber, C. I. (2021). Check your truth conditions! Clarifying the relationship between theories of causation and social science methods for causal inference. Sociological Methods & Research, 50(4), 16231659. https://doi.org/10.1177/0049124119826156CrossRefGoogle Scholar
Rosenthal, R. (1979). The File Drawer Problem and tolerance for null results. Psychological Bulletin, 86(3), 638641. https://doi.org/10.1037/0033-2909.86.3.638CrossRefGoogle Scholar
Rouse, S. V. (2017). The red badge of research (and the yellow, blue, and green badges, too). Psi Chi Journal of Psychological Research, 22(1), 29.CrossRefGoogle Scholar
Savage, C. J., & Vickers, A. J. (2009). Empirical study of data sharing by authors publishing in PLOS journals. PLOS ONE, 4(9), e7078. https://doi.org/10.1371/journal.pone.0007078CrossRefGoogle ScholarPubMed
Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90100. https://doi.org/10.1037/a0015108CrossRefGoogle Scholar
Schooler, J. (2011). Unpublished results hide the decline effect. Nature, 470(7335), 437438. https://doi.org/10.1038/470437aCrossRefGoogle ScholarPubMed
School Spirit Study Group, Sandra, A. H., Rowatt, T., Brooks, L., Magid, V., Stage, R., et al. (2004). Measuring school spirit: A national teaching exercise: The School Spirit Study Group. Teaching of Psychology, 31(1), 1821. https://doi.org/10.1207/s15328023top3101_5Google Scholar
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 13591366. https://doi.org/10.1177/0956797611417632CrossRefGoogle ScholarPubMed
Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 7680. https://doi.org/10.1177/1745691613514755CrossRefGoogle ScholarPubMed
Simons, D. J., Holcombe, A. O., & Spellman, B. A. (2014). An introduction to registered replication reports at Perspectives on Psychological Science. Perspectives on Psychological Science, 9(5), 552555. https://doi.org/10.1177/1745691614543974CrossRefGoogle ScholarPubMed
Smith, R. J. (2018). The continuing misuse of null hypothesis significance testing in biological anthropology. American Journal of Physical Anthropology, 166(1), 236245. https://doi.org/10.1002/ajpa.23399CrossRefGoogle ScholarPubMed
Spellman, B. A. (2012). Introduction to the special section on research practices. Perspectives on Psychological Science, 7(6), 655656. https://doi.org/10.1177/1745691612465075CrossRefGoogle Scholar
Świątkowski, W., & Dompnier, B. (2017). Replicability crisis in social psychology: Looking at the past to find new pathways for the future. International Review of Social Psychology, 30(1), 111124. https://doi.org/10.5334/irsp.66CrossRefGoogle Scholar
Uhlmann, E. L., Ebersole, C. R., Chartier, C. R., Errington, T. M., Kidwell, M. C., Lai, C. K., et al. (2019). Scientific Utopia: III. Crowdsourcing science. Perspectives on Psychological Science, 14(5), 711733. https://doi.org/10.1177/1745691619850561CrossRefGoogle ScholarPubMed
Vindrola-Padros, C., Andrews, L., Dowrick, A., Djellouli, N., Fillmore, H., Gonzalez, E. B., et al. (2020). Perceptions and experiences of healthcare workers during the COVID-19 pandemic in the UK. BMJ Open, 10(11), e040503. http://dx.doi.org/10.1136/bmjopen-2020-040503CrossRefGoogle ScholarPubMed
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B. Jr, et al. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917928. https://doi.org/10.1177/1745691616674458CrossRefGoogle Scholar
Wagge, J. R., Brandt, M. J., Lazarevic, L. B., Legate, N., Christopherson, C., Wiggins, B., & Grahe, J. E. (2019). Publishing research with undergraduate students via replication work: The collaborative replications and education project. Frontiers in Psychology, 10, 247. https://doi.org/10.3389/fpsyg.2019.00247CrossRefGoogle Scholar
Wallach, J. D., Boyack, K. W., & Ioannidis, J. P. (2018). Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLOS Biology, 16(11), e2006930. https://doi.org/10.1371/journal.pbio.2006930CrossRefGoogle ScholarPubMed
Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in psychology, 7. https://doi.org/10.3389/fpsyg.2016.01832CrossRefGoogle ScholarPubMed
Wiggins, B. J., & Christopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202217. https://doi.org/10.1037/teo0000137CrossRefGoogle Scholar
Wingen, T., Berkessel, J. B., & Englich, B. (2020). No replication, no trust? How low replicability influences trust in psychology. Social Psychological and Personality Science, 11(4), 454463. https://doi.org/10.1177/1948550619877412CrossRefGoogle Scholar
Zajonc, R. B., Murphy, S. T., & Inglehart, M. (1989). Feeling and facial efference: Implications of the vascular theory of emotion. Psychological Review, 96(3), 395416. https://doi.org/10.1037/0033-295X.96.3.395CrossRefGoogle ScholarPubMed
Zhang, X., Astivia, O. L. O., Kroc, E., & Zumbo, B. D. (2022). How to think clearly about the Central Limit Theorem. Psychological Methods, 28(6), 14271445. https://doi.org/10.1037/met0000448CrossRefGoogle ScholarPubMed
Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, 3120. https://doi.org/10.1017/S0140525X17001972CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×