Hostname: page-component-586b7cd67f-2brh9 Total loading time: 0 Render date: 2024-11-25T10:57:47.886Z Has data issue: false hasContentIssue false
  • English
  • Français

Making replication mainstream

Published online by Cambridge University Press:  25 October 2017

Rolf A. Zwaan ,
Alexander Etz ,
Richard E. Lucas  and
M. Brent Donnellan
Rolf A. Zwaan
Affiliation:
Department of Psychology, Education, and Child Sciences, Erasmus University Rotterdam, 3000 DR Rotterdam, The Netherlands. [email protected]://www.eur.nl/essb/people/rolf-zwaan
Alexander Etz
Affiliation:
Department of Cognitive Sciences, University of California, Irvine, CA 92697-5100. [email protected]://alexanderetz.com/
Richard E. Lucas
Affiliation:
Department of Psychology, Michigan State University, East Lansing, MI 48824. [email protected]://www.msu.edu/user/lucasri/
M. Brent Donnellan
Affiliation:
Department of Psychology, Texas A&M University, College Station, TX 77843. [email protected]://psychology.msu.edu/people/faculty/donnel59
Get access Rights & Permissions [Opens in a new window]

Abstract

Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component of science. A finding is elevated from single observation to scientific evidence when the procedures that were used to obtain it can be reproduced and the finding itself can be replicated. Recent replication attempts show that some high profile results – most notably in psychology, but in many other disciplines as well – cannot be replicated consistently. These replication attempts have generated a considerable amount of controversy, and the issue of whether direct replications have value has, in particular, proven to be contentious. However, much of this discussion has occurred in published commentaries and social media outlets, resulting in a fragmented discourse. To address the need for an integrative summary, we review various types of replication studies and then discuss the most commonly voiced concerns about direct replication. We provide detailed responses to these concerns and consider different statistical ways to evaluate replications. We conclude there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science.

Keywords

psychological researchreplicationreproducibilityresearch programs
Type
Target Article
Information
Behavioral and Brain Sciences , Volume 41 , 2018 , e120
Copyright
Copyright © Cambridge University Press 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

1.

Current address: Department of Psychology, Psychology Building, 316 Physics Road, Room 249A, Michigan State University, East Lansing, MI 48824.

References

Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, Š., Birch, S., Birt, A. R., Bornstein, B. H., Bouwmeester, S., Brandimonte, M. A., Brown, C., Buswell, K., Carlson, C., Carlson, M., Chu, S., Cislak, A., Colarusso, M., Colloff, M. F., Dellapaolera, K. S., Delvenne, J.-F., Di Domenico, A., Drummond, A., Echterhoff, G., Edlund, J. E., Eggleston, C. M., Fairfield, B., Franco, G., Gabbert, F., Gamblin, B. W., Garry, M., Gentry, R., Gilbert, E. A., Greenberg, D. L., Halberstadt, J., Hall, L., Hancock, P. J. B., Hirsch, D., Holt, G., Jackson, J. C., Jong, J., Kehn, A., Koch, C., Kopietz, R., Körner, U., Kunar, M. A., Lai, C. K., Langton, S. R. H., Leite, F. P., Mammarella, N., Marsh, J. E., McConnaughy, K. A., McCoy, S., McIntyre, A. H., Meissner, C. A., Michael, R. B., Mitchell, A. A., Mugayar-Baldocchi, M., Musselman, R., Ng, C., Nichols, A. L., Nunez, N. L., Palmer, M. A., Pappagianopoulos, J. E., Petro, M. S., Poirier, C. R., Portch, E., Rainsford, M., Rancourt, A., Romig, C., Rubínová, E., Sanson, M., Satchell, L., Sauer, J. D., Schweitzer, K., Shaheed, J., Skelton, F., Sullivan, G. A., Susa, K. J., Swanner, J. K., Thompson, W. B., Todaro, R., Ulatowska, J., Valentine, T., Verkoijen, P. P. J. L., Vranka, M., Wade, K. A., Was, C. A., Weatherford, D., Wiseman, K., Zaksaite, T., Zuj, D. V. & Zwaan, R. A. (2014) Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science 9:556–78.Google Scholar
Anderson, C. J., Bahník, Š., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., Della Penna, N., Estel, V., Fedor, A., Fitneva, S. A., Frank, M. C., Grange, J. A., Hartshorne, J. K., Hasselman, F., Henninger, F., van der Hulst, M., Jonas, K. J., Lai, C. K., Levitan, C. A., Miller, J. K., Moore, K. S., Meixner, J. M., Munafò, M. R., Neijenhuijs, K. I., Nilsonne, G., Nosek, B. A., Plessow, F., Prenoveau, J. M., Ricker, A. A., Schmidt, K., Spies, J. R., Steiger, S., Strohminger, N., Sullivan, G. B., van Aert, R. C. M., van Assen, M. A. L. M., Vanpaemel, W., Vianello, M., Voracek, M. & Zuni, K. (2016) Response to comment on “estimating the reproducibility of psychological science.” Science 351(6277):1037.Google Scholar
Baker, M. (2016) Is there a reproducibility crisis? Nature 533:452–54.Google Scholar
Barsalou, L. W. (2016) Situated conceptualization offers a theoretical account of social priming. Current Opinion in Psychology 12:6–11.Google Scholar
Baumeister, R. F. (2016) Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology 66:153–58. Available at: http://doi.org/10.1016/j.jesp.2016.02.003.Google Scholar
Bem, D. J. (2003) Writing the empirical journal article. In: The compleat academic: A career guide, 2nd edition, ed . Darley, J. M., Zanna, M. P. & Roediger, H. L. III, pp . 185–219. American Psychological Association.Google Scholar
Bem, D. J. (2011) Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 100:407–25.Google Scholar
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. & van ‘t Veer, A. (2014) The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology 50:217–24.Google Scholar
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. & Munafò, M. R. (2013) Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14(5):365–76.Google Scholar
Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmedj, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M. & Wu, H. (2016) Evaluating replicability of laboratory experiments in economics. Science 351(6280):1433–36.Google Scholar
Cesario, J. (2014) Priming, replication, and the hardest science. Perspectives on Psychological Science 9:40–48.Google Scholar
Chambers, C. (2017) The 7 deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press.Google Scholar
Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R. A., Aykutoğlu, B., Bahník, Š., Bowen, J. D., Bredbow, C. A., Bromberg, C., Caprariello, P. A., Carcedo, R. J., Carson, K. J., Cobb, R. J., Collins, N. L., Corretti, C. A., DiDonato, T. E., Ellithorpe, C., Fenrnández-Rouco, N., Fuglestad, P. T., Goldberg, R. M., Golom, F. D., Gündoğdu-Aktürk, E., Hoplock, L. B., Houdek, P., Kane, H. S., Kim, J. S., Kraus, S., Leone, C. T., Li, N. P., Logan, J. M., Millman, R. D., Morry, M. M., Pink, J. C., Ritchey, T., Root Luna, L. M., Sinclair, H. C., Stinson, D. A., Sucharyna, T. A., Tidwell, N. D., Uysal, A., Vranka, M., Winczewski, L. A. & Yong, J. C. (2016) Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro & Hannon (2002). Perspectives on Psychological Science 11(5):750–64.Google Scholar
Cohen, J. (1990) Things I have learned (so far). American Psychologist 45(12):1304.Google Scholar
Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., Mcllor, W. & Oxman, A. D. (1993) Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 269(21):2749–53.Google Scholar
Coyne, J. C. (2016) Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology 4:28. Available at: http://doi.org/10.1186/s40359-016-0134-3.Google Scholar
Crandall, C. S. & Sherman, J. W. (2016) On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology 66:93–99. Available at: http://doi.org/10.1016/j.jesp.2015.10.002.Google Scholar
Dunlap, K. (1926) The experimental methods of psychology. In: Psychologies of 1925, ed . Murchison, C., pp . 331–53. Clark University Press.Google Scholar
Ebersole, C. R., Axt, J. R. & Nosek, B. A. (2016b) Scientists’ reputations are based on getting it right, not being right. PLoS Biology 14(5):e1002460. Available at: https://doi.org/10.1371/journal.pbio.1002460.Google Scholar
Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P., Berger, S. A., Birt, A. R., Capezza, N., Carlucci, M., Crocker, C., Ferretti, T. R., Kibbe, M. R., Knepp, M. M., Kurby, C. A., Melcher, J. M., Michael, S. W., Poirier, C. & Prenoveau, J. M. (2016) Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science 11:158–71.Google Scholar
Errington, T. M., Iorns, E., Gunn, W., Tan, F. E., Lomax, J. & Nosek, B. A. (2014) An open investigation of the reproducibility of cancer biology research. Elife 3:e04333.Google Scholar
Etz, A. (2015, August 30) The Bayesian Reproducibility Project . Weblog post . Retrieved 23 August 2017 from: https://web.archive.org/web/20160407113631/http://alexanderetz.com:80/2015/08/30/the-bayesian-reproducibility-project/.Google Scholar
Etz, A. & Vandekerckhove, J. (2016) A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE 11(2):e0149794. Available at: http://doi.org/10.1371/journal.pone.0149794.Google Scholar
Etz, A. & Wagenmakers, E. J. (2017) JBS Haldane's contribution to the Bayes factor hypothesis test. Statistical Science 32(2):313–29.Google Scholar
Fabrigar, L. R. & Wegener, D. T. (2016) Conceptualizing and evaluating the replication of research results. Journal of Experimental Social Psychology 66:68–80.Google Scholar
Ferguson, C. J. & Brannick, M. T. (2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods 17(1):120–28. Available at: http://doi.org/10.1037/a0024445.Google Scholar
Ferguson, C. J. & Heene, M. (2012) A vast graveyard of undead theories publication bias and psychological science's aversion to the null. Perspectives on Psychological Science 7(6):555–61.Google Scholar
Fetterman, A. K. & Sassenberg, K. (2015) The reputational consequences of failed replications and wrongness admission among scientists. PLoS One 10(12):e0143723. Available at: https://doi.org/10.1371/journal.pone.0143723.Google Scholar
Fiedler, K. & Schwarz, N. (2015) Questionable research practices revisited. Social Psychological and Personality Science 7:45–52. Available at: http://doi.org/10.1177/1948550615612150.Google Scholar
Finkel, E. J., Eastwick, P. W. & Reis, H. T. (2015) Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology 108:275–97. Available at: http://doi.org/10.1037/pspi0000007.Google Scholar
Franco, A., Malhotra, N. & Simonovits, G. (2014) Publication bias in the social sciences: Unlocking the file drawer. Science 345(6203):1502–505.Google Scholar
Gelman, A. & Loken, E. (2014) The statistical crisis in science data-dependent analysis – a “garden of forking paths” – explains why many statistically significant comparisons don't hold up. American Scientist 102(6):460.Google Scholar
Gilbert, D. T., King, G., Pettigrew, S, & Wilson, T. D. (2016) Comment on “estimating the reproducibility of psychological science.” Science 351(6277):1037. Available at: http://doi.org/10.1126/science.aad7243.Google Scholar
Greenwald, A. G. (1975) Significance, nonsignificance, and interpretation of an ESP experiment. Journal of Experimental Social Psychology 11:180–91.Google Scholar
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., Bruyneel, S., Calvillo, D. P., Campbell, W. K., Cannon, P. R., Carlucci, M., Carruth, N. P., Cheung, T., Crowell, A., De Ridder, D. T. D., Dewitte, S., Elson, M., Evans, J. R., Fay, B. A., Fennis, B. M, Finley, A., Francis, Z., Heise, E., Hoemann, H., Inzlicht, M., Koole, S. L., Koppel, L., Kroese, F., Lange, F., Lau, K., Lynch, B. P., Martijn, C., Merckelbach, H., Mills, N. V., Michirev, A., Miyake, A., Mosser, A. E., Muise, M., Muller, D., Muzi, M., Nalis, D., Nurwanti, R., Otgaar, H, Philipp, M. C., Primoceri, P., Rentzsch, K., Ringos, L., Schlinkert, C., Schmeichel, B. J., Schoch, S. F., Schrama, M., Schütz, A., Stamos, A., Tinghög, G., Ullrich, J., vanDellen, M., Wimbarti, S., Wolff, W., Yusainy, C., Zerhouni, O. & Zwienenberg, M. (2016) A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science 11(4):546–73.Google Scholar
Hewitt, J. K. (2012) Editorial policy on candidate gene association and candidate gene-by-environment interaction studies of complex traits. Behavior Genetics 42(1):1–2.Google Scholar
Hofstee, W. K. B. (1984) Methodological decision rules as research policies: A betting reconstruction of empirical research. Acta Psychologica 56:93–109.Google Scholar
Hüffmeier, J., Mazei, J. & Schultze, T. (2016) Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology 66:81–92.Google Scholar
Ioannidis, J. P. (2005) Why most published research findings are false. PLoS Medicine 2(8):e124.Google Scholar
Jeffreys, H. (1961) Theory of probability. Oxford University Press.Google Scholar
John, L. K., Loewenstein, G. & Prelec, D. (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23(5):524–32.Google Scholar
Kahneman, D. (2003) Experiences of collaborative research. American Psychologist 58:723.Google Scholar
Kerr, N. L. (1998) HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2:196–217. Available at: http://doi.org/10.1207/s15327957pspr0203_4.Google Scholar
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B. Jr., Bahník, S., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., Hicks, J. A., Hovermale, J. F., Hunt, S. J., Hunstinger, J. R., IJerzman, H., John, M.-S., Joy-Gaba, J. A., Kappes, H. B., Krueger, L. E., Kurtz, J., Levitan, C. A., Mallett, R. K., Morris, W. L., Nelson, A. J., Nier, J. A., Packard, G., Pilati, R., Rutchick, A. M., Schmidt, K., Skorinko, J. L., Smith, R., Steiner, T. G., Storbeck, J., Van Swol, L. M., Thompson, D., van't Veer, A. E., Vaughn, L. A., Vranka, M., Wichman, A. L., Woodzicka, J. A. & Nosek, B. A. (2014a) Investigating variation in replicability: A “Many Labs” replication project. Social Psychology 45(3):142–52. Available at: http://doi.org/10.1027/1864-9335/a000178.Google Scholar
Kühberger, A., Fritz, A. & Scherndl, T. (2014) Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One 9(9):e105825.Google Scholar
Kunert, R. (2016) Internal conceptual replications do not increase independent replication success. Psychonomic Bulletin and Review 23(5):1631–38.Google Scholar
Lakatos, I. (1970) Falsification and the methodology of scientific research programmes. In: Criticism and the growth of knowledge, ed. Lakatos, I. & Musgrave, A. , pp . 91–196. Cambridge University Press.Google Scholar
LeBel, E. P., Berger, D., Campbell, L. & Loving, T. J. (2017) Falsifiability is not optional. Journal of Personality and Social Psychology 113:254–61.Google Scholar
Levelt Committee Noort Committee & Drent Committee (2012, November 28) Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Retrieved 21 August 2017 from www.tilburguniversity.edu/upload/3ff904d7-547b-40ae85febea38e05a34a_Final%20report%20Flawed%20Science.pdf.Google Scholar
Lupia, A. & Elman, C. (2014) Openness in political science: Data access and research transparency. PS – Political Science and Politics 47:19–42.Google Scholar
Ly, A., Etz, A., Marsman, M. & Wagenmakers, E. J. (2017) Replication Bayes factors from evidence updating. PsyArXiv preprints. Available at: https://psyarxiv.com/u8m2s/.Google Scholar
Ly, A., Verhagen, J. & Wagenmakers, E. J. (2016) Harold Jeffreys's default Bayes factor hypothesis tests: Explanation, extension, and application in psychology. Journal of Mathematical Psychology 72:19–32.Google Scholar
Lykken, D. T. (1968) Statistical significance in psychological research. Psychological Bulletin 70:151–59.Google Scholar
Makel, M. C., Plucker, J. A. & Hegarty, B. (2012) Replications in psychology research: How often do they occur? Perspectives on Psychological Science 7(6):537–42.Google Scholar
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W. & Wagenmakers, E.-J. (2015) The effect of horizontal eye movements on free recall: A preregistered adversarial collaboration. Journal of Experimental Psychology: General 144(1):e1–15. Available at: http://doi.org/10.1037/xge0000038.Google Scholar
Maxwell, S. E., Lau, M. Y. & Howard, G. S. (2015) Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist 70:487–98. Available at: http://dx.doi.org/10.1037/a0039400.Google Scholar
Meehl, P. E. (1990b) Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry 1:108–41. Available at: http://doi.org/10.1207/s15327965pli0102_1.Google Scholar
Mellers, B., Hertwig, R. & Kahneman, D. (2001) Do frequency representations eliminate conjunction effects? An exercise in adversarial collaboration. Psychological Science 12:269–75.Google Scholar
Morey, R. D. & Lakens, D. (2016) Why most of psychology is statistically unfalsifiable . Available at: https://github.com/richarddmorey/psychology_resolution/blob/master/paper/response.pdfGoogle Scholar
National Cancer Institute–National Human Genome Research Institute (NCI-NHGRI) Working Group on Replication in Association Studies (2007) Replicating genotype-phenotype associations. Nature 447:655–60.Google Scholar
Nosek, B. A. & Errington, T. M. (2017) Making sense of replications. eLife 6:e23383.Google Scholar
Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349(6251):aac4716. Available at: http://doi.org/10.1126/science.aac4716.Google Scholar
Pashler, H. & Harris, C. (2012) Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7:531–36.Google Scholar
Pashler, H. & Wagenmakers, E.-J. (2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science 7:528–30.Google Scholar
Popper, K. R. (1959/2002) The logic of scientific discovery, translation of Logik der Forschung . Routledge.Google Scholar
Rohrer, D., Pashler, H. & Harris, C. R. (2015) Do subtle reminders of money change people's political views? Journal of Experimental Psychology: General 144(4):e73.Google Scholar
Rosenthal, R. (1979) The “file drawer problem” and tolerance for null results. Psychological Bulletin 86(3):638–41.Google Scholar
Rotello, C. M., Heit, E. & Dubé, C. (2015) When more data steer us wrong: Replications with the wrong dependent measure perpetuate erroneous conclusions. Psychonomic Bulletin and Review 22:944–54.Google Scholar
Rothstein, H. R. & Bushman, B. J. (2012) Publication bias in psychological science: Comment on Ferguson and Brannick (2012). Psychological Methods 17:129–36.Google Scholar
Schmidt, F. L. & Oh, I.-S. (2016) The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology 4(1):32–37. Available at: http://dx.doi.org/10.1037/arc0000029.Google Scholar
Schmidt, S. (2009) Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13:90–100.Google Scholar
Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., Srinivasan, M., Tannenbaum, D., Bivolaru, E., Dana, J., Davis-Stober, C. P., du Plessis, C., Gronau, Q. F., Hafenbrack, A. C., Liao, E. Y., Ly, A., Marsman, M., Murase, T., Qureshi, I., Schaerer, M., Thornley, N., Tworek, C. M., Wagenmakers, E.-J., Wong, L., Anderson, T., Bauman, C. W., Bedwell, W. L., Brescoll, V., Canavan, A., Chandler, J. J., Cheries, E., Cheryan, S., Cheung, F., Cimpian, A., Clark, M. A., Cordon, D., Cushman, F., Ditto, P. H., Donahue, T., Frick, S. E., Gamez-Djokic, M., Hofstein Grady, R., Graham, J., Gu, J., Hahn, A., Hanson, B. E., Hartwich, N. J., Hein, K. Inbar, Y., Jiang, L., Kellogg, T., Kennedy, D. M., Legate, N., Luoma, T. P., Maibuecher, H., Meindl, P., Miles, J., Mislin, A., Molden, D. C., Motyl, M., Newman, G., Ngo, H. H., Packham, H., Ramsay, P. S., Ray, J. L., Sackett, A. M., Sellier, A.-L., Sokolova, T., Sowden, W., Storage, D., Sun, X., Van Bavel, J. J., Washburn, A. N., Wei, C., Wetter, E., Wilson, C. T., Darrous, S.-C. & Uhlmann, E. L. (2016) The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology 66:55–67.Google Scholar
Shanks, D. R., Vadillo, M. A., Riedel, B., Clymo, A., Govind, S., Hickin, N., Tamman, A. J. & Puhlmann, L. (2015) Romance, risk, and replication: Can consumer choices and risk-taking be primed by mating motives? Journal of Experimental Psychology: General 144(6):e142–58.Google Scholar
Silberzahn, R., Uhlmann, E. L., Martin, D., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T. J., Hederos, K., Heene, M., Hofelich Mohr, A. J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D. M., Lei, R., Lindsay, T. A., Liverani, S., Madan, C. R., Molden, D., Molleman, E., Morey, R. D., Mulder, L. B., Nijstad, B. R., Pope, N. G., Pope, B., Prenoveau, J. M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F. D., Sherman, M. F., Sommer, S. A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E.-J., Witkowiak, M., Yoon, S. & Nosek, B. A. (2017) Many analysts, one dataset: Making transparent how variations in analytical choices affect results. PsyArXiv Preprint. Available at: https://psyarxiv.com/qkwst/.Google Scholar
Simmons, J. P., Nelson, L. D. & Simonsohn, U. (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22:1359–66. Available at: http://doi.org/10.1177/0956797611417632.Google Scholar
Simons, D. J., Shoda, Y. & Lindsay, D. S. (2017) Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science 12:1123–28. Available at: http://doi.org/10.1177/1745691617708630.Google Scholar
Simonsohn, U. (2015) Small telescopes: Detectability and the evaluation of replication results. Psychological Science 26:559–69.Google Scholar
Simonsohn, U. (2016, March 3) [47] Evaluating replications: 40% full≠60% empty . Available at: https://web.archive.org/web/20170709184952/http://datacolada.org/47.Google Scholar
Smaldino, P. E. & McElreath, R. (2016) The natural selection of bad science. Royal Society Open Science 3:160384.Google Scholar
Smart, R. G. (1964) The importance of negative results in psychological research. Canadian Psychologist 5:225–32.Google Scholar
Spellman, B. A. (2015) A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science 10:886–99.Google Scholar
Sterling, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Journal of the American Statistical Association 54(285):30–34. Available at: http://doi.org/10.2307/2282137.Google Scholar
Sterling, T. D., Rosenbaum, W. L. & Weinkam, J. J. (1995) Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician 49:108–12.Google Scholar
Stroebe, W. & Strack, F. (2014) The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9:59–71. Available at: http://doi.org/10.1177/1745691613514450.Google Scholar
van Aert, R. C. & van Assen, M. A. (2017) Bayesian evaluation of effect size after replicating an original study. PLoS One 12(4):e0175302.Google Scholar
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J. & Reinero, D. A. (2016) Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America 113(23):6454–59.Google Scholar
Verhagen, A. J. & Wagenmakers, E.-J. (2014) Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General 143:1457–75.Google Scholar
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B. Jr., Albohn, D. N., Allard, E. S., Benning, S. D., Blouin-Hudon, E.-M., Bulnes, L. C., Caldwell, T. L., Calin-Jageman, R. J., Capaldi, C. A., Carfagno, N. S., Chasten, K. T., Cleeremans, A., Connell, L., DeCicco, J. M., Dijkstra, K, Fischer, A. H., Foroni, F., Hess, U., Holmes, K. J., Jones, J. L. H., Klein, O., Koch, C., Korb, S., Lewinski, P., Liao, J. D., Lund, S., Lupianez, J., Lynott, D., Nance, C. N., Oosterwijk, S., Ozdoğru, A. A., Pacheco-Unguetti, A. P., Pearson, B., Powis, C., Riding, S., Roberts, T.-A., Rumiati, R. I., Senden, M., Shea-Shumsky, N. B., Sobocko, K., Soto, J. A., Steiner, T. G., Talarico, J. M., van Allen, Z. M., Vandekerckhove, M., Wainwright, B., Wayand, J. F., Zeelenberg, R., Zetzer, E. E. & Zwaan, R. A. (2016a) Registered replication report: Strack, Martin & Stepper (1988). Perspectives on Psychological Science 11:917–28.Google Scholar
Wagenmakers, E.-J., Verhagen, A. J. & Ly, A. (2016b) How to quantify the evidence for the absence of a correlation. Behavior Research Methods 48:413–26.Google Scholar
Wrinch, D. & Jeffreys, H. (1921) XLII. On certain fundamental principles of scientific inquiry. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 42(249):369–90.Google Scholar
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K. & Zeelenberg, R. (2017) Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin and Review. Available at: http://doi.org/10.3758/s13423-017-1348-y.Google Scholar