Hostname: page-component-586b7cd67f-g8jcs Total loading time: 0 Render date: 2024-11-22T15:37:37.010Z Has data issue: false hasContentIssue false
  • English
  • Français

Exposing and overcoming the fixed-effect fallacy through crowd science

Published online by Cambridge University Press:  10 February 2022

Wilson Cyrus-Lai ,
Warren Tierney ,
Martin Schweinsberg  and
Eric Luis Uhlmann Open the ORCID record for Eric Luis Uhlmann [Opens in a new window]
Wilson Cyrus-Lai
Affiliation:
Organisational Behaviour Area, INSEAD, [email protected]; [email protected]
Warren Tierney
Affiliation:
Organisational Behaviour Area/Marketing Area, INSEAD, [email protected]
Martin Schweinsberg
Affiliation:
Martin Schweinsberg, Organisational Behaviour Area, ESMT Berlin, 10178, BerlinGermany. [email protected]
Eric Luis Uhlmann
Affiliation:
Organisational Behaviour Area, INSEAD, [email protected]; [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

By organizing crowds of scientists to independently tackle the same research questions, we can collectively overcome the generalizability crisis. Strategies to draw inferences from a heterogeneous set of research approaches include aggregation, for instance, meta-analyzing the effect sizes obtained by different investigators, and parsing, attempting to identify theoretically meaningful moderators that explain the variability in results.

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 45 , 2022 , e8
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Banaji, M. R. (2003). The opposite of a great truth is also true: Homage of Koan #7. In Jost, J., Prentice, D. & Banaji, M. R. (Eds.), The yin and yang of progress in social psychology: Perspectivism at work (pp. 127–140). Washington, DC: American Psychological Association.Google Scholar
Baribault, B., Donkin, C., Little, D. R., Trueblood, J. S., Oravecz, Z., Van Ravenzwaaij, D., … Vandekerckhove, J. (2018). Metastudies for robust tests of theory. Proceedings of the National Academy of Sciences, 115(11), 2607–2612.CrossRefGoogle ScholarPubMed
Bastiaansen, J. A., Kunkels, Y. K., Blaauw, F. J., Boker, S. M., Ceulemans, E., Chen, M., … Bringmann, L. F. (2020). Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology. Journal of Psychosomatic Research, 137, 110211.CrossRefGoogle ScholarPubMed
Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M. … Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582, 84–88.CrossRefGoogle ScholarPubMed
Brescoll, V., & Uhlmann, E. L. (2008). Can angry women get ahead? Status conferral, gender, and workplace emotion expression. Psychological Science, 19, 268–275.CrossRefGoogle Scholar
Landy, J. F., Jia, M., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., … Uhlmann, E. L. (2020). Crowdsourcing hypothesis tests: Making transparent how design choices shape research results. Psychological Bulletin, 146(5), 451–479.CrossRefGoogle ScholarPubMed
Liu, Y., Kale, A., Althoff, T., & Heer, J. (2020). Boba: Authoring and visualizing multiverse analyses. IEEE Transactions on Visualization and Computer Graphics, 27(2), 1753–1763.CrossRefGoogle Scholar
McGuire, W. J. (1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26(3), 446–456.CrossRefGoogle Scholar
McGuire, W. J. (1983). A contextualist theory of knowledge: Its implications for innovations and reform in psychological research. In Berkowitz, L. (Ed.), Advances in experimental social psychology (Vol. 16, pp. 1–47). New York, NY: Academic Press.Google Scholar
Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O., van Aert, R., van Assen, M., Liu, Y., … Uhlmann, E. (2021). Radical dispersion of effect size estimates when independent scientists operationalize and test the same hypothesis with the same data. Organizational Behavior and Human Decision Processes, 165, 228–249.CrossRefGoogle Scholar
Silberzahn, R., Uhlmann, E. L., Martin, D., Anselmi, P., Aust, F., Awtrey, E. … Nosek, B. N. (2018). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science, 1, 337–356.CrossRefGoogle Scholar
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712.CrossRefGoogle ScholarPubMed
Tierney, W., Cyrus-Lai, W., … Uhlmann, E. L. (in preparation). Who respects an angry woman? A pre-registered re-examination of the relationships between gender, emotion expression, and status conferral. Crowdsourced research project in progress.Google Scholar