Hostname: page-component-cd9895bd7-mkpzs Total loading time: 0 Render date: 2024-12-22T15:27:14.135Z Has data issue: false hasContentIssue false
  • English
  • Français

Addressing a crisis of generalizability with large-scale construct validation

Published online by Cambridge University Press:  10 February 2022

Jessica Kay Flake Open the ORCID record for Jessica Kay Flake [Opens in a new window] ,
Raymond Luong  and
Mairead Shaw
Jessica Kay Flake
Affiliation:
Department of Psychology, McGill University, Montreal, QCH3A 1G1, Canada. [email protected]; [email protected]; [email protected]; https://www.mcgill.ca/psychology/jessica-kay-flake; https://www.mcgill.ca/psychology/jessica-kay-flake
Raymond Luong
Affiliation:
Department of Psychology, McGill University, Montreal, QCH3A 1G1, Canada. [email protected]; [email protected]; [email protected]; https://www.mcgill.ca/psychology/jessica-kay-flake; https://www.mcgill.ca/psychology/jessica-kay-flake
Mairead Shaw
Affiliation:
Department of Psychology, McGill University, Montreal, QCH3A 1G1, Canada. [email protected]; [email protected]; [email protected]; https://www.mcgill.ca/psychology/jessica-kay-flake; https://www.mcgill.ca/psychology/jessica-kay-flake
Get access Rights & Permissions [Opens in a new window]

Abstract

Because of the misspecification of models and specificity of operationalizations, many studies produce claims of limited utility. We suggest a path forward that requires taking a few steps back. Researchers can retool large-scale replications to conduct the descriptive research which assesses the generalizability of constructs. Large-scale construct validation is feasible and a necessary next step in addressing the generalizability crisis.

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 45 , 2022 , e14
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23(7), 764–771. https://doi.org/10.1177/0956797611434537.CrossRefGoogle ScholarPubMed
Cloos, L. J. R., & Flake, J. K. (submitted). Lost in translation? Addressing measurement equivalence in large-scale replications. Department of Psychology, McGill University.Google Scholar
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957.CrossRefGoogle ScholarPubMed
Ebersole, C. R., Mathur, M. B., Baranski, E., Bart-Plange, D.-J., Buttrick, N. R., Chartier, C. R., … Nosek, B. A. (2020). Many labs 5: Testing pre-data-collection peer review as an intervention to increase replicability. Advances in Methods and Practices in Psychological Science, 3(3), 309–331. https://doi.org/10.1177/2515245920958687.CrossRefGoogle Scholar
Kane, M. T. (2013). Validating the interpretations and uses of test scores. The Journal of Educational Measurement, 50(1), 1–73.CrossRefGoogle Scholar
Moshontz, H., Campbell, L., Ebersole, C. R., Ijzerman, H., Urry, H. L., Forscher, P. S., … Chartier, C. R. (2018). The Psychological Science Accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science, 1(4), 501–515.CrossRefGoogle ScholarPubMed
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716.CrossRefGoogle Scholar
Shaw, M., Cloos, L. J. R., Luong, R., Elbaz, S., & Flake, J. K. (2020). Measurement practices in large-scale replications : Insights from Many Labs 2. Canadian Psychology, 61(4), 289–298. https://doi.org/10.1037/cap0000220.CrossRefGoogle Scholar
Wagenmakers, E. J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., … Zwaan, R. A. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. https://doi.org/10.1177/1745691616674458.CrossRefGoogle Scholar