Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-5cf477f64f-mgq6s Total loading time: 0 Render date: 2025-03-31T21:57:17.765Z Has data issue: false hasContentIssue false

15 - Methodological Issues in the Study of Implicit Attitudes

from Section IV - Improving Measurement and Theorizing About Implicit Bias

Published online by Cambridge University Press:  21 December 2024

By
James Jaccard
Edited by
Jon A. Krosnick ,
Tobias H. Stark  and
Amanda L. Scott
Jon A. Krosnick
Affiliation:
Stanford University, California
Tobias H. Stark
Affiliation:
Utrecht University, The Netherlands
Amanda L. Scott
Affiliation:
The Strategy Team, Columbus, Ohio
Get access

Summary

Eight methodological issues relevant to improving the quality of research on implicit attitudes are considered. These include (1) formulating and implementing strong psychometric models for implicit attitude measures, (2) using modern theories of explicit attitudes as a base from which to test key propositions about implicit attitudes, (3) using sound psychometric practices to assess explicit attitudes, (4) addressing the problem of endogeneity, (5) addressing ecological fallacies when pursuing aggregate analyses of implicit attitudes, (6) evaluating the magnitude of effect sizes, (7) using structural equation modeling in implicit attitude research, and (8) using proper moderation analysis and incremental explained variance analysis in meta-analyses. The central elements of each problem are described and recommendations for addressing them are provided.

Keywords

implicit attitude psychometricsimplicit versus explicit attitudesmodern attitude theorydata analysis for implicit attitudes
Type
Chapter
Information
The Cambridge Handbook of Implicit Bias and Racism , pp. 425 - 455
Publisher: Cambridge University Press
Print publication year: 2025

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Achen, C. (1987). Interpreting and Using Regression. New York, NY: SAGE Publications.Google Scholar
Baker, F., & Kim, S. (2004). Item Response Theory: Parameter Estimation Techniques. New York, NY: CRC Press.CrossRefGoogle Scholar
Bargh, J.A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype-activation on action. Journal of Personality and Social Psychology, 71, 230–244.CrossRefGoogle ScholarPubMed
Bentler, P., & Satorra, A. (2000). Hierarchical regression without phantom factors. Structural Equation Modeling, 7, 287–291.CrossRefGoogle Scholar
Blanton, H., & Jaccard, J. (2006a). Arbitrary metrics in psychology. American Psychologist, 61, 27–41.CrossRefGoogle ScholarPubMed
Blanton, H., & Jaccard, J. (2006b). Tests of multiplicative models in psychology: A case study using the unified theory of implicit attitudes, stereotypes, self-esteem, and self-concept. Psychological Review, 113, 155–169.CrossRefGoogle ScholarPubMed
Blanton, H., & Jaccard, J. (2008). Unconscious racism: A concept in pursuit of a measure. Annual Review of Sociology, 34, 277–297.CrossRefGoogle Scholar
Blanton, H., & Jaccard, J. (2017). You can’t assess the forest if you can’t assess the trees: Psychometric challenges to measuring implicit bias in crowds. Psychological Inquiry, 28, 249–257.CrossRefGoogle Scholar
Blanton, H., & Jaccard, J. (2021). Listening to measurement error: Lessons from the IAT. In Krosnick, J. A., Stark, T. H., & Scott, A. L. (Eds.). The Cambridge Handbook of Implicit Bias and Racism. Cambridge: Cambridge University Press.Google Scholar
Blanton, H., Jaccard, J., Gonzales, P. M., et al. (2006). Decoding the implicit association test: Implications for criterion prediction. Journal of Experimental Social Psychology, 42, 192–212.CrossRefGoogle Scholar
Bollen, K. A., & Diamantopoulos, A. (2017). In defense of causal–formative indicators: A minority report. Psychological Methods, 3, 581–596.CrossRefGoogle Scholar
Bond, C. F., Wiitala, W. L., & Richard, F. D. (2003). Meta-analysis of raw mean differences. Psychological Methods, 8, 406–418.CrossRefGoogle ScholarPubMed
Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71, 425–440.CrossRefGoogle ScholarPubMed
Buttrick, N., Axt, J., Ebersole, C., et al. (2020). Re-assessing the incremental predictive validity of Implicit Association Tests. Journal of Experimental Social Psychology, 88. On-line preview.CrossRefGoogle Scholar
Cheung, M., & Chan, W. (2005). Meta-analytic structural equation modeling: A two-stage approach. Psychological Methods, 10, 40–64.CrossRefGoogle ScholarPubMed
Cohen, J. (1989). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Erlbaum Associates.Google Scholar
Cohen, J., Cohen, P., West, S. G., et al. (2003). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. New York, NY: Erlbaum Associates.Google Scholar
Conrey, F. R., Sherman, J. W., Gawronski, B., et al. (2005). Separating multiple processes in implicit social cognition: The Quad-Model of implicit task performance. Journal of Personality and Social Psychology, 89, 469–487.CrossRefGoogle ScholarPubMed
Cronbach, L. (1976). Research on classrooms and schools: Formulation of questions, design and analysis. Occasional paper of the Stanford Evaluation Consortium. Palo Alto, CA: Stanford University. http://files.eric.ed.gov/fulltext/ED135801.pdfGoogle Scholar
Cronbach, L., & Furby, L. (1970). How should we measure “change”—or should we? Psychological Bulletin, 74, 68–80.CrossRefGoogle Scholar
Cronbach, L., Gleser, C., Nanda, H., et al. (1972). The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York, NY: Wiley.Google Scholar
Cummins, J., Hussey, I., & Hughes, S. (2019). The AMPeror’s new clothes: Performance on the Affect Misattribution Procedure is mainly driven by awareness of influence of the primes (online). https://doi.org/10.31234/osf.io/d5zn8CrossRefGoogle Scholar
Cunningham, W.A., Preacher, K.J., & Banaji, M.R. (2001). Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science, 12, 163–170.CrossRefGoogle ScholarPubMed
Dodaj, A. (2012). Social desirability and self-reports: Testing a content and response-style model of socially desirable responding. Europe’s Journal of Psychology, 8, 651–666.CrossRefGoogle Scholar
Draheim, C., Hicks, K. L., & Engle, R. W. (2016). Combining reaction time and accuracy: The relationship between working memory capacity and task switching as a case example. Perspectives on Psychological Science 11(1), 133–155.CrossRefGoogle ScholarPubMed
Draheim, C., Mashburn, C., Martin, J., et al. (2019). Reaction time in differential and developmental research: A review and commentary on the problems and alternatives.Psychological Bulletin, 145, 508–535.CrossRefGoogle ScholarPubMed
Drasgow, F., Chernyshenko, S., & Stark, S. (2010). 75 years after Likert: Thurstone was right. Industrial and Organziational Psychology, 3, 465–476.CrossRefGoogle Scholar
Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517.CrossRefGoogle Scholar
Fazio, R. H., Jackson, J. R., Dunton, B. C., et al. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69, 1013–1027.CrossRefGoogle Scholar
Firebaugh, G. (1978). A rule for inferring individual-level relationships from aggregate data. American Sociological Review, 43, 557–572.CrossRefGoogle Scholar
Fishbein, M., & Ajzen, I. (2010). Predicting and Changing Behavior: The Reasoned Action Approach. New York, NY: Psychology Press.Google Scholar
Fisher, W. A., Fisher, J. D., & Shuper, P. (2014). Social psychology and the fight against AIDS: An information-motivation-behavioral skills model for the prediction and promotion of health behavior change. Advances in Experimental Social Psychology, 50, 105–193.CrossRefGoogle Scholar
Forstmann, B. U., Tittgemeyer, M., Wagenmakers, E.-J., et al. (2011). The speed–accuracy tradeoff in the elderly brain: A structural model-based approach. Journal of Neuroscience, 31, 17242–17249.CrossRefGoogle Scholar
Furlow, C.F., & Beretvas, N. (2005). Meta-analytic methods of pooling correlation matrices for structural equation modeling under different patterns of missing data. Psychological Methods, 10, 227–254.CrossRefGoogle ScholarPubMed
Gawronski, B., & De Houwer, J. (2014). Implicit measures in social and personality psychology. In Reis, H. T., & Judd, C. M. (Eds.), Handbook of Research Methods in Social and Personality Psychology (2nd ed.). Cambridge: Cambridge University Press.Google Scholar
Gelman, A., Park, D., Shor, B., et al. (2008). Red State, Blue State, Rich State, Poor State. Princeton, NJ: Princeton University Press.Google Scholar
Gerrard, M., Gibbons, F. X., Houlihan, A. E., et al. (2008). A dual-process approach to health risk decision making: The prototype willingness model. Developmental Review, 28, 29–61.CrossRefGoogle Scholar
Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102, 4–27.CrossRefGoogle ScholarPubMed
Greenwald, A. G., & Banaji, M. R. (2017). The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72, 861–871.CrossRefGoogle ScholarPubMed
Greenwald, A. G., Banaji, M. R., & Nosek, B. A. (2015). Statistically small effects of the Implicit Association Test can have societally large effects. Journal of Personality and Social Psychology, 108, 553–561.CrossRefGoogle ScholarPubMed
Greenwald, A. G., McGhee, D., & Schwartz, J. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464–1480.CrossRefGoogle ScholarPubMed
Greenwald, A. G., Poehlman, T. A., Uhlmann, E. L., et al. (2009). Understanding and using the Implicit Association Test: Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 17–41.CrossRefGoogle ScholarPubMed
Hayduk, L. (2014a). Seeing perfectly fitting factor models that are causally misspecified: Understanding that close-fitting models can be worse. Educational and Psychological Measurement, 74, 905–926.CrossRefGoogle Scholar
Hayduk, L. (2014b). Shame for disrespecting evidence: The personal consequences of insufficient respect for structural equation model testing. BMC Medical Research Methodology, 14(124).Google Scholar
Heitz, R. P. (2014). The speed–accuracy tradeoff: History, physiology, methodology, and behavior. Frontiers in Neuroscience, 8, 150–172.CrossRefGoogle ScholarPubMed
Jaccard, J., & Bo, A. (2020). Making outcome measures more useful in clinical trials: Addressing the problem of arbitrary metrics. Under editorial review.Google Scholar
Jaccard, J., & Jacoby, J. (2020). Theory Construction and Model Building Skills: A Practical Guide for Social Scientists. New York, NY: Guilford Press.Google Scholar
Jaccard, J., & Levitz, N. (2015). Parent-based interventions to reduce adolescent problem behaviors: New directions for self-regulation approaches. In Oettingen, G. & Gollwitzer, P. (Eds.), Self-regulation in Adolescence. Cambridge: Cambridge University Press.Google Scholar
Jaccard, J., Turrisi, R. & Wan, C. (1990) Interaction Effects in Multiple Regression. New York, NY: SAGE Publications.Google ScholarPubMed
Jaccard, J., & Wan, C. K. (2006). A paradigm for studying the accuracy of self-reports of risk behavior relevant to AIDS: Empirical perspectives on stability, recall bias, and transitory influences. Journal of Applied Social Psychology, 25, 1559–1816.Google Scholar
Jost, J., Rudman, L., Blair, I, et al. (2009). The existence of implicit bias is beyond reasonable doubt: A refutation of ideological and methodological objections and executive summary of ten studies that no manager should ignore. Research in Organizational Behavior, 29, 39–69.CrossRefGoogle Scholar
Kim, J., & Ferree, G. (1981). Standardization in causal analysis. Sociological Methods and Research, 10, 187–210.CrossRefGoogle Scholar
Kline, R. (2011). Principles and Practice of Structural Equation Modeling. New York, NY: Guilford Press.Google Scholar
Kurdi, B., Seitchik, A. E., Axt, J. R., et al. (2019). Relationship between the Implicit Association Test and intergroup behavior: A meta-analysis. American Psychologist, 74, 569–586.CrossRefGoogle ScholarPubMed
Miller, J., & Ulrich, R. (2013). Mental chronometry and individual differences: Modeling reliabilities and correlations of reaction time means and effect sizes. Psychonomic Bulletin & Review, 20, 819–858.CrossRefGoogle ScholarPubMed
Mulaik, S. A. (2005). Looking back on the indeterminacy controversies in factor analysis. In A. Maydeu-Olivares, & J. McArdle, (Eds.), Contemporary Psychometrics. New York, NY: Erlbaum Associates, pp. 173–206.Google Scholar
Oswald, F.L., Mitchell, G., Blanton, H., et al. (2015). Using the IAT to predict ethnic and racial discrimination: Small effect sizes of unknown importance. Journal of Personality and Social Psychology, 108, 562–571.CrossRefGoogle Scholar
Payne, B. K., Cheng, C. M., Govorun, O., et al. (2005). An inkblot for attitudes: Affect misattribution as implicit measurement. Journal of Personality and Social Psychology, 89, 277–293.CrossRefGoogle ScholarPubMed
Payne, K,. & Lundberg, K. (2014). The affect misattribution procedure: Ten years of evidence on reliability, validity, and mechanisms. Social and Personality Psychology Compass, 8, 672–686.CrossRefGoogle Scholar
Payne, K., Niemi, L., & Doris, J. (2018). How to think about explicit bias. Scientific American (online). www.scientificamerican.com/article/how-to-think-about-implicit-bias/Google Scholar
Payne, B. K., Vuletich, H. A., & Lundberg, K. B. (2017). The bias of crowds: How implicit bias bridges personal and systemic prejudice. Psychological Inquiry, 28, 233–248.CrossRefGoogle Scholar
Pearl, J., & Mackenzie, D. (2018). The Book of Why. New York, NY: Basic Books.Google Scholar
Prochaska, J. O., & DiClemente, C. C. (1992). The transtheoretical approach. In Norcross, J. C. & Goldfried, M. R. (Eds.), Handbook of Psychotherapy Integration. New York, NY: Basic Books.Google Scholar
Raykov, T., Marcoulides, G. A., & Millsap, R. E. (2013). Factorial invariance in multiple populations: A multiple testing procedure. Educational and Psychological Measurement, 73, 713–727.CrossRefGoogle Scholar
Rhemtulla, M., van Bork, R., & Borsboom, D. (2020). Worse than measurement error: Consequences of inappropriate latent variable measurement models. Psychological Methods, 25, 30–45.CrossRefGoogle ScholarPubMed
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.CrossRefGoogle Scholar
Rudman, L. A. (2008). The validity of the Implicit Association Test is a scientific certainty. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 426–429.CrossRefGoogle Scholar
Sackett, P., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85, 112–118.CrossRefGoogle ScholarPubMed
Selvin, H. (1958). Durkheim’s suicide and problems of empirical research. American Journal of Sociology, 63, 607–619.CrossRefGoogle Scholar
Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752–760.CrossRefGoogle ScholarPubMed
Teige-Mocigemba, S., Becker, M., Sherman, J. et al. (2017). The affect misattribution procedure, Experimental Psychology, 64, 215–230.CrossRefGoogle ScholarPubMed
Wilcox, R., & Tian, T. (2011). Measuring effect size: A robust heteroscedastic approach for two or more groups. Journal of Applied Statistics, 38, 1350–1368.CrossRefGoogle Scholar
Wolsiefer, K., Westfall, J., & Judd, C. M. (2017). Modeling stimulus variation in three common implicit attitude tasks. Behavior Research, 49, 1193–1209.CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×