Hostname: page-component-78c5997874-dh8gc Total loading time: 0 Render date: 2024-11-05T09:31:42.387Z Has data issue: false hasContentIssue false
  • English
  • Français

Inattentive Responding in MTurk and Other Online Samples

Published online by Cambridge University Press:  28 July 2015

Avi Fleischer ,
Alan D. Mead  and
Jialin Huang
Avi Fleischer*
Affiliation:
Tetrics LLC, Chicago, Illinois
Alan D. Mead
Affiliation:
Talent Algorithms Inc., Chicago, Illinois
Jialin Huang
Affiliation:
Illinois Institute of Technology
*
Correspondence concerning this article should be addressed to Avi Fleischer, Tetrics LLC, Chicago, IL. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Extract

The focal article by Landers and Behrend (2015) makes the case that samples collected on microtask websites like Amazon's Mechanical Turk (MTurk) are inherently no better or worse than traditional samples of convenience from university students or organizations. We wholeheartedly agree. However, having successfully used MTurk and other online sources for data collection, we feel that the focal article was insufficient regarding the caution required in identifying inattentive respondents and the problems that can arise if such individuals are not removed from the dataset. Although we focus on MTurk, similar issues arise for most “low-stakes” assessments, including student samples, which seem to be increasingly collected online.

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 8 , Issue 2 , June 2015 , pp. 196 - 202
Copyright
Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Long Grove, IL: Waveland Press.Google Scholar
Armstrong, R. D., Stoumbos, Z. G., Kung, M. T., & Shi, M. (2007). On the performance of the lZ person-fit statistic. Practical Assessment, Research & Evaluation, 12. Retrieved from http://pareonline.net/pdf/v12n16.pdfGoogle Scholar
Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI–A. Journal of Personality Assessment, 68, 139–151.Google Scholar
Basic HHS Policy for Protection of Human Research Subjects, General Requirements for Informed Consent, 45 C.F.R. §46.116(a)(8) (2009).Google Scholar
Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.Google Scholar
Fleischer, A., & Mead, A. D. (2015). Detecting inattentive pilot test examinees in an MTurk sample. Paper presented at the 2015 NCME Annual Meeting, Chicago, IL.Google Scholar
Fleischer, A., Mead, A. D., & Neuhengen, J. (2014). Comparability of Big Five scores from online and traditional samples. Paper presented at the 29th annual Society of Industrial Organizational Psychologists, Honolulu, HI.Google Scholar
Huang, J. (2015). Does the negatively-worded item effect exist in personality measures? A meta-analysis (Unpublished doctoral dissertation). Illinois Institute of Technology, Chicago, IL.Google Scholar
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.Google Scholar
Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Lee, Y.-H., & Chen, H. (2011). A review of recent response-time analyses in educational testing. Psychological Test and Assessment Modeling, 53, 359–379.Google Scholar
Marquis, K. H., Marquis, M. S., & Polich, J. M. (1986). Response bias and reliability in sensitive topic surveys. Journal of the American Statistical Association, 81, 381–389.Google Scholar
McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470.Google Scholar
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.Google Scholar
Montgomery, K. C. (2007). Generation digital: Politics, commerce, and childhood in the age of the Internet. Cambridge, MA: MIT Press.CrossRefGoogle Scholar
Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. (2001). Attitudes toward surveys: Development of a measure and its relationship to respondent behavior. Organizational Research Methods, 4, 3–25.CrossRefGoogle Scholar
Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000). Employee attitude surveys: Examining the attitudes of noncompliant employees. Journal of Applied Psychology, 85, 284–293.Google Scholar
Schmitt, N., & Stults, D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9, 367–373.Google Scholar
Sliter, K. A., & Zickar, M. J. (2014). An IRT examination of the psychometric functioning of negatively worded personality items. Educational and Psychological Measurement, 74, 214–226.Google Scholar
Thompson, B. (1994, January). It is incorrect to say “the test is reliable”: Bad language habits can contribute to incorrect or meaningless research conclusions. Paper presented at the Annual Meeting of the Southwest Educational Research Association, San Antonio, TX. ERIC Number: ED367707.Google Scholar
Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186–191.Google Scholar
Wright, K. B. (2005). Researching Internet-based populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of Computer-Mediated Communication, 10 (3), 00. doi:10.1111/j.1083-6101.2005.tb00259.xGoogle Scholar