MTurk and Online Panel Research

doi:10.1017/9781009243957.024

20 - MTurk and Online Panel Research

from 3 - Methods for Understanding Consumer Psychology

Published online by Cambridge University Press: 30 March 2023

Joseph K. Goodman and

Edited by

Derek D. Rucker and

Cait Lamberton: Affiliation:
Wharton School, University of Pennsylvania
Derek D. Rucker: Affiliation:
Kellogg School, Northwestern University, Illinois
Stephen A. Spiller: Affiliation:
Anderson School, University of California, Los Angeles

Book contents

Get access

Summary

Online platforms such as Amazon’s Mechanical Turk (MTurk), CloudResearch, and Prolific have become a common source of data for behavioral researchers and consumer psychologists alike. This chapter reviews contemporary issues associated with online panel research, discussing first how the COVID-19 pandemic impacted the extent to which researchers use online panels and the workers participating on certain online panels. The chapter explores how factors like a TikTok video can impact who uses these online panels and why. A longitudinal study of researcher perceptions and data quality practices finds that many practices do not align with current recommendations. The authors provide several recommendations for researchers to conduct high-quality behavioral research online, including the use of appropriate prescreens before data collection, data analysis preregistration practices, and avoiding post-screens after data collection that are not preregistered. Finally, the authors recommend researchers thoroughly report details on recruitment, restrictions, completion rates, and any differences in dropout rates across conditions.

Keywords

online research MTurk philosophy of science consumer research methods data integrity crowdsourced research

Type: Chapter
Information: The Cambridge Handbook of Consumer Psychology , pp. 548 - 571

DOI: https://doi.org/10.1017/9781009243957.024 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2023

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47(4), 823–837.Google Scholar

Arechar, A. A., & Rand, D. G. (2021). Turking in the time of COVID. Behavior Research Methods, 53(6), 2591–2595.Google Scholar

Arndt, A. D., Ford, J. B., Babin, B. J., & Luong, V. (2022). Collecting samples from online services: How to use screeners to improve data quality. International Journal of Research in Marketing, 39(1), 117–133.Google Scholar

Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Current Directions in Psychological Science, 13(2), 149–154.Google Scholar

Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130.Google Scholar

Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.Google Scholar

Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022–2038.Google Scholar

Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12.Google Scholar

Charalambides, N. (2021, August 24). We recently went viral on TikTok - here’s what we learned. Prolific Blog, www.blog.prolific.co/we-recently-went-viral-on-tiktok-heres-what-we-learned Google Scholar

Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464–473.Google Scholar

Couper, M. (2008). Designing Effective Web Surveys, Vol. 75. Cambridge University Press.Google Scholar

Dennis, S. A., Goodson, B. M., & Pearson, C. A. (2020). Online worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening procedures. Behavioral Research in Accounting, 32(1), 119–134.Google Scholar

Dholakia, U. (2015, July 20). My Experience as an Amazon Mechanical Turk (MTurk) Worker. www.linkedin.com/pulse/my-experience-amazon-mechanical-turkmturk-worker-utpal-dholakia Google Scholar

Dreyfuss, E. (2018, August 17). A bot panic hits Amazon’s Mechanical Turk. Wired, www.wired.com/story/amazon-mechanical-turk-bot-panic Google Scholar

Eyal, P., David, R., Andrew, G., Zak, E., & Ekaterina, D. (2022). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54, 1643–1662.Google Scholar

Freitas, A. L., Gollwitzer, P., & Trope, Y. (2004). The influence of abstract and concrete mindsets on anticipating and guiding others’ self-regulatory efforts. Journal of Experimental Social Psychology, 40(6), 739–752.Google Scholar

Garcia, T. (2021, June 15). Consumers find shortages and higher prices as COVID-impacted supply chains shift for recovery. MarketWatch. www.marketwatch.com/story/consumers-find-shortages-and-higher-prices-as-covid-impacted-supply-chains-shift-for-recovery-11623425796 Google Scholar

Goodman, J. K., Cryder, C. E., Cheema, A. A. (2013). Data collection in a flat world: Strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26 (July), 213–224.Google Scholar

Goodman, J. K., & Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210.Google Scholar

Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902.Google Scholar

Hauser, D., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., & Litman, L. (2021). Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk. https://psyarxiv.com/48yxj/Google Scholar

Hauser, D. J., Paolacci, G., & Chandler, J. (2019). Common concerns with MTurk as a participant pool: Evidence and solutions. In Kardes, F. R., Herr, P. M., & Schwarz, N. (Eds.). Handbook of Research Methods in Consumer Psychology (pp. 319–337). Routledge.Google Scholar

Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407.Google Scholar

Hernandez, J. M. da, C., Wright, S. A., & Rodrigues, F. F. (2015). Attributes versus benefits: The role of construal levels and appeal type on the persuasiveness of marketing messages. Journal of Advertising, 44(3), 243–253.Google Scholar

Jacowitz, K. E., & Kahneman, D. (1995). Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21(11), 1161–1166.Google Scholar

Jacquemet, N., James, A. G., Luchini, S., Murphy, J. J., & Shogren, J. F. (2021). Do truth-telling oaths improve honesty in crowd-working? PLoS ONE, 16(1), e0244958.Google Scholar

Janiszewski, C., & van Osselaer, S. M. J. (2021). The benefits of candidly reporting consumer research. Journal of Consumer Psychology, 31(4), 633–646.Google Scholar

Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46(1), 141–155.Google Scholar

Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P. D., Jewell, R., & Winter, N. J. (2020). The shape of and solutions to the MTurk quality crisis. Political Science Research and Methods, 8(4), 614–629.Google Scholar

Kochhar, R. (2020, June 11). Unemployment rose higher in three months of COVID-19 than it did in two years of the Great Recession. Pew Research Center. www.pewresearch.org/fact-tank/2020/06/11/unemployment-rose-higher-in-three-months-of-covid-19-than-it-did-in-two-years-of-the-great-recession Google Scholar

Litman, L., Moss, A., Rosenzweig, C., & Robinson, J. (2021, January 28). Reply to MTurk, Prolific or panels? Choosing the right audience for online research. Available at SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3775075 Google Scholar

Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavioral Research Methods, 47, 519–528.Google Scholar

Loepp, E., & Kelly, J. T. (2020). Distinction without a difference? An assessment of MTurk Worker types. Research & Politics, 7(1), p. 1–8.Google Scholar

Lourenco, S. F., & Tasimi, A. (2020). No participant left behind: Conducting science during COVID-19. Trends in Cognitive Sciences, 24(8), 583–584.Google Scholar

Lu, L., Neale, N., Line, N. D., & Bonn, M. (2021). Improving data quality using Amazon Mechanical Turk through platform setup. Cornell Hospitality Quarterly, 19389655211025475.Google Scholar

Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014). Being a Turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing, Association for Computing Machinery, 224–235.Google Scholar

Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.Google Scholar

Moss, A., & Litman, L. (2018). After the bot scare: Understanding what’s been happening with data collection on MTurk and how to stop it. Retrieved February 4, 2019. www.cloudresearch.com/resources/blog/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it Google Scholar

Moss, A. J., Rosenzweig, C., Jaffe, S. N., Gautam, R., Robinson, J., & Litman, L. (2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms. https://doi.org/10.31234/osf.io/wr8ds Google Scholar

Moss, A. J., Rosenzweig, C., Robinson, J., & Litman, L. (2020). Demographic stability on Mechanical Turk despite COVID-19. Trends in Cognitive Sciences, 24(9), 678–680.Google Scholar

Mummolo, J., & Peterson, E. (2019). Demand effects in survey experiments: An empirical assessment. American Political Science Review, 113, 517–529.Google Scholar

Mutikani, L. (2021, September 8). U.S. job openings hit record high as employers struggle to find workers. Reuters, www.reuters.com/business/us-job-openings-rise-record-109-million-july-2021-09-08 Google Scholar

Palan, S., & Schitter, C. (2018). Prolific.ac – A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.Google Scholar

Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.Google Scholar

Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.Google Scholar

Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031.Google Scholar

Peyton, K., Huber, G. A., & Coppock, A. (2021). The generalizability of online experiments conducted during the COVID-19 pandemic. Journal of Experimental Political Science, 1–16.Google Scholar

Ramos, S. (2021, March 1). COVID-19’s impact felt by researchers. American Psychological Association, www.apa.org/science/leadership/students/covid-19-impact-researchers Google Scholar

Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLoS ONE, https://doi.org/10.1371/journal.pone.0226394 Google Scholar

Sandford, A. (2020, April 2). Coronavirus: Half of humanity on lockdown in 90 countries. Euronews. Archived from the original on May 19, 2020. Retrieved June 15, 2021. www.euronews.com/2020/04/02/coronavirus-in-europe-spain-s-death-toll-hits-10-000-after-record-950-new-deaths-in-24-hou#vuukle-comments-1065562 Google Scholar

Shapiro, D. N., Chandler, J., & Mueller, P. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.Google Scholar

Sheth, J. (2020). Impact of Covid-19 on consumer behavior: Will the old habits return or die? Journal of Business Research, 117, 280–283.Google Scholar

Simon, R. (2021, April 16). Covid-19’s toll on U.S. business? 200,000 Extra closures in pandemic’s first year. The Wall Street Journal, www.wsj.com/livecoverage/covid-2021-04-16/card/QKG4xgwU4HyCqMq6TiyI Google Scholar

Simmons, J., Nelson, L. D., & Simonsohn, U. (2021). Pre‐registration: Why and how. Journal of Consumer Psychology, 31(1), 151–162.Google Scholar

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547.Google Scholar

Stokel-Walker, C. (2018, August 10). Bots on Amazon’s Mechanical Turk are ruining psychology studies. New Scientist, www.newscientist.com/article/2176436-bots-on-amazons-mechanical-turk-are-ruining-psychology-studies Google Scholar

Su, I., & Ceci, S. (2021). “Zoom developmentalists”: Home-based videoconferencing developmental research during COVID-19. https://doi.org/10.31234/osf.io/nvdy6 Google Scholar

Wessling, K. S., Huber, J., Netzer, O. (2017). MTurk character misrepresentation: Assessment and solutions. Journal of Consumer Research, 44, 211–230.Google Scholar

Wright, S., Manolis, C., Brown, D., et al. (2012). Construal-level mind-sets and the perceived validity of marketing claims. Marketing Letters, 23(1), 253–261.Google Scholar

Wright, S. A., & Goodman, J. K. (2019). Mechanical Turk in consumer research: Perceptions and usage in marketing academia. In Kardes, F. R., Herr, P. M., & Schwarz, N. (Eds.). Handbook of Research Methods in Consumer Psychology (pp. 338–357). Routledge.Google Scholar

Yue, C., Gilbert, B., Zhu, Q., Kleeman, H., & Rea, S. C. (2020). Crowdsourcing as a tool for research: Methodological, fair, and political considerations. Bulletin of Science, Technology & Society, 40(3–4), 40–53.Google Scholar

Zhou, H., & Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111(4), 493–504.Google Scholar

Book contents

20 - MTurk and Online Panel Research

Summary

Keywords

Access options

References

Save book to Kindle

Save book to Dropbox

Save book to Google Drive