Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-jkksz Total loading time: 0 Render date: 2024-12-22T20:33:43.446Z Has data issue: false hasContentIssue false

2 - Mechanical Turk: A Versatile Tool in the Behavioral Scientist’s Toolkit

from Part I - Quantitative Data Collection Sources

Published online by Cambridge University Press:  12 December 2024

By
Aaron J. Moss ,
David Hauser ,
Cheskie Rosenzweig ,
Jonathan Robinson  and
Leib Litman
Edited by
John E. Edlund  and
Austin Lee Nichols
John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

Building a successful research career often requires being adept at the methods and tools of the time. For social and behavioral scientists today, that means navigating online participant platforms and the tools used to create online studies. In this chapter, we describe how Amazon’s Mechanical Turk (MTurk) can be leveraged as a source for participant recruitment. We provide a brief history of MTurk’s usage by researchers, describe the challenges researchers have faced with the site, and summarize the status of issues like data quality, sample representativeness, and ethics in online research. Along the way, we provide tips for how researchers can use MTurk to collect high-quality data and to start and advance a research career.

Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 2: Performing Research
 , pp. 24 - 47
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47(4), 823–837. https://doi.org/10.1177/0149206320969787CrossRefGoogle Scholar
Ahler, D. J., Roush, C. E., & Sood, G. (2021). The micro-task market for lemons: Data quality on Amazon’s Mechanical Turk. Political Science Research and Methods, 1–20. https://doi.org/10.1017/psrm.2021.57CrossRefGoogle Scholar
American National Election Studies. (2020). The ANES Guide to Public Opinion and Electoral Behavior. https://electionstudies.org/data-tools/anes-guide/anes-guide.html?chart=lib_con_identification_7_ptGoogle Scholar
Arechar, A. A., Gächter, S., & Molleman, L. (2018). Conducting interactive experiments online. Experimental Economics, 21(1), 99–131. https://doi.org/10.1007/s10683-017-9527-2CrossRefGoogle ScholarPubMed
Arechar, A. A., Kraft-Todd, G. T., & Rand, D. G. (2017). Turking overtime: How participant characteristics and behavior vary over time and day on Amazon Mechanical Turk. Journal of the Economic Science Association, 3(1), 1–11. https://doi.org/10.1007/s40881-017-0035-0CrossRefGoogle ScholarPubMed
Bai, H. (2018, August 8). Evidence that a large amount of low quality responses on MTurk can be detected with repeated GPS coordinates. MaxHuiBai.com. www.maxhuibai.com/blog/evidence-that-responses-from-repeating-gps-are-randomGoogle Scholar
Barr, J. (2011, June 23). Get better results with Amazon Mechanical Turk Masters. Amazon Web Services. https://aws.amazon.com/blogs/aws/amazon-mechanical-turk-master-workersGoogle Scholar
Benndorf, V., Moellers, C., & Normann, H.-T. (2017). Experienced vs. inexperienced participants in the lab: Do they behave differently? Journal of the Economic Science Association, 3(1), 12–25. https://doi.org/10.1007/s40881-017-0036-zCrossRefGoogle Scholar
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368. https://doi.org/10.1093/pan/mpr057CrossRefGoogle Scholar
Berry, C., Kees, J., & Burton, S. (2022). Drivers of data quality in advertising research: Differences across MTurk and professional panel samples. Journal of Advertising, 51(4), 1–15. https://doi.org/10.1080/00913367.2022.2079026CrossRefGoogle Scholar
Bohannon, J. (2011). Social science for pennies. Science, 334(6054), 307. https://doi.org/10.1126/science.334.6054.307CrossRefGoogle ScholarPubMed
Boynton, M. H., & Richman, L. S. (2014). An online daily diary study of alcohol use using Amazon’s Mechanical Turk. Drug and Alcohol Review, 33(4), 456–461. https://doi.org/10.1111/dar.12163CrossRefGoogle ScholarPubMed
Buchanan, E. A., & Hvizdak, E. E. (2009). Online survey tools: Ethical and methodological concerns of human research ethics committees. Journal of Empirical Research on Human Research Ethics, 4(2), 37–48. https://doi.org/10.1525/jer.2009.4.2.37CrossRefGoogle ScholarPubMed
Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. https://doi.org/10.1177/1745691610393980CrossRefGoogle Scholar
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on Psychological Science, 13(2), 149–154. https://doi.org/10.1177/1745691617706516CrossRefGoogle ScholarPubMed
Burleigh, T., Kennedy, R., & Clifford, S. (2018). How to screen out VPS and international respondents using qualtrics: A protocol. SSRN, 3265459. https://papers.ssrn.com/abstract=3265459CrossRefGoogle Scholar
Campbell, D. S., & Reiman, A.-K. (2022). Has social psychology lost touch with reality? Exploring public perceptions of the realism and consequentiality of social psychological research. Journal of Experimental Social Psychology, 98, 104255. https://doi.org/10.1016/j.jesp.2021.104255CrossRefGoogle Scholar
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017). Intertemporal differences among MTurk workers: Time-based sample variations and implications for online data collection. SAGE Open, 7(2), 215824401771277. https://doi.org/10.1177/2158244017712774CrossRefGoogle Scholar
Chandler, J. J., & Paolacci, G. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8(5), 500–508. https://doi.org/10.1177/1948550617698203CrossRefGoogle Scholar
Chandler, J. J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139. https://doi.org/10.1177/0956797615585115CrossRefGoogle ScholarPubMed
Chandler, J. J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022–2038. https://doi.org/10.3758/s13428-019-01273-7CrossRefGoogle ScholarPubMed
Chandler, J., Sisso, I., & Shapiro, D. (2020). Participant carelessness and fraud: Consequences for clinical research and potential solutions. Journal of Abnormal Psychology, 129(1), 49–55. https://doi.org/10.1037/abn0000479CrossRefGoogle ScholarPubMed
Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464–473. https://doi.org/10.1177/1948550619875149CrossRefGoogle Scholar
Clifford, S., & Jerit, J. (2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1(2), 120–131. https://doi.org/10.1017/xps.2014.5CrossRefGoogle Scholar
Converse, B. A., & Epley, N. (2007). With God on our side. TESS: Time-Sharing Experiments for the Social Sciences. www.tessexperiments.org/study/converse561Google Scholar
Coppock, A. (2019). Generalizing from survey experiments conducted on Mechanical Turk: A replication approach. Political Science Research and Methods, 7(3), 613–628. https://doi.org/10.1017/psrm.2018.10CrossRefGoogle Scholar
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLOS ONE, 8(3), e57410. https://doi.org/10.1371/journal.pone.0057410CrossRefGoogle Scholar
Dennis, J. M. (2001). Are internet panels creating professional respondents? Marketing Research, 13(2), 34–38.Google Scholar
Dennis, S. A., Goodson, B. M., & Pearson, C. (2020). Online worker fraud and evolving threats to the integrity of MTurk data: A discussion of virtual private servers and the limitations of IP-based screening procedures. Behavioral Research in Accounting, 32(1), 119–134.CrossRefGoogle Scholar
Difallah, D., Filatova, E., & Ipeirotis, P. (2018). Demographics and dynamics of Mechanical Turk workers. In Chang, Y. (ed.) Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (pp. 135–143). ACM. https://doi.org/10.1145/3159652.3159661CrossRefGoogle Scholar
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010, April). Are your participants gaming the system? Screening Mechanical Turk workers. In Kaye, J. (ed.), Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 2399–2402). ACM.Google Scholar
Edlund, J. E., Lange, K. M., Sevene, A. M., Umansky, J., Beck, C. D., & Bell, D. J. (2017). Participant crosstalk: Issues when using the Mechanical Turk. Tutorials in Quantitative Methods for Psychology, 13(3), 174–182. http://dx.doi.org/10.20982/tqmp.13.3.p174CrossRefGoogle Scholar
Edlund, J. E., Nichols, A. L., Okdie, B. M., Guadagno, R. E., Eno, C. A., Heider, J. D., et al. (2014). The prevalence and prevention of crosstalk: A multi-institutional study. Journal of Social Psychology, 154(3), 181–185. https://doi.org/10.1080/00224545.2013.872596CrossRefGoogle ScholarPubMed
Enyon, R., Fry, J., & Schroeder, R. (2016). The ethics of online research. In Fielding, N. G., Lee, R. M., & Blank, G. (eds.), The SAGE Handbook of Online Research Methods (pp. 19–37). SAGE Publications.Google Scholar
Fair Crowd Work. (n.d.). Amazon Mechanical Turk. http://faircrowd.work/platform/amazon-mechanical-turk (retrieved June 9, 2022).Google Scholar
Fordsham, N., Moss, A. J., Krumholtz, S., Roggina, T., Robinson, J., & Litman, L. (2019). Variation among Mechanical Turk Workers across time of day presents an opportunity and a challenge for research [preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/p8bnsCrossRefGoogle Scholar
Fort, K., Adda, G., & Bretonnel Cohen, K. (2011). Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37(2), 413–420. https://doi.org/10.1162/COLI_a_00057CrossRefGoogle Scholar
Gallo, J., & Gran-Ruaz, S. (2021, October 1). Racial trauma scale: Creative interview strategies employed in the development of a new clinical tool for measuring race-based stress and trauma. CloudResearch Innovations in Online Research Conference (virtual). www.youtube.com/watch?v=hKCK_dWfUfI&t=1568sGoogle Scholar
Gallup News. (2007, September 20). Party affiliation. https://news.gallup.com/poll/15370/Party-Affiliation.aspxGoogle Scholar
Garbinsky, E. N., Gladstone, J. J., Nikolova, H., & Olson, J. G. (2020). Love, lies, and money: Financial infidelity in romantic relationships. Journal of Consumer Research, 47(1), 1–24. https://doi.org/10.1093/jcr/ucz052CrossRefGoogle Scholar
Gray, M. L., & Suri, S. (2019). Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt.Google Scholar
Haggbloom, S. J., Warnick, R., Warnick, J. E., Jones, V. K., Yarbrough, G. L., Russell, T. M., et al. (2002). The 100 most eminent psychologists of the 20th century. Review of General Psychology, 6(2), 139–152. https://doi.org/10.1037/1089-2680.6.2.139CrossRefGoogle Scholar
Hall, M. P., Lewis, N. A. Jr., Chandler, J., & Litman, L. (2020). Conducting longitudinal research on Amazon Mechanical Turk. In Litman, L. & Robinson, J. (eds.), Conducting Online Research on Amazon Mechanical Turk and Beyond (pp. 198–216). SAGE Publications.Google Scholar
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018). A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1–14). ACM. https://doi.org/10.1145/3173574.3174023Google Scholar
Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S., & Robinson, J. (2022). Evaluating CloudResearch’s approved group as a solution for problematic data quality on MTurk. Behavioral Research Methods, 55(8), 3953–3964. https://doi.org/10.31234/osf.io/48yxjCrossRefGoogle Scholar
Hauser, D. J., Paolacci, G., & Chandler, J. (2019). Common concerns with MTurk as a participant pool: Evidence and solutions. In Kardes, F. R., Herr, P. M., & Schwarz, N. (eds.), Handbook of Research Methods in Consumer Psychology (pp. 319–337). Routledge.Google Scholar
Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. https://doi.org/10.3758/s13428-015-0578-zCrossRefGoogle ScholarPubMed
Hillyguys, D. S., Jackson, N., & Young, M. (2014). Professional respondents in non-probability online panels. In Callegaro, M. et al. (eds.), Online Panel Research: A Data Quality Perspective, 1st ed. (pp. 219–237). Wiley.CrossRefGoogle Scholar
Hitlin, P. (2016). Research in the crowdsourcing age, a case study. Pew Research Center. www.pewresearch.org/internet/2016/07/11/what-is-mechanical-turkGoogle Scholar
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. https://doi.org/10.1007/s10683-011-9273-9CrossRefGoogle Scholar
Huff, C., & Tingley, D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics, 2(3), 205316801560464. https://doi.org/10.1177/2053168015604648CrossRefGoogle Scholar
Kaplan, T., Saito, S., Hara, K., & Bigham, J. P. (2018, June 15). Striving to earn more: A survey of work strategies and tool use among crowd workers. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 6, 70–78. https://doi.org/10.1609/hcomp.v6i1.13327CrossRefGoogle Scholar
Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46(1), 141–155. https://doi.org/10.1080/00913367.2016.1269304CrossRefGoogle Scholar
Keller, L., Kabengele, M.-C., & Gollwitzer, P. M. (2021). The self-regulation of face touching – a preregistered experiment testing if-then plans as a means to promote COVID-19 prevention. Psychology & Health, 38(8), 1–19. https://doi.org/10.1080/08870446.2021.2005793Google ScholarPubMed
Kennedy, R., Clifford, S., Burleigh, T., Waggoner, P., & Jewell, R. (2018). How Venezuela’s economic crisis is undermining social science research – about everything. Washington Post, November 7, www.washingtonpost.com/news/monkey-cage/wp/2018/11/07/how-the-venezuelan-economic-crisis-is-undermining-social-science-research-about-everything-not-just-venezuela (retrieved May 27, 2022).Google Scholar
Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing user studies with Mechanical Turk. CHI ’08: Proceedings of the Twenty-Sixth Annual CHI Conference on Human Factors in Computing Systems (pp. 453–456). ACM. https://doi.org/10.1145/1357054.1357127CrossRefGoogle Scholar
Krupnikov, Y., & Levine, A. S. (2014). Cross-sample comparisons and external validity. Journal of Experimental Political Science, 1(1), 59–80. https://doi.org/10.1017/xps.2014.7CrossRefGoogle Scholar
Levay, K. E., Freese, J., & Druckman, J. N. (2016). The demographic and political composition of Mechanical Turk samples. SAGE Open, 6(1), 215824401663643. https://doi.org/10.1177/2158244016636433CrossRefGoogle Scholar
Litman, L. (2019, May). Composition of Online Participant Pools Moderates Effect Sizes of Experimental Manipulations. Association for Psychological Science.Google Scholar
Litman, L., & Robinson, J. (2020a). Conducting ethical online research: A data-driven approach. In Litman, L. & Robinson, J. (eds.), Conducting Online Research on Amazon Mechanical Turk and Beyond (pp. 234–263). SAGE Publications.Google Scholar
Litman, L., & Robinson, J. (2020b). Introduction. In Litman, L. & Robinson, J. (eds.), Conducting Online Research on Amazon Mechanical Turk and Beyond (pp. 1–26). SAGE Publications.Google Scholar
Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433–442. https://doi.org/10.3758/s13428-016-0727-zCrossRefGoogle ScholarPubMed
Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behavior Research Methods, 47(2), 519–528. https://doi.org/10.3758/s13428-014-0483-xCrossRefGoogle ScholarPubMed
Litman, L., Rosenzweig, C., Jaffe, S. N., Gautam, R., Robinson, J., & Moss, A. J. (2021). Bots or inattentive humans? Identifying sources of low-quality data in online platforms [preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/wr8dsCrossRefGoogle Scholar
Litman, L., Rosenzweig, C., & Moss, A. J. (2020, July 15). New solutions dramatically improve research data quality on MTurk. CloudResearch [blog]. www.cloudresearch.com/resources/blog/new-tools-improve-research-data-quality-mturkGoogle Scholar
Loepp, E., & Kelly, J. T. (2020). Distinction without a difference? An assessment of MTurk worker types. Research & Politics, 7(1), 205316801990118. https://doi.org/10.1177/2053168019901185CrossRefGoogle Scholar
Matsakis, L. (2018, March 22). A window into how YouTube trains AI to moderate videos. Wired. www.wired.com/story/youtube-mechanical-turk-content-moderation-aiGoogle Scholar
Matthijsse, S. M., de Leeuw, E. D., & Hox, J. J. (2015). Internet panels, professional respondents, and data quality. Methodology, 11(3), 81–88.CrossRefGoogle Scholar
McCredie, M. N., & Morey, L. C. (2019). Who are the Turkers? A characterization of MTurk workers using the Personality Assessment Inventory. Assessment, 26(5), 759–766. https://doi.org/10.1177/1073191118760709CrossRefGoogle ScholarPubMed
Mehrotra, D. (2020). Horror Stories From Inside Amazon’s Mechanical Turk. Gizmodo, January 28. https://gizmodo.com/horror-stories-from-inside-amazons-mechanical-turk-1840878041Google Scholar
Meyers, E. A., Walker, A. C., Fugelsang, J. A., & Koehler, D. J. (2020). Reducing the number of non-naïve participants in Mechanical Turk samples. Methods in Psychology, 3, 100032. https://doi.org/10.1016/j.metip.2020.100032CrossRefGoogle Scholar
Miller, J. D., Crowe, M., Weiss, B., Maples-Keller, J. L., & Lynam, D. R. (2017). Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon’s Mechanical Turk. Personality Disorders: Theory, Research, and Treatment, 8(1), 26–34. https://doi.org/10.1037/per0000191CrossRefGoogle ScholarPubMed
Moss, A. J. (2021, December 1). Five years of Mechanical Turk data in five figures. CloudResearch [blog]. www.cloudresearch.com/resources/blog/mechanical-turk-data-five-years-in-five-figuresGoogle Scholar
Moss, A. J. (2022, March 4). How CloudResearch and IARPA completed the largest longitudinal online research project ever. CloudResearch [blog]. www.cloudresearch.com/resources/blog/the-largest-longitudinal-online-research-projectGoogle Scholar
Moss, A. J., Rosenzweig, C., Robinson, J., Jaffe, S. N., & Litman, L. (2023). Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. Behavior Research Methods, 55(8), 4048–4067.CrossRefGoogle ScholarPubMed
Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138. https://doi.org/10.1017/XPS.2015.19CrossRefGoogle Scholar
Newman, A. (2019, November 15). I found work on an Amazon Website. I made 97 cents an hour. New York Times. www.nytimes.com/interactive/2019/11/15/nyregion/amazon-mechanical-turk.htmlGoogle Scholar
Nichols, A. L., & Edlund, J. E. (2020). Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants. International Journal of Social Research Methodology, 23(6), 625–638. https://doi.org/10.1080/13645579.2020.1719618CrossRefGoogle Scholar
Ogletree, A. M., & Katz, B. (2021). How do older adults recruited using MTurk differ from those in a national probability sample? International Journal of Aging and Human Development, 93(2), 700–721. https://doi.org/10.1177/0091415020940197CrossRefGoogle Scholar
Ophir, Y., Sisso, I., Asterhan, C. S. C., Tikochinski, R., & Reichart, R. (2020). The Turker blues: Hidden factors behind increased depression rates among Amazon’s Mechanical Turkers. Clinical Psychological Science, 8(1), 65–83. https://doi.org/10.1177/2167702619865973CrossRefGoogle Scholar
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. https://doi.org/10.1177/0963721414531598CrossRefGoogle Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. SSRN, 1626226. https://papers.ssrn.com/abstract=1626226CrossRefGoogle Scholar
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031. https://doi.org/10.3758/s13428-013-0434-yCrossRefGoogle ScholarPubMed
Pontin, J. (2007, March 25). Artificial intelligence, with help from the humans. New York Times. www.nytimes.com/2007/03/25/business/yourmoney/25Stream.htmlGoogle Scholar
Rand, D. G. (2018). Non-naïvety may reduce the effect of intuition manipulations. Nature Human Behaviour, 2(9), 602. https://doi.org/10.1038/s41562-018-0404-6CrossRefGoogle ScholarPubMed
Rivera, E. D., Wilkowski, B. M., Moss, A. J., Rosenzweig, C., & Litman, L. (2022). Assessing the efficacy of a participant-vetting procedure to improve data-quality on Amazon’s Mechanical Turk. Methodology, 18(2), 126–143. https://doi.org/10.5964/meth.8331CrossRefGoogle Scholar
Robinson, J., & Litman, L. (2020). Conducting a study on Mechanical Turk. In Litman, L. & Robinson, J. (eds.), Conducting Online Research on Amazon Mechanical Turk and Beyond (pp. 49–78). SAGE Publications.Google Scholar
Robinson, J., Litman, L., & Rosenzweig, C. (2020). Who are the Mechanical Turk workers? In Litman, L. & Robinson, J. (eds.), Conducting Online Research on Amazon Mechanical Turk and Beyond (pp. 121–147). SAGE Publications.Google Scholar
Robinson, J., Rosenzweig, C., Moss, A. J., & Litman, L. (2019). Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLOS ONE, 14(12), e0226394. https://doi.org/10.1371/journal.pone.0226394CrossRefGoogle ScholarPubMed
Rouse, S. V. (2020). Reliability of MTurk data from masters and workers. Journal of Individual Differences, 41(1), 30–36. https://doi.org/10.1027/1614-0001/a000300CrossRefGoogle Scholar
Ryan, T. J. (2018, August 12). Data contamination on Mturk [blog]. https://timryan.web.unc.edu/2018/08/12/data-contamination-on-mturkGoogle Scholar
Semuels, A. (2018, January 23). The internet is enabling a new kind of poorly paid hell. The Atlantic. www.theatlantic.com/business/archive/2018/01/amazon-mechanical-turk/551192Google Scholar
Simonton, D. K. (2000). Methodological and theoretical orientation and the long-term disciplinary impact of 54 eminent psychologists. Review of General Psychology, 4(1), 13–24. https://doi.org/10.1037/1089-2680.4.1.13CrossRefGoogle Scholar
Stagnaro, M., Pennycook, G., & Rand, D. G. (2018). Performance on the cognitive reflection test is stable across time. SSRN, 3115809. https://papers.ssrn.com/abstract=3115809CrossRefGoogle Scholar
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.CrossRefGoogle Scholar
Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99–113.CrossRefGoogle Scholar
TurkerView. (2019, November 18). Writer who never learned to drive works for Uber. Makes $0.97/hr. https://blog.turkerview.com/writer-who-never-learned-to-drive-works-for-uberGoogle Scholar
US Census Bureau. (2020, February 5). Marital status in the United States. www.census.gov/library/visualizations/interactive/marital-status-in-united-states.htmlGoogle Scholar
US Census Bureau. (2021, November 29). Census Bureau releases new estimates on America’s families and living arrangements. www.census.gov/newsroom/press-releases/2021/families-and-living-arrangements.htmlGoogle Scholar
US Census Bureau. (2022, February 24). Census Bureau releases new educational attainment data. www.census.gov/newsroom/press-releases/2022/educational-attainment.htmlGoogle Scholar
US Census Bureau. (n.d.). QuickFacts: United States. www.census.gov/quickfacts/fact/table/US/POP010220Google Scholar
Walters, K., Christakis, D. A., & Wright, D. R. (2018). Are Mechanical Turk worker samples representative of health status and health behaviors in the U.S.? PLOS ONE, 13(6), e0198835. https://doi.org/10.1371/journal.pone.0198835CrossRefGoogle ScholarPubMed
Wessling, K., Huber, J., & Netzer, O. (2017). MTurk character misrepresentation: Assessment and solutions. Journal of Consumer Research, 44(1), 211–230. https://doi.org/10.1093/jcr/ucx053CrossRefGoogle Scholar
Williamson, V. (2016). On the ethics of crowdsourced research. PS: Political Science & Politics, 49(01), 77–81. https://doi.org/10.1017/S104909651500116XGoogle Scholar
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2018). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25(5), 1968–1972. https://doi.org/10.3758/s13423-017-1348-yCrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×