Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-mkpzs Total loading time: 0 Render date: 2024-12-22T19:50:05.909Z Has data issue: false hasContentIssue false

11 - Conducting Surveys and Experiments on the Internet

from Part III - Deep Dives on Methods and Tools for Testing Your Question of Interest

Published online by Cambridge University Press:  12 December 2024

By
Chadly Stern  and
Jordan R. Axt
Edited by
Harry T. Reis ,
Tessa West  and
Charles M. Judd
Harry T. Reis
Affiliation:
University of Rochester, New York
Tessa West
Affiliation:
New York University
Charles M. Judd
Affiliation:
University of Colorado Boulder
Get access

Summary

Social and personality psychologists have conducted surveys and experiments online for nearly twenty-five years. Researchers have used the Internet to ask questions about a wide range of topics, including racial bias, personality development, and attitude change. The frequency of conducting internet research has increased over time and understanding how to conduct online research has become a critical skill for psychologists. This chapter provides a general introduction to conducting survey and experimental research online. We outline how researchers can host and program internet studies, as well as their options for recruiting participant samples. We also cover important issues that researchers should consider about data quality, representativeness, generalizability, and upholding ethical standards. Throughout the chapter we discuss practices and guidelines that we view as optimal at the current time, and direct readers to additional literature that can further inform their thinking.

Keywords

internet researchhosting studiesprogramming studiesparticipant recruitmentrepresentativenessgeneralizabilityresearch ethics
Type
Chapter
Information
Handbook of Research Methods in Social and Personality Psychology , pp. 244 - 265
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, C. A., Allen, J. J., Plante, C., Quigley-McBride, A., Lovett, A., and Rokkum, J. N. (2019). The MTurkification of social and personality psychology. Personality and Social Psychology Bulletin, 45(6), 842–850.CrossRefGoogle ScholarPubMed
Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.CrossRefGoogle ScholarPubMed
Bialek, M., and Pennycook, G. (2018). The cognitive reflection test is robust to multiple exposures. Behavior Research Methods, 50(5), 1953–1959.CrossRefGoogle ScholarPubMed
Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832.CrossRefGoogle ScholarPubMed
Bosnjak, M., Fiebach, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., and Sokol- Chang, R. I. (2022). A template for preregistration of quantitative research in psychology: Report of the joint psychological societies preregistration task force. American Psychologist, 77, 602–615.CrossRefGoogle ScholarPubMed
Brock, R. L., Barry, R. A., Lawrence, E., Dey, J., and Rolffs, J. (2012). Internet administration of paper-and-pencil questionnaires used in couple research: Assessing psychometric equivalence. Assessment, 19(2), 226–242.CrossRefGoogle ScholarPubMed
Buhrmester, M. D., Kwang, T., and Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5.CrossRefGoogle Scholar
Chandler, J. J., and Paolacci, G. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8(5), 500–508.CrossRefGoogle Scholar
Chandler, J., Paolacci, G., Peer, E., Mueller, P., and Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.CrossRefGoogle ScholarPubMed
Chandler, J., Rosenzweig, C., Moss, A. J., Robinson, J., and Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022–2038.CrossRefGoogle ScholarPubMed
Clifford, S., Jewell, R. M., and Waggoner, P. D. (2015). Are samples drawn from Mechanical Turk valid for research on political ideology? Research & Politics, 2(4), 2053168015622072, https://doi.org/10.1177%2F2053168015622072.CrossRefGoogle Scholar
Coppock, A. (2019). Generalizing from survey experiments conducted on Mechanical Turk: A replication approach. Political Science Research and Methods, 7(3), 613–628.CrossRefGoogle Scholar
Coppock, A., Leeper, T. J., and Mullinix, K. J. (2018). Generalizability of heterogeneous treatment effect estimates across samples. Proceedings of the National Academy of Sciences, 115(49), 12441–12446.CrossRefGoogle ScholarPubMed
Crump, M. J., McDonnell, J. V., and Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLOS ONE, 8(3), e57410, https://doi.org/10.1371/journal.pone.0057410.CrossRefGoogle Scholar
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.CrossRefGoogle Scholar
Fraley, R. C. (2004). How to Conduct Behavioral Research over the Internet: A Beginner’s Guide to HTML and CGI/Perl. Guilford Press.Google Scholar
Gosling, S. D., Vazire, S., Srivastava, S., and John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104.CrossRefGoogle ScholarPubMed
Hauser, D. J., and Schwarz, N. (2015). It’s a trap! Instructional manipulation checks prompt systematic thinking on “tricky” tasks. Sage Open, 5(2), 2158244015584617, https://doi.org/10.1177%2F2158244015584617.CrossRefGoogle Scholar
Hauser, D. J., and Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407.CrossRefGoogle ScholarPubMed
Hawkins, R. X. (2015). Conducting real-time multiplayer experiments on the web. Behavior Research Methods, 47(4), 966–976.CrossRefGoogle ScholarPubMed
Henrich, J., Heine, S. J., and Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.CrossRefGoogle ScholarPubMed
Ilagan, M. J., and Falk, C. F. (in press). Supervised classes, unsupervised mixing proportions: Detection of bots in a Likert-type questionnaire. Educational and Psychological Measurement, https://doi.org/10.1177/00131644221104220.CrossRefGoogle Scholar
Kees, J., Berry, C., Burton, S., and Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46(1), 141–155.CrossRefGoogle Scholar
Krantz, J. H., and Dalal, R. (2000). Validity of Web-based psychological research. In Birnbaum, M. H. (ed.) Psychological Experiments on the Internet. Academic Press.Google Scholar
Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., and Couper, M. (2004). Psychological research online: Report of Board of Scientific Affairs’ Advisory Group on the Conduct of Research on the Internet. American Psychologist, 59(2), 105–117.CrossRefGoogle Scholar
Litman, L., and Robinson, J. (2021). Conducting Online Research on Amazon Mechanical Turk and Beyond. Sage Academic Publishing.CrossRefGoogle Scholar
Mason, W., and Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23.CrossRefGoogle ScholarPubMed
Moss, A., and Litman, L. (2018). After the Bot Scare: Understanding What’s Been Happening with Data Collection on MTurk and How to Stop It, www.cloudresearch.com/resources/blog/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it.Google Scholar
Necka, E. A., Cacioppo, S., Norman, G. J., and Cacioppo, J. T. (2016). Measuring the prevalence of problematic respondent behaviors among MTurk, campus, and community participants. PLOS ONE, 11(6), e0157732, https://doi.org/10.1371/journal.pone.0157732.CrossRefGoogle ScholarPubMed
Nosek, B. A., Banaji, M. R., and Greenwald, A. G. (2002). E‐research: Ethics, security, design, and control in psychological research on the Internet. Journal of Social Issues, 58(1), 161–176.CrossRefGoogle Scholar
Nosek, B. A., Sriram, N., and Umansky, E. (2012). Presenting survey items one at a time compared to all at once decreases missing data without sacrificing validity in research with internet volunteers. PLOS ONE, 7(5), e36771, https://doi.org/10.1371/journal.pone.0036771.CrossRefGoogle Scholar
Oppenheimer, D. M., Meyvis, T., and Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.CrossRefGoogle Scholar
Palan, S., and Schitter, C. (2018). Prolific.ac: A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.CrossRefGoogle Scholar
Center, Pew Research (2021). Internet/Broadband Fact Sheet, www.pewresearch.org/internet/fact-sheet/internet-broadband/?menuItem=6ba9316e-006c-482d-be4b-69feb64c4be8.Google Scholar
Ping, H., Stoyanovich, J., and Howe, B. (2017). Datasynthesizer: Privacy-preserving synthetic datasets (June 2017). In Proceedings of the 29th International Conference on Scientific and Statistical Database Management, Association for Computing Machinery, New York.Google Scholar
Rhodes, M., Rizzo, M. T., Foster-Hanson, E., Moty, K., Leshin, R. A., Wang, M., … Ocampo, J. D. (2020). Advancing developmental science via unmoderated remote research with children. Journal of Cognition and Development, 21(4), 477–493.CrossRefGoogle ScholarPubMed
Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51(3), 515–530.CrossRefGoogle Scholar
Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.CrossRefGoogle ScholarPubMed
Skitka, L. J., and Sargis, E. G. (2006). The Internet as psychological laboratory. Annual Review of Psychology, 57, 529–555.CrossRefGoogle ScholarPubMed
Srivastava, S., John, O. P., Gosling, S. D., and Potter, J. (2003). Development of personality in early and middle adulthood: Set like plaster or persistent change? Journal of Personality and Social Psychology, 84, 1041–1053.CrossRefGoogle ScholarPubMed
Summerville, A., and Chartier, C. R. (2013). Pseudo-dyadic “interaction” on Amazon’s Mechanical Turk. Behavior Research Methods, 45(1), 116–124.CrossRefGoogle ScholarPubMed
Thai, S., and Page-Gould, E. (2018). ExperienceSampler: An open-source scaffold for building smartphone apps for experience sampling. Psychological Methods, 23(4), 729–739.CrossRefGoogle ScholarPubMed
Tissera, H., Mignault, M. C., and Human, L. J. (2023). “Zooming” in on positive and accurate metaperceptions in first impressions: Examining the links with social anxiety and liking in online video interactions. Journal of Personality and Social Psychology, 125(4), 852–873.CrossRefGoogle Scholar
Zhou, H., and Fishbach, A. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111(4), 493–504.CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×