Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-22T20:45:34.704Z Has data issue: false hasContentIssue false

4 - Prolific: Crowdsourcing Academic Online Research

from Part I - Quantitative Data Collection Sources

Published online by Cambridge University Press:  12 December 2024

By
Eyal Peer
Edited by
John E. Edlund  and
Austin Lee Nichols
John E. Edlund
Affiliation:
Rochester Institute of Technology, New York
Austin Lee Nichols
Affiliation:
Central European University, Vienna
Get access

Summary

Prolific is a website that offers researchers the ability to recruit and sample participants for online research. In contrast to earlier crowdsourcing platforms, such as Amazon Mechanical Turk (MTurk), it focuses primarily on academic and marketing research – typically done through online surveys and experiments. In this chapter, I aim to introduce this platform to researchers conducting online studies and to provide knowledge and practical advice on how to best use the platform for online research. The review includes explanations of how the site works, the composition of its pool of participants, the options available to researchers for sampling and recruiting participants online, how to achieve advanced abilities by connecting Prolific to research software (e.g., Qualtrics, Gorilla), and how to ensure high data quality when using Prolific. I then review the evidence on the current state of data quality on Prolific, suggesting that it can provide higher data quality than MTurk and also better than some commercial panels. I conclude with a summary of the advantages and disadvantages of using Prolific for online research and potential future developments in the platform that could promote more credible online research.

Type
Chapter
Information
The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
Volume 2: Performing Research
 , pp. 72 - 92
Publisher: Cambridge University Press
Print publication year: 2024

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self‐administered surveys. American Journal of Political Science, 58(3), 739–753.CrossRefGoogle Scholar
Chandler, J. J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130.CrossRefGoogle ScholarPubMed
Chandler, J. J., & Paolacci, G. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8(5), 500–508.CrossRefGoogle Scholar
Chandler, J. J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.CrossRefGoogle ScholarPubMed
Chandler, J. J., Rosenzweig, C., Moss, A. J., Robinson, J., & Litman, L. (2019). Online panels in social science research: Expanding sampling methods beyond Mechanical Turk. Behavior Research Methods, 51(5), 2022–2038.CrossRefGoogle ScholarPubMed
Douglas, B. D., Ewell, P. J., & Brauer, M. (2023). Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. PLOS ONE, 18(3), e0279720.CrossRefGoogle ScholarPubMed
Gerlach, P., Teodorescu, K., & Hertwig, R. (2019). The truth about lies: A meta-analysis on dishonest behavior. Psychological Bulletin, 145(1), 1–44.CrossRefGoogle Scholar
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.CrossRefGoogle Scholar
Goodman, J. K., & Paolacci, G. (2017). Crowdsourcing consumer research. Journal of Consumer Research, 44(1), 196–210.CrossRefGoogle Scholar
Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S., & Robinson, J. (2022). Evaluating CloudResearch’s approved group as a solution for problematic data quality on MTurk. Behavioral Research Methods, 55(8), 3953–3964. https://doi.org/10.31234/osf.io/48yxjCrossRefGoogle Scholar
Nichols, A. L., & Edlund, J. E. (2020). Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants. International Journal of Social Research Methodology, 23(6), 625-638.CrossRefGoogle Scholar
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.CrossRefGoogle Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision making, 5(5), 411–419.CrossRefGoogle Scholar
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.CrossRefGoogle Scholar
Peer, E., Rothschild, D., & Gordon, A. (2023). Behavioral Lab 3.0. Towards the next generation of online behavioral research [preprint]. PsyArXiv.Google Scholar
Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2022). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54, 1643–1662. https://doi.org/10.3758/s13428-021-01694-3CrossRefGoogle ScholarPubMed
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46(4), 1023–1031.CrossRefGoogle ScholarPubMed
Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.CrossRefGoogle Scholar
Thomas, K. A., & Clifford, S. (2017). Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77, 184–197.CrossRefGoogle Scholar
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453–458.CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×