Hostname: page-component-cd9895bd7-q99xh Total loading time: 0 Render date: 2024-12-22T10:51:54.008Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing the Correspondence between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research*

Published online by Cambridge University Press:  04 August 2014

Alexander Coppock  and
Donald P. Green
Get access Rights & Permissions [Opens in a new window]

Abstract

A small but growing social science literature examines the correspondence between experimental results obtained in lab and field settings. This article reviews this literature and reanalyzes a set of recent experiments carried out in parallel in both the lab and field. Using a standardized format that calls attention to both the experimental estimates and the statistical uncertainty surrounding them, the study analyzes the overall pattern of lab-field correspondence, which is found to be quite strong (Spearman's ρ = 0.73). Recognizing that this correlation may be distorted by the ad hoc manner in which lab-field comparisons are constructed (as well as the selective manner in which results are reported and published), the article concludes by suggesting directions for future research, stressing in particular the need for more systematic investigation of treatment effect heterogeneity.

Type
Original Articles
Information
Political Science Research and Methods , Volume 3 , Issue 1 , January 2015 , pp. 113 - 131
Copyright
Copyright © The European Political Science Association 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*Alexander Coppock is a PhD student, Department of Political Science, Columbia University, 420 West 118th Street, Suite 710, New York, NY 10025 ([email protected]). Donald P. Green is Professor of Political Science, Columbia University, 420 West 118th Street, Suite 710, New York, NY 10025 ([email protected]). We are grateful to Oliver Armantier, Peter Aronow, Lindsay Dolan, Uri Gneezy, John List, Jennifer Jerit, Andrej Tusicisny and two anonymous reviewers for helpful comments, Glenn Harrison and Nicholas Valentino for graciously providing replication data, and to Johannes Abeler and Ido Erev for providing supplementary information. To view supplementary material for this article, please visit http://dx.doi.org/10.1017/psrm.2014.10

References

Abeler, J., Marklein, F.. 2013. ‘Fungibility, Labels, and Consumption’. Unpublished manuscript.Google Scholar
Arceneaux, K.Nickerson, D.W.. 2009. ‘Who Is Mobilized to Vote? A ReAnalysis of 11 Field Experiments’. American Journal of Political Science 53(1):1–16.Google Scholar
Armantier, O.Boly, A.. 2013. ‘Comparing Corruption in the Laboratory and in the Field in Burkina Faso and in Canada’. The Economic Journal 123(573):1168–1187.CrossRefGoogle Scholar
Ashraf, N., Berry, J.Shapiro, J.M.. 2010. ‘Can Higher Prices Stimulate Product Use? Evidence from a Field Experiment in Zambia’. American Economic Review 100(December):2383–2413.Google Scholar
Baird, S., Ferreira, F., Özler, B., Woolcock, M.. 2013. ‘Relative Effectiveness of Conditional and Unconditional Cash Transfers for Schooling Outcomes in Developing Countries: A Systematic Review’. Technical report, Campbell Systematic Reviews.Google Scholar
Banerjee, A., Kumar, S., Pande, R., Su, F.. 2010. ‘Do Informed Voters Make Better Choices? Experimental Evidence from Urban India’. Unpublished manuscript.Google Scholar
Benz, M.Meier, S.. 2008. Do People Behave in Experiments as in the Field? Evidence from Donations. Experimental Economics 11(3):268–281.Google Scholar
Bond, R.Smith, P.B.. 1996. ‘Culture and Conformity: A Meta-analysis of Studies Using Asch's (1952b, 1956) Line Judgment Task’. Psychological Bulletin 119(1):111–137.Google Scholar
Bornstein, G., Erev, I.Rosen, O.. 1990. ‘Intergroup Competition as a Structural Solution to Social Dilemmas’. Social Behaviour 5(4):247–260.Google Scholar
Camerer, C.F. forthcoming. ‘The Promise and Success of Lab-Field Generalizability in Experimental Economics: A Critical Reply to Levitt and List’. In G. Frechette and A. Schotter (eds), Methods of Modern Experimental Economics. Oxford: Oxford University Press.Google Scholar
Castro, L., Scartascini, C.. 2013. ‘Tax Compliance and Enforcement in the Pampas: Evidence from a Field Experiment’. Washington, DC: Inter-American Development Bank, 2013.Google Scholar
Charness, G., Schram, A.. 2013. ‘Social and Moral Norms in Allocation Choices in the Laboratory’. Unpublished manuscript.Google Scholar
Chong, A., De La O, A.L., Karlan, D.S., Wantchekon, L.. 2010. ‘Information Dissemination and Local Governments Electoral Returns, Evidence from a Field Experiment in Mexico’. Unpublished manuscript.Google Scholar
Cohen, J.Dupas, P.. 2010. ‘Free Distribution or Cost-sharing? Evidence from a Randomized Malaria Prevention Experiment’. Quarterly Journal of Economics 125(1):1–45.CrossRefGoogle Scholar
Cook, T.D.Campbell, D.T.. 1979. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin.Google Scholar
Engel, C. 2011. ‘Dictator Games: a Meta Study’. Experimental Economics 14(4):583–610.CrossRefGoogle Scholar
Erev, I., Bornstein, G.Galili, R.. 1993. ‘Constructive Intergroup Competition as a Solution to the Free Rider Problem: A Field Experiment’. Journal of Experimental Social Psychology 29:463–478.Google Scholar
Falk, A.Heckman, J.J.. 2009. ‘Lab Experiments are a Major Source of Knowledge in the Social Sciences’. Science 326(5952):535–538.Google Scholar
Fellner, G., Sausgruber, R.Traxler, C.. 2013. ‘Testing Enforcement Strategies in the Field: Threat, Moral Appeal and Social Information’. Journal of the European Economic Association 11(3):634–660.Google Scholar
Gerber, A.S. 2004. ‘Does Campaign Spending Work?: Field Experiments Provide Evidence and Suggest New Theory’. American Behavioral Scientist 47(5):541–574.Google Scholar
Gerber, A.S.Green, D.P.. 2012. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton.Google Scholar
Gerber, A.S., Green, D.P.Larimer, C.W.. 2008. ‘Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment’. American Political Science Review 102(1):33–48.Google Scholar
Glennerster, R.Takavarasha, K.. 2013. Running Randomized Evaluations: A Practical Guide. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Gneezy, U., Haruvy, E.Yafe, H.. 2004. ‘The Inefficiency of Splitting the Bill’. The Economic Journal 114:265–280.Google Scholar
Gneezy, U.List, J.A.. 2006. ‘Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments’. Econometrica 74(5):1365–1384.Google Scholar
Gneezy, U., Niederle, M.Rustichini, A.. 2003. ‘Performance in Competitive Environments: Gender Differences’. The Quarterly Journal of Economics 118(3):1049–1074.CrossRefGoogle Scholar
Gneezy, U.Rustichini, A.. 2000. ‘Pay Enough or Don't Pay at All’. The Quarterly Journal of Economics 115(3):791–810.Google Scholar
Gneezy, U.Rustichini, A.. 2004. ‘Gender and Competition at a Young Age’. American Economic Review 94(2):377–381.Google Scholar
Green, D.P., Tusicisny, A. 2013. ‘Statistical Analysis of Results from Laboratory Studies in Experimental Economics: A Critique of Current Practice’. Unpublished manuscript.CrossRefGoogle Scholar
Großer, J.Schram, A. 2010. ‘Public Opinion Polls, Voter Turnout, and Welfare: An Experimental Study’. American Journal of Political Science 54(3):700–717.Google Scholar
Habyarimana, J., Humphreys, M., Posner, D.N.Weinstein, J.M.. 2009. Coethnicity: Diversity and the Dilemmas of Collective Action. New York: Russell Sage Foundation.Google Scholar
Harder, V.S., Stuart, E.A.Anthony, J.C.. 2010. ‘Propensity Score Techniques and the Assessment of Measured Covariate Balance to Test Causal Associations in Psychological Research’. Psychological Methods 15(3):234–249.CrossRefGoogle ScholarPubMed
Harrison, G.W.List, J.A.. 2004. ‘Field Experiments’. Journal of Economic Literature 42(4):1009–1055.Google Scholar
Harrison, G.W.List, J.A.. 2008. ‘Naturally Occurring Markets and Exogenous Laboratory Experiments: A Case Study of the Winners Curse’. The Economic Journal 118(528):822–843.Google Scholar
Henrich, J., Boyd, R., Bowles, S., Camerer, C.F., Fehr, E., Gintis, H.McElreath, R.. 2001. ‘In Search of Homo Economicus: Behavioral Experiments in 15 Small-Scale Societies’. American Economic Review 91(2):73–78.CrossRefGoogle Scholar
Henrich, J., Heine, S.J.Norenzayan, A.. 2010. ‘The Weirdest People in the World?’ Behavioral and Brain Sciences 33:61–83.Google Scholar
Hotz, V.J., Imbens, G.W.Mortimer, J.H.. 2005. ‘Predicting the Efficacy of Future Training Programs using Past Experiences at other Locations’. Journal of Econometrics 125(1–2):241–270.Google Scholar
Humphreys, M., Sanchez de la Sierra, R.van der Windt, P.. 2013. ‘Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration’. Political Analysis 21(1):1–20.Google Scholar
Humphreys, M., Weinstein, J. M.. 2010. ‘Policing Politicians: Citizen Empowerment and Political Accountability in Uganda’. Unpublished manuscript.Google Scholar
Jerit, J., Barabas, J.Clifford, S.. 2013. ‘Comparing Contemporaneous Laboratory and Field Experiments on Media Effects’. Public Opinion Quarterly 77(1):256–282.Google Scholar
Johnson, N.D.Mislin, A.A.. 2011. ‘Trust Games: A Meta-analysis’. Journal of Economic Psychology 32(5):865–889.Google Scholar
Kessler, J.Vesterlund, L.. forthcoming. ‘The External Validity of Laboratory Experiments: Qualitative rather than Quantitative Effects’. In G. Frechette and A. Schotter (eds), Methods of Modern Experimental Economics. Oxford: Oxford University Press.Google Scholar
King, E.B.Ahmad, A.S.. 2010. An Experimental Field Study Of Interpersonal Discrimination Toward Muslim Job Applicants. Personnel Psychology 63(4):881–906.Google Scholar
Lalancette, M.-F.Standing, L.. 1990. Asch Fails Again. Social Behavior and Personality 18(1):7–12.CrossRefGoogle Scholar
Levitt, S.D.List, J.A. 2007. ‘Viewpoint: On the Generalizability of Lab Behaviour to the Field’. Canadian Journal of Economics 40(2):347–370.Google Scholar
List, J.A. 2006. ‘The Behavioralist Meets the Market: Measuring Social Preferences and Reputation Effects in Actual Transactions’. Journal of Political Economy 114(1):1–37.Google Scholar
Milgram, S. 1963. ‘Behavioral Study of Obedience’. The Journal of Abnormal and Social Psychology 67(4):371–378.Google Scholar
Morton, R.B.Williams, K.C.. 2010. Experimental Political Science and the Study of Causality: From Nature to the Lab. New York: Cambridge University Press.Google Scholar
Oosterbeek, H., Sloof, R.van de Kuilen, G.. 2004. ‘Cultural Differences in Ultimatum Game Experiments: Evidence from a Meta-Analysis’. Experimental Economics 7(2):171–188.Google Scholar
Paluck, Elizabeth Levy. 2009. ‘Reducing Intergroup Prejudice and Conflict Using the Media: A Field Experiment in Rwanda’. Journal of Personality and Social Psychology 96(3):574–587.Google Scholar
Rondeau, D.List, J.A.. 2008. ‘Matching and Challenge Gifts to Charity: Evidence from Laboratory and Natural Field Experiments’. Experimental Economics 11(3):253–267.Google Scholar
Rosenthal, R. 1979. ‘The File Drawer Problem and Tolerance for Null Results’. Psychological Bulletin 86(3):638–641.Google Scholar
Schultz, W.P., Khazian, A.M.Zaleski, A.C.. 2008. ‘Using Normative Social Influence to Promote Conservation Among Hotel Guests’. Social Influence 3(1):4–23.Google Scholar
Sears, D.O. 1986. ‘College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature’. Journal of Personality and Social Psychology 51(3):515–530.Google Scholar
Shadish, W.R., Cook, T.D.Campbell, D.T.. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston, MA: Houghton, Mifflin and Company.Google Scholar
Shang, J., Reed, A.Croson, R.. 2008. ‘Identity Congruency Effects on Donations’. Journal of Marketing Research 45:351–361.Google Scholar
Stewart, R., Rooyen, C.V., Korth, M., Chereni, A., Rebelo Da Silva, N.de Wet, T.. 2012. Do Micro-credit, Micro-savings and Micro-leasing Serve as Effective Financial Inclusion Interventions Enabling Poor People, and Especially Women, to Engage in Meaningful Economic Opportunities in Low- and Middle-income Countries. London: EPPI-Centre, University of London.Google Scholar
Student. 1908. ‘Probable Error of a Correlation Coefficient’. Biometrika 6(2):302–310.Google Scholar
Valentino, N.A., Traugott, M.W.Hutchings, V.L.. 2002. ‘Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field’. Political Communication 19:29–48.Google Scholar
Supplementary material: Link

Coppock and Green Datasets

Datasets

http://dx.doi.org/10.7910/DVN/22808
Link
Supplementary material: Link

Coppock & Green Dataset

Dataset

http://dx.doi.org/10.7910/DVN/25543
Link
Supplementary material: Link

Coppock and Green code and data to replicate results

Code Ocean

https://doi.org/10.24433/CO.1998f084-503c-41b1-b86c-aa481346b7eb
Link
Supplementary material: File

Coppock and Green Supplementary Material

Appendix

Download Coppock and Green Supplementary Material(File)
File 122 KB