The illusion of learning from observational research

doi:10.1017/CBO9780511492174.012

12 - The illusion of learning from observational research

Published online by Cambridge University Press: 22 September 2009

Donald P. Green and

Edited by

Rogers M. Smith and

Alan S. Gerber: Affiliation:
Department of Political Science, Yale University
Donald P. Green: Affiliation:
Department of Political Science, Yale University
Edward H. Kaplan: Affiliation:
School of Management, Yale University
Ian Shapiro: Affiliation:
Yale University, Connecticut
Rogers M. Smith: Affiliation:
University of Pennsylvania
Tarek E. Masoud: Affiliation:
Yale University, Connecticut

Book contents

Get access

Summary

Introduction

Empirical studies of cause and effect in social science may be divided into two broad categories, experimental and observational. In the former, individuals or groups are randomly assigned to treatment and control conditions. Most experimental research takes place in a laboratory environment and involves student participants, but several noteworthy studies have been conducted in real-world settings, such as schools (Howell and Peterson 2002), police precincts (Sherman and Rogan 1995), public housing projects (Katz, Kling, and Liebman 2001), and voting wards (Gerber and Green 2000). The experimental category also encompasses research that examines the consequences of randomization performed by administrative agencies, such as the military draft (Angrist 1990), gambling lotteries (Imbens, Rubin, and Sacerdote 2001), random assignment of judges to cases (Berube 2002), and random audits of tax returns (Slemrod, Blumenthal, and Christian 2001). The aim of experimental research is to examine the effects of random variation in one or more independent variables.

Observational research, too, examines the effects of variation in a set of independent variables, but this variation is not generated through randomization procedures. In observational studies, the data generation process by which the independent variables arise is unknown to the researcher. To estimate the parameters that govern cause and effect, the analyst of observational data must make several strong assumptions about the statistical relationship between observed and unobserved causes of the dependent variable (Achen 1986; King, Keohane, and Verba 1994). To the extent that these assumptions are unwarranted, parameter estimates will be biased.

Type: Chapter
Information: Problems and Methods in the Study of Politics , pp. 251 - 273

DOI: https://doi.org/10.1017/CBO9780511492174.012 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2004

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Achen, Christopher H. 1986. The Statistical Analysis Of Quasi-Experiments. Berkeley: University of California Press

Angrist, Joshua A. 1990. “Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records.” American Economic Review 80(3): 313–36Google Scholar

Angrist, Joshua D., Imbens, Guido W., and Donald, B. Rubin. 1996. “Identification of Casual Effects Using Instrumental Variables.” Journal of the American Statistical Association 91(June): 444–55CrossRef Google Scholar

Berube, Danton. 2002. “Random Variations in Federal Sentencing as an Instrumental Variable.” Unpublished ms, Yale University

Box, George E. P. and G. C. Tiao. 1973. Baysian Inference in Statistical Analysis. Reading, MA: Addison-Wesley

Cook, Thomas D. and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin

Cook, Thomas D. and Monique R. Payne. 2002. “Objecting to the Objections to Using Random Assignment in Educational Research,” in Frederick Mosteller and Robert Boruch (eds.), Evidence Matters: Randomized Trials in Education Research. Washington, DC: Brookings Institution Press

Eldersveld, Samuel J. 1956. “Experimental Propaganda Techniques and Voting Behavior.” American Political Science Review 50 (March): 154–65CrossRef Google Scholar

Freund, John E. 1971. Mathematical Statistics, 2nd ed. New York: Prentice-Hall

Gerber, Alan S. and Donald, P. Green. 2000. “The Effects of Canvassing, Direct Mail, and Telephone Contact on Voter Turnout: A Field Experiment.” American Political Science Review 94: 653–63CrossRef Google Scholar

Green, Donald P. and Alan S. Gerber. 2003. “Reclaiming the Experimental Tradition in Political Science,” in The State of the Discipline IIIHelen Milner and Ira Katznelson (eds.), Washington, DC: American Political Science Association

Gosnell, Harold F. 1927. Getting-Out-The-Vote: An Experiment in the Stimulation of Voting. Chicago: University of Chicago Press

Heckman, James J. and Jeffrey, A. Smith. 1995. “Assessing the Case for Social Experiments.” Journal of Economic Perspectives 9 (Spring): 85–110CrossRef Google Scholar

Howell, William G. and Paul E. Peterson. 2002. The Education Gap: Vouchers and Urban Schools. Washington, DC: Brookings Institution Press

Imbens, Guido W., Donald, B. Rubin, and Bruce, I. Sacerdote. 2001. “Estimating the Effect of Unearned Income on Labor Earnings, Savings, and Consumption: Evidence from a Survey of Lottery Winners.” American Economic Review 91(4): 778–94CrossRef Google Scholar

Katz, Lawrence F., Jeffrey, R. Kling, and Jeffrey, B. Liebman. 2001. “Moving to Opportunity in Boston: Early Results of a Randomized Mobility Experiment.” Quarterly Journal of Economics 116: 607–54CrossRef Google Scholar

King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton: Princeton University Press

Kramer, Gerald H. 1970. “The Effects of Precinct-Level Canvassing on Voting Behavior.” Public Opinion Quarterly 34 (Winter): 560–72CrossRef Google Scholar

Levitt, Stephen D. 1997. “Using Electoral Cycles in police hiring to Estimate the Effect of Police on Crime.” American Economic Review 87(3): 270–90Google Scholar

McConahay, John B. 1982. “Self-Interest versus Racial Attitudes as Correlates of Anti-Busing Attitudes in Louisville: Is it the Buses or the Blacks?” Journal of Politics 44(3): 2–720CrossRef Google Scholar

Morton, Rebecca. 2002. “EITM: Experimental Implications of Theoretical Models.” The Political Methodologist 10(2): 14–16Google Scholar

Rosenstone, Steven J. and John Mark Hansen. 1993. Mobilization, Participation, and Democracy in America. New York: Macmillan Publishing Company

Sherman, Lawrence W. and Dennis, P. Rogan. 1995. “Deterrent Effects of Police Raids on Crack Houses: A Randomized, Controlled Experiment.” Justice Quarterly 12(4): 755–81CrossRef Google Scholar

Slemrod, J., Blumenthal, M., and Christian, C.. 2001. “Taxpayer Response to an Increased Probability of Audit: Evidence from a Controlled Experiment in Minnesota.” Journal of Public Economics 79(3): 455–83CrossRef Google Scholar

Weiss, Carol H. 2002. “What to Do until the Random Assigner Comes,” in Frederick Mosteller and Robert Boruch (eds.), Evidence Matters: Randomized Trials in Education Research. Washington, DC: Brookings Institution Press