Hostname: page-component-745bb68f8f-b6zl4 Total loading time: 0 Render date: 2025-01-08T04:52:23.221Z Has data issue: false hasContentIssue false
  • English
  • Français

Multivariate Continuous Blocking to Improve Political Science Experiments

Published online by Cambridge University Press:  04 January 2017

Ryan T. Moore
Ryan T. Moore*
Affiliation:
University of California—Berkeley and University of California—San Francisco, 50 University Hall, MC7360, Berkeley CA 94720-7360; Department of Political Science, Washington University in St. Louis, 241 Seigle Hall, Campus Box 1063, One Brookings Drive, St. Louis MO 63130. e-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Political scientists use randomized treatment assignments to aid causal inference in field experiments, psychological laboratories, and survey research. Political research can do considerably better than completely randomized designs, but few political science experiments combine random treatment assignment with blocking on a rich set of background covariates. We describe high-dimensional multivariate blocking, including on continuous covariates, detail its statistical and political advantages over complete randomization, introduce a particular algorithm, and propose a procedure to mitigate unit interference in experiments. We demonstrate the performance of our algorithm in simulations and three field experiments from campaign politics and education.

Type
Research Article
Information
Political Analysis , Volume 20 , Issue 4 , Autumn 2012 , pp. 460 - 479
Copyright
Copyright © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anderson, Michael L. 2008. Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and early training projects. Journal of the American Statistical Association 103: 1481–95.Google Scholar
Barnard, John, Frangakis, Constantine E., Hill, Jennifer L., and Rubin, Donald B. 2003. Principal stratification approach to broken randomized experiments. Journal of the American Statistical Association 98: 299–323.Google Scholar
Belfield, Clive R., Nores, Milagros, Barnett, Steve, and Schweinhart, Lawrence. 2006. The high/scope Perry Preschool Program. Journal of Human Resources 41: 162–90.Google Scholar
Boruch, Robert, May, Henry, Turner, Herbert, Lavenberg, Julia, Petrosino, Anthony, De Moya, Dorothy, Grimshaw, Jeremy, and Foley, Ellen. 2004. Estimating the effects of interventions that are deployed in many places. American Behavioral Scientist 47: 608–33.Google Scholar
Bowers, Jake. 2011. Making effects manifest in randomized experiments. In Cambridge handbook of experimental political science, eds. Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur, 459–80. Cambridge, UK: Cambridge University Press.Google Scholar
Bowers, Jake, Fredrickson, Mark, and Hansen, Ben. 2010. RItools: Randomization inference tools. R package version 0.1–11, http//www.jakebowers.org/RItools.html (accessed August 16, 2012).Google Scholar
Bullock, John G. 2011. Elite influence on public opinion in an informed electorate. American Political Science Review 105: 496–515.Google Scholar
Casella, George. 2008. Statistical design. New York: Springer.Google Scholar
Cochran, William G., and Rubin, Donald B. 1973. Controlling bias in observational studies: A review. Sankhya: The Indian Journal of Statistics, Series A 35: 417–46.Google Scholar
Donner, Allan, and Klar, Neil. 2000. Design and analysis of cluster randomization trials in health research. London: Arnold Publishers.Google Scholar
Duflo, Esther, Glennerster, Rachel, and Kremer, Michael. 2008. Using randomization in development economics research: A toolkit. In Handbook of Development Economics, ed. Paul Schultz, T., Vol. 4, 3895–962. Amsterdam: Elsevier, B.V. Google Scholar
Epstein, Lee, and King, Gary. 2002. The rules of inference. University of Chicago Law Review 69: 1–133.Google Scholar
Freedman, Laurence S., Gail, Mitchell H., Green, Sylvan B., and Corle, Donald K. 1997. The efficiency of the matched-pairs design of the Community Intervention Trial for Smoking Cessation (COMMIT). Controlled Clinical Trials 18: 131–9.Google Scholar
Gail, Mitchell H., Byar, David P., Pechacek, Terry F., and Corle, Donald K. 1992. Aspects of statistical design for the Community Intervention Trial for Smoking Cessation (COMMIT). Controlled Clinical Trials 13: 6–21.Google Scholar
Gerber, Alan S., Gimpel, James G., Green, Donald P., and Shaw, Daron R. 2011. How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized field experiment. American Political Science Review 105: 135–50.Google Scholar
Goldstein, Daniel G., Imai, Kosuke, Göritz, Anja S., and Gollwitzer, Peter M. 2010. Nudging turnout: Mere measurement and implementation planning of intentions to vote. Manuscript.Google Scholar
Gosnell, Harold F. 1927. Getting out the vote: An experiment in the stimulation of voting. Chicago, IL: University of Chicago Press.Google Scholar
Green, Donald P., and Kern, Holger L. Forthcoming 2012. Modeling heterogeneous treatment effects in survey experiments with Bayesian additive regression trees. Public Opinion Quarterly.Google Scholar
Greevy, Robert, Lu, Bo, Silber, Jeffrey H., and Rosenbaum, Paul. 2004. Optimal multivariate matching before randomization. Biostatistics 5: 263–75.Google Scholar
Hangartner, Dominik, and Moore, Ryan T. 2011. Generalizing and stabilizing the augmented inverse propensity weighted estimator. Proceedings of the Midwest Political Science Association Annual Meeting.Google Scholar
Hansen, Ben B. 2004. Full matching in an observational study of coaching for the SAT. Journal of the American Statistical Association 99: 609–18.Google Scholar
Hansen, Ben B., and Bowers, Jake. 2008. Covariate balance in simple, stratified, and clustered comparative studies. Statistical Science 23: 219–36.CrossRefGoogle Scholar
Hansen, Ben B., and Klopfer, Stephanie Olsen. 2006. Optimal full matching and related designs via network flows. Journal of Computational and Graphical Statistics 15: 609–27.Google Scholar
Heckman, James, Moon, Seong Hyeok, Pinto, Rodrigo, Savelyev, Peter, and Yavitz, Adam. 2009. A reanalysis of the high-scope Perry Preschool Program, Unpublished manuscript.Google Scholar
Heckman, James, Moon, Seong Hyeok, Pinto, Rodrigo, Savelyev, Peter, and Yavitz, Adam. 2010. The rate of return to the high-scope Perry Preschool Program. Journal of Public Economics 94: 114–28.Google Scholar
Ho, Daniel, Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2007. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis 15: 199–236.Google Scholar
Holland, Paul. 1986. Statistics and causal inference. Journal of the American Statistical Association 81(396): 945–60.Google Scholar
Horiuchi, Yusaku, Imai, Kosuke, and Taniguchi, Naoko. 2007. Designing and analyzing randomized experiments: Application to a Japanese election survey experiment. American Journal of Political Science 51: 669–87.Google Scholar
Hudgens, Michael G., and Elizabeth Halloran, M. 2008. Toward causal inference with interference. Journal of the American Statistical Association 103: 832–42.Google Scholar
Hyde, Susan. 2010. Experimenting in democracy promotion: International observers and the 2004 Presidential elections in Indonesia. Perspectives on Politics 8: 511–27.Google Scholar
Iacus, Stefano M., King, Gary, and Porro, Giuseppe. 2011. Multivariate matching methods that are monotonic imbalance bounding. Journal of the American Statistical Association 106: 345–61.Google Scholar
Iacus, Stefano M., King, Gary, and Porro, Giuseppe. 2012. Causal inference without balance checking: Coarsened exact matching. Political Analysis 20: 1–24.Google Scholar
Ichino, Nahomi, and Schündeln, Matthias. 2012. Deterring or displacing electoral irregularities? Spillover effects of observers in a randomized field experiment in Ghana. Journal of Politics 74: 292–307.Google Scholar
Imai, Kosuke, and van Dyk, David A. 2004. Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association 99: 854–66.Google Scholar
Imai, Kosuke, King, Gary, and Nall, Clayton. 2009. The essential role of pair-matching in cluster-randomized experiments, with application to the Mexican universal health insurance evaluation. Statistical Science 24: 29–53.CrossRefGoogle Scholar
Imai, Kosuke, King, Gary, and Stuart, Elizabeth A. 2008. Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A 171: 481–502.Google Scholar
Imai, Kosuke, Keele, Luke, Tingley, Dustin, and Yamamoto, Teppei. 2011. Unpacking the black box of causality: Learning about causal mechanisms from experimental and observational studies. American Political Science Review 105: 765–89.Google Scholar
Imbens, Guido W. 2011. Experimental design for unit and cluster randomized trials. Manuscript prepared for the International Initiative for Impact Evaluation.Google Scholar
Keele, Luke, and Morgan, Jason W. 2011. Stronger instruments by design. Poster presented at the 28th Annual Summer Meeting of the Society for Political Methodology.Google Scholar
King, Gary. 1995. Replication, replication. PS: Political Science and Politics 28: 444–52.Google Scholar
King, Gary, Gakidou, Emmanuela, Imai, Kosuke, Lakin, Jason, Moore, Ryan T., Nall, Clayton, Ravishankar, Nirmala, Vargas, Manett, María Téllez-Rojo, Martha, Eugenio Hernández Ávila, Juan, Ávila, Mauricio, and Hernández Hernández Llamas, Héctor. 2009. Public policy for the poor? A randomized assessment of the Mexican universal health insurance program. Lancet 373: 1447–54.Google Scholar
King, Gary, Gakidou, Emmanuela, Ravishankar, Nirmala, Moore, Ryan T., Lakin, Jason, Vargas, Manett, María Téllez-Rojo, Martha, Eugenio Hernández Ávila, Juan, Hernández Ávila, Mauricio, and Hernández Llamas, Héctor. 2007. A “politically robust” experimental design for public policy evaluation, with application to the Mexican universal health insurance program. Journal of Policy Analysis and Management 26: 479–509.Google Scholar
Lachin, John M. 1988. Properties of Simple Randomization in Clinical Trials. Controlled Clinical Trials 9: 312–26.Google ScholarPubMed
Linday, Linda A., Tsiouris, J. A., Cohen, Ira L., Shindledecker, Richard, and DeCresce, Robert. 2001. Famotidine treatment of children with autistic spectrum disorders: Pilot research using single-subject research design. Journal of Neural Transmission 108: 593–611.Google Scholar
Loewen, Peter John, and Rubenson, Daniel. 2011. For want of a nail: Negative persuasion in a party leadership race. Party Politics 17: 45–65.Google Scholar
Lu, Bo, Zanutto, Elaine, Hornik, Robert, and Rosenbaum, Paul R. 2001. Matching with doses in an observational study of a media campaign against drug abuse. Journal of the American Statistical Association 96: 1245–53.Google Scholar
Mason, Robert L., Gunst, Richard F., and Hess, James L. 1989. Statistical design and analysis of experiments: With applications to engineering and science. New York: Wiley.Google Scholar
Mebane, Walter R.J., and Sekhon, Jasjeet S. 1998. GENetic Optimization Using Derivatives (GENOUD).Google Scholar
Moore, Ryan T. 2012a. blockTools: Blocking, assignment, and diagnosing interference in randomized experiments. R package version 0.5–6, http://www.wustl.edu/software.blockTools.him (accessed August 16, 2012).Google Scholar
Moore, Ryan T. 2012b. Replication data for: Multivariate continuous blocking to improve political science experiments. http://hdl.handle.net/1902.1/18341, IQSS Dataverse Network [Distributor] V1 [Version].Google Scholar
Moore, Ryan T., and Moore, Sally A. 2012. Blocking for sequential political experiments. Manuscript in preparation.Google Scholar
Murray, David M. 1998. Design and analysis of group-randomized trials. New York: Oxford University Press.Google Scholar
National Cancer Institute. 1995. NCI Monograph #6: Community-based interventions for smokers: The COMMIT field experience. Technical Report 95-4028, National Institutes of Health.Google Scholar
Paluck, Elizabeth Levy, and Green, Donald P. 2009. Deference, dissent, and dispute resolution: An experimental intervention using mass media to change norms and behavior in Rwanda. American Political Science Review 103: 622–44.Google Scholar
Panagopoulos, Costas, and Green, Donald P. 2008. Field experiments testing the impact of radio advertisements on electoral competition. American Journal of Political Science 52: 156–68.Google Scholar
Pechacek, Terry F. 2006. Personal communication, Centers for Disease Control's Associate Director for Science for the Office on Smoking and Health, 5 December.Google Scholar
R Development Core Team. 2012. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.Google Scholar
Rosenbaum, Paul R. 2007. Interference between units in randomized experiments. Journal of the American Statistical Association 102: 191–200.Google Scholar
Rosenbaum, Paul R. 2010. Design of observational studies. New York: Springer.Google Scholar
Rousseeuw, Peter J. 1985. Multivariate estimation with high breakdown point. Mathematical Statistics and Applications 8: 283–97.Google Scholar
Rousseeuw, Peter J., and van Zomeren, Bert C. 1990. Unmasking multivariate outliers and leverage points. Journal of the American Statistical Association 85(411): 633–9.Google Scholar
Rubin, Donald B. 1980. Bias reduction using Mahalanobis-metric matching. Biometrics 36: 293–8.Google Scholar
Rubin, Donald B. 1990. Formal modes of statistical inference for causal effects. Journal of Statistical Planning and Inference 25: 279–92.Google Scholar
Rubin, Donald B. 2001. Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology 2: 169–88.Google Scholar
Sekhon, Jasjeet S. 2011. Multivariate and propensity score matching software with automated balance optimization: The matching package for R. Journal of Statistical Software 42: 1–52.Google Scholar
Sobel, Michael E. 2006. What do randomized studies of housing mobility demonstrate? Causal inference in the face of interference. Journal of the American Statistical Association 101(476): 1398–407.Google Scholar
Tamhane, Ajit C. 2009. Statistical analysis of designed experiments: Theory and applications. Hoboken, NJ: John C. Wiley & Sons.Google Scholar
Wantchekon, Leonard. 2003. Clientelism and voting behavior: Evidence from a field experiment in Benin. World Politics 55(3): 399–422.CrossRefGoogle Scholar
Supplementary material: PDF

Moore supplementary material

Supplementary Material

Download Moore supplementary material(PDF)
PDF 222.1 KB