Hostname: page-component-586b7cd67f-dlnhk Total loading time: 0 Render date: 2024-11-24T12:51:25.541Z Has data issue: false hasContentIssue false
  • English
  • Français

Analysis of Cluster-Randomized Experiments: A Comparison of Alternative Estimation Approaches

Published online by Cambridge University Press:  14 September 2007

Donald P. Green  and
Lynn Vavreck
Donald P. Green
Affiliation:
Yale University, Department of Political Science, 77 Prospect Street, New Haven, CT 06520-8209, e-mail: [email protected]
Lynn Vavreck*
Affiliation:
UCLA, Department of Political Science, 4289 Bunche Hall Box 951472, Los Angeles, CA 90095-1472
*
e-mail: [email protected] (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

Analysts of cluster-randomized field experiments have an array of estimation techniques to choose from. Using Monte Carlo simulation, we evaluate the properties of point estimates and standard errors (SEs) generated by ordinary least squares (OLS) as applied to both individual-level and cluster-level data. We also compare OLS to alternative random effects estimators, such as generalized least squares (GLS). Our simulations assess efficiency across a variety of scenarios involving varying sample sizes and numbers of clusters. Our results confirm that conventional OLS SEs are severely biased downward and that, for all estimators, gains in efficiency come mainly from increasing the number of clusters, not increasing the number of individuals within clusters. We find relatively minor differences across alternative estimation approaches, but GLS seems to enjoy a slight edge in terms of the efficiency of its point estimates and the accuracy of its SEs. We illustrate the application of alternative estimation approaches using a clustered experiment in which Rock the Vote TV advertisements were used to encourage young voters in 85 cable TV markets to vote in the 2004 presidential election.

Type
Research Article
Information
Political Analysis , Volume 16 , Issue 2 , Spring 2008 , pp. 138 - 152
Copyright
Copyright © The Author 2007. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: We thank Rock the Vote for permission to use their public service announcements in this field experiment. The authors are grateful to Alan Gerber for suggestions throughout the design phase of this project. We are also grateful to Dan Kotin and Margaret Coblentz, who worked with cable operators, distributed the advertisements, and assembled the data. We thank Terence Leong for his programming expertise. Replication materials are available on the Political Analysis Web site.

References

Angrist, Joshua D., and Lavy, Victor. 2002. The effect of high school matriculation wards: Evidence from randomized trials. National Bureau of Economic Research Working Paper 9389.Google Scholar
Arceneaux, Kevin. 2005. Using cluster randomized field experiments to study voting behavior. The Annals of The American Academy of Political and Social Science 601: 169–79.Google Scholar
Arellano, Manuel. 1987. Computing robust standard errors for within-groups estimators. Oxford Bulletin of Economics and Statistics 49: 431.Google Scholar
Cornfield, J. 1978. Randomization by group: a formal analysis. American Journal of Epidemiology 108: 100–2.Google Scholar
Donner, Allan, and Klar, Neil. 2000. Design and analysis of cluster randomization trials in health research. London: Arnold.Google Scholar
Donner, Allan, and Klar, Neil. 2004. Pitfalls of and controversies in cluster randomized trials. American Journal of Public Health 94: 416–22.Google Scholar
Eldersveld, Samuel J. 1956. Experimental propaganda techniques and voting behavior. American Political Science Review 50: 154–65.Google Scholar
Gerber, Alan S., and Green, Donald P. 2000. The effects of canvassing, direct mail, and telephone contact on voter turnout: A field experiment. American Political Science Review 94: 653–63.Google Scholar
Gerber, Alan S., and Green, Donald P. 2005. Correction to Gerber and Green (2000), replication of disputed findings, and reply to Imai (2005). American Political Science Review 99: 301–13.CrossRefGoogle Scholar
Green, Donald P., and Gerber, Alan S. 2002. Reclaiming the experimental tradition in political science. In Political science: The state of the discipline, ed. Milner, Helen V. and Katznelson, Ira, 805–32. 3rd ed. New York: W.W. Norton & Co. Google Scholar
Green, Donald P., and Gerber, Alan S. 2004. Get out the vote!: How to increase voter turnout. Washington, DC: Brookings Institution Press.Google Scholar
Green, Donald P., Kim, Soo Yeon, and Yoon, David. 2001. Dirty pool. International Organization 55: 441–68.CrossRefGoogle Scholar
Greene, William H. 2003. Econometric analysis. 5th ed. Upper Saddle River, NJ: Prentice-Hall.Google Scholar
Huber, P. J. 1967. The behavior of maximum likelihood estimates under non-standard conditions. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability 1: 223–33.Google Scholar
Hyde, Susan D. 2006. Foreign democracy promotion, norm development and democratization: Explaining the causes and consequences of internationally monitored elections. Unpublished doctoral thesis, Department of Political Science, University of California, San Diego.Google Scholar
Moulton, Brent. 1990. An illustration of a pitfall in estimating the effects of aggregate variables on micro units. Review of Economics and Statistics 72: 334.Google Scholar
Murray, D. M. 1998. Design and analysis of group-randomized trials. New York: Oxford University Press.Google ScholarPubMed
Nickerson, David Warwick. 2005. Measuring interpersonal influence. Unpublished doctoral thesis, Department of Political Science, Yale University.Google Scholar
Raudenbush, Stephen W. 1997. Statistical analysis and optimal design for cluster randomized trials. Psychological Methods 2: 173–85.Google Scholar
Rogers, W. H. 1993. Regression standard errors in clustered samples. Stata Technical Bulletin 13: 19–23.Google Scholar
Varnelli, Sherri P., Murray, David M., Jenega, Jessica B., and Blitstein, Jonathan L. 2004. Design and analysis of group-randomized trials: A review of recent practices. Evaluation Methods and Practice 94: 393–9.Google Scholar
Vavreck, Lynn, and Green, Donald P. 2006. Mobilizing voters through TV public service announcements: A field experiment. Unpublished report prepared for the Carnegie Corporation and presented at the MWPSA meeting, Chicago, IL.Google Scholar
Wantchekon, Leonard. 2003. Clientelism and voting behavior: Evidence from a field experiment in Benin. World Politics 55: 399–422.CrossRefGoogle Scholar
White, Halbert. 1980. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48: 817–30.Google Scholar
Williams, Rick L. 2000. A note on robust variance estimation for cluster-correlated data. Biometrics 56: 645–6.CrossRefGoogle ScholarPubMed
Wooldridge, Jeffrey M. 2003. Cluster-sample methods in applied econometrics. American Economic Review 93: 133–8.Google Scholar