Hostname: page-component-586b7cd67f-rcrh6 Total loading time: 0 Render date: 2024-11-28T02:10:08.110Z Has data issue: false hasContentIssue false
  • English
  • Français

Comparing Discrete Distributions: Survey Validation and Survey Experiments

Published online by Cambridge University Press:  04 January 2017

Kishore Gawande ,
Gina Yannitell Reinhardt ,
Carol L. Silva  and
Domonic Bearfield
Kishore Gawande*
Affiliation:
Bush School of Government, Texas A&M University, College Station, TX 77843-4220
Gina Yannitell Reinhardt
Affiliation:
Bush School of Government, Texas A&M University, College Station, TX 77843-4220 e-mail: [email protected]
Carol L. Silva
Affiliation:
Center for Applied Social Research, Department of Political Science, University of Oklahoma, 455 W. Lindsey, Room 205, Norman, OK 73019-2001 e-mail: [email protected]
Domonic Bearfield
Affiliation:
Bush School of Government, Texas A&M University, College Station, TX 77843-4220 e-mail: [email protected]
*
e-mail: [email protected] (corresponding author)
Get access Rights & Permissions [Opens in a new window]

Abstract

Field survey experiments often measure amorphous concepts in discretely ordered categories, with postsurvey analytics that fail to account for the discrete attributes of the data. This article demonstrates the use of discrete distribution tests, specifically the chi-square test and the discrete Kolmogorov—Smirnov (KS) test, as simple devices for comparing and analyzing ordered responses typically found in surveys. In Monte Carlo simulations, we find the discrete KS test to have more power than the chi-square test when distributions are right or left skewed, regardless of the sample size or the number of alternatives. The discrete KS test has at least as much power as the chi-square, and sometimes more so, when distributions are bi-modal or approximately uniform and samples are small. After deriving rules of usage for the two tests, we implement them in two cases typical of survey analysis. Using our own data collected after Hurricanes Katrina and Rita, we employ our rules to both validate and assess treatment effects in a natural experimental setting.

Type
Regular Articles
Information
Political Analysis , Volume 21 , Issue 1 , Winter 2013 , pp. 70 - 85
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Authors' note: We appreciate Matt Henderson's valuable assistance with coding the Internet-based survey. We thank Elisabeth Gerber, Jennifer Jerit, Jason Barabas, Janet Box-Steffensmeier, Andrew Sobel, Gary Miller, Randall Calvert, Andrew Martin, and Itai Sened for helpful comments. The editor, R. Michael Alvarez, and an anonymous referee suggested revisions that improved the article significantly. For replication data, code, and instructions, see Gawande et al. (2012). Supplementary materials for this article are available on the Political Analysis website.

References

Barabas, Jason, and Jerit, Jennifer. 2010. Are survey experiments externally valid? American Political Science Review 104(2): 226–42.Google Scholar
Berrens, Robert P., Bohara, Alok K., Jenkins-Smith, Hank, Silva, Carol, and Weimer, David L. 2003. The advent of Internet surveys for political research: A comparison of telephone and Internet samples. Political Analysis 11(1): 1–22.Google Scholar
Bourque, Linda B., Siegel, Judith M., and Shoaf, Kimberley I. 2002. Psychological distress following urban earthquakes in California. Prehospital and Disaster Medicine 17(2): 81–90.Google Scholar
Bourque, Linda B., Siegel, Judith M., Kano, Megumi, and Wood, Michele M. 2006. Morbidity and mortality associated with disasters. In Handbook of disaster research, eds. Rodriguez, Havidan, Quarantelli, Enrico L., and Dynes, Russell R., 97–112. New York: Springer.Google Scholar
Canner, Paul L. 1975. A simulation study of one- and two-sample Kolmogorov-Smirnov statistics with a particular weight function. Journal of the American Statistical Association 70: 209–11.Google Scholar
Clason, Dennis L., and Dormody, Thomas J. 1994. Analyzing data measured by individual Likert-type items. Journal of Agricultural Education 35(4): 31–35.CrossRefGoogle Scholar
Darling, Donald A. 1960. On the theorems of Kolmogorov-Smirnov. Theory of Probability and Its Applications 5(4): 356–61.CrossRefGoogle Scholar
Gawande, Kishore, Reinhardt, Gina Yannitell, Silva, Carol L., and Bearfield, Domonic A. 2012. Replication data for: Comparing discrete distributions: Surveys and survey experiments. IQSS Dataverse Network V1 [Version]. http://hdl.handle.net/1902.1/18921 (Accessed October 8, 2012).Google Scholar
Gleser, Leon Jay. 1985. Exact power of goodness-of-fit tests of Kolmogorov type for discontinuous distributions. Journal of the American Statistical Association 80: 954–58.Google Scholar
Hetherington, Marc, and Suhay, Elizabeth. 2011. Authoritarianism, threat, and Americans' support for the war on terror. American Journal of Political Science 55(3): 546–60.Google Scholar
Hilbe, Joseph M. 2007. Negative Binomial Regression. New York: Cambridge University Press.CrossRefGoogle Scholar
Hilbe, Joseph M. 2009. Logistic Regression Models. Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Hilbe, Joseph. 2010. Creation of synthetic discrete response regression models. Stata Journal 10(1): 104–24.Google Scholar
Horn, Susan D. 1977. Goodness-of-fit tests for discrete data: A review and an application to a health impairment scale. Biometrics 33: 237–47.Google Scholar
Jerit, Jennifer. 2009. How predictive appeals affect policy opinions. American Journal of Political Science 53(2): 411–26.CrossRefGoogle Scholar
Kempthorne, Oscar. 1967. The classical problem of inference-goodness-of fit. In: Proceedings of Fifth Berkeley Symposium On Mathematical Statistics and Probability, Vol. 1, 235–49. Berkeley: University of California Press.Google Scholar
Likert, Rensis. 1932. A technique for the measurement of attitudes. Archives of Psychology 140: 1–55.Google Scholar
Lilliefors, Hubert W. 1967. On the Kolmogorov-Smirnov test for normality with mean and variance unknown. Journal of the American Statistical Association 62: 399–402.Google Scholar
Lilliefors, Hubert W. 1969. On the Kolmogorov-Smirnov test for the exponential distribution with mean unknown. Journal of the American Statistical Association 64: 387–89.Google Scholar
Lott, Neal, Smith, Adam, Houston, Tamara, Shein, Karsten, and Crouch, Jake. 2012. Billion dollar U.S. weather/climate disasters, 1980 November 2011. National Climate Data Center.Google Scholar
Malhotra, Neil, and Margalit, Yotam. 2010. Short-term communication effects or longstanding dispositions? The public's response to the financial crisis of 2008. Journal of Politics 72(3): 852–67.Google Scholar
Massey, Frank J. 1950. The Kolmogorov-Smirnov test for goodness-of-fit. Journal of the American Statistical Association 46: 68–77.Google Scholar
MEPS (Medical Expenditures Panel Survey). 2007 MEPS-HC sample design and collection process. Agency for healthcare research and quality. Rockville, MD. http://www.meps.ahrq.gov/survey\_comp/hc\_data\_collection.jsp (Accessed December 2008).Google Scholar
Panchenko, Dmitry. 2003. Lecture notes on MIT OpenCourseWare site, Lectures 11 and 14. http://ocw.mit.edu/OcwWeb/Mathematics/18-443Fall-2006/LectureNotes/index.htm (Accessed August 2008).Google Scholar
Pettitt, A. N., and Stephens, M.A. 1977. The Kolmogorov-Smirnov goodness-of-fit statistic with discrete and grouped data. Technometrics 19: 207–10.Google Scholar
Quarantelli, Enrico L. 2002. The Disaster Research Center field studies of organized behavior in the crisis time period of disasters. In Methods of disaster research, ed. Stallings, Robert A., 94–126. Philadelphia: Xlibris.Google Scholar
Smirnov, Nikolai V. 1939. An estimate of divergence between empirical curves of a distribution in two independent samples. Bulletin Math. de l'Univ. de Moscou 2: 3–14 (in Russian).Google Scholar
Smith, Tom W., Marsden, Peter, Hout, Michael, and Kim, Jibum. 2011. General social surveys, 1972–2010: cumulative codebook. Chicago: National Opinion Research Center.Google Scholar
Wilcox, Rand R. 1997. Some practical reasons for reconsidering the Kolmogorov-Smirnov test. British Journal of Mathematical and Statistical Psychology 50: 9–20.Google Scholar
Witt, Evans, Best, Jonathan, and Rainie, Lee. 2008. Internet access and use: Does cell phone interviewing make a difference? Paper for the 2008 Conference of the American Association for Public Opinion Research. http://pewresearch.org/assets/pdf/cellphone-\linebreakpewinternet.pdf (Accessed July 2009).Google Scholar
Wood, Constance L., and Altavela, Michele M. 1978. Large-sample results for Kolmogorov-Smirnov statistics for discrete distributions. Biometrika 65: 235–39.Google Scholar
Supplementary material: File

Gawande et al. supplementary material

Appendix

Download Gawande et al. supplementary material(File)
File 45.1 KB