Hostname: page-component-586b7cd67f-rcrh6 Total loading time: 0 Render date: 2024-11-24T16:10:12.214Z Has data issue: false hasContentIssue false
  • English
  • Français

Qualitative Comparative Analysis: How Inductive Use and Measurement Error Lead to Problematic Inference

Published online by Cambridge University Press:  04 January 2017

Simon Hug
Simon Hug*
Affiliation:
Département de science politique et relations internationales, Faculté des sciences économiques et sociales, Université de Genève, 40 Bd du Pont d'Arve, 1211 Genève 4, Switzerland
*
e-mail: [email protected] (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

An increasing number of analyses in various subfields of political science employ Boolean algebra as proposed by Ragin's qualitative comparative analysis (QCA). This type of analysis is perfectly justifiable if the goal is to test deterministic hypotheses under the assumption of error-free measures of the employed variables. My contention is, however, that only in a very few research areas are our theories sufficiently advanced to yield deterministic hypotheses. Also, given the nature of our objects of study, error-free measures are largely an illusion. Hence, it is unsurprising that many studies employ QCA inductively and gloss over possible measurement errors. In this article, I address these issues and demonstrate the consequences of these problems with simple empirical examples. In an analysis similar to Monte Carlo simulation, I show that using Boolean algebra in an exploratory fashion without considering possible measurement errors may lead to dramatically misleading inferences. I then suggest remedies that help researchers to circumvent some of these pitfalls.

Type
Research Article
Information
Political Analysis , Volume 21 , Issue 2 , Spring 2013 , pp. 252 - 265
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Author's note: Comments by Matthew Gabel, Jonathan Katz, Gerald Schneider, and anonymous reviewers as well as the proofreading by Joanne Richards are gratefully appreciated. Thanks are also due to Bear Braumoeller and Gary Goertz, who provided guidance in implementing their recommendations for testing necessary conditions, and to Jeff Gill for encouraging me to submit this article to Political Analysis. Supplementary materials for this article are available on the Political Analysis Web site.

References

Achen, Christopher H. 2005. Two cheers for Charles Ragin. Studies in Comparative International Development 40: 27–32.CrossRefGoogle Scholar
Alesina, Alberto, Baqir, Reza, and Easterly, William. 1999. Public goods and ethnic divisions. Quarterly Journal of Economics 114: 1243–84.Google Scholar
Beger, Andreas, DeMeritt, Jacqueline H. R., Hwang, Wonjae, and Moore, Will H. 2009. The split population logit (SPopLogit): Modeling measurement bias in binary data. Department of Political Science, Florida State University, and Department of Political Science, University of Tennessee.Google Scholar
Bennett, Andrew. 2004. Case study methods: Design use and comparative advantages. In Models, numbers, and cases: Methods for studying international relations, eds. Sprinz, Detlef F. and Wolinsky-Nahmias, Yael, 19–55. Ann Arbor, MI: University of Michigan Press.Google Scholar
Bennett, Andrew. 2006. Stirring the frequentist pot with a dash of Bayes. Political Analysis 14(3): 339–44.CrossRefGoogle Scholar
Benoit, Kenneth R. 2005. How qualitative research really counts. Qualitative Methods Newsletter 3(1): 9–12.Google Scholar
Berg-Schlosser, Dirk, de Meur, Gisèle, Rihoux, Benoît, and Ragin, Charles C. 2008. Qualitative comparative analysis (QCA) as an approach. In Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, eds. Rihoux, Benoît and Ragin, Charles C., 1–18. Thousand Oaks, CA: Sage.Google Scholar
Bollen, Kenneth A., Entwistle, Barbara, and Alderson, Arthur S. 1993. Macrocomparative research methods. Annual Review of Sociology 19: 321–51.CrossRefGoogle Scholar
Brady, Henry E., Collier, David, and Seawright, Jason. 2006. Toward a pluralistic vision of methodology. Political Analysis 14(3): 353–68.CrossRefGoogle Scholar
Braumoeller, Bear F. 2003. Causal complexity and the study of politics. Political Analysis 11(3): 209–33.CrossRefGoogle Scholar
Braumoeller, Bear F., and Goertz, Gary D. 2000. The methodology of necessary conditions. American Journal of Political Science 44: 844–58.CrossRefGoogle Scholar
Braumoeller, Bear F., and Goertz, Gary D. 2002. Watching your posterior: Comment on Seawright. Political Analysis 10(2): 198–203.CrossRefGoogle Scholar
Caramani, Daniele. 2008. Introduction to the comparative method with Boolean algebra. Thousand Oaks, CA: Sage.Google Scholar
Clark, William Roberts, Gilligan, Michael J., and Golder, Matt. 2006. A simple multivariate test for asymmetric hypotheses. Political Analysis 14(3): 311–31.CrossRefGoogle Scholar
Clarke, Kevin A. 2002. The reverend and the ravens: Comment on Seawright. Political Analysis 10(2): 194–7.CrossRefGoogle Scholar
de Meur, Gisèle, Rihoux, Benoît, and Yamasaki, Sakura. 2008. Addressing the critiques of QCA. In Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, eds. Rihoux, Benoît and Ragin, Charles C., 147–65. Thousand Oaks, CA: Sage.Google Scholar
Dion, Douglas. 1998. Evidence and inference in the comparative case study. Comparative Politics 30(2): 127–45.CrossRefGoogle Scholar
Dusa, Adrian. 2006. QCA package for R version 0.1–3. Romanian Social Data Archive, University of Bucharest.Google Scholar
Dusa, Adrian. 2007. User manual for the QCA(GUI) package in R. Journal of Business Research 60(5): 576–86.CrossRefGoogle Scholar
Ebbinghaus, Bernhard. 2005. When less is more: Selection problems in large-N and small-N cross-national comparisons. International Sociology 20(2): 133–52.CrossRefGoogle Scholar
Ebbinghaus, Bernhard, and Visser, Jelle. 1999. When institutions matter: Union growth and decline in Western Europe, 1950–1995. European Sociological Review 15(2): 1–24.CrossRefGoogle Scholar
Gates, Scott, and Strand, Havard. 2004. Modeling the duration of civil wars: Measurement and estimation issues. Paper prepared for presentation at the Joint Session of Workshops of the ECPR, Uppsala.Google Scholar
Gelman, Andrew, Leenen, Iwin, Van Mechelen, Iven, De Boeck, Paul, and Poblome, Jeroen. 2010. Bridges between deterministic and probabilistic models for binary data. Statistical Methodology 7: 187–209.CrossRefGoogle Scholar
George, Alexander L., and Bennett, Andrew. 2005. Case studies and theory development. Cambridge, MA: MIT Press.Google Scholar
Goertz, Gary, and Mahoney, James. 2009. Scope in case study research. In Sage handbook of case-based methods, eds. Byrne, David and Ragin, Charles C., 307–17. Thousand Oaks, CA: Sage.Google Scholar
Goldthorpe, John H. 1997. Current issues in comparative macrosociology: A debate on methodological issues. Comparative Social Research 16: 1–26.Google Scholar
Grofman, Bernard, and Schneider, Carsten. 2009. An introduction to crisp-set QCA, with a comparison to binary logistic regression. Political Research Quarterly 62(4): 662–72.CrossRefGoogle Scholar
Hanushek, Eric A., and Jackson, John E. 1977. Statistical methods for social scientists. New York: Academic Press.Google Scholar
Hausman, Jerry, Abrevaya, Jason, and Scott-Morton, Fiona. 1998. Misclassification of the dependent variable in a discrete-response setting. Journal of Econometrics 87: 239–69.CrossRefGoogle Scholar
Hicks, Alexander, Misra, Joya, and Ng, Tang Nah. 1995. The programmatic emergence of the social security state. American Sociological Review 60(3): 329–49.CrossRefGoogle Scholar
Huang, Ronggui. 2011. QCA3: Yet another package for qualitative comparative analysis. R package version 0.0–5. http://cran.r-project.org/web/packages/QCA3/.Google Scholar
Hug, Simon. 2010. The effect of misclassifications in probit models. Monte Carlo simulations and applications. Political Analysis 18(1): 78–102.CrossRefGoogle Scholar
Hug, Simon. 2013. Replication data for Qualitative Comparative Analysis: How inductive use and measurement error lead to problematic inference. http://hdl.handle.net/1902.1/19489 IQSS Dataverse Network [Distributor] V1 [Version].Google Scholar
Kangas, Olli. 1994. Macrosociological and comparative methodology. On regression, qualititative comparisons, and cluster analysis in the politics of social security. In The Comparative political economy of the welfare state, eds. Janoski, Thomas and Hicks, Alexander M., 346–64. Cambridge, UK: Cambridge University Press.Google Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing social inquiry: Scientific inference in qualitative research. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Kittel, Bernhard, Obinger, Herbert, and Wagschal, Uwe. 2000. Wohlfahrtsstaaten im Internationalen Vergleich. Politisch-Institutionelle Faktoren der Entstehung und Entwicklungsdynamik. In Der gezügelte Wohlfahrtsstaat: Sozialpolitik in Australien, Japan, Schweiz, Kanada, Neuseeland and den Vereinigten Staaten, eds. Obinger, Herbert and Wagschal, Uwe, 329–64. Frankfurt: Campus.Google Scholar
Lieberson, Stanley. 1991. Small N's and big conclusions: An examination of the reasoning in comparative studies based on a small number of cases. Social Forces 70(2): 307–20.CrossRefGoogle Scholar
Little, Daniel. 1991. Varieties of social explanation: An introduction to the philosophy of social science. Boulder, CO: Westview.Google Scholar
Little, Daniel. 1995. Causal explanation in the social sciences. Southern Journal of Philosophy Supplement 34: 31–56.CrossRefGoogle Scholar
Maggetti, Martino, and Levi-Faur, David. Forthcoming. Dealing with errors in QCA. Political Research Quarterly.Google Scholar
Mahoney, James. 2007. Qualitative methodology and comparative politics. Comparative Political Studies 40(2): 122–44.CrossRefGoogle Scholar
Mahoney, James, and Goertz, Gary. 2006. A tale of two cultures: Contrasting quantitative and qualitative research. Political Analysis 14(3): 227–49.CrossRefGoogle Scholar
Mill, John Stuart. 1973. Of the four methods of experimental inquiry. In A system of logic, ratiocinative and inductive: Collected works of John Stuart Mill (Volume VII), ed. Stuart Mill, John, 388–406. Toronto: University of Toronto Press.Google Scholar
Mill, John Stuart. 1988. The logic of moral sciences. La Salle, IL: Open Court.Google Scholar
A, Maryjane, and Corduneanu-Huci, Cristina. 2003. Running uphill: Political opportunity in non-democracies. Comparative Sociology 2(4): 605–29.Google Scholar
Pearl, Judea. 2001. Causality. New York: Cambridge University Press.Google Scholar
Pennings, Paul. 2005. The diversity and causality of welfare state reforms explored with fuzzy-sets. Quality and Quantity 39(3): 317–39.CrossRefGoogle Scholar
Popper, Karl R. 1959. The logic of scientific discovery. New York: Harper & Row.Google Scholar
Ragin, Charles. 1987. The comparative method: Moving beyond qualitative and quantative strategies. Berkeley, CA: University of California Press.Google Scholar
Ragin, Charles. 1994. A qualitative comparative analysis of pension systems. In The comparative political economy of the welfare state, eds. Janoski, Thomas and Hicks, Alexander M., 320–45. Cambridge, MA: Cambridge University Press.Google Scholar
Ragin, Charles. 2000. Fuzzy-set social science. Berkeley, CA: University of California Press.Google Scholar
Ragin, Charles. 2005. Core versus tangential assumptions in comparative research. Studies in Comparative International Development 40: 33–8.CrossRefGoogle Scholar
Ragin, Charles. 2006. Set relations in social research: Evaluating their consistency and coverage. Political Analysis 14(3): 291–310.CrossRefGoogle Scholar
Ragin, Charles. 2008. Redesigning social inquiry: Fuzzy sets and beyond. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Ragin, Charles C., Berg-Schlosser, Dirk, and de Meur, Gisèle. 1996. Political methodology: Qualitative methods. In A new handbook of political science, eds. Goodin, Robert E. and Klingemann, Hans-Dieter, 749–68. Oxford: Oxford University Press.Google Scholar
Rihoux, Benoît. 2006a. Qualitative comparative analysis (QCA) and related systematic comparative methods: Recent advances and remaining challenges for social science research. International Sociology 21(5): 679–706.CrossRefGoogle Scholar
Rihoux, Benoît. 2006b. Two methodological worlds apart? Praises and critiques from a European comparativist. Political Analysis 14(3): 332–35.CrossRefGoogle Scholar
Rihoux, Benoît. 2008. Case-orientated configurational research using QCA. In The Oxford handbook of political methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David, 722–36. Oxford: Oxford University Press.Google Scholar
Rihoux, Benoit, and Lobe, Bojana. 2009. The case for qualitative comparative analysis (QCA). Adding leverage for thick cross-case comparison. In Sage handbook of case-based methods, eds. Byrne, David and Ragin, Charles C., 222–42. Thousand Oaks, CA: Sage.Google Scholar
Rihoux, Benoît, and Ragin, Charles C. 2008a. Introduction. In Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, eds. Rihoux, Benoît, and Ragin, Charles C., xvii–xxv. Thousand Oaks, CA: Sage.Google Scholar
Rihoux, Benoît, and Ragin, Charles C. 2008b. Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. Thousand Oaks, CA: Sage.Google Scholar
Schneider, Carsten Q., and Wagemann, Claudius. 2006. Reducing complexity in qualitative comparative analysis (QCA): Remote and proximate factors and the consolidation of democracy. European Journal of Political Research 45(5): 751–86.CrossRefGoogle Scholar
Schneider, Carsten Q., and Wagemann, Claudius. 2007. Qualitative comparative analysis (QCA) und fuzzy sets. Ein Lehrbuch für Anweder und jene, die es warden wollen. Opladen, Germany: Barbara Budrich Verlag Anton Hain.CrossRefGoogle Scholar
Schneider, Carsten Q., and Wagemann, Claudius. 2010. Standards of good practice in qualitative comparative analysis (QCA) and fuzzy-sets. Comparative Sociology 9(3): 397–418.CrossRefGoogle Scholar
Schrodt, Philip A. 2006. Beyond the linear frequentist orthodoxy. Political Analysis 14(3): 335–9.CrossRefGoogle Scholar
Seawright, Jason. 2002a. Testing for necessary and/or sufficient causation: Which cases are relevant? Political Analysis 10(2): 178–93.CrossRefGoogle Scholar
Seawright, Jason. 2002b. What counts as evidence? Reply. Political Analysis 10(2): 204–7.CrossRefGoogle Scholar
Seawright, Jason. 2005a. Assumptions, causal inference, and the goals of QCA. Studies in Comparative International Development 40(1): 39–42.CrossRefGoogle Scholar
Seawright, Jason. 2005b. Qualitative comparative analysis vis-à-vis regression. Studies in Comparative International Development 40(1): 3–26.CrossRefGoogle Scholar
Sekhon, Jasjeet S. 2005a. Probability tests require distribution. Qualitative Methods Newsletter 3(1): 30–1.Google Scholar
Sekhon, Jasjeet S. 2005b. Quality meets quantity: Case studies, conditional probability and counterfactuals. Perspectives on Politics 2(2): 281–93.Google Scholar
Shively, W. Phillips. 2006. Case selection: Insights from rethinking social inquiry. Political Analysis 14(3): 344–7.CrossRefGoogle Scholar
Skaaning, Svend-Erik. 2011. Assessing the robustness of crisp-set and fuzzy-set QCA results. Sociological Methods & Research 40: 391–408.CrossRefGoogle Scholar
Stokke, Olav Schram. 2004. Boolean analysis, mechanisms, and the study of regime effectiveness. In Regime consequences: Methodological challenges and research strategies, eds. Underdal, Arild and Young, Oran R., 87–120. Dordrecht, The Netherlands: Kluwer Academic Publishers.CrossRefGoogle Scholar
Waldner, David. 2005. It ain't necessarily so—or is it? Qualitative Methods Newsletter 3(1): 27–30.Google Scholar
Yamasaki, Sakura, and Rihoux, Benoît. 2008. A commented review of applications. In Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, eds. Rihoux, Benoit and Ragin, Charles C., 123–45. Thousand Oaks, CA: Sage.Google Scholar