Hostname: page-component-78c5997874-ndw9j Total loading time: 0 Render date: 2024-11-19T07:23:51.774Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing Data Quality: An Approach and An Application

Published online by Cambridge University Press:  15 September 2021

Kelly McMann ,
Daniel Pemstein ,
Brigitte Seim Open the ORCID record for Brigitte Seim [Opens in a new window] ,
Jan Teorell  and
Staffan Lindberg
Kelly McMann
Affiliation:
Political Science, Case Western Reserve University, Cleveland, OH44106, USA. Email: [email protected]
Daniel Pemstein
Affiliation:
Political Science and Public Policy & Challey Institute, North Dakota State University, Fargo, ND58102, USA. Email: [email protected]
Brigitte Seim*
Affiliation:
Public Policy, University of North Carolina, Chapel Hill, NC27599, USA. Email: [email protected]
Jan Teorell
Affiliation:
Political Science, Lund University, Lund, 221 00, Sweden. Email: [email protected]
Staffan Lindberg
Affiliation:
V-Dem Institute & Political Science, University of Gothenburg, Gothenburg, 413 27, Sweden. Email: [email protected]
*
Corresponding author Brigitte Seim
Get access Rights & Permissions [Opens in a new window]

Abstract

Political scientists routinely face the challenge of assessing the quality (validity and reliability) of measures in order to use them in substantive research. While stand-alone assessment tools exist, researchers rarely combine them comprehensively. Further, while a large literature informs data producers, data consumers lack guidance on how to assess existing measures for use in substantive research. We delineate a three-component practical approach to data quality assessment that integrates complementary multimethod tools to assess: (1) content validity; (2) the validity and reliability of the data generation process; and (3) convergent validity. We apply our quality assessment approach to the corruption measures from the Varieties of Democracy (V-Dem) project, both illustrating our rubric and unearthing several quality advantages and disadvantages of the V-Dem measures, compared to other existing measures of corruption.

Keywords

validityreliabilityBayesian IRTcorruption
Type
Article
Information
Political Analysis , Volume 30 , Issue 3 , July 2022 , pp. 426 - 449
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Edited by Jeff Gill

References

Adcock, R., and Collier, D.. 2001. “Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.” American Political Science Review 95(3):529–546.CrossRefGoogle Scholar
Arndt, C., and Oman, C.. 2006. Uses and Abuses of Governance Indicators. Paris: Development Centre Studies, OECD Publishing.Google Scholar
Benson, G. C., Maaranen, S. A., and Heslop, A.. 1978. Political Corruption in America. Lexington: D.C. Heath and Company.Google Scholar
Bolck, A., Croon, M., and Jagenaars, J.. 2004. “Estimating Latent Structure Models with Categorical Variables: One-Step Versus Three-Step Estimators.” Political Analysis 12(1):3–27.CrossRefGoogle Scholar
Bollen, K. A. 1980. “Issues in the Comparative Measurement of Political Democracy.” American Sociological Review 45:370–390.CrossRefGoogle Scholar
Bollen, K. A. 1989. Structural Equations with Latent Variables. New York: Wiley.CrossRefGoogle Scholar
Bollen, K. A. 1990. “Political Democracy: Conceptual and Measurement Traps.” Studies in Comparative International Development 25:7–24.CrossRefGoogle Scholar
Bollen, K. A., and Paxton, P.. 2000. “Subjective Measures of Liberal Democracy.” Comparative Political Studies 33(1):58–86.CrossRefGoogle Scholar
Bowman, K., Lehoucq, F., and Mahoney, J.. 2005. “Measuring Political Democracy Case Expertise, Data Adequacy, and Central America.” Comparative Political Studies 38(8):939–970.CrossRefGoogle Scholar
Campbell, D. T., and Fiske, D. W.. 1959. “Convergent and Discriminant Validation by the Multitrait-Multimethod Matrix.” Psychological Bulletin 56(2):81.CrossRefGoogle ScholarPubMed
Carmines, E. G., and Zeller, R. A.. 1979. Reliability and Validity Assessment. Beverly Hills, CA: Sage.CrossRefGoogle Scholar
Chikulo, B. C. 2000. “Corruption and Accumulation in Zambia.” In Corruption and Development in Africa: Lessons from Country Case Studies, edited by Hope, K. R. Sr., and Chikulo, B., 161–182. London: Palgrave Macmillan.CrossRefGoogle Scholar
Collier, D., LaPorte, J., and Seawright, J.. 2012. “Putting Typologies to Work: Concept Formation, Measurement, and Analytic Rigor.” Political Research Quarterly 65(1):217–232.CrossRefGoogle Scholar
Collier, D., and Levitsky, S.. 1997. “Democracy with Adjectives.” World Politics 49(3):430–451.CrossRefGoogle Scholar
Coppedge, M., et al. 2011. “Conceptualizing and Measuring Democracy: A New Approach.” Perspectives on Politics 9(2):247–267.CrossRefGoogle Scholar
Coppedge, M., et al. 2015a. V-Dem Country-Year Dataset v4. Varieties of Democracy (V-Dem) Project.Google Scholar
Coppedge, M., et al. 2020. Varieties of Democracy: Measuring Two Centuries of Political Change. New York: Cambridge University Press.CrossRefGoogle Scholar
Coppedge, M., et al. 2017. V-Dem Country-Year Dataset v7.1. Varieties of Democracy (V-Dem) Project.Google Scholar
Coppedge, M., et al. 2015a. V-Dem Country-Year Dataset v4. Varieties of Democracy (V-Dem) Project.Google Scholar
Coppedge, M., et al. 2015b. Varieties of Democracy: Codebook v4. Varieties of Democracy (V-Dem) Project.Google Scholar
Dahlström, C., Lapuente, V., and Teorell, J.. 2012. “Public Administration Around the World.” In Good Government. The Relevance of Political Science, edited by Holmberg, S. and Rothstein, B., 40–67. Cheltenham, UK: Edward Elgar.Google Scholar
Donchev, D., and Ujhelyi, G.. 2014. “What Do Corruption Indices Measure?” Economics & Politics 26(2):309–331.CrossRefGoogle Scholar
Donnelly, M. J., and Pop-Eleches, G.. 2018. “Income Measures in Cross-National Surveys: Problems and Solutions.” Political Science Research and Methods 6(2):355–363.CrossRefGoogle Scholar
Fariss, C. J. 2014. “Respect for Human Rights has Improved Over Time: Modeling the Changing Standard of Accountability.” American Political Science Review 108(2):297–318.CrossRefGoogle Scholar
Galtung, F. 2006. “Measuring the Immeasurable: Boundaries and Functions of (Macro) Corruption Indices.” In Measuring Corruption, edited by Galtung, F., and Sampford, C., 101–130. Aldershot, UK: Ashgate.Google Scholar
Gerring, J. 2012. Social Science Methodology: A Unified Framework, 2nd ed. Cambridge, UK: Cambridge University Press.Google Scholar
Gingerich, D. W. 2013. “Governance Indicators and the Level of Analysis Problem: Empirical Findings from South America.” British Journal of Political Science 43(3):505–540.CrossRefGoogle Scholar
Grossman, M. 2003. Political Corruption in America: An Encyclopedia of Scandals, Power, and Greed. Santa Barbara: ABC-CLIO.Google Scholar
Harrington, D. 2008. Confirmatory Factor Analysis. Oxford: Oxford University Press.CrossRefGoogle Scholar
Hawken, A., and Munck, G. L.. 2009a. “Do You Know Your Data? Measurement Validity in Corruption Research.” Working Paper, School of Public Policy, Pepperdine University.Google Scholar
Hawken, A., and Munck, G. L.. 2009b. “Measuring Corruption: A Critical Assessment and a Proposal.” In Perspectives on Corruption and Human Development, vol. 1, edited by Rajivan, A. K. and Gampat, R., 71–106. New Delhi, India: Macmillan India for UNDP.Google Scholar
Hayes, A. F., and Krippendorff, K.. 2007. “Answering the call for a standard reliability measure for coding data.” Communication Methods and Measures 1(1):77–89.CrossRefGoogle Scholar
Herrera, Y. M., and Kapur, D.. 2007. “Improving data quality: Actors, incentives, and capabilities.” Political Analysis 15(4):365–386.CrossRefGoogle Scholar
Huckfeldt, R., and Sprague, J.. 1993. “ Citizens, Contexts, and Politics.” In Political Science: The State of the Discipline II , edited by Finifter, A. W., 281–303. Washington, DC: American Political Science Association.Google Scholar
Jeong, G.-H. 2018. “Measuring Foreign Policy Positions of Members of the US Congress.” Political Science Research and Methods 6(1):181–196.CrossRefGoogle Scholar
Johnson, V. E., and Albert, J. H.. 1999. Ordinal Data Modeling. New York: Springer.CrossRefGoogle Scholar
Kaufmann, D., and Kraay, A.. 2002. “Growth Without Governance.” World Bank Policy Research Working Paper no. 2928.Google Scholar
Kennedy, J. 2014. “International Crime Victims Survey.” The Encyclopedia of Criminology and Criminal Justice.CrossRefGoogle Scholar
Knack, S. 2007. “Measuring Corruption: A Critique of Indicators in Eastern Europe and Central Asia.” Journal of Public Policy 27(3):255–291.CrossRefGoogle Scholar
Lambsdorff, J. G. 2007. “The Methodology of the Corruption Perceptions Index 2007.” Transparency International (TI) and the University of Passau.Google Scholar
Lindstaedt, R., Proksch, S.-O., and Slapin, J. B.. 2016. “When Experts Disagree: Response Aggregation and Its Consequences in Expert Surveys.” Working paper.Google Scholar
Marcus, G. E., Neuman, W. R., and MacKuen, M. B.. 2017. “Measuring Emotional Response: Comparing Alternative Approaches to Measurement.” Political Science Research and Methods 5(4):733–754.CrossRefGoogle Scholar
Marquardt, K. L. 2019. “How and How Much Does Expert Error Matter? Implications for Quantitative Peace Research.” V-Dem Working Papers Series no. 84.CrossRefGoogle Scholar
Marquardt, K. L., and Pemstein, D.. 2018. “IRT Models for Expert-Coded Panel Data.” Political Analysis 26(4):431–456.CrossRefGoogle Scholar
Martinez i Coma, F., and van Ham, C.. 2015. “Can Experts Judge Elections? Testing the Validity of Expert Judgments for Measuring Election Integrity.” European Journal of Political Research 54(2):305–325.CrossRefGoogle Scholar
Mbao, M. 2011. “Prevention and Combating of Corruption in Zambia.” Comparative and International Law Journal of Southern Africa 44(2):255–274.Google Scholar
McMann, K., Pemstein, D., Seim, B., Teorell, J., and Lindberg, S.. 2021a. Replication Data for: Assessing Data Quality: An Approach and An Application. Version v1. https://doi.org/10.24433/CO.0269024.v1.CrossRefGoogle Scholar
McMann, K., Pemstein, D., Seim, B., Teorell, J., and Lindberg, S.. 2021b. Replication Data for: Assessing Data Quality: An Approach and An Application. https://doi.org/10.7910/DVN/BXV4AT, Harvard Dataverse, V1.CrossRefGoogle Scholar
McMann, K., Seim, B., Teorell, J., and Lindberg, S.. 2020. “Why Low Levels of Democracy Promote Corruption and High Levels Diminish It.” Political Research Quarterly 73(4):893–907.CrossRefGoogle Scholar
Menes, R. 2003. “Corruption in Cities: Graft and Politics in American Cities at the Turn of the Twentieth Century.” NBER Working Paper no. 9990.CrossRefGoogle Scholar
Mislevy, R. 1991. “Randomization-Based Inference about Latent Variables from Complex Samples.” Psychometrika 56(2):177–196.CrossRefGoogle Scholar
Morin-Chassé, A., Bol, D., Stephenson, L. B., and Labbé St-Vincent, S.. 2017. “How to Survey About Electoral Turnout? The Efficacy of the Face-Saving Response Items in 19 Different Contexts.” Political Science Research and Methods 5(3):575–584.CrossRefGoogle Scholar
Mudde, C., and Schedler, A.. 2010. “Introduction: Rational Data Choice.” Political Research Quarterly 63(2):410–416.CrossRefGoogle Scholar
Munck, G. L., and Verkuilen, J.. 2002. “Conceptualizing and measuring democracy: Evaluating alternative indices.” Comparative Political Studies 35(1):5–34.Google Scholar
Pemstein, D., et al. 2020. “The V-Dem Measurement Model: Latent Variable Analysis for Cross-National and Cross-Temporal Expert-Coded Data.” V-Dem Working Paper No. 21, 5th edition.CrossRefGoogle Scholar
Pemstein, D., Marquardt, K., Tzelgov, E., Wang, Y.-t., and Miri, F.. 2015. “Latent Variable Models for the Varieties of Democracy Project.” Varieties of Democracy Institute Working Paper Series no. 21.Google Scholar
Pemstein, D., Meserve, S. A., and Melton, J.. 2010. “Democratic Compromise: A Latent Variable Analysis of Ten Measures of Regime Type.” Political Analysis 18(4):426–449.CrossRefGoogle Scholar
Reeves, T. C. 2000. Twentieth-Century America: A Brief History. New York: Oxford University Press.Google Scholar
Reise, S. P., Widaman, K. F., and Pugh, R. H.. 1993. “Confirmatory Factor Analysis and Item Response Theory: Two Approaches for Exploring Measurement Invariance.” Psychological Bulletin 114(3):552.CrossRefGoogle ScholarPubMed
Reuning, K., Kenwick, M. R., and Fariss, C. J.. 2019. “Exploring the Dynamics of Latent Variable Models.” Political Analysis 27(4):503–517.CrossRefGoogle Scholar
Rose-Ackerman, S. 1999. Corruption and Government: Causes, Consequences, and Reform. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Sartori, G. 1970. “Concept Misformation in Comparative Politics.” American Political Science Review 64(4):1033–1053.CrossRefGoogle Scholar
Schedler, A. 2012. “Judgment and Measurement in Political Science.” Perspectives on Politics 10(1):22–36.CrossRefGoogle Scholar
Schnakenberg, K. E., and Fariss, C. J.. 2014. “Dynamic Patterns of Human Rights Practices.” Political Science Research and Methods 2(1):1–31.CrossRefGoogle Scholar
Seawright, J., and Collier, D.. 2014. “Rival Strategies of Validation: Tools for Evaluating Measures of Democracy.” Comparative Political Studies 47(1):111–138.CrossRefGoogle Scholar
Shadish, W., Cook, T. D., and Campbell, D. T.. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.Google Scholar
Steenbergen, M. R., and Marks, G.. 2007. “Evaluating expert judgments.” European Journal of Political Research 46(3):347–366.CrossRefGoogle Scholar
Szeftel, M. 2000. “Eat With Us: Managing Corruption and Patronage Under Zambia’s Three Republics, 1964–99.” Journal of Contemporary African Studies 18(2):207–224.CrossRefGoogle Scholar
Tanner, M. A. 1993. Tools for Statistical Inference: Methods for the Exploration of Posterior Distributions and Likelihood Functions, 2nd Edn. New York, NY: Springer Verlag.CrossRefGoogle Scholar
Thomas, M. A. 2010. “What Do the Worldwide Governance Indicators Measure?” European Journal of Development Research 22(1):31–54.CrossRefGoogle Scholar
Treisman, D. 2000. “The Causes of Corruption: A Cross-National Study.” Journal of Public Economics 76(3):399–457.CrossRefGoogle Scholar
Treisman, D. 2007. “What Have We Learned About the Causes of Corruption from Ten Years of Cross-National Empirical Research?” Annual Review of Political Science 10:211–244.CrossRefGoogle Scholar
Van Donge, J. K. 2009. “The Plundering of Zambian Resources by Frederick Chiluba and His Friends: A Case Study of the Interaction between National Politics and the International Drive Towards Good Governance.” African Affairs 108(430):69–90.CrossRefGoogle Scholar
Woodiwiss, M. 1988. Crimes, Crusades, and Corruption: Prohibitions in the United States, 1900–1987. Lanham, MD: Rowman & Littlefield Publishers.Google Scholar
Zeller, R. A., and Carmines, E. G.. 1980. Measurement in the Social Sciences: The Link between Theory and Data. Cambridge: Cambridge University Press.Google Scholar
Supplementary material: Link

McMann et al. Dataset

Dataset

https://doi.org/10.7910/DVN/BXV4AT
Link
Supplementary material: PDF

McMann et al. supplementary material

McMann et al. supplementary material

Download McMann et al. supplementary material(PDF)
PDF 388.7 KB