Hostname: page-component-586b7cd67f-rcrh6 Total loading time: 0 Render date: 2024-11-28T07:05:43.914Z Has data issue: false hasContentIssue false
  • English
  • Français

Computerized Adaptive Testing for Public Opinion Surveys

Published online by Cambridge University Press:  04 January 2017

Jacob M. Montgomery  and
Josh Cutler
Jacob M. Montgomery*
Affiliation:
Department of Political Science, Washington University in St. Louis, Campus Box 1063, One Brookings Drive, St Louis, MO 63130-4899
Josh Cutler
Affiliation:
Department of Political Science, Duke University, 326 Perkins Library, Box 90204, Durham, NC 27708
*
e-mail: [email protected] (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

Survey researchers avoid using large multi-item scales to measure latent traits due to both the financial costs and the risk of driving up nonresponse rates. Typically, investigators select a subset of available scale items rather than asking the full battery. Reduced batteries, however, can sharply reduce measurement precision and introduce bias. In this article, we present computerized adaptive testing (CAT) as a method for minimizing the number of questions each respondent must answer while preserving measurement accuracy and precision. CAT algorithms respond to individuals' previous answers to select subsequent questions that most efficiently reveal respondents' positions on a latent dimension. We introduce the basic stages of a CAT algorithm and present the details for one approach to item selection appropriate for public opinion research. We then demonstrate the advantages of CAT via simulation and empirically comparing dynamic and static measures of political knowledge.

Type
Research Article
Information
Political Analysis , Volume 21 , Issue 2 , Spring 2013 , pp. 172 - 192
Copyright
Copyright © The Author 2013. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Authors' note: We are grateful for helpful comments provided by Martin Elff, Sunshine Hillygus, Walter Mebane, Brendan Nyhan, and two anonymous reviewers. A previous version of this article was presented at the 2012 Annual Meeting of the Midwest Political Science Association, the 2012 Saint Louis Area Methods Meeting, and the 2012 Summer Methods Meeting. Supplementary materials for this article are available on the Political Analysis Web site.

References

Anderson, A., Basilevsky, A., and Hum, D. 1983. Missing data: A review of the literature. In Handboook of survey research, eds. Rossi, P. H., Wright, J. D., and Anderson, A. B., 415–81. New York: Academic Press.Google Scholar
Bafumi, J., Gelman, A., Park, D. K., and Kaplan, N. 2005. Practical issues in implementing and understanding Bayesian ideal point estimation. Political Analysis 13(2): 171–87.Google Scholar
Bafumi, J., and Herron, M. C. 2010. Leapfrog representation and extremism: A study of American voters and their members in Congress. American Political Science Review 104(3): 519–42.CrossRefGoogle Scholar
Bailey, M. A. 2007. Comparable preference estimates across time and institutions for the court, Congress, and presidency. American Journal of Political Science 51(3): 433–48.Google Scholar
Baker, F. B., and Kim, S.-H. 2004. Item response theory: Parameter estimation techniques. New York: Marcel Dekker.Google Scholar
Barabas, J. 2002. Another look at the measurement of political knowledge. Political Analysis 10(2): 209–22.Google Scholar
Berinsky, A. J., Huber, G. A., and Lenz, G. S. 2012. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 20(3): 329–50.CrossRefGoogle Scholar
Brewer, P. R. 2003. Values, political knowledge, and public opinion about gay rights. Public Opinion Quarterly 67(3): 173–201.Google Scholar
Burchell, B., and Marsh, C. 1992. The effect of questionnaire length on survey response. Quality & Quantity 26(3): 233–44.Google Scholar
Cacioppo, J. T., and Petty, R. E. 1984. The efficient assessment of need for cognition. Journal of Personality Assessment 48(3): 306–7.Google Scholar
Choi, S. W., and Swartz, R. J. 2009. Comparison of CAT item selection criteria for polytomous items. Applied Psychological Measurement 33(6): 419–40.Google Scholar
Clinton, J. D., and Meirowitz, A. 2001. Agenda constrained legislator ideal points and the spatial voting model. Political Analysis 9(3): 242–59.Google Scholar
Clinton, J. D., and Meirowitz, A. 2003. Integrating voting theory and roll call analysis: A framework. Political Analysis 11(4): 381–96.Google Scholar
Clinton, J., Jackman, S., and Rivers, D. 2004. The statistical analysis of roll call voting: A unified approach. American Political Science Review 98(2): 355–70.Google Scholar
Crawford, S. D., Couper, M. P., and Lamias, M. J. 2001. Web surveys: Perceptions of burden. Social Science Computer Review 19(2): 146–62.Google Scholar
DeBell, M. 2012. Harder than it looks: Coding political knowledge on the ANES. Paper presented at the 2012 meeting of the Midwest Political Science Association, Chicago, IL.Google Scholar
Delli Carpini, M. X., and Keeter, S. 1993. Measuring political knowledge: Putting first things first. American Journal of Political Science 37(4): 1179–206.Google Scholar
Delli Carpini, M. X., and Keeter, S. 1996. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.Google Scholar
Dodd, B. G., De Ayala, R., and Koch, W. R. 1995. Computerized adaptive testing with polytomous items. Applied Psychological Measurement 19(1): 5–22.Google Scholar
Embretson, S. E., and Reise, S. P. 2000. Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Feldman, S., and Huddy, L. 2005. Racial resentment and white opposition to race-conscious programs: Principles or prejudice? American Journal of Political Science 49(1): 168–83.Google Scholar
Forbey, J. D., and Ben-Porath, Y. S. 2007. Computerized adaptive personality testing: A review and illustration with the MMPI-2 computerized adaptive version. Psychological Assessment 19(1): 14–24.Google Scholar
Galesic, M., and Bosnjak, M. 2009. Effects of questionnaire length on participation and indicators of response quality in Web surveys. Public Opinion Quarterly 73(2): 349–60.Google Scholar
Gerber, A. S., Huber, G. A., Doherty, D., Dowling, C. M., and Ha, S. E. 2010. Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review 104(01): 111–33.Google Scholar
Gibson, J. L., and Caldeira, G. A. 2009. Knowing the Supreme Court? A reconsideration of public ignorance of the high court. Journal of Politics 71(2): 429–41.Google Scholar
Gillion, D. Q. 2012. Re-defining political participation through item response theory. Unpublished paper.Google Scholar
Gomez, B. T., and Wilson, J. M. 2001. Political sophistication and economic voting in the American electorate: A theory of heterogeneous attribution. American Journal of Political Science 45(4): 899–914.Google Scholar
Gosling, S. D., Rentfrow, P. J., and Swann, W. B. 2003. A very brief measure of the big-five personality domains. Journal of Research in Personality 37(6): 504–28.Google Scholar
Heberlein, T. A., and Baumgartner, R. 1978. Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review 43(4): 447–62.Google Scholar
Herzog, A. R., and Bachman, J. G. 1981. Effects of questionnaire length on response quality. Public Opinion Quarterly 45(4): 549–59.CrossRefGoogle Scholar
Hol, A. M., Vorst, H. C., and Mellenbergh, G. J. 2007. Computerized adaptive testing for polytomous motivation items: Administration mode effects and a comparison with short forms. Applied Psychological Measurement 31(5): 412–29.Google Scholar
Jackman, S. 2001. Multidimensional analysis of roll call data via Bayesian simulation: Identification, estimation, inference, and model checking. Political Analysis 9(3): 227–41.Google Scholar
Kingsbury, G., and Weiss, D. J. 1983. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New horizons in testing: Latent trait test theory and computerized adaptive testing, ed. Weiss, D. J. New York: Academic Press.Google Scholar
Krosnick, J. A. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5(3): 213–36.CrossRefGoogle Scholar
Krosnick, J. A. 1999. Survey research. Annual Review of Psychology 50: 537–67.CrossRefGoogle ScholarPubMed
Krosnick, J. A., Holbrook, A. L., Berent, M. K., Carson, R. A. B. T., Hanemann, W., Kopp, R. J., Mitchell, C., Cameron, R., Presser, S., Ruud, P. A., Smith, V., Moody, W. R., Green, M. C., and Conaway, M. 2002. The impact of “no opinion“ response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly 66(3): 371–403.CrossRefGoogle Scholar
Lord, F. M. 1980. Applications of item response theory to practical testing problems. Hillsdale, NJ: L. Erlbaum Associates.Google Scholar
Lord, F., and Novick, M. R. 1968. Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
Lupia, A. 2006. How elitism undermines the study of voter competence. Critical Review 18 (1–3): 217–32.Google Scholar
Lupia, A. 2008. Procedural transparency and the credibility of election surveys. Electoral Studies 27(4): 732–9.CrossRefGoogle Scholar
Luskin, R. C. 1987. Measuring political sophistication. American Journal of Political Science 31(4): 856–99.Google Scholar
Luskin, R. C., and Bullock, J. G. 2011. “Don't know” means “don't know”: DK responses and the public's level of political knowledge. Journal of Politics 73(2): 547–57.CrossRefGoogle Scholar
Martin, A. D., and Quinn, K. M. 2002. Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953–1999. Political Analysis 10(2): 134–53.Google Scholar
Matthews, R. A., Kath, L. M., and Barnes-Farrell, J. L. 2010. A short, valid, predictive measure of work-family conflict: Item selection and scale validation. Journal of Occupational Health Psychology 15(1): 75–90.Google Scholar
Mondak, J. J., and Anderson, M. R. 2004. The knowledge gap: A reexamination of gender-based differences in political knowledge. Journal of Politics 66(2): 492–512.Google Scholar
Mondak, J. J., and Davis, B. C. 2001. Asked and answered: Knowledge levels when we will not take “don't know” for an answer. Political Behavior 23(3): 199–224.Google Scholar
Mondak, J. J. 2001. Developing valid knowledge scales. American Journal of Political Science 45(1): 224–38.Google Scholar
Montgomery, J. M., and Cutler, J. 2012. Replication data for: Computerized adaptive testing for public opinion surveys. http://hdl.handle.net/1902.1/19381 IQSS Dataverse Network.Google Scholar
Piazza, T., Sniderman, P. M., and Tetlock, P. E. 1989. Analysis of the dynamics of political reasoning: A general-purpose computer-assisted methodology. Political Analysis 1(1): 99–119.Google Scholar
Podsakoff, P. M., and MacKenzie, S. B. 1994. An examination of the psychometric properties and nomological validity of some revised and reduced substitutes for leadership scales. Journal of Applied Psychology 79(5): 702–13.Google Scholar
Poole, K. T. 2005. Spatial models of parliamentary voting. New York: Cambridge University Press.Google Scholar
Prior, M., and Lupia, A. 2008. Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science 52(1): 19–183.Google Scholar
Prior, M. 2012. Visual political knowledge: A different road to competence. Unpublished paper.Google Scholar
Richins, M. L. 2004. The material values scale: Measurement properties and development of a short form. Journal of Consumer Research 31(1): 209–19.Google Scholar
Russell, S. S., Spitzmüller, C., Lin, L. F., Stanton, J. M., Smith, P. C., and Ironson, G. H. 2004. Shorter can also be better: The abridged job in general scale. Educational and Psychological Measurement 64(5): 878–93.Google Scholar
Segall, D. O. 2002. Confirmatory item factor analysis using Markov chain Monte Carlo estimation with applications to online calibration in CAT. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.Google Scholar
Segall, D. O. 2005. Computerized adaptive testing. In Encyclopedia of social measurement, ed. Kempf-Leonard, K., Vol. 1, 429–38. Oxford, UK: Elsevier.Google Scholar
Segall, D. O. 2010. Principles of multidemensional adaptive testing. In Elements of adaptive testing, eds. van der Linden, W. J. and Glas, C. A. W., 57–76. New York: Springer.Google Scholar
Sheatsley, P. 1983. Questionnaire construction and item writing. In Handboook of survey research, eds. Rossi, P. H., Wright, J. D., and Anderson, A. B., 195–230. New York: Academic Press.Google Scholar
Singh, J., Howell, R. D., and Rhoads, G. K. 2007. Designs for Likert-type data: An approach for implementing marketing surveys. Journal of Marketing Research 19(1): 12–24.Google Scholar
Sniderman, P. M., Brody, R. A., and Tetlock, P. E. 1991. Reasoning and choice: Explorations in political psychology. New York: Cambridge University Press.Google Scholar
Sniderman, P. M., Piazza, T., Tetlock, P. E., and Kendrick, A. 1991. The new racism. American Journal of Political Science 35(2): 423–47.Google Scholar
Stanton, J. M., Sinar, E. F., Balzer, W. K., and Smith, P. C. 2002. Issues and strategies for reducing the length of self-report scales. Personnel Psychology 55(1): 167–94.Google Scholar
Thompson, E. R. 2012. A brief index of affective job satisfaction. Group & Organization Management 37(3): 275–307.Google Scholar
Treier, S., and Hillygus, D. S. 2009. The nature of political ideology in the contemporary electorate. Public Opinion Quarterly 73(4): 679–703.Google Scholar
Treier, S., and Jackman, S. 2008. Democracy as a latent variable. American Journal of Political Science 52(1): 201–17.Google Scholar
van der Linden, W. J. 1998. Bayesian item selection criteria for adaptive testing. Psychometrika 63(2): 201–16.Google Scholar
van der Linden, W. J. 1999. Empirical initialization of the trait estimator in adaptive testing. Applied Psychological Measurement 23(1): 21–29.Google Scholar
van der Linden, W. J. 2008. Using response times for item selection in adaptive testing. Journal of Educational and Behavioral Statistics 33(1): 5–20.Google Scholar
van der Linden, W. J. 2010. Constrained adaptive testing with shadow tests. In Elements of adaptive testing, eds. van der Linden, W. J. and Glas, C. A. W., 31–56. New York: Springer.Google Scholar
van der Linden, W. J., and Pashley, P. J. 2010. Elements of adaptive testing. New York: Springer.Google Scholar
Verba, S., Schlozman, K. L., and Brady, H. E. 1995. Voice and equality: Civic voluntarism in American politics. Cambridge, MA: Harvard University Press.Google Scholar
Wainer, H. 1990. Introduction and history. In Computerized Adaptive Testing: A Primer, ed. Wainer, H. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Waller, N. G., and Reise, S. P. 1989. Computerized adaptive personality assessment: An illustration with the absorption scale. Journal of Personality and Social Psychology 57(6): 1051.Google Scholar
Weiss, D. J. 1982. Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement 6(4): 473–92.Google Scholar
Weiss, D. J., and Kingsbury, G. G. 1984. Application of computerized adaptive testing to educational problems. Journal of Educational Measurement 21(4): 361–75.Google Scholar
Xu, X., and Douglas, J. 2006. Computerized adaptive testing under nonparamteric IRT models. Psychometrika 71(1): 121–37.Google Scholar
Yammarino, F. J., Skinner, S. J., and Childers, T. L. 1991. Understanding mail survey response behavior: A meta-analysis. Public Opinion Quarterly 55(4): 613–639.Google Scholar
Zaller, J. R. 1992. The nature and origins of mass opinion. New York: Cambridge University Press.Google Scholar
Supplementary material: PDF

Montgomery and Cutler supplementary material

Appendix

Download Montgomery and Cutler supplementary material(PDF)
PDF 51.6 KB