Knowledge of the Court In Open- and Closed-Ended Questions

To fix ideas, we focus on knowledge of the US Supreme Court. What Americans seem to know about it depends a lot on the questions that we ask and the ways in which we ask them. For example, the chief justice may be the most prominent member of the most prominent court, but few can recall the exact job of ‘William Rehnquist’ or ‘John Roberts’ when asked in an open-ended question. Answers to this question in the American National Election Studies are illustrative: only 8 per cent of respondents in the 2000 ANES and 13 per cent in the 2012 ANES gave an answer that included ‘chief justice’ and either ‘Supreme Court’ or ‘United States’. But this is both a tough question and a tough standard for what counts as correct.Footnote ² By our reading, an additional 25 per cent of respondents in 2000 and 18 per cent of respondents in 2012 gave answers that were nearly correct.Footnote ³

Perhaps it is unsurprising, then, that many more people can choose the correct name when asked to identify the chief justice in a closed-ended question. In one 2003 survey, for example, 63 per cent of respondents correctly identified Rehnquist as the current chief justice in response to a closed-ended question (Prior Reference Prior2005); eight years later, 69 per cent of people answering a closed-ended ANES question correctly identified Roberts as the chief justice. Results like these led some to suspect that differences in correct-response rates to open- and closed-ended questions stem from fundamental differences between the two types of questions (e.g., Gibson and Caldeira Reference Gibson and Caldeira2009, 431–35). In particular, open-ended questions test ability to recall correct answers, while closed-ended questions instead test ability to recognize the correct answers in a limited set of options (Tedin and Murray Reference Tedin and Murray1979). And because open-ended questions lack response options, they offer no frame of reference to anchor respondents’ thinking. Coupled with research that suggests the superiority of closed-ended knowledge measures (e.g., Mondak Reference Mondak2001), results like these have led scholars to infer that the public knows more about the Supreme Court – and perhaps much else – than many have suggested (Gibson and Caldeira Reference Gibson and Caldeira2009, 432–33).

To further explore the extent to which conventional open-ended questions indicate lower levels of political knowledge than their closed-ended counterparts, we collected the universe of national-sample studies, conducted in or before 2017, that included closed-ended questions about the name of the chief justice.Footnote ⁴ Table 1 shows, for each study, the percentage of respondents who identified Rehnquist or Roberts as the chief justice. In every case, the percentage correct is larger than the ‘exactly correct’ response rate from the corresponding open-ended ANES question. To better understand the sources of these differences, a close look at Table 1 is in order.Footnote ⁵

Table 1. Variation in responses to closed-ended ‘Chief Justice’ questions

Note: EGSS = Evaluations of Government and Society Study. The table has two tiers: in the first tier (top two rows), the correct answer is ‘William Rehnquist’; in the second tier, it is ‘John Roberts’. Within each tier, studies are sorted by the percentage of respondents who answered correctly.

The table reports data from closed-ended questions, and it shows that the variation in correct-response rates is great – even though these questions have all been designed to measure knowledge of the same fact. For example, across the six surveys conducted during John Roberts' term, the correct-response rates range from 28 per cent to 69 per cent. This 41-point difference is comparable to the 39-point difference that Gibson and Caldeira (Reference Gibson and Caldeira2009, 435) find between correct-response rates for open- and closed-ended questions about John Roberts.Footnote ⁶

What explains this wide variation among questions that all use a closed-ended format? Increased public knowledge of the relatively new chief probably plays some role. In the two Pew surveys, more respondents correctly identified Roberts as the chief in 2012 (34 per cent) than in 2010 (28 per cent). But this within-survey-firm difference is dwarfed by the within-year difference between the 2012 Pew survey (34 per cent) and the 2012 ANES study (69 per cent).

Survey mode may also play a role. But the size of its role is hard to determine, partly because it is confounded with the year in which the interview was conducted. Moreover, the within-mode differences are large. In particular, the studies that generated the lowest and highest percentages of correct responses were both conducted by telephone.

Instead, Table 1 suggests that variation in correct-response rates may have more to do with the number and difficulty of the response options that respondents receive. In some sense, this is not news: much has been written about the role of response options in survey questions, and no scholar will deny that ‘response options matter’. But for all that has been written about levels of political knowledge, the extent to which measurement of those levels depends on the choice of response options has rarely been mentioned. Even the most prominent works in this field typically do not broach the topic (e.g., Delli Carpini and Keeter Reference Delli Carpini and Keeter1996; Luskin Reference Luskin1987). Instead, previous studies have focused on inherent differences between closed- and open-ended questions: for example, their implications for guessing and the way that open-ended questions test recall while closed-ended questions test recognition (e.g., Tedin and Murray Reference Tedin and Murray1979).

While Table 1 is suggestive, the questions it describes differ in many ways – not just with respect to response options. For example, the questions were asked in different years, and they appeared at different positions within different kinds of surveys. To isolate the effects of the number and difficulty of response options, we conducted a new experiment in which we held question wording constant while randomly assigning some people to see an open-ended question and others to see closed-ended versions of the same question. No previous published studies of political knowledge have isolated the effects of response options in this way.Footnote ⁷

A New Experiment: Exploring the Role of Response Options

We fielded an experiment from 14–30 March 2017. Our subjects were a national sample of 2,080 US adults. Following our pre-analysis plan, we restrict our analyses to the 1,961 subjects for whom we have complete demographic information. The Appendix shows that relaxing this restriction makes no substantive difference. Subjects were Survey Sampling International panelists, and they were asked six questions: about the number of justices who serve on the court, the length of their terms, the number of women who serve on the court, and the names of the chief justice and Senate majority leader.Footnote ⁸

For each question, subjects were assigned to one of five conditions: ‘short easy’, ‘short difficult’, ‘long easy’, ‘long difficult’, and open-ended. In the ‘long’ conditions, subjects saw five response options: the correct option and four others. In the ‘short’ conditions, they saw the correct option and two others that were randomly selected from the full set of four incorrect options. In both the long and short conditions, subjects also saw a ‘don't know’ response option (Cor and Sood Reference Cor and Sood2016, esp. 229n6; Jessee Reference Jessee2017; Luskin and Bullock Reference Luskin and Bullock2011; Sturgis, Allum and Smith Reference Sturgis, Allum and Smith2008). Assignment varied across questions; for example, a subject might be in the short-easy condition for one question, but in the long-difficult condition for another.

We classified response options as easy or difficult by considering the extent to which they were likely to seem correct. For example, our difficult-and-incorrect response options for the chief justice question were William Rehnquist, Earl Warren, Clarence Thomas, and Antonin Scalia. Two of these men served as chief justice before John Roberts; the two others served on the court with Roberts. By contrast, our easy-and-incorrect response options were Theodore Olson, Mark Rockefeller, Homer Stille Cummings, and J. Harvie Wilkinson, III. None of these men have ever been prominent figures in public life.

All of these response options have been used in other research. And for all other questions, we used response options that might plausibly be used in other surveys.

We asked six additional questions for which we manipulated only the number of response options. These questions were about judicial review, the limits of Supreme Court power, how justices are chosen and removed from office, the consequences of tie votes, and public access to oral arguments. For each question, some subjects were assigned to see five response options, and others were assigned to see three options. The incorrect responses in the three-option condition were randomly selected from those in the five-option condition.

We use these data to examine three hypotheses. Two are simple: we anticipate that correct-response rates decline as the number and difficulty of response options increases. But our main hypothesis is more complex, as it involves comparing two differences:

1. (a) the difference between correct-response rates to questions that have a short-easy set of response options and questions that have a long-difficult set of response options, and

2. (b) the difference between correct-response rates to questions that have a long-difficult set of response options and questions that are open-ended.

We expected that difference (a) would be greater than difference (b). That is, we expected that correct-response rates would depend more on the contrast between different types of closed-ended questions than on inherent differences between open- and closed-ended questions. This hypothesis is contrary to much thinking about open- and closed-ended questions. It is especially contrary to the claim of Gibson and Caldeira (Reference Gibson and Caldeira2009, 435), who attribute the large differences in correct-response rates that they observe to the inherent difference between closed- and open-ended questions, and who ‘strongly doubt’ that variation in response options alone can account for such differences. Our hypotheses were preregistered, and our pre-analysis plan can be found at https://doi.org/10.7910/DVN/HU04WI.

Results

Figure 1 displays the correct-response rates from each experimental condition, on average (black lines) and for each question (grey lines). And Table 2 depicts estimates from regressions of answers, coded correct (1) or incorrect (0), on indicator variables that represent different experimental conditions.

Figure 1. Correct response rates by response-option set

Note: ROs = response options. Grey lines indicate correct-response rates for individual questions, and the black line in each panel indicates the average correct-response rate. Respondents are from SSI (n = 1,961).

Table 2. Effects of number and difficulty of response options

Note: cell entries are OLS estimates and standard errors. The outcomes are answers to the questions: correct = 1, incorrect = 0. The listed predictors are also coded 0 or 1. All regressions include fixed effects for each question. The baseline condition for Column 1 is ‘three response options’; for Column 2, ‘easy response options’; and for Columns 3 and 4, ‘open-ended questions’. Standard errors are clustered at the respondent level.

Panels 1 and 2 of the figure show that correct-response rates decline with both the number and the difficulty of response options. And in Table 2, the first regression shows that offering five response options, rather than three, reduces the correct-response rate by 12 percentage points. The second regression shows that offering difficult rather than easy response options has an even larger effect: it reduces correct-response rates by 24 percentage points.Footnote ⁹

To generalize, one may imagine two batteries that include the ‘same’ questions but different response options. All of the response options might plausibly be used in actual surveys – but how much people seem to know may depend hugely on which battery one chooses. As best we can tell, this point has not been demonstrated before in studies of political knowledge. (See Ahler and Goggin Reference Ahler and Goggin2017 for the most closely related work that we know.)

The third regression in Table 2 speaks to our main concern: the difference between different kinds of closed-ended questions, relative to the difference between open-ended questions and difficult closed-ended questions. With this regression, we estimate a model of the form:

(1)$$correct_{{\rm ij}} = {\rm \alpha} + {\rm \beta }_1\left( {{ three \; easy}} \right)_{{\rm ij}} +\; {\rm \beta }_2\left( {{five \; difficult}} \right)_{{\rm ij}} + \; {\rm }\lambda _{\rm j} + {\rm \varepsilon }_{{\rm ij}}$$

where (three easy)_ij and (five difficult)_ij indicate the condition to which subject i was assigned for question j. The reference category in this regression is the open-ended condition. Per our pre-analysis plan, data from the ‘three difficult’ and ‘five easy’ conditions were excluded from this analysis. Appendix Table A13 shows that including these conditions makes no material difference to the results.

The results show that, relative to the open-ended condition, assignment to the ‘five difficult’ condition increases correct-response rates by 6 percentage points. But relative to the same open-ended condition, assignment to the ‘three easy’ condition increases correct-response rates by 35 points. The difference between correct-response rates in the ‘three easy’ and ‘five difficult’ conditions, 35 − 6 = 29 points, far exceeds the 6-point difference between correct-response rates in the ‘five difficult’ and open-ended conditions. The difference of differences is 29 − 6 = 23 points (95 per cent CI: [17 per cent, 26 per cent]). Given the evidence presented here, estimates of knowledge levels depend far more on differences between different kinds of closed-ended questions than on inherent differences between closed- and open-ended questions.

But perhaps it seems that we have stacked the deck in favor of such a finding. After all, the comparison of differences that we have just described involves the difference between open-ended questions and the most difficult closed-ended questions. One might find another difference of differences more revealing – a comparison of:

1. (a) the difference between correct-response rates to questions that have a short-easy set of response options and questions that have a long-difficult set of response options, to

2. (c) the difference between correct-response rates to all closed-ended questions and questions that are open-ended.

Table 2 speaks to this comparison as well. We know from its third regression that (a) is 35 – 6 = 29 percentage points. And the final regression shows us that (c) is 21 percentage points. Thus, the difference in correct-response rates between different kinds of closed-ended questions exceeds even the average difference between correct-response rates to closed- and open-ended questions. The difference of differences is 29 – 21 = 8 percentage points (95 per cent CI: [3 per cent, 10 per cent]). This result illustrates even more strongly that differences in estimated knowledge levels depend less on the choice of open- or closed-ended questions than on the number and difficulty of response options.Footnote ¹⁰

These results also bear on the recall-recognition distinction in knowledge measurement (e.g., Tedin and Murray Reference Tedin and Murray1979). Prior research suggests that open-ended questions measure recall while closed-ended questions measure recognition. Table 2 shows that although this distinction is sensible, it is not always substantial. Under difficult conditions, the proportion of people who can recognize the correct answer is similar to the proportion who can recall it.