Do incentives for accuracy reduce motivated reasoning among numerate partisans?

Scholars disagree about how worldview conflict interacts with scientific literacy to shape motivated numeracy. On the one hand, subjects highest in numeracy appear to exhibit more bias when interpreting uncongenial data (Kahan et al., Reference Kahan, Peters, Dawson and Slovic2017), consistent with findings that cognitive ability does not impede in-group bias (Stanovich et al., Reference Stanovich, West and Toplak2013) and that receiving new evidence exacerbates biases in experts (Baekgaard et al., Reference Baekgaard, Christensen, Dahlmann, Mathiasen and Petersen2017).

On the other hand, Mérola and Hitt (Reference Mérola and Hitt2016) demonstrate that more scientifically literate subjects are more persuaded by data sponsored by an opposing party, suggesting that higher numeracy may provide immunity against the congeniality bias.Footnote ² This is consistent with experimental research on persuasion (Redlawsk et al., Reference Redlawsk, Civettini and Emmerson2010) and studies on citizens’ updating from observing economic reality (Parker-Stephen, Reference Parker-Stephen2013), which suggest voters may update from uncongenial information.

This latter research is also indirectly reinforced by two sets of findings around incentives for accuracy and “motivated responding,” which indicate that the congeniality effect may have been overstated. First, the congeniality bias drops if individuals are motivated to answer accurately by monetary payments or appeals to accuracy (Bullock et al., Reference Bullock, Gerber, Hill and Huber2015; Prior et al., Reference Prior, Sood and Khanna2015). Partisans’ biased reasoning appears to be driven (in part) by the lack of stimulus for accuracy. These studies do not, however, test whether incentives lessen motivated numeracy,Footnote ³ instead focusing on bias in recalling facts or interpreting verbal information. This research, by contrast, tests whether incentives reduce motivated interpretation of data, a critical question to answer if we are to understand how individuals form evidence-based opinions.

Second, other studies differentiate “motivated learning” from “motivated responding” (Bisgaard, Reference Bisgaard2019; Khanna and Sood, Reference Khanna and Sood2018), where the former describes absorbing congenial facts more readily than uncongenial information and the latter – giving incorrect but congenial answers even after correctly processing the worldview-conflicting information.Footnote ⁴ Our study speaks to the difference in these concepts as follows. To measure motivated responding, Khanna and Sood (Reference Khanna and Sood2018)Footnote ⁵ announce incentives for accuracy after respondents were shown numerical tasks to examine motivated responding. Our research announces incentives before respondents are shown in table. This design allows us to answer to what extent incentives reduce motivated learning when interpreting uncongenial data.

In summary, evidence indicates that incentives for accuracy should lessen the congeniality bias. We now examine how incentives may interact with cognition and ideology to shape motivated numeracy.

Hypotheses

Replication of prior results is key to understanding contradictory findings. Replications may be direct (i.e., testing whether the original findings are true under the same conditions, using as-similar-as-possible design and analysis) or conceptual (i.e., retesting the underlying ideas to specify the conditions under which the same results would be found, potentially using different design or analysis) (Chambers, Reference Chambers2017). Our study conceptually replicates and extends Kahan et al. (Reference Kahan, Peters, Dawson and Slovic2017) by modifying treatments from and adding incentives for accuracy.

Conceptual replication

Prior research replicated Kahan et al.’s (Reference Kahan, Peters, Dawson and Slovic2017) findings directly and conceptually. First, Kahan and Peters (Reference Kahan and Peters2017) directly retested their own results in a new sample and found the same results. Second, Khanna and Sood (Reference Khanna and Sood2018) found partially consistent results in a conceptual replication; they did not, however, revisit the initial question of how numeracy interacts with ideology. Third, Baker et al.’s (Reference Baker, Patel, Von Gunten, Valentine and Scherer2020) conceptual replication finds that motivated numeracy depends only on partisanship, not on scientific literacy; this result contradicts both Kahan et al.’s (Reference Kahan, Peters, Dawson and Slovic2017) and Mérola and Hitt’s (Reference Mérola and Hitt2016) contrasting findings.Footnote ⁶

These mixed findings emphasize the need for conceptual replication that will revisit the questions of (1) whether accuracy is higher when interpreting congenial data, and (2) whether this effect is amplified among numerate individuals.

Extension

How do incentives for accurate learning influence motivated numeracy? Prior research reveals that rewarding accuracy reduces the congeniality bias (Bullock et al., Reference Bullock, Gerber, Hill and Huber2015; Prior et al., Reference Prior, Sood and Khanna2015), when respondents are asked to recall facts or interpret passages.

But do incentives reduce motivated processing of numerical data rather than verbal information? Khanna and Sood (Reference Khanna and Sood2018) employed incentives for accuracy when solving numerical tasks and found mixed results: monetary rewards reduced the congeniality bias among some but not all partisans. Given the limitations of Khanna and Sood’s (Reference Khanna and Sood2018) sample,Footnote ⁷ it is difficult to determine how incentives shaped motivated numeracy in partisans vs. moderates and in high- vs. low-numeracy subjects. We thus test the impact of incentives for accuracy in a sample with sufficient variation in numeracy and ideology.

Additionally, Khanna and Sood (Reference Khanna and Sood2018) focus on motivated responding as opposed to learning, announcing rewards for accuracy after respondents were given the numerical task. In our extension of Kahan et al. (Reference Kahan, Peters, Dawson and Slovic2017), we focus on the impact of incentives on motivated learning from numerical data; therefore, we announce incentives for accuracy before respondents are shown the task.

Our expectations for introducing an incentives condition are twofold. First, if incentives lessen motivated numeracy, then slightly higher stakes of giving a correct answer may alleviate motivated learning from data. Second, theoretically, incentives should increase the accuracy of high-numeracy individuals in both congenial and uncongenial conditions. This means that with incentives for accuracy, the congeniality bias, that is, the difference between correct answers in congenial and uncongenial treatment conditions for a given ideology, should not increase with numeracy.

Registration

The research design was peer-reviewed in this journal from August 2020–January 2021, and the embargoed OSF pre-registration (https://osf.io/935xb) was released in August 2021. No explorative tests were done on the data.

Deviations from pre-registration

The post-analysis peer-review process uncovered two mistakes in the pre-registered regression equations, and these deviations from the pre-registered design do not substantively alter the results. Table 1 lists said changes, section “Deviations from pre-registration” of the appendix also includes the original analysis that follows the pre-registered design without deviations.

Table 1. Deviations from Pre-registration

Case

From March 2020–August 2021, the U.S. (4.2% of the world population) experienced 18% of COVID-19 cases and 14% of COVID-19-related deaths (Ritchie et al. Reference Ritchie, Mathieu, Rodés-Guirao, Appel, Giattino, Ortiz-Ospina, Hasell, Macdonald, Beltekian and Roser2021). Despite the evidence that masks curb the spread of the virus (Ingraham, Reference Ingraham2020), in 2020, then President Trump’s and his allies’ dismissive statements about masks politicized masking further (Miller et al., Reference Miller, Colvin and Madhani2020), such that Republicans reported most reluctance to wear masks (Pew Research Center, 2020). Out of nineteen “much more Republican” states based on the 2016 election margin, seventeen did not introduce mask mandates (Markowitz, Reference Markowitz2020).

This survey was fielded in May 2021, when mask mandates remained politicized. Republican-led states opted to lift mask mandates earlier, notwithstanding low vaccination rates in those states. In May, eleven Republican governors lifted mask mandates, while House Speaker Pelosi indicated that the chamber would keep its mandate despite new CDC guidelines, sparking a GOP backlash (Elfrink, Reference Elfrink2021). In May–August 2021, eight states (all with Republican governors) prohibited local governments from issuing mask mandates, while ten states (all with Democratic governors) required masks in schools (Vestal Reference Vestal2021).

Experimental conditions

The study randomly assigned participants to either the no-incentives or incentives treatment. Within each group, respondents were randomly assigned to one of two data tasks shown in Figure 1. Participants were asked to judge, based on the table, whether cities that had a mask mandate were more likely to experience an increase/decrease in COVID-19 cases. The description of the tables clarified that the numbers represent cases since the mandate was implemented. The correct answers for conditions 1 and 2 are mutually opposite. Cell values remained the same across conditions, which differed only in their column headings. This covariance-detection task represents a difficult numeracy problem because one needs to consider ratios (Baker et al., Reference Baker, Patel, Von Gunten, Valentine and Scherer2020).

Figure 1. Experimental Conditions.

Participants’ numeracy was captured by six questions, randomized to appear either before or after the contingency table; the analysis section discusses order effects.

Ethics

The study was approved by the Institutional Review Board (#IRB_00132903). Participants were asked to provide consent before the survey and could withdraw at any time during the survey. The survey specified that the data were hypothetical and did not use deception.

Incentives

Participants received $1 for correctly answering the data tasks in Figure 1 in the incentives condition. Additionally, all respondents were motivated to answer the numeracy questions correctly: one of six numeracy questions was randomly selected for each participant and if the participant’s answer was correct in that question, they received $1. Qualtrics sent rewards to respondents as gift cards.

Operationalization

The appendix includes the survey instrument.

Independent variables

Numeracy _i is the number of correct answers the participant i gives to the six questions aimed at measuring numeracy (Lipkus et al., Reference Lipkus, Samsa and Rimer2001; Peters et al., Reference Peters, Hibbard, Slovic and Dieckmann2007; Schwartz et al., Reference Schwartz, Woloshin, Black and Welch1997).Footnote ⁸ Items on disease/infection were removed considering their relevance to the treatment.

Congenial _i is a continuous measure of attitude-consistent message that captures the degree of congeniality of the contingency table’s data with one’s ideology. This variable is constructed using, first, the continuous measure of ideology (Conservative _i ) and, second, two binary indicators of the condition to which a respondent was assigned (Covid_decreases _i or Covid_increases _i ). Using Cronbach’s α, two 7-point Likert scales of party affiliation and ideology form an aggregate scale, Conservative _i , where negative values indicate liberal Democrats and positive values indicate conservative Republicans (α = 0.795 which is similar to the α value of 0.83 in Kahan et al. Reference Kahan, Peters, Dawson and Slovic2017). Conservative _i is identical to the measure of ideology in Kahan et al. (Reference Kahan, Peters, Dawson and Slovic2017). Second, conservatives received uncongenial data in Covid_decreases _i and liberals – in Covid_increases _i . The Congenial _i variable follows the formula:

$$Congenial = {{\;Conservative*\left( {Covid\_increases - Covid\_decreases} \right)\;\;\;\;{\rm{if}}\;Conservative \ge 0} \over { - Conservative*\left( {Covid\_decreases - Covid\_increases} \right)\;\;{\rm{if}}\;Conservative < {\rm{ }}0}}$$

For a conservative respondent (Conservative _i >0), the Congenial _i variable is positive in the Covid_increases _i condition and negative in the Covid_decreases _i condition, while for a liberal (Conservative _i <0), Congenial _i is positive in Covid_decreases _i and negative in Covid_increases _i condition. The absolute magnitude of Congenial _i increases with the strength of one’s ideology.

Incentive _i is a binary indicator of whether a participant is assigned to that condition.

Statistical model

A linear probability model estimates the parameters of equations (1)–(4). Equations (1) and (2) are estimated on unincentivized observations to test hypotheses 1 and 2; equations (3) and (4) test hypotheses 3 and 4, utilizing the full sample. The appendix also includes the logistic regression models that estimate these equations.

(1) $${\rm{Correc}}{{\rm{t}}_i} = {\beta _0} + {\beta _1}{\rm{Congenia}}{{\rm{l}}_i} + {\beta _2}{\rm{Numerac}}{{\rm{y}}_i} + {\beta _3}{\rm{Numeracy}}_i^2 + {\varepsilon _i}$$

(2) \begin{align}{\rm{Correc}}{{\rm{t}}_i} = {\beta _0}{\rm{ }} &+ {\beta _1}{\rm{Congenia}}{{\rm{l}}_i} + {\beta _2}{\rm{Numerac}}{{\rm{y}}_i} + {\beta _3}{\rm{Numerac}}{{\rm{y}}_i} \times {\rm{Congenia}}{{\rm{l}}_i}{\rm{ }}\\ & + {\beta _4}{\rm{Numeracy}}_i^2 \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _5}{\rm{Numeracy}}_i^2 + {\varepsilon _i}\end{align}

(3) $${\rm{Correc}}{{\rm{t}}_i} = {\beta _0} + {\beta _1}{\rm{Incentiv}}{{\rm{e}}_i} + {\varepsilon _i}$$

(4) \begin{align}{\rm{Correc}}{{\rm{t}}_i} &= {\rm{ }}{\beta _0} + {\beta _1}{\rm{Numerac}}{{\rm{y}}_i} + {\beta _2}{\rm{Congenia}}{{\rm{l}}_i} + {\beta _3}{\rm{Numerac}}{{\rm{y}}_i} \times {\rm{Congenia}}{{\rm{l}}_i}{\rm{ }}\\ &\quad + {\beta _4}{\rm{Numeracy}}_i^2 + {\beta _5}{\rm{Numeracy}}_i^2 \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _6}{\rm{Incentiv}}{{\rm{e}}_i}{\rm{ }}\\ &\quad + {\beta _7}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _8}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numerac}}{{\rm{y}}_i}{\rm{ }}\\ &\quad + {\beta _9}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numeracy}}_i^2 + {\beta _{10}}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numerac}}{{\rm{y}}_i}{\rm{ }}\\ &\quad \times {\rm{Congenia}}{{\rm{l}}_i} + {\beta _{11}}{\rm{Incentiv}}{{\rm{e}}_i} \times {\rm{Numeracy}}_i^2 \times {\rm{Congenia}}{{\rm{l}}_i} + {\varepsilon _i}\end{align}

Sample

No pilot data were collected. Qualtrics fielded the survey.

Per the pre-registered power analysis, the required sample size is 3,000 with 1/3 of observations unincentivized and 2/3 – incentivized; Qualtrics collected 1,016 (512 and 504 in each contingency table condition) unincentivized responses and 2,034 incentivized (992 and 1,042 in each table condition), a total of 3,050 responses.

Balance on covariates

Per the pre-registered power calculations, respondents were randomly assigned to either a no-incentive or incentive treatment with a probability of 1/3 and 2/3, respectively. Respondents were also assigned randomly to conditions Covid Increases vs. Covid Decreases with a probability of 1/2. While assignment was random, some respondents were removed after failing the manipulation check and/or other quality controls outlined in the “Data exclusion […]” section. To verify that passing the exclusion criteria did not correlate with any observable attributes, we obtained differences-in-means between each pair of treatments (see the appendix). None of the differences-in-means is statistically discernible from 0 at the 5% level; the treatment groups are, therefore, balanced on observable attributes.

Multivariate analyses

The impact of congeniality and numeracy on accuracy among unincentivized respondents (Hypothesis 2)

Hypothesis 2 posits that, among unincentivized respondents, the congeniality bias increases with one’s numeracy. The coefficient estimate for the multiplicative interaction term Numeracy×Congenial is positive (only marginally significant in model 3), indicating that increases in numeracy improve accuracy even as congeniality increases. Following Kahan et al. (Reference Kahan, Peters, Dawson and Slovic2017), we also include the squared Numeracy term; the coefficient estimate for the interaction Numeracy ² ×Congenial is not statistically different from zero. We use predicted probability densities (Figure 2) and differences in the predicted congeniality bias for various numeracy levels (Table 3) to further understand if Numeracy _i has a range of values for which Congenial _i generates a statistically meaningful impact among unincentivized respondents.

Figure 2. Predicted Probabilities of Correctly Interpreting the Data.

Note: Density distributions derived via MC simulation from logistic regression that estimates equation 4 (output is shown in the appendix), when Congenial is set at –1 SD (i.e., respondents facing data contradicting their beliefs), at mean (i.e., ideological moderates on the ideology-party affiliation spectrum), and +1 SD (i.e., data are consistent with beliefs), and numeracy set at -1SD (1 out of six correct questions) for “low numeracy” and +1SD (4.35 out of six correct questions) for “high numeracy.” Numeracy is centered at “0” for ease of interpretation. +/-1SD Numeracy value is +/-1.654, and numeracy squared term is 2.736.

Section “Inferiority tests” of the appendix shows that we cannot conclude that a meaningful effect of the interaction terms on accuracy is absent.

The impact of incentives on accuracy (Hypothesis 3)

Hypothesis 3 expects that incentivized participants are more likely to correctly interpret the contingency table compared to those unincentivized. Models 5 and 6 of Table 2 test this expectation, estimating parameters in equation (3), using all observations. The coefficient estimates on Incentive _i in models 3 and 4 are substantively and statistically negligible; we thus fail to reject the null hypothesis of monetary incentives having no impact on accuracy.

Table 2. The Impact of Numeracy and Congeniality on Accuracy (Unincentivized Participants)

Standard errors in parentheses.

Note: Linear Probability Model with heteroscedasticity robust standard errors.

Control variables in the regression are age, gender, race, education, and voting 2016.

* p <0.10, ** p <0.05, *** p <0.01

Table 3. The Impact of Incentives, Numeracy, and Congeniality on Accuracy (All Participants)

Standard errors in parentheses.

Note: Linear Probability Model with heteroscedasticity robust standard errors.

Control variables in the regression are age, gender, race, education, and voting 2016.

* p <0.10, ** p <0.05, *** p< 0.01.

Table 4. Differences in the Predicted Congeniality Bias Between Less and More Numerate Individuals at Various Levels of Numeracy

Note: Congeniality Bias = Pr(correct = 1|congenial = −1 SD) − Pr(correct = 1|congenial = 1 SD); Numeracy is set at either +/−1 SD or +/−1.5 SD or +/−2 SD.