1 Introduction
The dual-process model of the mind has inspired a surge of recent research attempting to explain moral judgment (Reference Greene, Sommerville, Nystrom, Darley and CohenGreene, Sommerville, Nystrom, Darley & Cohen, 2001; Reference Greene, Morelli, Lowenberg, Nystrom and CohenGreene, Morelli, Lowenberg, Nystrom & Cohen, 2008; Reference Paxton, Ungar and GreenePaxton, Ungar & Greene, 2012; Reference Trémolière, De Neys and BonnefonTrémolière, De Neys & Bonnefon, 2012) and religious belief (Reference Gervais and NorenzayanGervais & Norenzayan, 2012; Reference Pennycook, Cheyne, Seli, Koehler and FugelsangPennycook, Cheyne, Seli, Koehler & Fugelsang, 2012; Reference Pennycook, Ross, Koehler and FugelsangPennycook, Ross, Koehler & Fugelsang, 2016; Reference Shenhav, Rand and GreeneShenhav, Rand & Greene, 2012; Reference Yilmaz, Karadöller and SofuogluYilmaz, Karadöller & Sofuoglu, 2016). The present research is concerned with the application of this model to the domain of political ideology, which is also an increasingly popular research topic. For instance, individual differences in political orientation and ideological beliefs have been associated with cognitive style (e.g., Reference Jost, Glaser, Kruglanski and SullowayJost, Glaser, Kruglanski & Sulloway, 2003; Jost, Sterling & Stern, in press). More specifically, there is an emerging literature on the analytic-intuitive thought differences across ideological groups (Reference Brandt, Evans and CrawfordBrandt, Evans & Crawford, 2015; Reference Deppe, Gonzalez, Neiman, Jacobs, Pahlke, Smith and HibbingDeppe et al., 2015; Reference Eidelman, Crandall, Goodman and BlancharEidelman, Crandall, Goodman & Blanchar, 2012; Reference Iyer, Koleva, Graham, Ditto and HaidtIyer, Koleva, Graham, Ditto & Haidt, 2012; Reference Kahan and CorbinKahan, 2013; Reference LandyLandy, 2016; Reference Piazza and SousaPiazza & Sousa, 2014; Reference Pennycook, Cheyne, Seli, Koehler and FugelsangPennycook et al., 2012; Reference Sterling, Jost and PennycookSterling, Jost & Pennycook, 2016; Reference Talhelm, Haidt, Oishi, Zhang, Miao and ChenTalhelm, Haidt, Oishi, Zhang, Miao & Chen, 2015; Reference Van Berkel, Crandall, Eidelman and BlancharVan Berkel, Crandall, Eidelman & Blanchar, 2015; Reference Yilmaz and SaribayYilmaz & Saribay, 2016, 2017).
A general observation in this line of research has been the negative correlation between the single-item self-placement political orientation measure (where typically 1 = liberal/left and 7 = conservative/right) and scores on the Cognitive Reflection Test (CRT; Reference FrederickFrederick, 2005), a test used to measure analytic thinking tendency (Reference Deppe, Gonzalez, Neiman, Jacobs, Pahlke, Smith and HibbingDeppe et al., 2015; Reference Pennycook, Cheyne, Seli, Koehler and FugelsangPennycook et al., 2012; Reference Iyer, Koleva, Graham, Ditto and HaidtIyer et al., 2012; Reference Talhelm, Haidt, Oishi, Zhang, Miao and ChenTalhelm et al., 2015; Reference Yilmaz and SaribayYilmaz & Saribay, 2016). However, using these measures, Kahan (2013) was not able to replicate this finding in a representative sample of U.S. participants. More recently, Landy (2016) replicated this finding in only one of three samples (see also Baron, 2017; Reference Kahan and CorbinKahan & Corbin, 2016; Reference Piazza and SousaPiazza & Sousa, 2014). What explains these contradictory findings?
The dual-process model originally argues that human mental operations are supported by two different set of mental processes. The evolutionarily older set (Type 1) provides automatic, intuitive, and fast responses whereas the newer set (Type 2) provides analytic, controlled, and slower responses (Reference EvansEvans, 2003; Reference Evans and StanovichEvans & Stanovich, 2013; Reference Morewedge and KahnemanMorewedge & Kahneman, 2010). However, this model has been criticized because it assumes that the automatic activations of intuitive processes are blocked sequentially by reflection (e.g., Reference Baron, Scott, Fincher and MetzBaron, Scott, Fincher & Metz, 2015; Bialek & De Neys, 2016; Reference KleinKlein, 2011; Reference ThompsonThompson, 2009; Reference Trémolière and BonnefonTrémolière & Bonnefon, 2014; Reference Sinayev and PetersSinayev & Peters, 2015). This suspect assumption is also evoked in discussions of the relationship between CRT performance and various psychological tendencies. For instance, Baron et al. (2015) argued that this sequential assumption cannot explain the relationship between CRT performance and moral judgment. In the present research, we refrain from drawing inferences regarding this theoretical discussion. However, this critique bears importance for investigations of the cognitive style-conservatism relationship because most past studies of the latter relationship have relied on the CRT to operationalize cognitive style.
There have been other criticisms directed at the CRT. For instance, Reference Sinayev and PetersSinayev and Peters (2015) found that higher scores on the CRT have more to do with numeracy skills than high-effort thought. Other researchers have shown that there is wide familiarity with the CRT in test populations and this may affect scores (Reference HaighHaigh, 2016; Reference Stieger and ReipsStieger & Reips, 2016). These criticisms have led to the development of alternative measures of ACS, such as CRT–2 (Reference Thomson and OppenheimerThomson & Oppenheimer, 2016).
In addition to overreliance on the CRT, most past studies have used the single-item self-placement measure of political orientation. The limitations of this measure have been discussed extensively (see Feldman & Johnson, 2014). For instance, people who are both socially and economically conservative; socially conservative but economically liberal; and socially liberal but economically conservative all tend to place themselves toward the conservative side of the single-item scale. When given the chance, some of these people identify with ideological labels other than conservative. The best known example may be libertarians: Feldman and Johnson (2014) have shown that approximately 20% of self-identified conservatives are actually libertarians who are conservative economically but not socially. Taking these observations into account, Pennycook et al. (2012) distinguished fiscal and social orientations and showed that self-placement on only the social dimension was significantly related to cognitive style. Likewise, Deppe et al. (2015) measured reactions to various social and economic policy issues in addition to general orientation and showed that socially liberal participants relied more on reflection while economic orientation generally appeared unrelated to cognitive style. Reference Yilmaz and SaribayYilmaz and Saribay (2016) replicated these findings using similar measures in a non-Western sample.
Contrary to the findings just summarized, some studies associate a neoliberal orientation (economic conservatism) with intuitive cognitive style. For instance, Sterling et al. (2016) found that participants who gave stronger support for neoliberal policies tend to be higher on bullshit receptivity, which is linked to intuitive thought. In addition, a recent meta-analysis (Jost et al., in press) reported a significant and negative relationship between CRT scores and economic attitudes, even though its effect size was smaller than that for the relationship between CRT scores and social conservatism.
As seen above, the literature offers mixed findings. It is clear that there is need for further research using operationalizations of ACS other than the CRT as well as more nuanced measures of political orientation. On the basis of this observation, we asked an Amazon Mechanical Turk sample to complete measures of ACS that included the CRT, two other problem-solving tasks, a measure of attitudes toward self-critical and reflective thinking (Actively Open Minded Thinking Scale; AOT), and measures of social and economic conservatism. We expected that social conservatism would be related (negatively) to ACS while economic conservatism would not. However, in light of recent criticisms of the CRT, we suspected that it may fail to reveal this expected pattern or otherwise behave differently than other measures of ACS that do not suffer from the same limitations.
2 Method
2.1 Participants
Five hundred and twenty-three participants were recruited via Amazon Mechanical Turk. Participants who did not complete the full battery and those with an IP outside of the U.S.A. were excluded from analyses. This resulted in 426 participants (Mage = 38.67, SDage = 13.81, 235 female, 160 male). 37.1% of the participants identified themselves as Democrat (n = 158). Eighty five were Republican (20%), 96 (22.5%) Independent, 12 Libertarian (2.8%), nine Socialist (2.1%), four Anarchist (0.9%), 18 Other (4.2%), nine “Don’t know/prefer not to say” (2.1%), and 35 missing.
2.2 Materials and Procedure
Participants were directed to an online survey that took approximately 25 minutes to complete. Except for demographic questions (which were always presented at the end in fixed order), the order of the measures listed below and the order of the items within each measure were randomized. The survey was implemented using PsyToolkit (www.psytoolkit.org; Stoet, 2010, 2017).
ACS.
We used three different measures of ACS (see Appendix A for the entire set of items). The first one was the CRT (Reference FrederickFrederick, 2005), which has been used in many other studies for the same purpose. Participants were given three test items each with an intuitive (but wrong) answer. Arriving at the correct answer requires relatively high-effort thinking. Those with a stronger tendency to think analytically can override the intuitive response and produce the logically correct answer. Open-ended responses were coded as correct (1) or incorrect (0) and averaged to produce a total CRT score (Cronbach’s α = .75).
The second ACS measure was CRT–2, developed by Reference Thomson and OppenheimerThomson and Oppenheimer (2016) as an alternative to CRT to overcome the problems of growing participant familiarity and emphasis on numeracy (see the Introduction). Unlike original CRT items, CRT–2 items are based on verbal questions. Open-ended responses were coded as correct (1) or incorrect (0) and averaged to produce a total CRT–2 score (Cronbach’s α = .64).
As the third measure of ACS, we used Baserate-conflict problems (BRC; see Reference Pennycook, Cheyne, Seli, Koehler and FugelsangPennycook et al., 2012). In these problems, as in the CRT, there is an intuitive (wrong) and analytic (correct) answer. Each problem presents a conflict between baserate information that appeals to Type 2 thinking (and helps to produce the correct answer) and misleading stereotypic content that appeals to Type 1 thinking. Analytic thought presumably enables participants to override the appeal of the misleading stereotypic information and take into account the baserate thereby helping to produce the correct answer. Participants chose among two answers and responses were recorded as correct (1) or incorrect (0) and averaged to produce the BRC score (Cronbach’s α = .75).
All measures of ACS were converted to POMP (percent of maximum possible) scores (Reference Cohen, Cohen, Aiken and WestCohen, Cohen, Aiken & West, 1999) to standardize them into a common scale ranging from 0 to 100. We then combined CRT, CRT–2, and BRC POMP scores into a single composite ACS score (Cronbach’s α = .76). We also report separate analyses for three different ACS measures.
Actively Open-Minded Thinking Scale.
This seven item scale was used by Reference Haran, Ritov and MellersHaran, Ritov and Mellers (2013) and measures the value placed on to be open to revisiting one’s beliefs when contradictory evidence is available (see Appendix B). Responses were collected on a 9-point Likert-type scale (1 = strongly disagree; 9 = strongly agree). This scale does not measure reflectiveness in the sense of measures like the CRT. It concerns only the direction of thinking (Cronbach’s α for this study = .69)Footnote 1. AOT responses were averaged and converted to a POMP score.
Conservatism.
We used several measures of conservatism. We asked participants their self-reported political (i.e., general), social, and economic orientations with the single-item self-placement question (“How would you place your political views generally speaking?” “When it comes to social issues, how liberal or conservative are you?”, “When it comes to economic issues, how liberal or conservative are you?”). The response format for these three questions was a Likert-type scale ranging from 0 (extremely liberal) to 10 (extremely conservative). In addition, we used a 12-item Social (Cronbach α for this study = .85) and Economic (Cronbach α for this study = .62) Conservatism scale developed by Everett (2013; see Appendix C). In this scale, participants rate how positive or negative they feel about different policy issues (such as Abortion or Limited government) on a feeling thermometer (0 = negative, 100 = positive). Social and Economic items were averaged separately. All the questions were also converted to POMP scores and averaged into two composite variables representing social (social orientation self-placement and Everett’s social conservatism subscale) and economic conservatism (economic orientation self-placement and Everett’s economic conservatism subscale). The general political orientation question was analyzed separately and we also report correlations for the separate scales.
Demographic Variables.
Participants were asked to report their gender (0 = male, 1 = female), age (in years), education level (1 = less than a high school degree, 6 = Graduate degree), income (total earned in 2016: 1 = $15,000 or less, 7 = over $100,000), and political affiliation. Education and income were converted to POMP scores.
3 Results
As expected, composite ACS was negatively correlated with the composite social conservatism score (r = –.166, p = .001) and general political orientation (r = –.116, p = .019), but not with the composite economic conservatism score (r = –.065, p = .183). Among demographics, female gender (r = –.101, p = .046), education level (r = .249, p < .001), and income (r = .113, p = .027) are significantly correlated with ACS.Footnote 2
Table 1 displays the correlations between the separate cognitive style measures and conservatism. When we separately analyzed three ACS measures, CRT did not significantly correlate with any of the conservatism measures including social conservatism (r = –.061, p = 218). However, both CRT–2 and BRC did significantly correlate with social conservatism (r = –.141, p = .004; r = –.167, p = .001, respectively) and general political orientation (r = –.129, p = .010; r = –.103, p = .039, respectively), but not with economic conservatism (r = –.066, p = .188; r = –.070, p = .159, respectively).
Table 1. Correlations between the cognitive style and conservatism measures.
Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. Polit. Orient. = Political Orientation; Social Orient. = Social Orientation; SECS Social Conserv. = Social Conservatism subscale of Everett’s (2013) scale; Econ. Orient. = Economic Orientation; SECS Econ. Conserv. = Economic Conservatism subscale of Everett’s (2013) scale; BRC = Baserate conflict; CRT = Cognitive Reflection Test; CRT–2 = Thomson & Oppenheimer’s (2016) newer Cognitive Reflection Test; AOT = Actively Open-Minded Thinking Scale.
On the other hand, AOT was negatively correlated with all conservatism measures including social (r = –.483, p < .001) and economic conservatism (r = –.292, p < .001; see Table 1). In other words, these results show that conservatism correlates negatively with cognitive style. However, economic conservatism correlates less well (if at all) than social conservatism. In addition, the original CRT correlates less well than other ACS measures whereas the AOT correlates more strongly.
For descriptive purposes, we provide in Tables 2 and Tables 3 the correlations among demographic variables and conservatism and cognitive style variables.
Table 2. Correlations between the demographic and conservatism measures.
Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. See Table 1 for abbreviations.
Table 3. Correlations between the demographic and cognitive style measures.
Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. See Table 1 for abbreviations.
One may think that the lack of correlations between CRT and political variables is due to the low reliability of the CRT. However, CRT did not fare particularly low in reliability compared to the other ACS measures (see Method). In addition, we examined the correlations between the cognitive style measures and found that the CRT correlated reasonably well with the other ACS measures and the AOT (see Table 4). This suggests that it is unlikely that the CRT in particular suffered from a reliability issue. Finally, we corrected the correlations between the cognitive style measures and political variables for measurement error (see Reference SpearmanSpearman, 1904)Footnote 3. These figures are shown in Table 5 and, while the disattenuated correlations are higher than the raw correlations shown in Table 1, they also indicate the separate ACS measures and economic conservatism are negatively but weakly related, if at all. Social conservatism, however, is more strongly related to ACS measures.
Table 4. Correlations amongst the cognitive style measures.
Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed. See Table 1 for abbreviations.
Table 5. Disattenuated correlations between the cognitive style and conservatism measures.
Note: a correlation of 0.1 is significant at p<.05, 0.13 at p<.01, two-tailed.
4 Discussion
As far as we know, our study was the first to use multiple measures of all the key constructs involved: ACS and political orientation. The present findings contrast somewhat with those from Jost et al.’s (in press) meta-analysis in that they revealed a solid relationship between ACS and social conservatism but only a weak relationship with economic conservatism. Moreover, of the three ACS measures — CRT, CRT–2 and BRC — the original CRT was the weakest, with no significant correlations with political measures. Thus, the conclusion reached about the relationship between ACS and political orientation on the basis of our data depended largely on the measure used.
Unfortunately, there is no way to check whether the same would be true of earlier studies, since most of them used only the CRT. This is evident from Jost et al.’s (in press) meta-analysis, which is dominated by the CRT. This situation warrants caution as reliance solely on the CRT has recently been criticized on an empirical basis (Reference BaronBaron et al., 2015). Taking into account such criticism, we employed a more verbally-oriented version of CRT (CRT–2 and partially BRC problems), as well, and it indeed yielded qualitatively different results.
Additionally, we used the AOT, a measure of attitudes toward self-critical and reflective thinking concerning the value placed on revising existing beliefs in light of incoming information, and found that it was negatively related to both social and economic conservatism. This suggests that economic conservatism is related to reflectiveness but mostly in the sense operationalized by the AOT rather than the sense of reflection/impulsivity focal to CRT, CRT–2, and BRC problems.
Overall, the findings are in line with some past research and contrary to others and thus, point to the need for continued research on the topic. For instance, the present findings contrast with analyses reported by Deppe et al. (2015) using only the CRT showing generally negative relationships between CRT scores and social conservatism. Deppe et al. (2015) attempted (but failed) to activate analytic thought experimentally whereas the correlations above are comparable only to their control condition. When we focus only on the control conditions in Deppe et al. (2015), there is a significant negative correlation between the original CRT and economic conservatism only in one of four studies. On further analyses of the same data by Baron (2015), including both the control condition and the (ineffective) experimental condition, there was a relatively strong and significant positive correlation between economic conservatism and the CRT in what was probably the most representative of the U.S. population, yet still a (smaller) negative correlation in one of the studies using Amazon Mechanical Turk. It thus seems that populations also differ in the relationship between the CRT and conservatism.
Similarly, the present findings are also not in line with our previous findings from Turkey showing a negative correlation between CRT scores and general political orientation (Reference Yilmaz and SaribayYilmaz & Saribay, 2016). However, they are consistent with Kahan’s (2013) and Landy’s (2016) research, showing a lack of relationship between CRT scores and self-placement on the single-item political orientation scaleFootnote 4. Finally, there is also partial overlap between our findings and those of Sterling et al. (2016) in that they found a negative relationship between the Need for Cognition–which is similar to AOT–and fiscal (vs. social) conservatism; and a negative relationship between their Heuristics and Biases measurement–which is similar to our ACS measures– and social (vs. fiscal) conservatism. Thus, it appears so far from these efforts that ACS and trait reflectiveness (e.g., AOT, Need for Cognition) may have different relationships with political attitudes. Baron et al. (2015) argue that CRT-type measures and AOT-type measures tap different cognitive styles (see also Baron, 2017). Baron et al. (2015; see also Baron, 1995) specifically claim that thinking can be characterized in two dimensions–amount and direction–and that CRT measures amount of thinking. CRT performance can be characterized in terms of the desire to spend more time on a task in order to produce more accurate answers. In contrast, AOT concerns the desire to spend time considering arguments countering a preferred position and openness on that position (see Reference BaronBaron, 2017). Baron (2017) argues that the desire to spend time for increased accuracy on CRT-type measures is related to the desire to spend time proving one’s position is superior to countering positions on political and moral issues. Those scoring high on the AOT can instead be characterized as being more willing to spend time processing counter-arguments. This may explain the stronger correlation between political conservatism and AOT versus other ACS measures. Future research should further probe this possibility and the underlying reasons for these differential relationships between social/economic conservatism as well as aim for increased clarity regarding the different senses of reflectiveness.
All in all, the present findings suggest that ACS is related to social, but not economic, conservatism. Two out of three operationalizations of ACS supported this pattern of relationship. The exception was the widely-used CRT which had no significant relationship with political orientation, either social or economic. This adds to previous criticisms of the CRT and suggests that, at the least, whenever it is used in this domain, it should be supplemented with other ACS measures not based on numeracy. In addition, trait reflectiveness is seen to show a distinct pattern of responses compared to ACS. Thus, different conceptualizations of reflectiveness should be distinguished and, in this sense, more sophisticated analyses of the cognitive style-political orientation link should be conducted in future research.
Appendix A: ACS Measures
CRT
1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? (cents)
2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? (minutes)
3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? (days)
CRT–2
1. If you’re running a race and you pass the person in second place, what place are you in?
2. A farmer had 15 sheep and all but 8 died. How many are left?
3. Emily’s father has three daughters. The first two are named April and May. What is the third daughter’s name?
4. How many cubic feet of dirt are there in a hole that is 3’ deep x 3’ wide x 3’ long?
BRC
1. In a study 1000 people were tested. Among the participants there were 5 engineers and 995 lawyers. Jack is a randomly chosen participant of this study. Jack is 36 years old. He is not married and is somewhat introverted. He likes to spend his free time reading science fiction and writing computer programs. What is most likely?
- Jack is a lawyer
- Jack is an engineer
2. In a study 1000 people were tested. Among the participants there were 4 kindergarten teachers and 996 executive managers. Lilly is a randomly chosen participant of this study. Lilly is 37 years old. She is married and has 3 kids. Her husband is a veterinarian. She is committed to her family and always watches the daily cartoon shows with her kids. What is most likely?
- Lilly is an executive manager
- Lilly is a kindergarten teacher
3. In a study 1000 people were tested. Among the participants there were 996 boys and 4 girls. Kelly is a randomly chosen participant of this study. Kelly is 13 years old. Kelly’s favourite subject is art. Kelly’s favourite things to do are shopping and having sleepovers with friends to gossip about other kids at school. What is most likely?
- Kelly is a girl
- Kelly is a boy
Appendix B: Actively Open-Minded Thinking Scale
1. Abandoning a previous belief is a sign of good character.
2. People should always take into consideration evidence that goes against their beliefs.
3. Beliefs should always be revised in response to new information or evidence.
4. Changing your mind is a sign of weakness. (R)
5. Intuition is the best guide in making decisions. (R)
6. It is important to persevere in your beliefs even when evidence is brought to bear against them. (R)
7. One should disregard evidence that conflicts with one’s established beliefs. (R)
Items marked with "(R)" are reverse-scored. Item 5 was not included in the analyses (see Footnote 1).
Appendix C: Social and Economic Conservatism Scale
1. Abortion (R)
2. Limited government
3. Military and national security
4. Religion
5. Welfare benefits (R)
6. Gun ownership
7. Traditional marriage
8. Traditional values
9. Fiscal responsibility
10. Business
11. The family unit
12. Patriotism
Items marked with "(R)" are reverse-scored. Social Conservatism items are those numbered 1, 3, 4, 7, 8, 11, 12. Economic Conservatism items are those numbered 2, 5, 6, 9, 10.
Open access
