1 Introduction
The whole problem of the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts. (attributed to Bertrand Russell)
Are wiser people more doubtful, or does experiencing doubt make one wiser? This is an old debate, and thinkers as far back as those in ancient Greece have weighed in on this fundamental question. In the opinion of arguably the wisest man in Greece, Socrates, the feature which makes one wise is recognizing the limits of one’s knowledge: “I am wiser than this man, for neither of us appears to know anything great and good; but he fancies he knows something, although he knows nothing; whereas I do not know anything, so I do not fancy I do.” (Apology, 21d). The key feature of wisdom in the opinion of one great ancient thinker is to recognize what one knows, does not know, and adapt behavior to be in line with these limitations. Experts in an topic may provide a useful measuring stick against which non-experts can compare their understanding. Indeed, the degree to which we are willing to defer to the opinion of experts demonstrates the wisdom that comes with understanding the limits of our knowledge.
Nevertheless, people often disregard the opinion of experts in favor of their own unlearned intuition, or the opinion of people similarly unknowledgeable to themselves. That is, people should defer more to experts than to lay opinion but, puzzlingly, they often do not (Reference Johnston and BallardJohnston & Ballard, 2016). What underlies this behavior, and more pressingly, how can we help people weight the opinion of those with demonstrated expertise more heavily when making decisions?
The highly specialized world of today should dictate that decisions of epistemic authority, choosing when to think for oneself versus deferring to experts, would usually favor deferring (Reference PiersonPierson, 1994), especially when we lack necessary background to understand information we receive (Reference KeilKeil, 2010). People tend to behave, however, in a manner that suggests that experts possess an authority on decisions (i.e., how to do things), but not necessarily on beliefs and values (i.e., which things to do) (Reference ZagzebskiZagzebski, 2012). For example, people might defer to experts on how to efficiently trade with a foreign country, but not on whether that country should or should not be traded with. In the latter case people tend to be influenced by the opinions of the general public as much as if not more than the opinions of professional economists (Reference Johnston and BallardJohnston & Ballard, 2016). That is, people appear to find the views of their peers just as convincing as those of experts when considering how to adjust their normative beliefsFootnote 1 in response to new information.
Why are experts not more influential than the average citizen when it comes to normative belief adjustment? One idea is that humans have an intuitive tendency to conflate the knowledge of others with their own (Reference Rabb, Fernbach and SlomanRabb et al., 2019). People explicitly recognize the division of cognitive labor (Reference Bromme, Kienhues, Porsch, Bendixen and FeuchtBromme et al., 2010; Reference KitcherKitcher ,1990), that is, they understand, even at a young age, that people differ in their levels of obtained expertise (Reference Keil, Stein, Webb, Billings and RozenblitKeil et al., 2008; Reference Landrum and MillsLandrum & Mills, 2015). When tasked to judge their own understanding of a complex phenomenon, people judge it to be greater when also instructed that experts fully understand the phenomenon, compared to when experts do not fully understand the phenomenon (Reference Sloman and RabbSloman & Rabb, 2016). This effect may arise because people tend to implicitly conflate their markers of who is possessive of such technical knowledge with their own actual knowledge of the topic (Reference Rabb, Fernbach and SlomanRabb et al., 2019). So, even though humans can identify economists as having privileged knowledge about an intricate process (e.g., the effects of international trade on citizenry), the economists’ very possession of such knowledge leads many non-experts to mistakenly believe they understand it too (at in least in some part). An economist (expert) in this case has little then to offer to non-experts in terms of specialized knowledge because they already feel as if they possess the knowledge.
There is no shortage of cases in which people overestimate how much they know about a particular topic (Reference Dunning, Johnson, Ehrlinger and KrugerDunning et al., 2003; Reference Fernbach, Rogers, Fox and SlomanFernbach, Rogers, et al., 2013; Reference Fernbach, Sloman, Louis and ShubeFernbach, Sloman, et al., 2013; Reference KeilKeil, 2003; Reference Kruger and DunningKruger & Dunning, 1999; Reference Moore and HealyMoore & Healy, 2008; Reference Rozenblit and KeilRozenblit & Keil, 2002) even in the context of economics (Reference Ortoleva and SnowbergOrtoleva & Snowberg, 2015). A collection of these cases is reflected in the tendency to believe that one understands (and is capable of explaining in detail) both inherently complex as well as ostensibly simple phenomena. This has been called the “Illusion of Explanatory Depth” (Reference Rozenblit and KeilRozenblit & Keil, 2002). In these instances, one reason people may mistake their superficial knowledge for in-depth knowledge of a phenomenon is because they internally fail to distinguish between their markers for that knowledge and the exact knowledge it marks (Reference Rabb, Fernbach and SlomanRabb et al., 2019). So, in cases where people hold an illusion of explanatory depth, they are unlikely to credit experts with possessing privileged information that they themselves do not possess. One consequence could be that people fail to revise their normative beliefs more to opinion from experts than opinion from random members of the public.
Importantly, the illusion of explanatory depth can be easily exposed. When asked to explain the mechanics of a process in detail, people become aware of the gaps in their knowledge of the causal structure and are then confronted with the actual limits of their expertise. (This effect may be analogous to making people aware of known unknowns, see Reference Walters, Fernbach, Fox and SlomanWalters et al., 2016.) This leads to a recalibration of their perceived knowledge, as people tend to adjust their understanding claim downward. Such a process has been demonstrated to apply to everyday objects like the mechanics underpinning the function of toilets and toasters (e.g., Reference Rozenblit and KeilRozenblit & Keil, 2002), and to complex social policies like immigration and trade (e.g., Reference Fernbach, Rogers, Fox and SlomanFernbach et al., 2013; Reference Vitriol and MarshVitriol & Marsh, 2018). Moreover, people may not even have to explicitly generate an explanation for the illusion of knowledge to be exposed as simply reflecting on how well one can explain the mechanistic process of how something works reduces overestimation of knowledge (Reference Johnson, Murphy and MesserJohnson et al., 2016). This result suggests that having to provide a casual explanation for some phenomenon reveals the gaps in one’s own knowledge. Recognizing the limits of one’s own knowledge can have downstream effects such as reducing political extremism (Reference Fernbach, Sloman, Louis and ShubeFernbach et al., 2013) or, speculatively, increasing the tendency to privilege expert consensus when given an opportunity to change our opinion on a matter of economic policy.
In this research we tested whether overestimation of knowledge can explain how people revise their normative beliefs given expert and laymen consensus. We hypothesized that inducing a feeling of ignorance might be an efficient method for getting people to rely on more valid sources of information (i.e., from experts) over less valid ones (i.e., from the public). In particular, if people believe they already understand something to a much greater extent than they really do, they may not appreciate the vast difference in expertise between laypeople and experts. Thus, it is possible that participants will find the utility of expert opinion to be equivalent to that of members of the public unless their illusions of explanatory depth have been exposed. We propose that lowering confidence in perceived understanding by exposing an illusion of explanatory depth would increase the perceived utility of experts by making people aware that their markers for the knowledge (e.g., economists know X) were not representative of their actual knowledge (e.g., I know X), and thus be more willing to credit people who are likely to possess that specialized knowledge of X. We would then expect this to lead to greater normative belief revision in response to expert, rather than lay opinion
We devised five studies to test the claim that exposing an illusion of knowledge will increase the influence of experts. The first study replicated the main finding of Reference Johnston and BallardJohnston and Ballard (2016) that people fail to adjust their normative beliefs more to expert rather than lay consensus. The second study introduced the explanation paradigm: participants were asked to provide a mechanistic, step-by-step explanation for exactly how something worked. This procedure led to greater normative belief revision in response to experts, as opposed to lay opinion. The third study replicated the results of second and provided evidence for the claim that undermining an illusion of even topic-irrelevant knowledge can lead to greater normative belief revision in response to expert rather than lay opinion. The fourth study replicated this finding across five different economic issues. The fifth study, also using the expanded set of issues, included a control condition that again replicated the main finding that people do not revise more to experts than to lay people.
2 Study 1
We first attempted to replicate the original finding of Reference Johnston and BallardJohnston and Ballard (2016) that people fail to revise normative beliefs to be more in line with expert opinion, instead preferring the opinions of lay people. However, we made an important design change from the original work: we implemented a pre-post design to test whether the findings are consistent across a within-subjects manipulation. Further, participants in our study responded to more than one economic issue.
2.1 Method
The materials and data for each study can be found on the Open Science Framework here: https://osf.io/2pzbe/.
2.1.1 Participants
We recruited 204Footnote 2 participants via Mechanical Turk who were required to be United States citizens above the age of 18 and have a HIT approval rating of at least 90%. No other recruitment restrictions were applied. All studies reported in this paper followed this restriction criteria. Participants were mostly white (79%), male (60%), had obtained at least some level of post-secondary education (83%), and were between the ages of 18 to 69 (M = 33.12, SD = 9.54).
2.1.2 Procedure
A brief overview of the procedure of this study (and all studies reported in the paper) can be found in Table 1 below. The design of the study was a 2 Time (Pre-Consensus Judgment/ Post-Consensus Judgment) by 2 Source of Consensus (Economists/ General Public) mixed design. Time was a within-subjects factor while Source of Consensus was a between-subjects factor. In the study, participants were asked to rate their agreement with an economic issue statement twice each for two separate issues. For the first agreement rating, participants were presented the statement plainly (e.g., “Trade with China makes most Americans better off”) and asked to rate their agreement on a 1 (strongly disagree) to 5 (strongly agree) scale with 3 representing uncertainty. This judgment was labelled the Pre-Consensus Judgment. The participants were then presented with the “consensus information” (described below, also see Table 2) said to be from their assigned source (either Professional Economists or General Public). With this information present, the participants then re-rated their agreement with the statement. This judgment was labelled the Post-Consensus Judgment. After providing the agreement judgments for one statement, they repeated this process for the other statement. Finally, each participant responded to two “trust in economists” questions.
Table 1: Overview of the procedural steps for each study.
Table 2: Economic issue statements and corresponding consensus information.
The economic statements used in this experiment were two selected from Reference Johnston and BallardJohnston and Ballard (2016): a “gold standard” statement and a “trade with China” statement. Table 2 contains all the economic issues used throughout the presented work and their accompanying consensus information. As suggested by the original authors, the key distinction between these two statements is the prior beliefs held by participants. Reference Johnston and BallardJohnston and Ballard (2016) found that most of their sample had an opinion regarding the benefit of trading with China on the US economy, but few had prior opinions on whether the US should or should not be on a gold standard.
The consensus information was provided in terms of responses to the same statement participants were judging but said to have been made by 100 members with varying political preferences of their assigned source (either Professional Economists or General Public). The economic statements and the levels of consensus used by Reference Johnston and BallardJohnston and Ballard (2016) (and adapted for the current work) were taken from the Initiative on Global Markets’ (IGM) panel of economists, consensus on each issue represents the opinions of actual economists and the diversity of opinions for each issue is unique (Table 2).
For the “trust in economists” questions, the first assessed the extent to which the participant trusted the opinions of professional economists when thinking about economic policy issues. The second assessed the extent to which the participant thought that members of Congress should rely on the opinions of professional economists when crafting public policy on economic issues.
2.2 Results
To test whether participants’ normative beliefs were revised in accordance with consensus information and whether the source of the information mattered, we assessed the difference in agreement judgments across Time. In other words, we examined the change in agreement with the economic statement from the Pre-Consensus judgment to the Post-Consensus judgment. For both statements, we found a main effect of Time such that there was a shift in agreement with the statement (consistent with consensus information) after having been provided consensus information (both p’s < .001). However, we did not find a Source of Consensus by Time interaction for either statement (both p’s > .135), indicating that participants did not exhibit greater change in agreement to the opinion of experts compared to the opinion of laypeople. Table 3 contains the descriptive statistics and inferential test statistics for this study. Together these results demonstrate that while people do revise their normative beliefs in accordance with consensus information, the source of the consensus (Professional Economists or General Public) appears to play no role in such revision, replicating the results of Reference Johnston and BallardJohnston and Ballard (2016).
Table 3: Descriptive statistics and inferential tests of main analyses of Study 1.
Note. Each agreement judgement was made on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale with 3 representing uncertainty. For the Trade With China issue, consensus information agreed with the issue, while for the Gold Standard issue, consensus information disagreed with the issue.
We next analyzed the role of trust. Just over half of respondents (55%) stated that when thinking about economic policy issues they trust the opinions of professional economists (to varying degrees). Similarly, 67% of respondents agreed that members of Congress should rely on the opinions of professional economists when crafting public policy. Importantly, each form of stated trust in experts was not related to change in agreement for either issue (all p’s > .348), nor was it predictive of change in agreement in response to (expert or non-expert) consensus information for either issue (all p’s > .394).
2.3 Discussion
Study 1 replicated the finding of Reference Johnston and BallardJohnston and Ballard (2016) that people fail to revise their normative beliefs more to expert than lay opinion. Moreover, we found no evidence that trust in the opinion of economists influences such updating behavior. We now turn to answer the question of why in Study 2 — why do people behaviorally fail to privilege the opinion of experts over the public? One possible reason is that people overestimate how much they know and therefore undervalue the opinions of experts. People might revise more to the opinion of experts if they were less confident in how much they think they know.
The data of Study 1 suggest that people understand the value of experts in an abstract sense (as a majority of the participants reported trusting the opinions of professional economists when making their own economic decisions), however, this was not reflected in behavior where they would be expected to give expert opinion greater weight than the opinion of non-experts. Here, people may be implicitly failing to disqualify themselves – and, by extension, their fellow members of the public – as experts. They may be aware of the value that experts provide but unaware that they are conflating the expert’s knowledge with their own. That is to say, on these economic issues people implicitly consider themselves to be experts. Participants could understand that an expert is an expert, but in this case may not believe that the experts’ specialized knowledge exceeds their own and other members of the public. Exposing the illusion of explanatory depth could increase the salience of the difference between the topical knowledge of ordinary individuals and experts, and could thus increase normative belief revision in response to expert opinion to a greater degree than to public opinion. We conducted a second study to test this prediction.
3 Study 2
People tend to fail to correctly assess how much they really know about how the world works. Often, we think we can explain even ordinary phenomena (e.g., how recycling works) in more detail than we really can. When asked to mechanistically explain how something works in full detail, however, we become aware of our apparent lack of knowledge, and often experience humility at our overconfident assessment of our knowledge (Reference Rozenblit and KeilRozenblit & Keil, 2002). Importantly, recognition of our lack of knowledge happens without being provided any external feedback on the explanations provided. That is, without being told we do not know as much as we think we do, we realize it entirely by ourselves. Known as the illusion of explanatory depth, this paradigm reveals the false beliefs that many of us have regarding our knowledge of a topic.
It is this paradigm that was implemented in Study 2 as an attempt to make participants more aware of the discrepancy of the knowledge they possess, and that of an expert in economics.
3.1 Method
3.1.1 Participants
Three hundred and ninety-nine participants were recruited via Mechanical Turk and were mostly white (77%), male (56%), had obtained at least some level of post-secondary education (86%), and were between the ages of 18 to 68 (M = 35.36, SD = 10.51). In addition to the recruitment restrictions outlined in Study 1, potential recruits were also barred if they had participated in the previous study. Furthermore, we limited participation to unique IP addresses such that only one participant per IP address could complete the study.
3.1.2 Procedure
The procedure of this study expanded upon the procedure of Study 1. Table 1 provides a brief overview of the procedural steps of the study. In this study, participants rated their agreement with a single economic issue twice. Prior to providing their agreement judgments, each participant completed an illusion of explanatory depth exercise analogous to Reference Rozenblit and KeilRozenblit and Keil (2002). Here, participants would be given a topic (i.e., the impact that trading with China has on the US economy). First, participants were asked to rate how well they thought they understood the topic. This rating was made on a 1 (little understanding) to 7 (thorough understanding) scale that participants were provided instructions on how to use. Second, they were asked to explain in as much detail as possible how their topic (i.e., trading with China affects the US economy) worked. Finally, each participant rated their understanding of the topic again.
Once each participant finished the Writing Task (rate understanding, generate explanation, re-rate understanding), they would proceed to provide their agreement with the economic statement. Similar to Study 1, participants would rate their agreement with the economic issue. Then they would provide this judgment once again although the second time featured consensus information from their randomly assigned source (see Table 2 for the issue statement and its corresponding consensus). Unlike Study 1, all participants provided only their judgment for the “Trade With China” issue. This also meant that each participant’s Writing Task asked them to rate their understanding of and explain the impact of trading with China on the US economy.
One key detail was that half of the participants were asked to make a third agreement judgment with the economic issue statement. However, this judgment occurred before completing the Writing Task (this judgment is referred to as the “Pre-Writing” judgment) as opposed to after like the other two agreement judgments. Following the procedure used by Reference Fernbach, Rogers, Fox and SlomanFernbach et al., (2013) who demonstrated that exposing an illusion of knowledge can reduce position extremism on political issues (although this has not been consistently demonstrated, see Reference Voelkel, Brandt and ColomboVoekel et al., 2018) this additional judgment would allow us to examine whether exposing an illusion of knowledge would reduce position extremism on economic issues.
3.2 Results
We first assessed whether exposing an illusion of knowledge would decrease participant’s position extremity in their previously held economic beliefs. To this end, we tested whether the extremity of the Pre-Writing judgment (that half, n = 198, of the sample provided) was greater than the extremity of the Pre-Consensus judgment. To conduct this analysis we created an index of Polarity that was expressed as the absolute distance of one’s opinion from the “uncertain” response (the middle of the scale) for the Pre-Writing (M = 0.93, SD = 0.59) and Pre-Consensus judgments (M = 0.88, SD = 0.62). We found no significant decrease in Polarity after being asked to generate a mechanistic explanation (t(197) = 1.12, p = .132).Footnote 3
Next, we tested the effect of exposing an illusion of explanatory depth on normative belief revision in response to consensus information. Table 4 contains the descriptive statistics for each agreement judgment. We found a main effect of Time, such that people changed their agreement in response to receiving consensus information regardless of the source (F(1, 396) = 59.82, p < .001).Footnote 4 We also found a Source of Consensus by Time interaction (F(1, 396) = 14.12, p < .001). Further analysis revealed that participants still changed their agreement in response to the opinion of laypeople (t(206) = 3.11, p = .002, d = 0.22), but exhibited far greater change in response to the opinion of professional economists t(190) = 7.40, p < .001, d = 0.54). Figure 1 contains graphical depictions of this analysis as well as comparable analyses for Studies 3–5.
Table 4: Descriptive statistics for the agreement judgments of Study 2.
Note. Each agreement judgement was made on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale with 3 representing uncertainty. The Pre-Writing judgment contains only half of the sample (n = 198) while the Pre-Consensus and Post-Consensus judgments contain the entire sample. In this case, only the Trade With China issue was rated and consensus information agreed with the issue.
Figure 1: Main analyses graphs for studies 2–5 demonstrating change in agreement with the economic issue after being provided consensus information after completing the Writing Task. In each case, greater agreement with the issue reflected the opinion of the consensus information. In Studies 2 and 3 only the Trade With China issue was provided. In Studies 4 and 5 one of five possible economic issues were provided (the corresponding figures collapse across issue). In Study 3, the Unrelated How writing task asked participants to explain how modern recycling worked in the U.S. city. The Related How writing task asked participants to explain how trading with China affects the U.S. economy. In Study 4, the Unrelated How writing task asked participants to explain how a helicopter takes flight. The Related How writing task asked participants to explain how their assigned economic issue worked. The Related Why writing task asked participants to explain why they held their position on the economic issue. In Study 5, the Control writing task had participants reproduce a block of text that was displayed as an image. The Related How writing task asked participants to explain how their assigned economic issue worked. Error bars in each graph represent ±1 standard error of the mean.
3.3 Discussion
In Study 2, we found that, after being asked to explain the mechanisms of foreign trade, people became far more influenced by the opinions of economists than those of laypeople. While the participants still adjusted their normative beliefs to both sources of consensus, they did so to a far greater extent when presented with economist opinion than with lay opinion.
We next sought to explore why exposing an illusion of knowledge led to an increase in receptivity to expert opinion with a third study. To this end we generated two competing explanations. The first hypothesis suggests that exposure made participants aware of how little they know about the particular economic issue (the effects of foreign trade). As such, they were more willing to revise their beliefs to be in line with experts who likely possessed topic-relevant knowledge. The second hypothesis suggests that exposing the illusion of knowing induced a general feeling of ignorance in participants, and in turn, made them less convinced of their general expertise in any topic. Thus, they would be more influenced by the opinions of experts than by the opinions of their peers. If the second explanation is true (an induction of ignorance), failing to explain any issue would produce a similar willingness to revise their normative beliefs. If the first explanation is true (lack of topic-relevant knowledge), however, we should observe no revision after failing to explain an irrelevant issue (e.g., how modern recycling works). The next study aimed to replicate the findings of Study 2 while testing these two competing explanations.
4 Study 3
Study 3 attempted to replicate the previous study’s findings and to further test whether the content of the to-be-explained material in the explanation paradigm mattered. The question was whether is it necessary to make a participant experience a feeling of ignorance on a specific topic (in this case an economics topic), or, alternatively, is failing to explain a complicated procedure on any topic enought for participants to privilege the opinion of experts? To do so, we added a writing condition where participants would explain the recycling process of a modern U.S. city rather than the mechanisms of foreign trade. We believed that recycling is a topic that would be familiar enough to subjects to appear superficially simple while being complex in nature. As such, we deemed it a likely candidate to produce an overestimation of knowledge.Footnote 5
4.1 Method
4.1.1 Participants
We recruited 401 participants via Mechanical Turk with the same restrictions previously used in Study 2. Participants must not have participated in either Study 1 or 2 to enter this study. Respondents were mostly white (77%), male (55%), had obtained at least some level of post-secondary education (88%), and were between the ages of 18 to 77 (M = 35.96, SD = 11.31).
4.1.2 Procedure
The procedure followed that of Study 2 except for two changes. (Table 1 provides a brief overview of the procedural steps all studies.) First, every participant (rather than half, as in Study 2) made Pre-Writing agreement judgment. That is, each participant mad three agreement judgments total, one before the Writing Task and two after. Second, in an unrelated-content writing condition, half of the participants ratee their understanding of recycling and explain how it works in a modern US city, instead of writing about the impact of trading with China on the US economy. So, not only were participants randomly assigned as to which Source of Consensus they would receive (economists or the public) they were also randomly assigned, orthogonality, a Writing Task (related or unrelated).
To summarize the procedure. Participants first rated their agreement with the Trade with China economic issue (Pre-Writing Judgment). They then completed the Writing Task regarding a related or unrelated topic. Then they provided their agreement with the economic issue again (Pre-Consensus Judgment). Finally, they provided their agreement rating for a third time except this time they did so with the consensus information from their randomly assigned source present (Post-Consensus Judgment).
4.2 Results
We first tested whether each Writing Task reduced position extremity. To do so we again created a Polarity index to measure the average degree of distance from uncertainty in the Pre-Writing (M = 0.92, SD = 0.61) and Pre-Consensus judgments (M = 0.87, SD = 0.63). We found a significant reduction in Polarity after completing the writing task (F(1, 398) = 6.33, p = .012). Further, the Writing Task by Polarity interaction was not significant (F(1, 398) < 1). It thus seems that both writing topics (related and unrelated to trade with China) had a similar effect on reducing position extremity.
We then tested whether people revised their normative beliefs differentially, dependent on both the source of the consensus information and what topic they explained. That is, we examined if people gave additional weight to the opinion of experts when put through the explanation paradigm, as in Study 2, and whether this paradigm was required to be topic-relevant or not. We did not find a significant Time by Writing Task by Source of Consensus interaction (F(1, 397) < 1; see Figure 1), indicating that the pattern of agreement change to the source of the information was not significantly different across explanations. In other words, regardless of whether participants explained how trading with China impacts the US economy or how recycling in a modern US city works, their subsequent normative belief revision to consensus information was similar. As such, these explanation conditions were collapsed across to provide a higher-powered analysis of whether participants revised their normative beliefs more to experts than to laypeople.
Consistent with Study 2, we found a significant Time by Source of Consensus interaction, such that people changed their agreement more in response to expert opinion than to public opinion (F(1, 397) = 12.96, p < .001; see Table 5 for the descriptive statistics of the agreement judgments). Further analyses revealed that participants significantly changed their agreement to expert opinion (t(203) = 6.97, p < .001, d = 0.98), while they did not do so to public opinion (t(196) = 1.65, p = .102). After having an illusion of knowledge exposed, participants revised their normative beliefs on an economic issue to a far greater extent when presented with the opinions of professional economists than with the (same) opinions of the general public.
Table 5: Descriptive statistics for the main judgments of Study 3 by cell.
Note. Each agreement judgement was made on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale with 3 representing uncertainty. The Unrelated How writing task asked participants to explain how modern recycling worked in the U.S. city. The Related How writing task asked participants to explain how trading with China affects the U.S. economy.
4.3 Discussion
This study replicated the finding that after attempting to explain an economic issue mechanistically, people revise their opinion of that economic issue more when they receive consensus information from economists (experts) than when they receive consensus of the general public. We also tested competing hypotheses targeting whether the illusion of knowledge exposed needs to be topic-relevant or not. We found that the effect of explaining on normative belief revision occurred regardless of whether the written explanation was about the exact issue (trading with China) or an unrelated issue (recycling in a U.S. city). We also found that position extremity was decreased after explaining how something works, regardless of the topic of that explanation, consistent with the findings of Reference Fernbach, Rogers, Fox and SlomanFernbach et al., (2013).
One interpretation of the results of Study 3 is that, when individuals are presented with the opinions of experts and given the chance to update their normative beliefs, they do not credit the experts with possessing privileged information (or at least possessing information that the general public does not). Instead, they may believe that since the experts possesses that knowledge, they do too. When made aware of their lack of both topic-relevant and -irrelevant knowledge, people change their minds to a greater extent to expert than to public opinion. So, we suggest that exposing an illusion of knowledge shifts an individual’s mental model of what knowledge an expert possesses relative to themselves, leading them to revise their normative beliefs more in response to consensus from experts than from random members of the general public. We refer to this as inducing a feeling of ignorance. People may ordinarily maintain a feeling that they are generally more knowledgeable than they truly are on all topics, which the exposure to the explanation paradigm undermines by making their ignorance directly salient to them.
A shortcoming of the past two studies employing the writing paradigm is a lack of a true control group without induction of a feeling of ignorance. Based on Studies 2 and 3, we cannot claim that exposing ignorance led to greater belief revision than a group without the feeling of ignorance induced. To address this limitation, however, we conducted a cross-study analysis to test whether there was significantly more adjustment to experts than laypeople in the second and third studies compared to the first, treating the first study as a control condition. We found a significant Time by Source of Consensus by Study interaction, F(2, 997) = 4.99, p = .007. Further probing of this interaction revealed that Studies 2 and 3 each featured significantly greater agreement change in response to expert opinion than to public opinion in comparison to Study 1. Moreover, Study 2 and 3 were not significantly different from each other in this manner.Footnote 6 However, as this test was an internal meta-analysis of non-pre-registered studies, caution should be applied when interpreting this result (see Reference Vosgerau, Nelson, Simonsohn and SimmonsVosgerau et al., 2019). As a result, we introduced control conditions for the following two studies.
Another valid criticism of the studies conducted so far is the lack of variability in economic issue stimuli. We are unable to rule out the possibility that our results depend on something idiosyncratic to this specific issue, trade with China. To make a broader claim we need to demonstrate the effect across multiple economic issues. In addition to a control condition, the next two studies attempt to address this problem of stimulus sampling (Reference Wells and WindschitlWells & Windschitl, 1999).
5 Study 4
To address the concern of stimulus sampling, this study attempted to replicate Study 3’s findings across several economic issues. The issues selected were the five used in the original work by Reference Johnston and BallardJohnston and Ballard (2016). Further, as it is possible to suggest that the “unrelated” explanation condition in the previous study (how recycling works in a US city) is not unrelated enough, that is, it is an issue that a professional economist could have a knowledgeable opinion on, we changed the topic to be explained. The new unrelated writing task would feature a topic used in early explanatory depth research: how a helicopter takes flight (Reference KeilKeil, 2003). Also, in an attempt to address the issue of having no control condition in the previous two studies, we included a condition we hypothesized would work as a control: explaining why you hold the belief you do about the issue, rather than how it works. This method is based off the condition implemented by Reference Fernbach, Rogers, Fox and SlomanFernbach et al., (2013) who used it to demonstrate that explaining how rather than why leads to a decrease in political extremism. If we find that economic extremism is reduced by how but not why, then this could represent a valid control condition. If explaining why also reduces position extremism then it is very unlikely it would produce a belief revision effect discrepant from the how conditions. In sum, participants would be writing about one of: how their one economic issue works, why they hold the opinion they do of that economic issue, or how a helicopter takes flight.
5.1 Method
5.1.1 Participants
We recruited 1000 participants via Mechanical Turk for this study. Participants must not have completed any of the previous studies to participate. In addition to the recruitment restrictions applied for Studies 2 and 3, we also blocked responding from suspicious geolocations associated with bot farms via the Turk Prime feature (Reference Litman, Robinson and AbberbockLitman, Robinson & Abberbock, 2017). Respondents were mostly white (70%), of evenly mixed gender (49% male), had obtained at least some level of post-secondary education (71%), and were between the ages of 18 to 77 (M = 36.92, SD = 11.86). Prior to analysis we excluded 12 participants for either failing to write anything and/or failing to respond to any of the three agreement judgments. This left 284 participants in the related how condition, 376 participants in the unrelated how condition, and 329 related why condition.
5.1.2 Procedure
Table 1 provides a brief overview of the procedural steps of the study. This study’s procedure was nearly identical to Study 3, as each participant provided an agreement judgment, completed the Writing Task, and then provided their agreement two more times. Two changes were made to the Writing Task. First, a Related Why condition was added. In this condition participants explained why they held the position on the economic issue that they did (e.g., why they agreed that trading with China makes most Americans better off). Second, the Unrelated How writing condition had its topic changed from how recycling works in a modern US city to how a helicopter takes flight. So, participants in this experiment would either rate their understanding of and explain how their economic issue worked, why they held the position on the economic issue that they did, or how an unrelated issue works.
The pool of possible economic statements was expanded from 1 to 5, but each participant was randomly assigned to respond to only a single issue (three times). As the opinions of each economic issue were provided by professional economists, the diversity and levels of agreement (and disagreement) for each issue is unique (Table 2). However, for the purposes of analyses, each issue was coded such that a higher score reflected greater agreement with the consensus information.
5.2 Results
We first tested whether explaining how but not why reduced position extremity. In other words, we examined whether Polarity was reduced from the Pre-Writing to Pre-Consensus agreement judgments and whether this varied as a function of Writing Task (why vs. how). We found a main effect of Writing Task such that position extremity was reduced from the Pre-Writing judgment (M = 0.84, SD = 0.66) to the Pre-Consensus judgment (M = 0.80, SD = 0.68). Importantly, we did not find a significant Writing Task by Polarity interaction, F(2, 988) < 1, suggesting that the observed reduction in position extremity did not differ across the various Writing Task conditions.Footnote 7 As a result, one should not expect there to be a difference in belief revision based on these writing conditions.
Next, we tested whether generating a written explanation would lead to greater revision to the opinion of experts compared to laypeople. Table 6 contains the descriptive statistics pertinent to this analysis. We found a Source of Consensus by Time interaction (F(1, 985) = 12.69, p < .001)Footnote 8, suggesting that after generating a written explanation, participants revised more to the opinion of experts than the opinion of laypeople. Consistent with the previously described Polarity results, we found no evidence that the observed effect varied as a function of Writing Task. The three-way Source of Consensus by Time by Writing Task interaction was not significant (F(2, 985) < 1; Figure 1). This result demonstrates that after writing, regardless of what participants wrote, they proceeded to revise their normative beliefs to be more consistent with the opinions of experts than the opinions of laypeople
Table 6: Descriptive statistics for the main judgments of Study 4 by cell.
Note. Means and (Standard Deviation) are provided in the table. Each agreement judgement was made on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale with 3 representing uncertainty. The Unrelated How writing task asked participants to explain how a helicopter takes flight. The Related How writing task asked participants to explain how their assigned economic issue worked. The Related Why writing task asked participants to explain why they held their position on the economic issue.
The lack of difference between the originally planned written control condition (writing about why they hold their belief) and the written experimental conditions (writing about how the economic issue works or how a helicopter takes flight) is potentially problematic for the account we are presenting. So, we decided to further explore whether the condition we intended to serve as a control condition truly did. With the benefit of hindsight, we realized that when queried for reasons why someone holds a position on an economic issue, they may start attempting to explain how it works instead. In complex and technical economic issues like the ones presented to participants here, it may be the case that a consideration of “why” will tend to reduce to an explanation of “how”. For instance, it would be difficult to find an explanation to the question “why do you believe a ship floats on the water” without necessarily appealing to its underlying mechanisms. As a result, this may be why the written why condition reduced position extremity – because participants were writing about how it works in their natural explanation of why they believed what they believe. To explore this possibility we had two independent, hypothesis-blind coders read each participant’s explanation and categorized them as an attempt to: explain how something works, why someone believed in something, or reported stating “I don’t know” or wrote nonsense. The results of the coding analysis revealed that the distribution of responses between the how and why writing conditions were nearly identical. That is, for one coder, 38% of participants in the related how condition explained “why” they believed what they believed, and 48% of them explained “how” it worked. This is compared to the 42% who wrote “why” they believed it and 43% who wrote “how” it worked in the related why condition.Footnote 9 This is direct evidence that participants found it difficult to specifically write about how something worked or why they believed it. In addition to the main effect of Polarity (position extremity) reduction, the results of the independent coders suggest that the substance of what was being written about in the Related How and Related Why Writing Task conditions was essentially the same thing. Therefore, we believe it as appropriate to treat the why condition in this study as a further experimental condition.
5.3 Discussion
Across five different economic issues, each with a unique level of consensus, we replicated the finding that puncturing an illusion of knowledge (inducing a feeling of ignorance) leads to greater normative belief revision in response to the opinion of experts than the opinion of laypeople. In addition, we found further evidence for the generality of the effect that inducing a feeling of ignorance has on normative belief updating, as even generating a written explanation about an irrelevant topic (i.e., how a helicopter takes flight) led to the downstream revision effect.
While Study 4 helped address the concern of stimulus sampling, the lack of a true control condition remained an issue. As a result, we decided to run one more study that would contain a dedicated control condition.
6 Study 5
6.1 Method
6.1.1 Participants
We recruited 653 participants via Mechanical Turk for this study. In addition to the recruitment restrictions implemented in Study 4, potential participants could not have previously completed any of Studies 1–4. Respondents were mostly white (75%), an even mix of gender (51% men), had obtained at least some level of post-secondary education (67%), and were between the ages of 18 to 75 (M = 36.33, SD = 11.06). Prior to analysis we removed all participants who wrote nothing (1% of the sample). This left 246 participants in the experimental condition and 403 participants in the control condition.Footnote 10
6.1.2 Procedure
This study’s procedure was nearly identical to Study 4 as each participant provided an agreement judgment, completed the Writing Task, and then provided their agreement two more times (see Table 1 for a brief overview of the procedural steps of this study). The only modifications to the procedure were to the Writing Task as it was reduced to contain only two conditions. The Related How condition where participants would explain how their economic issue worked remained unchanged. The Unrelated How and Related Why conditions were removed and replaced with a Control condition. In this condition, participants would copy the text from a descriptive passage that was in image form (to prevent copy and pasting). The length of the descriptive passage was approximately equivalent to the amount of writing entered for an average written explanation in the previous studies.
6.2 Results
We first tested whether explaining how an economic issue worked led to a greater reduction in position extremity across the Pre-Writing to Pre-Consensus judgments, compared to writing out text displayed in an image. We did not find a significant reduction in Polarity from the Pre-Writing (M = 0.84, SD = 0.67) to Pre-Consensus judgments (M = 0.84, SD = 0.70, F(1, 648) < 1). Furthermore, we did not find a significant Writing Task by Polarity interaction (F(1, 648) = 1.82, p = .177). Thus the experimental condition did not exhibit a greater reduction in position extremity across judgments compared to the control condition (as overall, no reduction in position extremity was observed).
We then tested whether participants in the control condition changed their agreement more to expert versus lay opinion and whether this difference was distinguishable from the explanation condition. Table 7 contains the descriptive statistics pertinent to this analysis. The Time by Source of Consensus by Writing Task interaction was not significant (F(1, 646) < 1 (see Figure 1).Footnote 11 As indicated by the position extremity results, this result suggests that those in the explanation condition did not revise significantly more to experts versus the public compared to those in the control condition. However, we found a main effect of Source of Consensus such that people changed their agreement with the economic statement more to expert opinion than public opinion (F(1, 646) = 4.60, p = .032).Footnote 12
Table 7: Descriptive statistics for the main judgments of Study 5 by cell.
Note. Means and (Standard Deviation) are provided in the table. Each agreement judgement was made on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale with 3 representing uncertainty. The Control writing task had participants reproduce a block of text that was displayed as an image. The Related How writing task asked participants to explain how their assigned economic issue worked.
In an attempt to determine whether the control condition in this study replicated the results of Study 1 and of Reference Johnston and BallardJohnston and Ballard (2016), we examined whether those in the control condition exhibited greater change in agreement in response to expert versus lay consensus after having completed the copying-text Writing Task. Consistent with these previous findings, the Time by Writing Task interaction was not significant for this group (F(1, 402) = 28.42, p = .187). This result suggests that participants in the control condition did not privilege the opinion of experts over laypeople when provided the opportunity for normative belief revision.
6.3 Discussion
Contrary to Studies 2–4, Study 5 failed to replicate the effect that after generating a mechanistic explanation for how something works, people will revise their normative beliefs more to expert consensus than public consensus. However, when looking only at the control condition, we replicated the finding of Study 1 and of Reference Johnston and BallardJohnston and Ballard (2016) that people fail to privilege the opinion of experts over the opinion of laypeople. Apparently, Study 5 represents a case of the experimental manipulation failing to work. However, we aimed to provide the most comprehensive test for our claim that exposing an illusion of knowledge leads to greater normative belief revision to experts than when no illusion of knowledge is punctured. To do this we compiled all the data from our experiments and computed the main analyses of interest.
7 Internal Meta-analysis
The compilation of Studies 1–5 produced a dataset that contained responses from 2,862 unique participants. For the purposes of analyses, each participant was grouped to either the Experimental (n = 2,050) or Control (n = 812) condition. The Experimental condition consisted of each participant in Studies 2–4 who completed an experimental writing condition (which was all of them). This meant that participants who explained how their economic issue worked (Related How), how recycling in a modern US city worked (Unrelated How), how a helicopter takes flight (Unrelated How), or why they held their stance on the economic issue (Related Why), were compiled into the same group. The Experimental condition also featured the participants from Study 5 who completed the Related How Writing Task. The Control condition comprised of the participants from Study 1 and those in the copying-text condition of Study 5. This meant that the Control condition in this dataset represented participants in either an “active” (Study 5) or “passive” (Study 1) control condition. Table 8 shows the main results.
Table 8: Descriptive statistics of the main analysis for the compiled data set.
Note. Means and (Standard Deviation) are provided in the table. Each agreement judgement was made on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale with 3 representing uncertainty. Consensus information reflected agreement (and as such, higher scores). So higher scores on the Post-Consensus judgment reflect greater change in agreement toward the consensus information.
With this compiled dataset we tested our main hypothesis: whether exposing an illusion of knowledge leads to greater normative belief revision in response to expert versus public consensus than when an illusion of knowledge is not punctured. We conducted this analysis first by looking at respondents who received the Trade With China issue, but the results reported below are robust when accounting for all issues.
We found a significant Time by Source of Consensus by Writing Task interaction (F(1, 1335) = 9.45 , p = .002; see Figure 2).Footnote 13 Footnote 14 To unpack this interaction we tested whether there was greater agreement change when provided expert consensus compared to public consensus within each Writing Task condition (Control and Experimental). When examining the Control condition, we did not find a significant Source of Consensus by Time interaction (F(1, 810) < 1), demonstrating that these participants did not change their agreement more to the consensus of experts than the consensus of laypeople. When examining the Experimental condition, we found a significant Source of Consensus by Time interaction (F(1, 1052) = 34.74, p < .001). Further analyses revealed that after having an illusion of knowledge exposed, participants changed their agreement in response to the opinion of laypeople (t(527) = 5.62, p < .001, d = 0.15), but changed far more in response to the opinion of experts (t(525) = 11.78, p < .001, d = 0.51). Collectively, these results support the conclusion that, in the absence of any manipulations exposing gaps in knowledge, people do not revise their normative beliefs more to expert opinion than lay opinion. However, when an illusion of knowledge is exposed, people revise far more to the experts.
Figure 2: Main analyses graphs unpacking the significant Source of Consensus by Time by Writing Task three-way interaction for the compiled data set containing studies 1 – 5. The figure demonstrates that after having an illusion of knowledge exposed (Experimental Condition, n = 2,050) individuals change their agreement in accordance with consensus information to a greater extent when that consensus information is said to have come from professional economists compared to members of the public. When an illusion of knowledge is not exposed (Control Condition, n = 812) people do not revise more to experts than members of the public. Error bars in each graph represent +/- 1 standard error of the mean.
8 General Discussion
The present research focused on how people revise their normative beliefs in response to the opinions of experts (professional economists) compared to the opinions of the general public. Study 1 replicated the finding that people adjust their normative beliefs in response to consensus information but do not adjust more to economists’ opinion than lay opinion. Studies 2 and 3 showed that when an illusion of explanatory depth is exposed, people revise their normative beliefs far more in response to learning the opinion of experts. In addition, Study 3 found that exposing the illusion of explanatory depth is not topic-bound and that its exposure may induce a general feeling of ignorance that leads to the downstream effect of normative belief revision. Study 4 generalized the effect of the writing manipulation across five different economic issues each with its own unique level of consensus and provided further evidence that it is a general feeling of ignorance (rather than awareness of a lack of topic-relevant knowledge) that creates the revision effect. Finally, Study 5 featured a control condition that also replicated the main finding of Reference Johnston and BallardJohnston and Ballard (2016) and Study 1. Collapsing across all studies provides strong evidence for the contention that one reason people do not privilege the opinion of experts is because people think that they, and by extension their fellow members of the public, know more than they really do.
Given the vast complexity of the world it is impossible for any individual to know absolutely everything. Moreover, compared to what they could know, a given individual knows nearly nothing. Individuals must rely on the knowledge of others if they want to obtain and maintain an accurate model of the world. Through a web of epistemic dependence people store their knowledge of the world in others (Reference HardwigHardwig 1985; Reference WagenknechtWagenknecht, 2015). One way individuals achieve this is through transactive memory (Reference Wegner, Mullen and GoethalsWegner, 1987; Reference Wegner, Erber and RaymondWegner et al., 1991), whereby they encode into memory not what the exact details of a phenomenon are, but rather markers for who is likely to hold that information. However, individuals can mistake knowing where that information might be stored with actually understanding the information (Reference Sloman and RabbSloman & Rabb, 2016). This perhaps leads to an illusion of explanatory depth in which people believe they can explain phenomena to a far greater extent than they truly can (Reference Rabb, Fernbach and SlomanRabb et al., 2019). Our work is consistent with this model of human knowledge. If individuals believe they possess the knowledge of experts, there is little reason to update their beliefs more in response to experts than to the public. They may implicitly be asking themselves, “What does an expert know that I do not?” Thus, while people do revise their beliefs to consensus information somewhat, their updating behavior suggests they fail to discriminate between experts and other random members of the public (Reference CoppockCoppock, 2018; Reference Johnston and BallardJohnston & Ballard, 2016).
Our findings suggest that confronting failure to generate a coherent explanation of a phenomenon leads people to become aware that they are mistaking their markers of knowledge with actual knowledge. When then provided information from more valid (experts) and less valid (general public) sources of knowledge, people update their beliefs more to valid sources. We found this to occur even when the explanation failure concerned a topic unrelated to the topic of the subsequent belief revision task. One question that arises from this is why people revise more in response to experts (and not just to any given opinion)? We have generated two possible explanations for the agreement-updating behavior following the induction of a feeling of ignorance. One possibility is that in this state, a person may ignore the information presented from a source (rather than contrast what they know versus what the source is saying), and instead simply update toward those who more closely match their markers for who should hold that sort of knowledge. This is broadly consistent with evidence that suggests people are cognitive misers and use simple heuristics to avoid resource-intensive reflective processes (Reference Dawes, Carroll and PayneDawes, 1976; Reference KeilEvans & Stanovich, 2013; Reference Gilovich, Griffin and KahnemanGilovich et al., 2002; Reference StanovichStanovich, 2009). Another possibility is that people flexibly integrate what knowledge is being presented with who is presenting it. An individual may not willingly update their beliefs in resonse to an expert (the who) whose opinion (the what) is drastically different from the individual’s superficial knowledge of the topic. This integration of both types of information is consistent with research demonstrating that humans are “good Bayesians” in a variety of domains (e.g., argumentation, Harris, et al., 2015; probability judgment, Reference Krynski and TenenbaumKrynski & Tenenbaum, 2007; Reference Turpin, Meyers, Walker, Bialek, Stolz and FugelsangTurpin et al., 2020). These two contrasting accounts are good candidates for future research.
Our work has implications for the behavioral consequences of overestimating one’s knowledge. Much recent research has provided timely examples of potentially insidious effects. For example, extreme opposition to genetically modified foods has been linked to an increase in perceived understanding and a decrease in objective knowledge about science (Reference Fernbach, Light, Scott, Inbar and RozinFernbach et al., 2019). In addition, people who occupy extreme positions (as opposed to moderate) on both the political left and right experience more certainty about their domain-specific knowledge of an event independent of their actual knowledge of it (which in terms of the 2016 European Union refugee crisis was not greater than that of moderates: Reference van Prooijen, Krouwel and Emmervan Prooijen et al., 2017). People who report knowing as much or more than doctors and scientists about the causes of autism are highest among those with low levels of actual knowledge about the causes of autism (Reference Motta, Callaghan and SylvesterMotta et al., 2018). While exposing an illusion of explanatory depth has been demonstrated to reduce position extremism (Reference Fernbach, Sloman, Louis and ShubeFernbach et al., 2013), our results suggest that in addition to lowering perceived understanding, people may also be more willing to change their minds when presented with information from sources they deem valid. However, we are hesitant to claim the generalizability of our findings as we have only yet presented evidence for its effectiveness within the domain of economics.
If wisdom comes with recognizing the limits of one’s knowledge, and the privileging of expert opinion indicates that one does recognize these limits, then the results of these studies indicate that experiencing doubt can indeed make us wiser. The realization that we know much less than we thought seems to trigger a change in behavior which causes individuals to weight the opinion of experts over that of lay people. It seems that without this experience of self-doubt many of us too often resemble the self-certain “fools and fanatics” lamented by the late Bertrand Russell.
Open access
