Introduction
Misinformation about COVID-19 on social media has been a public concern during the COVID-19 pandemic. How can public health authorities communicate to mitigate the spread of misinformation? One line of research on countering COVID-19 misinformation suggests that subtly nudging people to think about accuracy reduces misinformation sharing (Pennycook et al. Reference Pennycook, McPhetres, Zhang, Lu and Rand2020). The strength of such interventions is their fast and frugal nature. At the same time, they are premised on the idea that people can discern between true and false, but “that when deciding what to share on social media, people are often distracted from considering the accuracy of the content” (Pennycook et al. Reference Pennycook, Epstein, Mosleh, Arechar, Eckles and Rand2021). As such, a key drawback of accuracy nudge interventions is that they simply remind people about accuracy, but leave individuals to rely on their own knowledge without providing tools or competences to deal with misinformation. The frugal nature of accuracy nudges also implies that their effects are small (Pennycook and Rand Reference Pennycook and Rand2022), making recent studies question their effectiveness during the COVID-19 pandemic (Rathje et al. Reference Rathje, Roozenbeek, Traberg, Van Bavel and Van Der Linden2022; Roozenbeek et al. Reference Roozenbeek, Freeman and Van Der Linden2021; Gavin et al. Reference Gavin, McChesney, Tong, Sherlock, Foster and Tomsa2022; Pretus et al. Reference Pretus, Javeed, Hughes, Hackenburg, Tsakiris, Vilarroya and Van Bavel2023).
Other interventions move beyond nudging accuracy motivations by seeking to equip people with better capabilities for identifying and avoiding sharing misinformation (Lee Reference Lee2018; Guess et al. 2020; Hertwig and Grüne-Yanoff Reference Hertwig and Grüne-Yanoff2017; Sheeran and Orbell Reference Sheeran and Orbell2000; Sheeran et al. Reference Sheeran, Aubrey and Kellett2007; Van Der Linden Reference Van Der Linden2022). This is consistent with research on risk communication that argues that feelings of competence are key in order to motivate people to respond effectively to risks by engaging in protective behaviors (Jørgensen et al. Reference Jørgensen, Bor and Petersen2021b). People respond effectively when they are provided with trustworthy information about a threat, provided with actionable advice on how to respond to the threat and assured that this response will be efficient against the threat (Rogers Reference Rogers1975; Maddux and Rogers Reference Maddux and Rogers1983; Rippetoe and Rogers Reference Rippetoe and Rogers1987).
In this manuscript, we test the effectiveness of interventions that nudge accuracy to ones that additionally provide capabilities. Specifically, we test an accuracy nudge as well as two video-based, real-world interventions circulated by the Danish National Health Authority on social media in January 2021 during the COVID-19 pandemic. The accuracy nudge subtly primed people to think about their motivation to share accurate headlines, while the videos – a 15-second and a 3-minute intervention – provided capabilities through concrete instructions on how to avoid sharing COVID-19 related to misinformation.
For study 1, we predicted that all three interventions would decrease false headline sharing, increase real headline sharing, and increase sharing discernment (i.e., the relative sharing of real compared to false headlines). The accuracy nudge, the 15-second intervention and the 3-minute intervention all significantly increased sharing discernment. Only the 3-minute intervention, however, directly and significantly decreased false headline sharing, but did not alter real headline sharing. Neither the 15-second intervention nor the accuracy nudge had a statistically significant effect on either false or real headline sharing. Consistent with a capability perspective, study 2 showed that the 3-minute intervention increased participants’ sense of self-efficacy in dealing with online misinformation. The intervention did not influence other aspects often highlighted in research on risk communication, specifically, participants’ sense of the threat from misinformation and the effectiveness of remedies against misinformation. Overall, these results suggest that when health authorities communicate elaborate and actionable advice on how to avoid sharing COVID-19 misinformation, such communication can reduce the spread of false headlines and enhance people’s sense of personal competence.
Two approaches to misinformation interventions
We examine two types of interventions to reduce misinformation sharing. One type of intervention is “accuracy nudges” which has received significant research interest (Pennycook et al. Reference Pennycook, Epstein, Mosleh, Arechar, Eckles and Rand2021,Reference Pennycook, McPhetres, Zhang, Lu and Rand2020; Roozenbeek et al. Reference Roozenbeek, Freeman and Van Der Linden2021; Rathje et al. Reference Rathje, Roozenbeek, Traberg, Van Bavel and Van Der Linden2022; Gavin et al. Reference Gavin, McChesney, Tong, Sherlock, Foster and Tomsa2022; Pretus et al. Reference Pretus, Javeed, Hughes, Hackenburg, Tsakiris, Vilarroya and Van Bavel2023). The psychological assumption behind accuracy nudges is that people are motivated to share accurate content on social media and are capable of distinguishing between true and false content. Yet, accuracy concerns do often not drive online sharing behavior because people are distracted from accuracy toward a desire to share emotionally engaging content and receive positive social feedback from friends on their sharing. Thus, reminding people to pay attention to accuracy through nudging should decrease the sharing of misinformation. Some research shows that subtle accuracy nudges where people are asked to rate the accuracy of a few news headlines decrease subsequent sharing of false headlines on social media (Pennycook et al. Reference Pennycook, Epstein, Mosleh, Arechar, Eckles and Rand2021, Reference Pennycook, McPhetres, Zhang, Lu and Rand2020).
Another type of intervention is capability interventions that seeks to mitigate misinformation sharing by building competences or resistance against the rhetorical techniques and strategies that are used to mislead people through misinformation (Roozenbeek and Van Der Linden Reference Roozenbeek and Van Der Linden2022; Guess et al. Reference Guess, Lyons, Persily and Tucker2020; Badrinathan Reference Badrinathan2021; Van Der Linden Reference Van Der Linden2022; Lee Reference Lee2018; Mo Jones-Jang et al. Reference Mo Jones-Jang, Mortensen and Liu2021). While these interventions are often rooted in distinct theoretical frameworks – such as inoculation theory or digital media literacy – the common denominator is that they go beyond merely priming accuracy motivations by providing concrete tools, actionable advice, or psychological competences to mitigate misinformation sharing. In other words, while the assumption behind accuracy nudges is that people already have the competence to avoid sharing misinformation and simply are in need of motivation-oriented reminders, capability interventions go beyond the motivational component of accuracy nudges: They build capabilities by providing education (i.e., increasing knowledge) and training (i.e., imparting skills and tools) to avoid misinformation sharing. Where accuracy nudges rely on prompting a pre-existing motivation for accuracy, capability interventions aim to build reflective motivation where citizens contemplate and plan how to implement advice behaviorally.
Overview of studies
In two studies, we conduct a preregistered test of a 15-second and a 3-minute capability-oriented intervention from the Danish Health Authorities as well as an accuracy nudge (Pennycook et al. Reference Pennycook, McPhetres, Zhang, Lu and Rand2020) and compare them to a control group. Study 1 tested the effect of the interventions on sharing of false and real headlines while Study 2 assessed the effect of the interventions on self-efficacy, response efficacy, and threat appraisal. Footnote 1 Whenever we report additional analyses that were not preregistered, we label them as exploratory. Table 1 provides an overview of the data collection.
Table 1. Overview of data collection
Note: †We conducted a pretest to validate the headlines for study 1 and study 2. Details can be found in Section C of the Supplementary Material.
In both studies, each participant was assigned to one of four conditions. In the control condition, participants were not exposed to any treatment prior to the respective dependent measures of Study 1 and Study 2. In the accuracy nudge condition, participants rated the accuracy of a single headline (unrelated to COVID-19) framed as a pretest mimicking prior studies of accuracy nudges (Pennycook et al. Reference Pennycook, McPhetres, Zhang, Lu and Rand2020; Roozenbeek et al. Reference Roozenbeek, Freeman and Van Der Linden2021). In the 15-second condition and the 3-minute condition, participants were shown videos titled “Can you trust what you read?” containing guidance on how to recognize and avoid sharing COVID-19 misinformation. Footnote 2 Specifically, the actionable advice in the video is summed up in three questions that one should ask oneself when facing novel information on social media: (1) Who is saying it? (2) How many are saying it? (3) Is the content too far out? Besides the 3-minute video being longer than the 15-second video, there are two major differences between the interventions. First, while the 15-second video only contains text, the 3-minute video includes both text and audio, which makes the content more immersive. Second, the 3-minute video provides more elaborate advice. In other words, the advice provided by the 3-minute intervention is more concrete and actionable in terms of providing a plan for implementation, which is conducive to behavior change (Pearce et al. Reference Pearce, Lindekilde, Parker and Rogers2019; Sommestad et al. Reference Sommestad, Karlzén and Hallberg2015). Through collaboration with the Danish Health Authority, we have been given permission to use the actual videos which were shared on Facebook in 2020 and 2021. Footnote 3
The studies were conducted in Denmark during in the Summer and Winter of 2021 during the COVID-19 pandemic. Denmark is characterized by high levels of interpersonal and institutional trust and low levels of political polarization. These factors materialized during the onset of the pandemic, as compliance with and support for governmental responses were high, while polarization was low, in contrast to other countries where governmental responses were more disputed (Lindholt et al. Reference Lindholt, Jørgensen, Bor and Petersen2021; Van Bavel et al. Reference Van Bavel, Cichocka, Capraro, Sjåstad, Nezlek, Pavlović, Alfano, Gelfand, Azevedo, Birtel, Cislak, Lockwood, Ross, Abts, Agadullina, Aruta, Besharati, Bor, Choma and Boggio2022; Jørgensenet al. Reference Jørgensen, Bor and Petersen2021b,Reference Jørgensen, Bor, Lindholt and Petersena). Furthermore, it was widely accepted, even among political elites, that the COVID-19 virus constituted a significant public health threat that required public collaboration and responsiveness to contain. As policies and messages aimed at countering the COVID-19 pandemic are more effective when backed by cross-partisan coalitions of political elites (Flores et al. Reference Flores, Cole, Dickert, Eom, Jiga-Boy, Kogut, Loria, Mayorga, Pedersen, Pereira, Rubaltelli, Sherman, Slovic, Västfjäll and Van Boven2022), public health communication is more likely to be effective in Denmark, compared to countries where the nature of the COVID-19 pandemic as a health crisis was disputed. We further elaborate on the implications for the generalizability of the findings in the discussion section.
Study 1
We preregistered the following hypotheses for study 1. We expected that the interventions would reduce false headline sharing, and thus, we predicted that compared to the control condition, all three interventions decrease sharing of false headlines, but not real headlines about COVID-19 on social media (H1). Conversely, the intervention could work by increasing real headline sharing, and thus, we predicted that compared to the control condition, all three interventions increase sharing of real headlines, but not false headlines about COVID-19 on social media (H2). Furthermore, the interventions might prompt a relative increase in real versus false headlines, and thus, we predicted that compared to the control condition, the interventions increase sharing discernment (H3). Footnote 4 Furthermore, we preregistered a range of robustness analyses of the treatment effects across covariates. We predicted that the treatment effect of 15-second and 3-minute interventions on citizens’ likelihood of sharing false headlines is lower for respondents who have low trust in public institutions and government handling of the pandemic and low scores on cognitive reflection and attention to social comparison information, compared to respondents with high scores on these variables (H4). Next, we predicted a significant interaction between attention to social comparison information and all of the three interventions compared to the control condition on the willingness to share both real and false headlines, such that the effect is stronger for people who score high on attention to social comparison information (H5), and, finally, that the effect of all three interventions decays gradually with number of rating tasks completed (H6). We report briefly on all preregistered hypotheses in the main text and provide more details in Section A of the appendix.
Data were collected in collaboration with the market research institute YouGov through a two-wave panel among a national sample of the Danish population. YouGov sampled from their internet panels and employed quota sampling to match population characteristics on age, gender, region, and education. The power analysis suggested that 940 participants were required to have 90% power to replicate effect sizes from previous research on accuracy nudges (Pennycook et al. Reference Pennycook, Epstein, Mosleh, Arechar, Eckles and Rand2021) (see preregistration for formal power calculation). We recruited 2,541 participants between July 2 and 13, 2021 for the first wave. 2,232 participants (88% of the original sample) completed the second wave between August 2 and 23. In the first wave, we collected psychological correlates of participants, while in the second wave, participants were exposed to a survey experiment. We did not record any post-treatment attrition in wave 2.
To assess the effectiveness of the interventions in study 1, participants completed a news-sharing task consisting of 15 real and 15 false headlines. Participants were informed that they would be exposed to a range of articles from the past year concerning COVID-19. We opted for headlines instead of full articles because people often share articles on social media without reading the full article (Gabielkov et al. Reference Gabielkov, Ramachandran, Chaintreau and Legout2016). The headlines were presented one at a time in random order, and respondents were asked whether they were willing to share them.
The main outcome is headline sharing measured through a standard item: “If you were to see the above article on social media, how likely would you be to share it?” (1: Extremely unlikely, 2: Moderately unlikely, 3: Slightly unlikely, 4: Slightly likely, 5: Moderately likely, 6: Extremely likely; re-scaled to 0-1) (Pennycook et al. Reference Pennycook, McPhetres, Zhang, Lu and Rand2020). Studies validating the measure suggest that people report higher sharing intentions for headlines in surveys that do indeed receive more shares on Twitter (Mosleh et al. Reference Mosleh, Pennycook and Rand2021). We use the outcome to measure both real (M = 0.26, SD = 0.32) and false (M = 0.14, SD = 0.26) headline sharing respectively. Furthermore, we use it to assess sharing discernment (M = 0.13, SD = 0.20) which is defined as the difference in sharing intentions between real and false headlines where a higher discernment score indicates that people share more real relative to false news. Footnote 5
Results
We conduct the analyses using OLS regressions with standard errors clustered on subject and headline. Fig. 1 shows the effect of each intervention in three separate models with false and real headline sharing as well as sharing discernment, respectively, as the dependent variable (scaled 0-1) Footnote 6 .
Figure 1. Willingness to share real and false headlines.
Note: Points are OLS estimates with 95% confidence interval bars based on clustered standard errors at the respondent level and headline level. The panels display estimates based on regressions of the interventions on false (n = 32,480) and real (n = 32,480) headline sharing as well as sharing discernment (n = 2,232) all re-scaled to 0-1.
Did the interventions decrease false headline sharing (H1)? The 3-minute video intervention significantly decreased willingness to share false headlines (b = −0.055, 95% CI = [−0.078, −0.031], p < 0.001, d = −0.222), while the 15-second video intervention (b =−0.002 95% CI = [−0.027, 0.023], p < 0.88, d = −0.008) and the accuracy nudge (b =−0.017, 95% CI = [−0.041, 0.008], p < 0.18, d = −0.064) did not, as shown in Fig. 1. In other words, only the 3-minute video intervention decreased false headline sharing.
Did the interventions increase real headline sharing (H2)? The 3-minute video intervention (b = 0.028, 95% CI = [−0.002, 0.052], p < 0.07, d = 0.089), the 15-second video intervention (b = 0.025, 95% CI = [−0.002, 0.052], p < 0.08, d = 0.081) and the accuracy nudge (b =0.026, 95% CI = [−0.001, 0.054], p < 0.06, d = 0.084) did not significantly affect sharing of real headlines compared to the control condition.
Did the interventions increase sharing discernment (H3)? All three interventions significantly increased sharing discernment. In other words, both the accuracy nudge (b = 0.043, 95% CI = [0.021, 0.065], p < 0.001, d = 0.230), the 15-second video intervention (b = 0.027, 95% CI = [0.005, 0.049], p < 0.02, d = 0.145), and the 3-minute video intervention (b = 0.082, 95% CI = [0.059, 0.106], p < 0.001, d =0.416) increased the relative sharing of real compared to false headlines.
We preregistered a range of robustness analyses of the treatment effects across covariates. We did not find that the treatment effect on sharing was significantly moderated by cognitive reflection, need for cognition, trust in government or health authorities (H4), and attention to social comparison (H5). Neither did the treatment effect significantly decay over time (H6) (See Section F in the appendix for details).
Study 2
In Study 1, we established that the 3-minute video intervention decreased sharing of false headlines. In Study 2, we used predictions derived from protection motivation theory to probe why the intervention worked. Protection motivation theory proposes that people protect themselves against risks – in our case believing in and sharing false headlines on social media – based on appraisals of (1) the threat from the risk and (2) their ability to cope with the risk (Rogers Reference Rogers1975; Maddux and Rogers Reference Maddux and Rogers1983; Floyd et al. Reference Floyd, Prentice-Dunn and Rogers2000; Sommestad et al. Reference Sommestad, Karlzén and Hallberg2015; Pearce et al. Reference Pearce, Lindekilde, Parker and Rogers2019). Threat appraisals reflect the severity of the situation, the likelihood of a threat materializing, and individual vulnerability to the threat. Coping appraisals consist of two factors. The first factor is perceived “self-efficacy,” understood as one’s ability to carry out the recommended action and follow the advice successfully. The second factor is the perceived “response efficacy,” understood as an individual’s expectation that carrying out the recommended action and following the given advice will keep one safe from the threat. Research on protection motivation theory suggests that feelings of self-efficacy are, in general, the most important factor behind motivations to engage in protective behavior (Norman et al. Reference Norman, Boer, Seydel and Mullan2015). Consistent with this, prior work on protection motivations in the context of the COVID-19 pandemic in Denmark has shown that compliance with public health authority advice is most strongly affected by self-efficacy (Jørgensen et al. Reference Jørgensen, Bor and Petersen2021b). For study 2, we preregistered that compared to the control condition, the 3-minute intervention would increase threat appraisals (H7), feelings of self-efficacy (H8), and feelings of response efficacy (H9). Footnote 7 As exploratory analyses, we also report the effects of the 15-second intervention and the accuracy nudge on these outcomes. Footnote 8
To determine sample size, we conducted a two-sided t-test power calculation as noted in the preregistration. Given 500 participants in each group, significance level at 0.05 an effect size of d = 0.19 can be estimated with power = 0.9, and d = 0.16 with power = 0.8. A national sample of 2,012 participants quota sampled to match population characteristics on age, gender, region, and education was collected in Denmark between December 17, 2021, and December 23, 2021, by YouGov.
Instead of the news-sharing task used in Study 1, the outcome measure in Study 2 was a battery of six protection motivation items that were combined to three measures (threat appraisal, self-efficacy, and response efficacy). Besides the change of outcome, the experimental protocol was the same.
Threat appraisal, self-efficacy, and response efficacy were measured in a battery of six items where each factor was measured as the mean of its two corresponding items. The following items were included. Threat appraisal (M = 0.55, SD = 0.21): (1) “I am exposed in terms of false information regarding COVID-19,” (2) “False information regarding COVID-19 is a threat to the Danish society.” Self-efficacy (M = 0.80, SD = 0.22): (3) “It is easy for me to avoid spreading false information about COVID-19,” (4) “I am confident that I can avoid spreading false information about COVID-19 if I want to.” Response efficacy (M = 0.73, SD = 0.24): (5) “If I avoid falling for false information, I will be in greater security during the Corona epidemic,” (6) “If I avoid spreading false information, I take part in protecting others against COVID-19.” For each item, participants were asked to what extent they agreed or disagreed with the statements on a scale from 1, “Strongly disagree,” to 7, “Strongly agree,” and re-scaled to vary between 0 and 1.
Results
Fig. 2 shows the effect of the interventions on threat appraisal, self-efficacy, and response efficacy.
Figure 2. Effect of interventions on threat appraisal, self-efficacy, and response efficacy.
Note: Points are OLS estimates with 95% confidence interval bars based on clustered standard errors at the respondent level from three regressions. Each panel represents a regression of the treatment conditions on the respective pmt measure as the dependent variable. All regressions are based on samples of 2,012 respondents.
Did the interventions affect threat appraisal (H7)? No. Neither the accuracy nudge (b = 0.01, 95% CI = [−0.02;0.03], p < 0.531, d =0.039), the 15-second intervention (b = −0.02, 95% CI = [−0.04,0.01], p < 0.210, d = 0.078) or the 3-minute intervention (b = 0.01, 95% CI = [−0.02;0.03], p < 0.626, d = 0.031) significantly affected threat appraisal. Footnote 9
Did the interventions affect self-efficacy (H8)? The 3-minute video intervention significantly increased self-efficacy (b = 0.06, 95% CI = [0.03;0.09], p < 0.001, d = 0.274), while there was no significant effect of the 15-second video intervention (b = 0.01, 95% CI = [−0.02,0.04], p < 0.529, d = 0.039) or the accuracy nudge (b = 0.01, 95% CI = [−0.02;0.04], p < 0.556, d = 0.037). This suggests that the 3-minute intervention boosts citizens’ feelings of competence when they face COVID-19 misinformation.
Did the interventions affect response efficacy (H9)? No, neither the accuracy nudge (b = −0.02, 95% CI = [−0.05;0.01], p < 0.194, d = 0.081), the 15-second video intervention (b = 0.02, 95% CI = [−0.01,0.04], p < 0.288, d = 0.066), or the 3-minute video intervention (b = 0.03, 95% CI = [−0.01;0.06], p < 0.104, d = 0.103) significantly affected response efficacy.
In sum, across the 3 interventions, we only found evidence of a statistically significant effect of the 3-minute intervention on self-efficacy. Across study 1 and study 2, this suggests that the 3-minute intervention both increase people’s personal feelings of being competent in terms of avoiding sharing misinformation (in study 2) and reduce people’s sharing of false headlines (study 1).
Discussion
Are misinformation interventions effective against misinformation? The analyses showed that while the accuracy nudge and a 15-second capability-oriented intervention significantly increased sharing discernment – that is, the relative sharing of real vs. false headlines – they did not have a significant effect on neither false or real headline sharing compared to the control condition. The 3-minute capability-oriented intervention significantly increased sharing discernment and self-efficacy and reduced false headline sharing. In sum, we found mixed support for effectiveness of short capability-oriented messages and accuracy nudges against misinformation. These results add to recent academic research on the effectiveness of short messages or nudges on misinformation sharing (Rathje et al. Reference Rathje, Roozenbeek, Traberg, Van Bavel and Van Der Linden2022; Roozenbeek et al. Reference Roozenbeek, Freeman and Van Der Linden2021; Gavin et al. Reference Gavin, McChesney, Tong, Sherlock, Foster and Tomsa2022; Pretus et al. Reference Pretus, Javeed, Hughes, Hackenburg, Tsakiris, Vilarroya and Van Bavel2023; Pennycook and Rand Reference Pennycook and Rand2022). Notably, the 3-minute intervention was effective both in terms of boosting people’s feelings of competence as well as reducing false headline sharing. The results expand our knowledge of the effectiveness of misinformation interventions by deploying different interventions in the same experimental framework. To assess the effectiveness of interventions, future research could adopt a similar approach in which they assess multiple interventions within the same experimental framework. Furthermore, while several studies assess whether interventions affect the belief in and sharing of misinformation (Pennycook et al. Reference Pennycook, McPhetres, Zhang, Lu and Rand2020; Roozenbeek et al. Reference Roozenbeek, Freeman and Van Der Linden2021; Guess et al. Reference Guess, Lyons, Persily and Tucker2020; Badrinathan Reference Badrinathan2021; Bode and Vraga Reference Bode and Vraga2018; Jensen et al. Reference Jensen, Ayers and Koskan2022), few studies explicitly address the potential underlying mechanisms for why people alter their behavior (Lin et al. Reference Lin, Pennycook and Rand2022; Altay et al. Reference Altay, Hacquin and Mercier2020). The finding that the most effective intervention also influenced participants’ feelings of self-efficacy is consistent with the general finding in the risk communication literature that such feelings are key for motivating protective behavior (Norman et al. Reference Norman, Boer, Seydel and Mullan2015).
In assessing these conclusions, it is worth noting important limitations of the study. While the study contributes by assessing interventions against misinformation beyond American or British samples (see also Badrinathan Reference Badrinathan2021; Guess et al. Reference Guess, Lyons, Persily and Tucker2020; Gavin et al. Reference Gavin, McChesney, Tong, Sherlock, Foster and Tomsa2022), it is important to note that the results are limited to a context where people are particularly responsive to government interventions. Danes held more trust in authorities, were more compliant, were less polarized, and were more concerned in the first stages of the COVID-19 pandemic than most other countries (Lindholt et al. Reference Lindholt, Jørgensen, Bor and Petersen2021; Lieberoth et al. Reference Lieberoth, Lin, Stöckli, Han, Kowal, Gelpi, Chrona, Tran, Jeftić, Rasmussen, Cakal and Milfont2021). As a consequence, the Danes may be more responsive to messages from public health authorities. In other contexts with a higher degree of political polarization and skepticism regarding, for example, the threat of COVID-19, the effectiveness of interventions from public authorities may be more limited. To examine this proposition, future studies could use a comparative approach to assess the effectiveness of communication from public authorities across countries with varying degrees of trust, political polarization, and compliance with advice from authorities.
Practitioners should note that the interventions are mitigation strategies, rather than addressing the root cause of misinformation sharing. First, the effects of these types of interventions are small (Pennycook and Rand Reference Pennycook and Rand2022). For instance, we included a shortened sharing task in study 2 and observed a smaller effect size than in study 1 for all interventions as the control condition was negatively affected by the protection motivation theory items asking questions like “False information regarding COVID-19 is a threat to the Danish society.”Footnote 10 This in line with the literature suggesting that small nudges can boost sharing discernment (Pennycook et al. Reference Pennycook, Epstein, Mosleh, Arechar, Eckles and Rand2021). Nevertheless, the ephemeral effect sizes serve as an important reminder for researchers and practitioners alike that the effects of misinformation interventions often are small (Pennycook and Rand Reference Pennycook and Rand2022), diminish over time (Carnahan et al. Reference Carnahan, Bergan and Lee2021; Carey et al. Reference Carey, Guess, Loewen, Merkley, Nyhan, Phillips and Reifler2022), and thus require repeated intervention in practice (Ecker et al., Reference Ecker, Lewandowsky, Cook, Schmid, Fazio, Brashier, Kendeou, Vraga and Amazeen2022). Second, while a relatively small share of people share misinformation (Guess and Lyons Reference Guess, Lyons, Persily and Tucker2020; Guess et al. Reference Guess, Nagler and Tucker2019; Cinelli et al. Reference Cinelli, Pelicon, Mozetic, Quattrociocchi, Novak and Zollo2021; Grinberg et al. Reference Grinberg, Joseph, Friedland, Swire-Thompson and Lazer2019), recent studies suggest that social and political goals serve as important motivations (Uscinski et al. Reference Uscinski, Enders, Seelig, Klofstad, Funchion, Caleb Everett, Premaratne and Murthi2021; Osmundsen et al. Reference Osmundsen, Bor, Vahlstrup, Bechmann and Petersen2021; Petersen et al. Reference Petersen, Osmundsen and Arceneaux2023; Rathje et al. Reference Rathje, Roozenbeek, Van Bavel and Van Der Linden2023; Pickup et al. Reference Pickup, Stecuła and Van Der Linden2022) and some interventions may be less effective for people with certain political allegiances (Rathje et al. Reference Rathje, Roozenbeek, Traberg, Van Bavel and Van Der Linden2022). In Section F of the appendix, we conduct exploratory analysis of whether the interventions have heterogeneous treatment effects across trust in government, trust in health authorities, cognitive reflection, need for closure, attention to social comparison information, age, gender, income, education, and partisanship. In line with previous research, we find that partisanship predicts false headline sharing (Osmundsen et al. Reference Osmundsen, Bor, Vahlstrup, Bechmann and Petersen2021), yet the effect of the 3-minute intervention on false headline sharing is consistent across partisanship. Furthermore, we do not find heterogeneous treatment effects for a range of other potentially relevant individual differences (specifically, cognitive reflection, need for closure, or attention to social comparison information, trust in government, or health authorities). In sum, while social, political, and accuracy motivations may shape people’s overall propensity to share false headlines, the results suggest that the effect of the 3-minute video intervention is consistent. In other words, while interventions that provide competences do not address the root cause of misinformation sharing such as political motivations, they can be a reliable mitigation strategy with consistent effects across sub-populations.
The experimental design does not allow us to disentangle the effects of the length of the videos from the comprehensiveness of the advice. To be clear, our interpretation is that the 3-minute intervention is more effective in reducing false headline sharing and boosting self-efficacy, because it provides more elaborate guidance both in terms of providing information, tools, and devising actions to avoid sharing misinformation, compared to the 15-second intervention that only provides brief advice. Yet, further studies could benefit from testing interventions of similar length and assessing their efficacy.
The measures used in this study are based on sharing intentions, not actual online behavior, which clearly warrants concerns about whether the experimental results generalize beyond the experimental context. While sharing intentions is a standard way of measuring the experimental effects of misinformation interventions and it has been shown that sharing intentions correlates with actual sharing on social media (Mosleh et al. Reference Mosleh, Pennycook and Rand2020), researchers are not able to address this caveat until social media platforms are willing to share data and conduct field experiments on their platforms. In this regard, we encourage other scholars to replicate these findings, including through field experiments, and urge social media platforms, relevant public authorities, and the like to redouble their efforts in collaborating with researchers.
In conclusion, this study has tangible policy implications for public health authorities. These findings suggest that elaborate public health communication on social media can be an effective tool for health authorities during a crisis to counter the circulation of misinformation. We demonstrated that communicating concrete advice on how to avoid sharing misinformation reduces false headline sharing and increases feelings of competence in the public.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/XPS.2024.2
Data availability
The data, code, and any additional materials required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network, at: doi: https://doi.org/10.7910/DVN/8UQV11 (Rasmussen et al. Reference Rasmussen, Lindekilde and Bang Petersen2023)
Acknowledgements
The authors want to thank Jay van Bavel and members of his lab, Jon Roozenbeek, Clara Pretus, and members of the ROPH Project for helpful feedback at various stages of the project. We are grateful for feedback received during APSA 2022 and the Aarhus’22 Conference on Online Hostility and Bystanders. The authors thank Christian Jonasen Noer and Lea Pradella for excellent research assistance. Please direct correspondence pertaining to this article to Jesper Rasmussen: [email protected].
Funding statement
This study was funded by the Carlsberg Foundation with Grant CF20-0044 to M.B.P.
Competing interests
During the preparation of this study, Michael Bang Petersen received salary from the Danish Health Authority as a member of the National Vaccination Council, a council of independent experts advising the Danish Health Authority on matters of vaccination. The Danish Health Authority had no role in the execution of the present research. Jesper Rasmussen and Lasse Lindekilde declare no competing interests.
Ethics statement
The authors affirm that this article adheres to APSA’s Principles and Guidance on Human Subject Research. The studies were exempt by Danish law from formal review by an Institutional Review Board. As per Section 14(2) of the act underlying the Danish National Research Ethics Committee, “notification of questionnaire surveys … to the system of research ethics committee system is only required if the project involves human biological material.” Informed consent was obtained from all participants. Upon completion of the surveys, respondents received extensive debriefing and were informed that some of the headlines were false, but have nevertheless circulated on Facebook in 2020 and 2021. The ones we deemed false were fact-checked by Danish or International fact-checkers, and we provided a complete list of all the headlines used in the study and whether they were deemed true or false. The survey vendor, YouGov, compensated participants with reward points that can be redeemed for cash. See Section E of the appendix for an elaborate discussion of ethics.
Consent to participate
Informed consent was obtained from all individual participants included in the study.
Pre-registrations
Study 1: https://osf.io/akybg?view_only=7ca987412b6d464c982f93a31d95df19
Study 2: https://osf.io/uqxv9?view_only=7ca987412b6d464c982f93a31d95df19
Open access
