Hostname: page-component-78c5997874-mlc7c Total loading time: 0 Render date: 2024-11-08T16:27:51.513Z Has data issue: false hasContentIssue false

For and Against Brexit: A Survey Experiment of the Impact of Campaign Effects on Public Attitudes toward EU Membership

Published online by Cambridge University Press:  05 February 2018

Rights & Permissions [Opens in a new window]

Abstract

What are the lessons of the 2016 referendum on UK membership of the European Union (EU) regarding the effects of message framing? This article reports findings from an innovative online survey experiment based on a two-wave panel design. The findings show that, despite the expectation that campaign effects are generally small for high-salience issues – such as Brexit – the potential for campaign effects was high for the pro-EU frames. This suggests that within an asymmetrical information environment – in which the arguments for one side of an issue (anti-EU) are ‘priced in’, while arguments for the other side (pro-EU) have been understated – the potential for campaign effects in a single direction are substantial. To the extent that this environment is reflected in other referendum campaigns, the potential effect of pro-EU frames may be substantial.

Type
Articles
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Cambridge University Press 2018

Referendums on a country’s relations with the European Union (EU) are relatively common. The 23 June 2016 referendum on whether the United Kingdom should remain in or leave the EU was the 55th such referendum in Europe since 1972, and the 30th since 2000. In some respects, the UK’s ‘remain-or-leave’ referendum was similar to other recent EU referendums.Footnote 1 The vote was triggered by a party in government that was divided on Europe. Whereas most centrist voters generally supported the pro-EU side, most voters on the radical right and a significant portion on the radical left supported the anti-EU side. The referendum was also preceded by growing public support for a radical right and ‘hard’ Eurosceptic party, the UK Independence Party (UKIP).Footnote 2

In other respects, though, the 2016 UK referendum was different. Whereas most other EU referendums have been on reforms of the EU treaties, or on whether a country should join the EU or adopt the euro single currency, the 2016 referendum was only the third on whether a country should remain in or leave the EU. The only other such ‘high-stakes’ referendums were an earlier referendum in the UK, in 1975, when 67 per cent voted to stay in the then European Communities, and the Greenland referendum in 1982, when 54 per cent opted to leave the European Communities. Following the UK 2016 referendum, though, we may see similar in/out referendums in other member states. Several radical right parties in other EU states are campaigning for a ‘Cameron-style’ referendum, for example.

We use the 2016 referendum to achieve two broad goals. First, we test theories of campaign effects in a context that is extraordinary from the perspective of most persuasion and framing studies, which typically focus on low-salience issues.Footnote 3 This may be explained by the fact that persuasion and framing affects are expected to be weak for long-standing issues on which individuals have been given the time and information necessary to form and solidify their opinions.Footnote 4 Little opinion change is expected for crystallized attitudes.Footnote 5 Leaving or remaining in the EU is the type of high-salience issue for which we would expect crystallized attitudes and weak campaign effects. Even before the referendum, voters considered the issue of Europe and the UK’s EU membership to be the leading issue facing the country.Footnote 6 However, the UK referendum also represented a situation where substantial campaign effects were possible, because of the information asymmetry that existed prior to the campaign. On the one hand, before the referendum campaign, consumers of UK media had been exposed to at least forty years of Eurosceptic coverage.Footnote 7 Public concern over immigration and dissatisfaction with established politics had been targeted by UKIP, which won the 2014 European Parliament elections and secured nearly 13 per cent of the vote in the 2015 general election.Footnote 8 On the other hand, arguments regarding the positive effects of EU membership were far less prominent as they received a fraction of the media attention. This created a situation in which argument frames that emphasized previously under-stressed considerations might have had an effect despite the high salience of the issue. It is particularly unusual that positive – but not negative – messages had the potential to influence opinions. Negativity bias usually leads to greater mental attention to negative arguments relative to positive ones, leading to larger persuasion and framing effects.Footnote 9

Persuasion and framing effects in the context of the 2016 referendum may also have been larger than expected due to weak partisan cuing. When an issue is highly salient, each political party typically takes a clear position. When this happens, it is difficult to move partisans away from their partisan-consistent position. Hence, both persuasion and framing effects are expected to be small or non-existent. Yet the 2016 Brexit referendum represented an unusual situation in which cues from the two major parties were unclear. Longstanding divisions within the incumbent Conservative Party were reflected in the fact that while 185 Conservative MPs campaigned for ‘Remain’, 138 declared support for ‘Leave’. This divide extended to the very top of the party, as at least six Cabinet ministers campaigned for Leave, including the Mayor of London and Conservative Member of Parliament (MP) Boris Johnson and Justice Secretary Michael Gove, alongside the UKIP leader, Nigel Farage, and ten Labour MPs.Footnote 10 While an overwhelming majority of Labour MPs were remain supporters, the party leader, Jeremy Corbyn, was lukewarm in his support, both because he was known to be critical of the EU and because he did not want to be seen to support the Cameron government. As a result, at the beginning of the campaign, which is when we undertook our study, the Labour leadership’s position was unclear.

In sum, the context of the 2016 EU referendum in the UK allows us to test persuasion and framing effects in a context in which the high salience of the issue would typically lead us to expect little to no effects, particularly for positive arguments, but the information asymmetry and the weak partisan cues of the environment lead us to expect strong effects. This is an atypical situation in which to test persuasion and framing effects, and leads to some unique theoretical expectations. Empirically testing these expectations provides a strong test of what we believe we know about campaign effects.

Even though the context of the Brexit referendum is extraordinary from the perspective of testing campaign effects, it does have some features in common with possible future in/out referendums. This allows us to achieve a second goal: to study the power of arguments used in the UK referendum to shed light not only on what happened in 2016 but also on what could happen in similar referendums in the future. In particular, our context allows us to examine whether relatively unfamiliar ‘pro-EU’ arguments that stress the benefits of EU membership can sway public attitudes in the face of well-established ‘anti-EU’ arguments that have been prominent in the UK and other EU nations.

We pursue these two goals via an online survey experiment, conducted between September and November 2015, just before the official referendum campaign began. The experiment included two innovative elements. First, it was designed to allow us to test the effects of different combinations of real-world arguments against each other during an actual campaign on public support for leaving/remaining in the EU. Secondly, the experiment was based on a ‘before and after’ design, using a two-wave panel, which allows us to examine how individuals changed their opinions in response to a set of arguments. In the next section we set out our theoretical expectations, after which we explain the design of our experiment and present the results. We conclude with a discussion of the implications of our results for the campaign effects literature and for future referendums.

Our main findings are that: (1) ‘pro-EU’ arguments on their own (could have) significantly increased support for remaining in the EU while ‘anti-EU’ arguments, which for decades have dominated the political and media debate about EU membership, and which we suggest were therefore ‘priced in’, had less potential to impact support for either side; (2) the effect of pro-EU arguments was mitigated when contrasted with anti-EU arguments, limiting the potential effectiveness of pro-EU arguments in an environment of competing pro-/anti-messages; (3) the effect of both pro- and anti-EU arguments was greatest among those whose prior beliefs were consistent with the argument and (4) the effect of pro-EU arguments was greatest among those with lower political knowledge (less education).

THEORY

Media, politicians and campaigns can portray information in different ways, stressing certain evaluations or parts of an argument.Footnote 11 Our research on the effect of different arguments on EU membership is rooted in existing research on persuasion, and how the ‘framing’ of issues can affect public opinion and vote choices generally, as well as research on how such framing can shape attitudes towards the EU specifically.

It is well established in political science that frames select and organize information on issues, give meaning, attribute positive or negative values, and can influence attitudes. Frames supply a readily comprehensible basis on which to think about the issue and how to justify what should be done about it.Footnote 12 A framing effect occurs when individuals who are exposed to different frames arrive at different positions on the issue, depending on the priority given to various considerations in the frame; effects can also be persistent.Footnote 13

When we consider the effects of different arguments that are made for remaining in or leaving the EU, we view these arguments as frames and are interested in the effects of the different considerations raised in the different frames.Footnote 14 There are a number of psychological processes that determine whether a consideration has an effect on opinion. In testing the effects of different arguments, we manipulate two of these: availability and accessibility.Footnote 15 For a consideration to have an effect, it must be available in an individual’s memory and it must be accessible (front of mind) to be applied to the current situation.

Framing studies typically argue that they manipulate the accessibility of different considerations by presenting research participants with different arguments.Footnote 16 However, there exists a debate about whether these framing studies manipulate the availability or accessibility of considerations. That is, whether exposing individuals to frames makes new information available or makes information that was already available to them accessible.Footnote 17 The effect of the latter (accessibility) is typically considered a framing effect and that of the former (availability) a persuasion effect. We remain somewhat agnostic about this distinction, and argue that when we expose respondents to our frames, this has the potential to make both new information available and pre-existing information accessible. Given the salience of the issue of remaining in or leaving the EU leading up to the referendum, we expect that most arguments were already available and our arguments primarily manipulated accessibility. In other words, any detected effects are framing effects. However, given the pre-existing asymmetry in the volume of claims made regarding the negative and positive consequences of remaining in the EU, it is possible that the pro-EU arguments increased availability. In other words, pro-EU arguments may have also produced a persuasion effect.

Consistent with a campaign in which two camps advocate diametrically opposed positions (that is, whether to remain in, or leave, the EU), we view the relevant arguments as ‘valence frames’. As Lecheler and de Vreese observe, this valence alludes to one of the most fundamental aspects of political discourse, namely that political elites try to influence public reactions to an issue by emphasizing the positive or negative aspects of it, such as the positive or negative features of EU membership.Footnote 18 As past studies observe, with regard to framing and the EU, ‘[t]he basic finding of two decades of research is that political communications that frame European integration in terms of its benefits increase public support for European integration, while messages that frame it in terms of costs, risks or cultural threats lower support for European integration’.Footnote 19

Indeed, a growing number of experimental studies investigate frames as they relate to EU issues and how they affect public attitudes. Abbarno and Zapryanova randomly exposed respondents in the UK and Bulgaria to a ‘cultural threat’ or ‘democratic deficit’ frame, and found that both reduced support for EU integration.Footnote 20 Schuck and de Vreese demonstrate how news media frequently use ‘risk’ and ‘opportunity’ frames when discussing the issue of EU enlargement, frames that are shown to have significant effects on citizens that are, in turn, moderated by political knowledge.Footnote 21 Studying attitudes toward Turkey’s possible EU membership, de Vreese et al. show how negative frames yielded stronger effects than positive frames, though more politically sophisticated citizens were more affected by positive frames, providing some evidence that attitudes can be shaped by the framing of the issue in interaction with individual characteristics.Footnote 22 Overall, pro- and anti-EU economic, cultural and political frames have the potential to increase or decrease support for remaining in the EU.

The magnitude of a framing effect depends on the context. We argue that one way in which context matters is that if certain considerations are already highly available and accessible to the individual, exposure to frames that contain those considerations will have little effect. Under such circumstances, those considerations are already ‘priced in’ – already accessible and available. This is particularly relevant to a situation in which one side of an argument may have dominated the dialogue for an extended period of time. Given Britain’s long tradition of Euroscepticism, we expect that new pro-EU frames that are not ‘priced in’ to the debate will have a greater effect on vote choice than older and more familiar anti-EU frames. On that basis, we hypothesize that pro-EU frames will increase support for remaining in the EU while anti-EU frames will have little effect.

Another way in which context could matter is particularly relevant to a campaign in which two opposing sides make competing claims. The effect of a frame may differ depending upon whether the individual is simultaneously being exposed to competing frames. When individuals receive competing frames, they may choose the side that is consistent with their prior beliefs.Footnote 23 However, if an individual contrasts competing frames, this can cause him or her to take an intermediate position. The effect of one frame is mitigated by the opposing frame, resulting in weak effects.Footnote 24 We anticipate that both of these effects are relevant. We hypothesize that exposure to competing frames mitigates the effects of framing. At the same time, frames will have stronger effects if they are consistent with prior beliefs.Footnote 25 Therefore, we hypothesize that pro-EU frames will be more effective among people who previously expressed positive EU sentiment and/or those who identify as Labour supporters, while anti-EU frames will be more effective among people who previously expressed negative EU sentiment and/or who are Conservative identifiers.Footnote 26

The magnitude of a framing effect also depends on other characteristics of the recipient of the frame. Consistent with extant research, we expect that exposure to frames will have stronger effects on the less knowledgeable by making considerations available and accessible that were previously not accessible, something found in studies of framing and EU issues.Footnote 27 Among the more knowledgeable these considerations are likely to already be highly available and accessible, and so little effect is expected from exposure to frames that contain these considerations. Lecheler and de VreeseFootnote 28 find evidence of this type of moderating effect for knowledge when examining the effects of news framing on support for EU enlargement. Using education as a proxy for knowledge,Footnote 29 we hypothesize that framing effects will be stronger among the less educated.

This consequently leads us to the following hypotheses:

  1. 1. Anti-EU frames, which are likely to already be highly available and accessible (priced in), will have less of an effect than pro-EU frames.

  2. 2. In a context in which there are competing frames, the effect of each frame will be mitigated, resulting in weak effects.

  3. 3. Frames consistent with prior beliefs will have stronger effects.

  4. 4. The effect of frames will be greatest on those with less knowledge/education.

EXPERIMENTAL DESIGN

We test these hypotheses in an innovative, online population-based experiment that combines the strengths of laboratory experiments and national surveys.Footnote 30 By randomly assigning a sample of the adult population to different treatments we can identify the causal effects of being exposed to different types of arguments on support for EU membership. Conducting our experiment on a sample designed to reflect the national population, we can also estimate the magnitude of these effects on the population as a whole.Footnote 31

The experiment was designed to simulate the ‘real-world’ campaign environment by exposing respondents to a combination of different frames, in the form of vignettes, about the UK’s EU membership, and which they might realistically receive in the referendum campaign, due to start shortly after our experiment. Our experiment reflects the real world in two ways. First, we use vignettes that include competing frames with different vignettes placing different weights on the two sides of the argument. Chong and DruckmanFootnote 32 note that there is little literature on the effect of competing frames. Sniderman and Theriault, and Brewer and GrossFootnote 33 examine competing frames but only with equal exposure to both. Of course, in reality arguments are rarely equal in exposure. Secondly, our experimental vignettes were inspired by real-world arguments used by the two sides of the debate only a few months before the referendum. Following past research on the effect of information on EU attitudes,Footnote 34 our vignettes include cultural, economic and political messages. Also following past researchFootnote 35 these messages were based on leaflets, speeches or websites associated with the pro-EU/Remain or the anti-EU/Leave campaigns, thereby maximizing the generalizability of the results to the referendum. The arguments were tailored so that the length of each was approximately equal with roughly equivalent amounts of information. The primary variation across treatments was the direction of the argument (that is, pro-EU or anti-EU) rather than the amount of information. Our vignettes purposely did not include the source of the considerations presented. This allowed us to estimate the effects of the considerations uncontaminated by cueing from the source. The vignettes are shown in Appendix I.

Our experiment used a 2×2×2 full factorial design, in which each respondent was exposed to a vignette composed of: (1) a ‘pro’ or ‘anti’ EU cultural frame; plus (2) a ‘pro’ or ‘anti’ EU economic frame; plus (3) a ‘pro’ or ‘anti’ EU political frame. One-ninth of respondents were assigned to each of the eight possible treatment groups and one-ninth was assigned to a control group, which was not exposed to a vignette.Footnote 36 Because every treatment contains three frames, we are able to estimate the impact of different combinations of arguments holding the number of frames constant.

The full factorial design allows us to estimate the effect of: arguments containing frames all in the same direction (all pro- or all anti-EU); arguments with competing frames (some pro- and some anti-EU); and arguments with different weights placed on the two sides (a varying number of pro-EU and anti-EU arguments). This, in turn, allows us to test our hypotheses regarding the effects of different types and combinations of frames. The experiment was conducted before the date of the referendum was confirmed and the conclusion of EU reform negotiations, hence minimizing external contamination.

The population-based experiment was conducted as a two-wave panel study conducted by YouGov. The first wave was conducted between 24 September and 1 October 2015 and the second wave was conducted two weeks later between 16 October and 9 November 2015. The first wave included 6,074 completions. With a retention rate of 87.8 per cent, the second wave sample included 5,333 respondents, with approximately 590 respondents per group.Footnote 37 Further information about the sample, including balance tests, can be found in Appendix II. Once data were collected we proceeded with a difference-in-differences analysis. This allows us to estimate the average change in referendum vote intention before and after exposure to a vignette for individuals in the eight treatment groups (the eight argument combinations) compared to the change in referendum vote intention for individuals in the control group.

In the first wave, vote intention was measured using a standard referendum voting intention question: ‘How would you vote if the referendum on Britain’s membership of the EU were held tomorrow? (Remain in the EU/Leave the EU/Don’t Know)’. In the second wave, respondents were assigned to one of the vignettes (eight treatment groups) or to the control group. Respondents were then immediately asked the same vote intention question.

The panel structure allows us to measure referendum vote intention before and after exposure to the vignettes. This provides us with a within-subject design, allowing us to estimate the change in opinion, for each individual, due to exposure to a vignette relative to those who were not exposed to a vignette (the control group). For example, we estimate the average change in referendum vote intention for individuals exposed to a vignette with three pro-EU frames compared to the change in referendum vote intention for individuals in the control group. We also did this for individuals exposed to vignettes with the following combination of arguments: two pro/one anti, one pro/two anti and three anti. The within-subject design also provides greater power to determine the effects than a between-subjects design.Footnote 38

The two-week period between Waves 1 and 2 was designed to prevent respondents from thinking about their reported vote intention (in Wave 1) when reporting their vote intention (in Wave 2) after exposure to the vignette. This reduced the probability of suppressing the effect of a vignette as respondents attempt to appear consistent. Contrasting the change for each treatment group against a control allows us to control for any universal changes in vote intention in the intervening two weeks.

RESULTS

In the first wave, 38.2 per cent of respondents indicated they would vote to remain in the EU, 43.9 per cent indicated they would vote to leave, and 17.9 per cent indicated that they did not know or would not vote. In the second wave, among those in the control group (who were not exposed to any arguments), these results changed slightly to 37.1 per cent, 44.0 per cent and 18.8 per cent, respectively. To illustrate the magnitudes of the shifts across the two waves, Appendix Figure A1 shows the ‘flows’ from Remain/Leave/Don’t Know in Waves 1 and 2 for the control group and the two most extreme treatment groups: those exposed to three pro-EU arguments and those exposed to three anti-EU arguments.

To quantify and test the statistical significance of the average change in referendum vote intention before and after respondents were exposed to a particular combination of the arguments, we estimate the following:

$$\Delta vote_{i} \, {\equals}\, \alpha {\plus}\beta treatment_{i} {\plus}{\epsilon}_{i} $$

where $\Delta vote_{i} $ is the change in vote intention from Waves 1 to 2 for respondent i; and $treatment_{i} $ is a binary variable equalling 0 if the respondent was in the control group and 1 if the respondent was exposed to that particular combination of arguments. Respondents exposed to a different combination of arguments were not included in this particular estimate. Separate equations were estimated for changes in vote between Remain and some other vote (Leave or no vote), and for changes in vote between Leave and some other vote, and between no vote and some other vote. In each case, the coefficient β is the estimate of the average change in vote intention for those exposed to the combination of arguments, relative to the control group. This is our estimate of the effect of the vignette. Doing this for each of the eight possible combinations of arguments gives us an estimate of the effect of each of the eight vignettes.

Figure 1 plots the estimated effects of each vignette on Remain and Leave vote intention (the full model estimates are provided in Appendix V).Footnote 39 Exposure to three pro-EU arguments produced a statistically significant (p<0.05) increase in support for remaining in the EU and a decrease in support for leaving the EU. The increase in the Remain vote was 5.2 percentage points and the decrease in the Leave vote was 3.6 percentage points. There was no corresponding statistically significant effect for the three anti-EU arguments. None of the vignettes that contained any combination of anti-EU arguments produced a statistically significant effect on referendum voting intention. Meanwhile, in addition to a positive effect from the three pro-EU arguments, the ‘pro-cultural, pro-economic, anti-political’ vignette produced a statistically significant change in the vote. It increased public support for Remain and decreased support for Leave by 3.1 percentage points. Other vignettes that included pro- and anti-EU arguments did not produce a statistically significant effect.

Fig. 1. Difference-in-differences effects of the treatments Note: the figure shows the effect of a treatment on the change in support for Remain (Leave) relative to the change in support for Remain (Leave) in the control group. The full results are in Appendix Table A4.

Having determined that some vignettes that included pro-EU arguments produced statistically significant results, we then specifically estimated the effects of different vignettes depending on the number of pro- versus anti-EU arguments. We separately estimated the effects of all vignettes with three pro-EU arguments, vignettes with two pro- and one anti-EU argument, vignettes with one pro- and two anti-EU, and vignettes with three anti-EU arguments. Doing so allowed us to further examine the differential effect of pro- and anti-EU frames, and to compare the effect of exposing respondents to non-competing frames (all negative or all positive) to that of exposing respondents to competing frames with different weights placed on each side of the argument. Figure 2 reveals again how the vignette with three pro-EU arguments produced a statistically significant effect, as did vignettes with two pro- and one anti-EU arguments (our full results are reported in Appendix V). These latter vignettes increased public support for remaining in the EU by a statistically significant 2.3 percentage points. Their effect on support for Leave was not statistically significant. Overall, vignettes that were predominantly pro-EU in tone had a statistically significant effect on referendum vote choice, while vignettes that conveyed anti-EU messages did not.Footnote 40 This is consistent with our first hypothesis.

Fig. 2. Difference-in-differences effects of the relative volume of arguments Note: the figure shows the average effect of the relative volume of Pro/Anti EU arguments on the change in support for Remain (Leave) relative to the change in support for Remain (Leave) in the control group. The full results are in Appendix Table A6.

Does this mean that anti-EU arguments had no effect? We cannot reject the null hypothesis that anti-EU arguments had no effect, while we can reject the null hypothesis that pro-EU arguments had no effect. To test whether this is because anti-EU arguments have no effect, rather than because we do not have the power to detect their effect, we provide a test of the following null hypothesis: vignettes with two or more pro-EU arguments have no more of an effect (in the positive direction) than those with two or more anti-EU arguments (in the negative direction). This is a test of the difference in the magnitudes of anti-EU vignette effects against the pro-EU vignette effects. At the 0.05 level (one-sided), we can reject this null hypothesis for the proportion indicating that they would vote Remain (z score = 0.046; p = 0.04).Footnote 41 For the Remain vote, we have evidence that the magnitude of the effect of pro-EU arguments was greater than the effect of anti-EU arguments and, in fact, no evidence that anti-EU arguments had any effect. Overall, this is strong evidence for our suggestion that anti-EU arguments (on average) had no effect while pro-EU arguments did.

Result 1: consistent with our first hypothesis that priced-in considerations will not have an effect, anti-EU frames have less of an effect than pro-EU frames.

Figure 2 also reveals that while a purely pro-EU vignette affected the proportion indicating they would vote Remain or Leave, a vignette containing two pro-EU frames and one anti-EU frame had less of an effect. Further, a vignette containing only one pro-EU frame and two anti-EU frames had no effect. The effect of the pro-EU frames was reduced when competing anti-EU frames were combined in the vignette.Footnote 42

Result 2: consistent with our second hypothesis, when there are competing frames, the effect of each frame is mitigated, resulting in weak effects.

Next, to examine the effect of prior attitudes on the effectiveness of pro- and anti-EU frames, we conducted two tests. First, we examined how effects differed among those who previously declared moderate feelings towards the EU, those who said they held extremely negative feelings and those with extremely positive feelings. The left panel of Figure 3 plots the distribution of EU attitudes by those who indicated that they intended to vote Remain, Leave or Don’t Know in Wave 1. Attitudes were measured on a 0 to 10 scale with 10 indicating the most positive and 0 indicating the most negative attitudes towards the EU. The negative end of the spectrum (0–3) was almost entirely populated by those who intended to vote Leave and the positive end of the spectrum (7–10) was almost entirely populated by those who intended to vote Remain. The moderate values (4–6) were a mixture of all three positions, with the centre dominated by those who did not know how they would vote.

Fig. 3. Extremists and moderates Note: the left-hand figure shows the distribution of respondents on a 0–10 Anti-/Pro-EU scale. The right-hand figure shows the average effect of each group of treatments on the change in support for Remain/Don’t Know/Leave relative to the change in support for Remain/Don’t Know/Leave in the control group. EU moderates were coded at points 3, 4, 5, 6, 7; Anti-EU extremists at points 0, 1, 2, 3; and Pro-EU extremists at points 7, 8, 9, 10. The full results are in Appendix Table A12.

On the basis of this distribution, we defined three groups of individuals based on their EU attitudes: anti-EU extremists, EU moderates and pro-EU extremists.Footnote 43Figure 3 presents the estimated effects of the vignettes on these three groups of individuals. We see that, as before, anti-EU arguments did not have any effect. However, either three or two pro-EU arguments increased the Remain vote among pro-EU extremists and EU moderates but not among the anti-EU extremists. This increase in the Remain vote came largely through a decrease in those indicating they did not know how they would vote. As expected, those with priors consistent (or at least not inconsistent) with the frames were most affected by them.

Somewhat counter to expectations, there was some effect of the pro-EU frames on anti-EU extremists. This effect was to decrease the Leave vote by increasing those who did not know how they would vote. What we are observing is a limitation on the effect of a frame depending on the strength of the individual’s priors. Those who had the most negative EU attitudes could be moved from a Leave vote to an undecided position, and those with the most positive EU attitudes could be moved from an undecided position to a Remain vote. However, individuals with extreme (pro- or anti-EU) attitudes were unlikely to be moved all the way from a Remain to a Leave vote or vice versa.

Our second test of how prior attitudes moderate the effectiveness of pro- and anti-EU frames examined how the effects of the vignettes differed by partisan identity. Figure 4 Panel A shows the effects of the vignettes on respondents who self-identified as Conservative or Labour Party identifiers. Conservative identifiers were not affected by pro-EU arguments; they were only moved away from Remain and towards Leave by the combination of three anti-EU messages. Labour identifiers, however, were moved by pro-EU arguments but not anti-EU arguments. The combination of either two or three pro-EU arguments moved Labour supporters away from saying they don’t know how they would vote and towards voting Remain. This is consistent with our expectations that those who identified as Labour would have prior attitudes that made them more susceptible to pro-EU frames, and that those who identified as Conservative would have prior attitudes that made them more susceptible to anti-EU frames.

Fig. 4. Difference-in-differences effects by subgroup (A). By party support (B). By age when finished education Note: each figure shows the average effect of each group of treatments on the change in support for Remain/Don’t Know/Leave relative to the change in support for Remain/Don’t Know/Leave in the control group. The full results are in Appendix Table A13.

Finding 3: in support of our third hypothesis, frames consistent with prior beliefs have stronger effects.

Finally, we examined education as a moderator of the effects of the vignettes. We made the assumption that those with higher levels of education would, on average, have high levels of political knowledge.Footnote 44 As shown in Figure 4, Panel B, we found that three or two pro-EU arguments increased the Remain vote only among those with the lowest levels of education (<17 years).Footnote 45 There were no other effects for the higher levels of education, except for a decrease in Remain and an increase in Leave due to three anti-EU arguments among the middle education category (17–19 years). To the extent that education correlates with level of political knowledge, it appears as though those with lower levels of political knowledge were more likely to be affected by frames that made information available and accessible.

Finding 4: consistent with Hypothesis 4, the effect of both pro- and anti-EU frames is strongest for those with the least knowledge/education.

CONCLUSIONS

In this article we have sought to build and expand upon existing research on persuasion and framing effects, on public attitudes toward the EU, voting behaviour and referendums. Our experimental design allowed us to test for the effect of ‘real-world’ arguments on both sides of the referendum campaign that attempted to influence the vote through an array of pro-EU or anti-EU messages. Our main finding is that, in one of the most Eurosceptic states in the EU, pro-EU arguments had the potential to significantly increase support for remaining in the EU, while anti-EU arguments had less potential to impact support for either remaining or leaving. Our results suggest that in more recent years the well-rehearsed arguments about the perceived costs, risks and threats from the EU became ‘priced in’ to the national debate about continued EU membership. Having experienced several decades of a Eurosceptic tabloid press, an established current of Conservative Euroscepticism and the rise of the Eurosceptic radical right UKIP, it is perhaps not surprising that participants in our survey were less strongly influenced by prominent Eurosceptic arguments than hitherto less well-known messages about the benefits of EU membership. In contrast, our results suggest that communicating the claimed benefits and advantages of EU membership to an electorate that had long been noted for its instinctive Euroscepticism might have had a significant impact on the overall vote.

In terms of testing theories of campaign effects, our results suggest that even though campaign effects are generally expected to be small when the issue(s) at stake is(are) highly salient, under conditions of information asymmetry coupled with low partisan cuing, the potential for a campaign effect in a single direction is substantial. The one-sidedness of the potential effect is important because it is under these circumstances that a campaign can have a decisive effect on the outcome. Our results are also important because they demonstrate that the high salience of the issues at stake in a campaign does not necessarily mean that all information is already available and accessible and that the electorate is immune to the way in which an argument is framed. Important campaign effects are still possible in referendums on highly salient issues.

In terms of shedding light on the power of arguments used in the UK referendum and similar referendums in the future, our results suggest that certain groups might have been especially receptive to pro-EU frames, notably Labour Party supporters, people under 26 (see Appendix VI), people with lower levels of education, and those undecided about their EU referendum vote. Our results also suggest that pro-EU arguments weakened opposition to the EU among the most fervently committed Eurosceptics and increased support for Remain among more moderate voters. While we find little evidence that pro-EU arguments resulted in ‘direct switching’ between the two camps, our results suggest that the campaign to keep the UK in the EU held important advantages, if it had focused on a clear and consistent pro-EU message.

Given that public attitudes toward EU membership were highly polarized and finely balanced before the referendum, such that the way the respective cases were framed could have made all the difference, our results suggest that it might have been a mistake for the Remain campaign to focus primarily on the potential economic costs of leaving the EU – which the Leave campaign dismissed as ‘Project Fear’ – rather than making a positive case for remaining in the EU. That said, it is important to note that the effectiveness of pro-EU arguments is mitigated when they compete with anti-EU considerations. Therefore, a pro-EU campaign would have had to dominate the anti-EU message in order to be effective. Further, it must be acknowledged that a Conservative Party leadership that had been critical of the EU for three decades could not have credibly led such a strong pro-EU Remain campaign. This credibility trap might be absent in other EU member states facing possible in/out referendums, where elites could perhaps make pro-EU economic, political or cultural arguments. However, anti-EU arguments may still be effective in countries that have not experienced the UK’s long tradition of Euroscepticism.

Overall, an important implication of our work is that in the context of information asymmetry and weak partisan cues, it is valuable to identify which arguments have become commonplace and which have not. Commonplace arguments may be effective at holding public opinion in place, but will be ineffective at shifting opinion during a referendum campaign. Arguments that have been typically understated have the potential to deliver a substantial impact.

Footnotes

*

School of Politics and International Relations, University of Kent (email: [email protected]); Department of Government, London School of Economics and Political Science (email: [email protected]); Department of Political Science, Simon Fraser University (email: [email protected]). This research is funded by the Economic and Social Research Council’s ‘UK In a Changing Europe’ programme, with some additional funding from the University of Kent and the London School of Economics. The authors would like to thank J. Scott Matthews and Sara B. Hobolt for their helpful comments. The article also benefited from the comments made by those in attendance at the Nuffield Politics Seminar, the 2016 European Political Science Association meeting and the LSE Political Behaviour seminar. Data replication sets are available in Harvard Dataverse at: https://dx.doi.org/doi:10.7910/DVN/PZB01G and online appendices at https://doi.org/10.1017/S0007123417000667.

1 For example, see Hug Reference Hug2002 and Hobolt Reference Hobolt2009.

2 Clarke, Goodwin, and Whiteley Reference Clarke, Whiteley and Goodwin2017.

3 Hopkins and Mummolo Reference Hopkins and Mummolo2017.

6 Data from the Ipsos-MORI Issues Index tracker. Available at https://www.ipsos.com/ipsos-mori/en-uk/eu-and-brexit-seen-most-important-issue-facing-britain?language_content_entity=en-uk (accessed 17 July 2017).

8 Goodwin and Milazzo Reference Goodwin and Milazzo2015.

10 The estimates of how MPs and ministers voted is taken from ‘EU vote: Where the cabinet and other MPs stand’, BBC News, 22 June 2016. Available at http://www.bbc.co.uk/news/uk-politics-eu-referendum-35616946 (accessed 3 January 2017).

11 Schuck and de Vreese Reference Schuck and de Vreese2006.

12 Sniderman and Theriault Reference Sniderman and Theriault2004.

13 Druckman and Nelson (Reference Druckman and Nelson2003, 730); on frame persistence, see Lecheler and de Vreese (Reference Lecheler and de Vreese2011).

14 Chong and Druckman Reference Chong and Druckman2007; see also Berinsky and Kinder Reference Berinsky and Kinder2006.

15 Bless, Fielder, and Strack Reference Bless, Klaus and Strack2004.

16 Chong and Druckman Reference Chong and Druckman2007.

18 de Vreese and Boomgaarden Reference de Vreese and Boomgaarden2003; Lecheler and de Vreese Reference Lecheler and de Vreese2011.

19 Abbarno and Zapryanova Reference Abbarno and Zapryanova2013, 583; also see de Vreese Reference de Vreese2004; de Vreese Reference de Vreese2005; Gabel and Scheve Reference Gabel and Scheve2007; Medrano Reference Medrano2003; Schuck and de Vreese Reference Schuck and de Vreese2006.

20 Abbarno and Zapryanova Reference Abbarno and Zapryanova2013; cf. McLaren Reference McLaren2002; Werts, Scheepers, and Lubbers Reference Werts, Scheepers and Lubbers2012.

21 Schuck and de Vreese Reference Schuck and de Vreese2006.

22 de Vreese, Boomgaarden, and Semetko Reference de Vreese, Boomgaarden and Semetko2011.

23 Sniderman and Theriault 2004.

24 See Chong and Druckman Reference Chong and Druckman2007.

25 Druckman 2001.

26 Hobolt Reference Hobolt2007; Van Elsas and Van Der Brug Reference Van Elsas and Van Der Brug2015.

27 See Chong and Druckman (Reference Chong and Druckman2007, 639); in terms of EU issues see Schuck and de Vreese (Reference Schuck and de Vreese2006).

28 Lecheler and de Vreese Reference Lecheler and De Vreese2010.

31 Previous studies that use an experimental design to study EU attitudes include: Maier and Rittberger (Reference Maier and Rittberger2008); Schuck and de Vreese (Reference Schuck and De Vreese2011) who use lab experiments; Abbarno and Zapryanova (Reference Abbarno and Zapryanova2013); Adriaansen, van Praag and de Vreese (Reference Adriaansen, van Praag and de Vreese2012); Bruter (Reference Bruter2009); Kumlin (Reference Kumlin2011); Lecheler and de Vreese (Reference Lecheler and De Vreese2010); and Schuck and de Vreese (Reference Schuck and de Vreese2012), who used survey experiments. De Vreese (Reference de Vreese2004) used a two-wave framing experiment.

32 Chong and Druckman Reference Chong and Druckman2007.

33 Brewer and Gross Reference Brewer and Gross2005; Sniderman and Theriault 2004.

34 Maier and Rittberger Reference Maier and Rittberger2008.

35 Maier et al 2012.

36 In our design, each respondent is assigned to only one treatment (or to the control group), unlike a conjoint design, in which the individual is exposed to a series of treatments. Such a design would be inappropriate for our purposes as the vignettes do not reference a hypothetical and so once a respondent is exposed to one vignette, we cannot reasonably ask them to forget the arguments to which they were just exposed in order to determine the effect of exposing them to a new vignette.

37 Group sizes range from 551 to 652. See Appendix Table A6 for a breakdown of the group sizes.

39 The results do not change if we drop all respondents in the eight treatment groups who took less than 90 seconds to complete the whole test and therefore may have not read the vignettes carefully.

40 If we examine the effects of each argument separately (averaging over the other arguments with which it is combined), we see that each of the pro-EU arguments increases the level of public support for Remain (p < 0.1) – see Appendix Table A6. In the case of the political pro-EU argument, there was also a decrease in support for Leave. Overall though, there is little to distinguish the effects of the different pro-EU arguments. The main distinction is between pro- and anti-EU arguments, with little evidence that the latter had an effect.

41 We cannot reject the null hypothesis for the proportion indicating they would vote to leave (z score=−0.011; p = 0.62) or who did not know how to vote (z score=−0.035; p = 0.245). To reduce standard errors, we include in the analysis the following pre-treatment variables: age, social class, region, party ID and EU attitudes in Wave 1.

42 It is also possible that the weaker effect is due to fewer pro-EU arguments.

43 For the estimation we overlapped the categories, so that 0–3 = anti-EU extremists, 3–7 = EU moderates, and 7–10 = pro-EU extremists. We did this so that the three categories were broadly the same size and also because it is unclear whether positions 3 and 7 are ‘moderate’ or ‘extreme’.

45 Due to larger standard errors, only the vignette with three pro-EU arguments results in a statistically significant decrease in the Leave vote among this group.

References

Abbarno, Aaron J., and Zapryanova, Galina M.. 2013. Indirect Effects of Eurosceptic Messages on Citizen Attitudes Toward Domestic Politics. JCMS: Journal of Common Market Studies 51 (4):581597.Google Scholar
Adriaansen, Maud L., van Praag, Philip, and de Vreese, Claes H.. 2012. A Mixed Report: The Effects of Strategic and Substantive News Content on Political Cynicism and Voting. European Journal of Communication Research 37 (2):691708.Google Scholar
Berinsky, Adam J., and Kinder, Donald R.. 2006. Making Sense of Issues Through Media Frames: Understanding the Kosovo Crisis. Journal of Politics 68 (3):640656.CrossRefGoogle Scholar
Bless, Herbert, Klaus, Fiedler, and Strack, Fritz. 2004. Social cognition: How individuals construct social reality. Psychology Press, 2004.Google Scholar
Brewer, Paul R., and Gross, Kimberly. 2005. Values, Framing, and Citizens’ Thoughts About Policy Issues: Effects on Content and Quantity. Political Psychology 26 (6):929948.CrossRefGoogle Scholar
Bruter, Michael. 2009. Time Bomb? The Dynamic Effect of News and Symbols on the Political Identity of European Citizens. Comparative Political Studies 42 (12):14981536.CrossRefGoogle Scholar
Carpini, Michael X. Delli, and Keeter, Scott. 1997. What Americans Know About Politics and Why It Matters. New Haven, CT: Yale University Press.Google Scholar
Chong, Dennis, and Druckman, James. 2007. Framing Public Opinion in Competitive Democracies. American Political Science Review 101 (4):637655.CrossRefGoogle Scholar
Chong, Dennis, and Druckman, James. 2010. Dynamic Public Opinion: Communication Effects Over Time. American Political Science Review 104 (4):663680.CrossRefGoogle Scholar
Clarke, Harold, Whiteley, Paul, and Goodwin, Matthew J.. 2017. Brexit: Why Britain Voted to Leave the European Union. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Daddow, Oliver. 2012. The UK Media and ‘Europe’: From Permissive Consensus to Destructive Dissent. International Affairs 88 (6):12191236.CrossRefGoogle Scholar
Druckman, James N. 2001. On the Limits of Framing Effects: Who Can Frame? Journal of Politics 63 (4):10411066.CrossRefGoogle Scholar
Druckman, James N., and Nelson, Kjersten R.. 2003. Framing and Deliberation: How Citizens' Conversations Limit Elite Influence. American Journal of Political Science 47 (4):729745.CrossRefGoogle Scholar
de Vreese, Claes H. 2004. The Effects of Frames in Political Television News on Issue Interpretation and Frame Salience. Journalism and Mass Communication Quarterly 81 (1):3652.CrossRefGoogle Scholar
de Vreese, Claes H.. 2005. News Framing: Theory and Typology. Information Design Journal and Document Design 13 (1):5162.Google Scholar
de Vreese, Claes H., and Boomgaarden, Hajo G.. 2003. Valenced News Frames and Public Support for the EU: Linking Content Analysis and Experimental Data. Communications 3 (4):361381.Google Scholar
de Vreese, Claes H., Boomgaarden, Hajo G., and Semetko, Holli A.. 2011. (In)direct Framing Effects: The Effects of News Media Framing on Public Support for Turkish Membership in the European Union. Communication Research 38 (2):179205.CrossRefGoogle Scholar
Gabel, Matthew J., and Scheve, Kenneth. 2007. Estimating the Effect of Elite Communications on Public Opinion Using Instrumental Variables. American Journal of Political Science 51:10131028.CrossRefGoogle Scholar
Goodwin, Matthew, Hix, Simon, and Pickup, Mark. 2017. “Replication Data for: For and Against Brexit: A Survey Experiment of the Impact of Campaign Effects on Public Attitudes toward EU Membership”, doi: 10.7910/DVN/PZB01G, Harvard Dataverse, V1, UNF:6:1UYwcxNhPuyuk9MJADWryw==.CrossRefGoogle Scholar
Goodwin, Matthew, and Milazzo, Caitlin. 2015. UKIP: Inside the Campaign to Redraw the Map of British Politics. Oxford: Oxford University Press.Google Scholar
Hobolt, Sara B. 2007. Taking Cues on Europe? Voter Competence and Party Endorsements in Referendums on European Integration. European Journal of Political Research 46 (2):151182.CrossRefGoogle Scholar
Hobolt, Sara B.. 2009. Europe in Question: Referendums on European Integration EU & European Institutions. Oxford: Oxford University Press.CrossRefGoogle Scholar
Hopkins, Daniel J., and Mummolo, Jonathan. 2017. Assessing the Breadth of Framing Effects. Quarterly Journal of Political Science 12 (1):3757.CrossRefGoogle Scholar
Hug, Simon. 2002. Voices of Europe: Citizens, Referendums, and European Integration. New York: Rowman & Littlefield.Google Scholar
Keren, Gideon. 2014. Between- Or Within-Subjects Design: A Methodological Dilemma. In A Handbook for Data Analysis in the Behaviorial Sciences: Volume 1: Methodological Issues Volume 2: Statistical Issues, edited by Gideon Keren and Charles Lewis, 257272. Mahwah, NJ: Lawrence Erlbaum Associates Inc.CrossRefGoogle Scholar
Kumlin, Staffan. 2011. Claiming Blame and Giving Credit? Unintended Effects of How Government and Opposition Frame the Europeanization of Welfare. European Union Politics 12 (4):575595.CrossRefGoogle Scholar
Lecheler, Sophie, and De Vreese, Claes H.. 2010. Framing Serbia: The Effects of News Framing on Public Support for EU Enlargement. European Political Science Review 2 (1):7393.CrossRefGoogle Scholar
Lecheler, Sophie, and de Vreese, Claes. 2011. Getting Real: The Duration of Framing Effects. Journal of Communication 61 (5):959983.CrossRefGoogle Scholar
Leeper, Thomas J., and Slothuus, Rune. 2015. Can Citizens Be Framed? How Information, Not Emphasis, Changes Opinions. Unpublished Manuscript.Google Scholar
Lenz, Gabriel S. 2009. Learning and Opinion Change, not Priming: Reconsidering the Priming Hypothesis. American Journal of Political Science 53 (4):821837.CrossRefGoogle Scholar
Luskin, Robert C. 1990. Explaining Political Sophistication. Political Behavior 12 (4):331361.CrossRefGoogle Scholar
Maier, Jürgen, and Rittberger, Berthold. 2008. Shifting Europe’s Boundaries: Mass Media, Public Opinion and the Enlargement of the EU. European Union Politics 9 (2):243267.CrossRefGoogle Scholar
Maier, Michaela, Adam, Silke, and Maier, Jürgen. 2012. The Impact of Identity and Economic Cues on Citizens’ EU Support: An Experimental Study on the Effects of Party Communication in the Run-up to the 2009 European Parliament Elections. European Union Politics 13 (4):580603.CrossRefGoogle Scholar
McLaren, Lauren M. 2002. Public Support for the European Union: Cost/Benefit Analysis or Perceived Cultural Threat? The Journal of Politics 64 (2):551566.CrossRefGoogle Scholar
Medrano, Juan Diez. 2003. Framing Europe: Attitudes to European Integration in Germany, Spain, and the United Kingdom. Princeton, NJ: Princeton University Press.Google Scholar
Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.Google Scholar
Schuck, Andreas R. T., and de Vreese, Claes. 2006. Between Risk and Opportunity: News Framing and Its Effects on Public Support for EU Enlargement. European Journal of Communication 21 (1):532.CrossRefGoogle Scholar
Schuck, Andreas R. T., and De Vreese, Claes H.. 2011. Public Support for Referendums: The Role of the Media. West European Politics 34 (2):181207.CrossRefGoogle Scholar
Schuck, Andreas R. T., and de Vreese, Claes H.. 2012. When Good News Is Bad News: Explicating the Moderated Mediation Dynamic Behind the Reversed Mobilization Effect. Journal of Communication 62 (1):5777.CrossRefGoogle Scholar
Sniderman, Paul M, and Theriault, Sean M.. 2004. The Structure of Political Argument and the Logic of Issue Framing. Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change, pp. 133165.CrossRefGoogle Scholar
Soroka, S. N. 2014. Negativity in Democratic Politics: Causes and Consequences. Cambridge University Press.CrossRefGoogle Scholar
Tesler, M. 2015. Priming Predispositions and Changing Policy Positions: An Account of When Mass Opinion Is Primed or Changed. American Journal of Political Science 59 (4):806824.CrossRefGoogle Scholar
Van Elsas, Erika, and Van Der Brug, Wouter. 2015. The Changing Relationship Between Left–Right Ideology and Euroscepticism, 1973–2010. European Union Politics 16 (2):194215.CrossRefGoogle Scholar
Werts, Han, Scheepers, Peer, and Lubbers, Marcel. 2012. Euro-Scepticism and Radical Right-Wing Voting in Europe, 2002–2008: Social Cleavages, Socio-Political Attitudes and Contextual Characteristics Determining Voting for the Radical Right. European Union Politics 14 (2):183205.CrossRefGoogle Scholar
Zaller, John. 1992. The Nature and Origins of Mass Opinion. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Figure 0

Fig. 1. Difference-in-differences effects of the treatments Note: the figure shows the effect of a treatment on the change in support for Remain (Leave) relative to the change in support for Remain (Leave) in the control group. The full results are in Appendix Table A4.

Figure 1

Fig. 2. Difference-in-differences effects of the relative volume of arguments Note: the figure shows the average effect of the relative volume of Pro/Anti EU arguments on the change in support for Remain (Leave) relative to the change in support for Remain (Leave) in the control group. The full results are in Appendix Table A6.

Figure 2

Fig. 3. Extremists and moderates Note: the left-hand figure shows the distribution of respondents on a 0–10 Anti-/Pro-EU scale. The right-hand figure shows the average effect of each group of treatments on the change in support for Remain/Don’t Know/Leave relative to the change in support for Remain/Don’t Know/Leave in the control group. EU moderates were coded at points 3, 4, 5, 6, 7; Anti-EU extremists at points 0, 1, 2, 3; and Pro-EU extremists at points 7, 8, 9, 10. The full results are in Appendix Table A12.

Figure 3

Fig. 4. Difference-in-differences effects by subgroup (A). By party support (B). By age when finished education Note: each figure shows the average effect of each group of treatments on the change in support for Remain/Don’t Know/Leave relative to the change in support for Remain/Don’t Know/Leave in the control group. The full results are in Appendix Table A13.

Supplementary material: Link

Goodwin et al Dataset

Link
Supplementary material: PDF

Goodwin et al. supplementary material 1

Appendix

Download Goodwin et al. supplementary material 1(PDF)
PDF 1.1 MB