1. Populism and its effectiveness

Although populism is still a contested concept, many scholars across disciplines have increasingly adopted a minimum “ideational” definition of populism over the past decade. According to this definition, populism is primarily a set of ideas depicting society as divided into two homogeneous and antagonistic groups—the “good” people and the “corrupt” elites—and emphasizing that politics should reflect the general will of the people (Mudde, Reference Mudde2004; Hawkins et al., Reference Hawkins, Carlin, Littvay and Kaltwasser2018; Mudde and Kaltwasser, Reference Mudde and Kaltwasser2018). Unlike classical ideologies such as socialism or nationalism, populism lacks programmatic content and can be combined with a wide range of “hosting” ideologies from across the left–right spectrum (Mudde and Kaltwasser, Reference Mudde and Kaltwasser2018).

As populist parties from the left and right have gained significant voter support worldwide in the past decades, there have been significant research efforts to explain the reasons behind the electoral success of populist parties and candidates. Some scholars have investigated various individual attitudes, such as dissatisfaction with representative democracy, and structural factors, such as financial and immigration crises, that might fuel the demand for populism (Akkerman et al., Reference Akkerman, Mudde and Zaslove2013; Bakker et al., Reference Bakker, Rooduijn and Schumacher2016; Kustov, Reference Kustov2023). However, since populism is always attached to other ideologies in practice, it has been difficult to disentangle its independent effects in observational studies.

The two recent studies that have managed to separate “thin populism” from the policy positions of populist parties in conjoint experiments find that policy positions, in and of themselves, have much larger effects on respondents’ vote choice than populism (Neuner and Wratil, Reference Neuner and Wratil2022; Silva et al., Reference Silva, Neuner and Wratil2022). However, we still lack empirical evidence on the interaction between the components of populism and their hosting ideologies. Building on this new strand of experimental research, we seek to further disentangle the effect of populism from the hosting ideology, as well as explicitly test the possible interaction effect between the two.

Although parties or voters can only be considered “populist” if they express or hold all of the core components of populism (Hawkins et al., Reference Hawkins, Carlin, Littvay and Kaltwasser2018; Wuttke et al., Reference Wuttke, Schimpf and Schoen2020), when one considers real-world political campaigns, the presence of all dimensions is much less common than the various combinations of two dimensions (Engesser et al., Reference Engesser, Ernst, Esser and Büchel2017). Similarly, only the most politically sophisticated voters hold consistent populist or anti-populist ideology (Spruyt et al., Reference Spruyt, Rooduijn and Zaslove2021). In other words, it is possible that it might be beneficial for political parties and candidates to only utilize or emphasize certain combinations of populism's main components.

Several recent experimental studies have investigated the effects of these components of populism on voter support and mobilization. Some scholars find that the morally charged anti-elitist rhetoric is the most appealing aspect of populist candidates, especially among those who are low in agreeableness (Bos et al., Reference Bos, Schemer, Corbu, Hameleers, Andreadis, Schulz, Schmuck, Reinemann and Fawzi2020; Bakker et al., Reference Bakker, Schumacher and Rooduijn2021; Silva et al., Reference Silva, Neuner and Wratil2022). At the same time, other scholars find that people-centrism is the most appealing populist aspect (Neuner and Wratil, Reference Neuner and Wratil2022). However, it is still unclear whether each of these components is more effective in the presence of other components and certain policy positions. This paper seeks to fill this gap by explicitly testing the interactional effects between the components of populism.

2. Data and methods

In April 2022, we administered a nationally diverse US online survey experiment on Amazon MTurk (N = 1004) with quality controls recommended in the literature (Kennedy et al., Reference Kennedy, Clifford, Burleigh, Waggoner, Jewell and Winter2020). Similar to analogous online surveys, our respondents were slightly more likely to be young, male, white, and educated than the general population. Thirty-six percent of our respondents identified as Republicans and 58 percent as Democrats (including leaners). Fifty-six percent of our respondents also could be categorized as displaying populist attitudes. For full descriptive statistics comparing our sample to a probability-based benchmark, see Table A3.

While the US political system offers much less opportunity for organized populist parties than countries with proportional representation, it still provides ample opportunities for populist candidacies (Lee, Reference Lee2019). Indeed, populist campaigns have been common among political candidates across all parties in the US (Bonikowski and Gidron, Reference Bonikowski and Gidron2016; Hawkins and Kaltwasser, Reference Hawkins and Kaltwasser2018; Dai and Kustov, Reference Dai and Kustov2022). Therefore, the US case arguably provides a good opportunity to create realistic candidate profiles with a combination of various components of populism and policy positions in a single experiment.

The survey, which was fielded as a part of the broader omnibus study, first presented our respondents with a conjoint task vignette and four pairs of candidate choices, and, following a few unrelated questions, asked about their demographic characteristics and populist attitudes. Our conjoint experiment asked our respondents to choose their preferred candidate from four realistic US House primary candidate profile pairs for their party with randomized attributes. We focus on hypothetical primary contests since any choice of candidates in the general elections would come down to the candidates’ partisan identity, which we would like to avoid since it is not our primary variable of interest (Kirkland and Coppock, Reference Kirkland and Coppock2018).

Compared to previous conjoint experiments on the topic that introduce populism in the form of candidates’ listed policy priorities (Neuner and Wratil, Reference Neuner and Wratil2022; Silva et al., Reference Silva, Neuner and Wratil2022), our study embeds the components of populism in the form of candidates’ campaign message features. Populist ideas embedded in campaign messages arguably make the overall conjoint procedure more realistic, and better resemble the information voters receive during elections. Our approach also enables us to better disentangle the potential electoral effects of populism and its components with various policy content from this content itself. We operationalize populism along the dimensions of people-centrism, anti-elitism, and anti-pluralism. In addition to populist messages, we specify the non-populist attributes using the direct ideological opposites of populism, such as elitism and pluralism (for a detailed discussion on the operationalization of populism, see Appendix B).

We also include policy positions (immigration and economy) and background candidate characteristics. All candidate attributes were randomly selected from those discrete predefined levels. The main binary outcome was a forced choice from a given pair of candidate profiles (Hainmueller et al., Reference Hainmueller, Hopkins and Yamamoto2014). The list of all conjoint attributes and their possible levels is presented in Appendix B Table B1. Each theoretically distinct dimension of populist and non-populist elements had four distinct manifestations paraphrased from real-world political campaigns. This helped us ensure the relative realism of our experiment despite the fictitious nature of our candidates (Brutger et al., Reference Brutger, Kertzer, Renshon, Tingley and Weiss2022).

To test our main empirical expectation regarding the effectiveness of populist rhetoric, we follow the conventional empirical approach and estimate average marginal component effects (AMCEs) of various attributes on candidate choice using simple linear regression with robust standard errors clustered by respondent. The AMCE represents the average difference in the probability of being chosen when comparing two different attribute values (e.g., a candidate who worked as a lawyer compared to a candidate who worked as a firefighter) where the average is taken over all other possible attribute combinations. To ensure our results are not driven by our choice of reference categories and to make valid subgroup comparisons, we also provide the estimates of marginal means in addition to AMCEs (Leeper et al., Reference Leeper, Hobolt and Tilley2020).

To simplify the presentation of our results, our main specification recodes pro-immigration and liberal economic stances as party-congruent for those who identify as Democrats and anti-immigration and conservative economic stances as party-congruent for those who identify as Republicans (with 5 percent of pure Independent respondents dropped from the analysis). For our subgroup analyses, in addition to partisan identification, we measure the respondents’ populist attitudes using a previously validated set of questions from Akkerman et al. (Reference Akkerman, Mudde and Zaslove2013) and Wuttke et al. (Reference Wuttke, Schimpf and Schoen2020) adjusted for the US context. In line with Wuttke et al. (Reference Wuttke, Schimpf and Schoen2020), we define “populists” as only those respondents who at least weakly endorse all core components of populism. For the full list of items, see Appendix C. For power analysis, see Appendix D.

3. Analysis and results

Our main results are summarized in Figures 1–3 below. Table A1 provides the detailed coefficients for the models in Figures 1–3 (model 1). Figures A1–3 provide an original coding of policy positions and an alternative coding of our two complex populist-related conjoint treatments as the four simpler, mutually exclusive binary attributes with no change in the underlying results. For other robustness checks, see Appendix E.

Figure 1. Effects of using various features of populist rhetoric on candidate choice. The plot shows the AMCE and marginal mean estimates of the randomly assigned profile and speech attributes on candidates’ probability of being selected. Estimates are based on the baseline OLS model of the original MTurk sample. Bars represent 95 percent CIs. Robust standard errors are clustered by respondent.

Figure 2. Effects of using various features of populist rhetoric by populist attitudes. The plot shows the AMCE and marginal mean estimates of the randomly assigned profile and speech attributes on candidates’ probability of being selected by respondents’ populist attitudes. Estimates are based on the baseline OLS model of the original MTurk sample. Bars represent 95 percent CIs. Robust standard errors are clustered by respondent.

Figure 3. Effects of using various features of populist rhetoric by partisanship (original coding). The plot shows the AMCE and marginal mean estimates of the randomly assigned profile and speech attributes on candidates’ probability of being selected by respondents’ partisanship. Estimates are based on the baseline OLS model of the original MTurk sample. Bars represent 95 percent CIs. Robust standard errors are clustered by respondent.

Overall, as can be seen from Figures 1 and A1, populist rhetoric does not have a statistically significant effect on candidate choice. At the same time, the effects of policy stances are both statistically and substantively significant. A candidate with a party-congruent stance on immigration and the economy has a respectively 7–8 percentage point higher probability of being selected. Importantly, this holds true regardless of the use of populist rhetorical elements in their campaign message (see Tables A1 and 1). Unlike populist rhetoric, these sizable effects of either economic or immigration issue positioning are also robust to Holm–Bonferroni correction for multiple comparisons across all 128 coefficients and models tested in this paper (p<0.01). The magnitude of these effects is also comparable to the estimates for congruent issue positioning uncovered in the previous candidate choices experiments (e.g., Graham and Svolik, Reference Graham and Svolik2020; Silva et al., Reference Silva, Neuner and Wratil2022).

Table 1. Effects of policy and populist rhetoric on vote choice

The table shows the AMCEs of the randomly assigned profile and speech attributes on candidates’ probability of being selected. Robust standard errors clustered by respondent are given in parentheses, *p < 0.05; **p < 0.01; ***p < 0.001.

Of course, the lack of statistical significance or the failure to reject the null hypothesis does not necessarily imply that populist rhetoric has no effect. To test whether there is a practical null effect, we follow the recent methodological literature and consider the “two one-sided test” (TOST) procedure (Lakens et al., Reference Lakens, Scheel and Isager2018). In particular, we test for equivalence against the AMCE interval of ±0.04, which is the minimum effect size that can be considered of substantive political significance given our experimental design. Since the observed confidence intervals for all possible populist features compared to non-populist features are fully contained in this interval, we consider this as evidence for the null practical effect of populist rhetoric.

We then test whether the effects differ among certain subgroups of voters such as among those who hold populist attitudes (Figures 2, A2, and A6). Interestingly, we see that such people are somewhat less sensitive to economic policy. Nonetheless, populists are not more responsive to populist rhetoric than non-populist voters. This holds true across various operationalizations of populist attitudes (see Appendix A).

To further test whether there is an interaction effect between populism and various policy positions, we interact recoded populism variable with the recoded party congruent positions and the original immigration and economic policy positions (see Table 1). Populist messages are the ones that contain all of the anti-pluralist, people-centric, and moralized anti-elitist attributes. We also report results from models with interaction terms between each of the binary components of populism and policy positions in Table A2. We find no significant interaction effects between populist rhetoric and policy positions (regardless of their operationalization). There are also no interaction effects between any of populism's components and policy positions. All in all, the populist rhetoric of any form does not seem to impact the electoral effectiveness or attractiveness of immigration and economic policies.

We also break down our results by partisanship (Figures 3 and A3). Overall, there is some indication that Democrats are relatively more sensitive to policy positions vis-a-́vis populist rhetoric. Specifically, unlike those who lean Democrat, the choices of Republican respondents appear to be almost equally driven by candidates’ economic positions and anti-establishment rhetoric. However, none of these subgroup differences are statistically significant when one accounts for multiple comparisons.