Survey experiments have become a central methodology across the social sciences. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, due to the expense of population-based samples, much research relies on convenience samples (e.g. students, online opt-in samples). The emergence of affordable, but non-representative online samples has reinvigorated debates about the external validity of experiments. We conduct two studies of how experimental treatment effects obtained from convenience samples compare to effects produced by population samples. In Study 1, we compare effect estimates from four different types of convenience samples and a population-based sample. In Study 2, we analyze treatment effects obtained from 20 experiments implemented on a population-based sample and Amazon's Mechanical Turk (MTurk). The results reveal considerable similarity between many treatment effects obtained from convenience and nationally representative population-based samples. While the results thus bolster confidence in the utility of convenience samples, we conclude with guidance for the use of a multitude of samples for advancing scientific knowledge.

Experimentation is an increasingly popular method among political scientists. While experiments are highly advantageous for creating internally valid conclusions, they are often criticized for being low on external validity. Critical to questions of external validity are the types of subjects who participate in a given experiment, with scholars typically arguing that samples of adults are more externally valid then student samples. Despite the vociferousness of such arguments, these claims have received little empirical treatment. In this paper we empirically test for key differences between student and adult samples by conducting four parallel experiments on each of the three samples commonly used by political scientists. We find that our student and diverse, national adult sample behave consistently and in line with theoretical predictions once relevant moderators are taken into account. The same is not true for our adult convenience sample.

Increasingly, experimental research is being conducted on the Internet in addition to the laboratory. Online experiments are more convenient for subjects and researchers, but we know little about how the choice of study location affects data quality. To investigate whether respondent behavior differs across study location, we randomly assign subjects to participate in a study in a laboratory or in an online setting. Contrary to our expectations, we find few differences between participants in terms of the level of attention and socially desirable responding. However, we find significant differences in two areas: the degree of self-reported distractions while completing the questionnaire and the tendency to consult outside sources for answers to political knowledge questions. We conclude that when the greater convenience (and higher response rates) of online experiments outweighs these disadvantages, Internet administration of randomized experiments represent an alternative to laboratory administration.

In the past decade, the body of research using experimental approaches to investigate the responsiveness of elected officials has grown exponentially. Given this explosion of work, a systematic assessment of these studies is needed not only to take stock of what we have learned so far about democratic responsiveness, but also to inform the design of future studies. In this article, I conduct the first meta-analysis of all experiments that examine elite responsiveness to constituent communication. I find that racial/ethnic minorities and messages sent to elected officials (as opposed to non-elected) are significantly less likely to receive a response. A qualitative review of the literature further suggests that some of these inequalities in responsiveness are driven by personal biases of public officials, rather than strategic, electoral considerations. The findings of this study provide important qualifications and context to prominent individual studies in the field.

Online misinformation has become a constant; only the way actors create and distribute that information is changing. Advances in artificial intelligence (AI) such as GPT-2 mean that actors can now synthetically generate text in ways that mimic the style and substance of human-created news stories. We carried out three original experiments to study whether these AI-generated texts are credible and can influence opinions on foreign policy. The first evaluated human perceptions of AI-generated text relative to an original story. The second investigated the interaction between partisanship and AI-generated news. The third examined the distributions of perceived credibility across different AI model sizes. We find that individuals are largely incapable of distinguishing between AI- and human-generated text; partisanship affects the perceived credibility of the story; and exposure to the text does little to change individuals’ policy views. The findings have important implications in understanding AI in online misinformation campaigns.

Affective polarization – partisans’ dislike and distrust of those from the other party – has reached historically high levels in the United States. While numerous studies estimate its effect on apolitical outcomes (e.g., dating and economic transactions), we know much less about its effects on political beliefs. We argue that those who exhibit high levels of affective polarization politicize ostensibly apolitical issues and actors. An experiment focused on responses to COVID-19 that relies on pre-pandemic, exogenous measures of affective polarization supports our expectations. Partisans who harbor high levels of animus towards the other party do not differentiate the “United States’” response to COVID-19 from that of the Trump administration. Less affectively polarized partisans, in contrast, do not politicize evaluations of the country’s response. Our results provide evidence of how affective polarization, apart from partisanship itself, shapes substantive beliefs. Affective polarization has political consequences and political beliefs stem, in part, from partisan animus.

The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods.

Misinformation can be very difficult to correct and may have lasting effects even after it is discredited. One reason for this persistence is the manner in which people make causal inferences based on available information about a given event or outcome. As a result, false information may continue to influence beliefs and attitudes even after being debunked if it is not replaced by an alternate causal explanation. We test this hypothesis using an experimental paradigm adapted from the psychology literature on the continued influence effect and find that a causal explanation for an unexplained event is significantly more effective than a denial even when the denial is backed by unusually strong evidence. This result has significant implications for how to most effectively counter misinformation about controversial political events and outcomes.

Do politicians engage in ethnic and racial favoritism when conducting constituency service? This article presents results from a replication field experiment with local South African politicians that tested for racial bias in responsiveness to requests about public goods provision. The experiment represents an adaptation of similar experiments conducted in the United States, extending the design to a different institutional environment, albeit one with a similar racially-charged history. Although one might suppose that politicians in South Africa would seek to avoid racial bias given the recent transition to full democracy, I find that South African politicians—both black and white—are more responsive to same-race constituents than to other-race constituents. Same-race bias is evident in both the dominant and the main opposition political parties. Moreover, politicians are not particularly responsive to anyone. Implications for the further study of democratic responsiveness are discussed.

Political elites sometimes seek to delegitimize election results using unsubstantiated claims of fraud. Most recently, Donald Trump sought to overturn his loss in the 2020 US presidential election by falsely alleging widespread fraud. Our study provides new evidence demonstrating the corrosive effect of fraud claims like these on trust in the election system. Using a nationwide survey experiment conducted after the 2018 midterm elections – a time when many prominent Republicans also made unsubstantiated fraud claims – we show that exposure to claims of voter fraud reduces confidence in electoral integrity, though not support for democracy itself. The effects are concentrated among Republicans and Trump approvers. Worryingly, corrective messages from mainstream sources do not measurably reduce the damage these accusations inflict. These results suggest that unsubstantiated voter-fraud claims undermine confidence in elections, particularly when the claims are politically congenial, and that their effects cannot easily be mitigated by fact-checking.

Partisan identity shapes social, mental, economic, and physical life. Using a novel dataset, we study the consequences of partisan identity by examining the immediate impact of electoral loss and victory on happiness and sadness. Employing a quasi-experimental regression discontinuity model we present two primary findings. First, elections strongly affect the immediate happiness/sadness of partisan losers, but minimally impact partisan winners. This effect is consistent with psychological research on the good-bad hedonic asymmetry, but appears to dissipate within a week after the election. Second, the immediate happiness consequences to partisan losers are relatively strong. To illustrate, we show that partisans are affected two times more by their party losing the 2012 U.S. Presidential Election than both respondents with children were to the Newtown shootings and respondents living in Boston were to the Boston Marathon bombings. We discuss implications regarding the centrality of partisan identity to the self and its well-being.

In this essay, we closely examine three aspects of the Reporting Guidelines for this journal, as described by Gerber et al. (2014, Journal of Experimental Political Science 1(1): 81–98) in the inaugural issue of the Journal of Experimental Political Science. These include manipulation checks and when the reporting of response rates is appropriate. The third, most critical, issue concerns the committee's recommendations for detecting errors in randomization. This is an area where there is evidence of widespread confusion about experimental methods throughout our major journals. Given that a goal of the Journal of Experimental Political Science is promoting best practices and a better understanding of experimental methods across the discipline, we recommend changes to the Standards that will allow the journal to play a leading role in correcting these misunderstandings.

Weighting techniques are employed to generalize results from survey experiments to populations of theoretical and substantive interest. Although weighting is often viewed as a second-order methodological issue, these adjustment methods invoke untestable assumptions about the nature of sample selection and potential heterogeneity in the treatment effect. Therefore, although weighting is a useful technique in estimating population quantities, it can introduce bias and also be used as a researcher degree of freedom. We review survey experiments published in three major journals from 2000–2015 and find that there are no standard operating procedures for weighting survey experiments. We argue that all survey experiments should report the sample average treatment effect (SATE). Researchers seeking to generalize to a broader population can weight to estimate the population average treatment effect (PATE), but should discuss the construction and application of weights in a detailed and transparent manner given the possibility that weighting can introduce bias.

Branding of foreign aid may undermine government legitimacy in developing countries when citizens see social services being provided by external actors. We run a survey experiment on a sample of Indian respondents. All subjects learn about an HIV/AIDS program; treated subjects learn that it was foreign-funded. We find null results that, along with existing results in the literature obtained from observational data, call into question the view that foreign-funded service delivery interferes with the development of a fiscal contract between the state and its citizens.

Audit experiments are used to measure discrimination in a large number of domains (Employment: Bertrand et al. (2004); Legislator responsiveness: Butler et al. (2011); Housing: Fang et al. (2018)). Audit studies all have in common that they estimate the average difference in response rates depending on randomly varied characteristics (such as the race or gender) of a requester. Scholars conducting audit experiments often seek to extend their analyses beyond the effect on response to the effects on the quality of the response. Response is a consequence of treatment; answering these important questions well is complicated by post-treatment bias (Montgomery et al., 2018). In this note, I consider a common form of post-treatment bias that occurs in audit experiments.

Politicians' personal beliefs and backgrounds strongly influence politics and policy. But why do individuals with particular beliefs and backgrounds tend to run for office and become politicians? This paper argues that parties and interest groups strategically shape the candidate pool from which voters choose by mobilizing certain individuals to run for office, much like they strategically shape the electorate by mobilizing like-minded individuals to vote. Supporting this view, I first unearth decades of previously disparate evidence suggesting that candidate mobilization efforts are widespread. I then present results from an experiment embedded in an actual candidate mobilization effort that finds encouragement to run for office can meaningfully increase interest in candidacy. Implications and opportunities for further research are discussed.

Can African politicians play the ethnic card? Ethnicity matters for a host of outcomes in Africa, but debate remains about the extent to which it motivates the African voter. In experimental settings, we know that ethnicity shapes political support for hypothetical candidates. This paper offers an experimental test of the extent to which ethnicity shapes political support for actual, real-world politicians. Relying on Benin’s mixed-ethnicity President, this paper proposes a survey experiment that measures the independent effect of coethnic cues in boosting support across both coethnic groups. The results reveal that coethnic cues work: the same political actor can draw support from two different ethnic groups based solely on subtle ethnic cues.

Citizens’ concerns about data privacy and data security breaches may reduce the adoption of COVID-19 contact tracing mobile phone applications, making them less effective. We implement a choice experiment (conjoint experiment) where participants indicate which version of two contact tracing apps they would install, varying the apps’ privacy-preserving attributes. Citizens do not always prioritise privacy and prefer a centralised National Health Service system over a decentralised system. In a further study asking about participants’ preference for digital-only vs human-only contact tracing, we find a mixture of digital and human contact tracing is supported. We randomly allocated a subset of participants in each study to receive a stimulus priming data breach as a concern, before asking about contact tracing. The salient threat of unauthorised access or data theft does not significantly alter preferences in either study. We suggest COVID-19 and trust in a national public health service system mitigate respondents’ concerns about privacy.

The extent to which citizens comply with newly enacted public health measures such as social distancing or lockdowns strongly affects the propagation of the virus and the number of deaths from COVID-19. It is however very difficult to identify non-compliance through survey research because claiming to follow the rules is socially desirable. Using three survey experiments, we examine the efficacy of different ‘face-saving’ questions that aim to reduce social desirability in the measurement of compliance with public health measures. Our treatments soften the social norm of compliance by way of a short preamble in combination with a guilty-free answer choice making it easier for respondents to admit non-compliance. We find that self-reported non-compliance increases by up to +11 percentage points when making use of a face-saving question. Considering the current context and the importance of measuring non-compliance, we argue that researchers around the world should adopt our most efficient face-saving question.

Scholars from across the social sciences argue that identities – such as race, ethnicity, and gender – are highly influential over individuals’ attitudes, actions, and evaluations. Experiments are becoming particularly integral for allowing identity scholars to explain how these social attachments shape our political behavior. In this letter, we draw attention to how identity scholars should approach the common practice of assessing moderators, measuring control variables, and detecting effect heterogeneity using covariates. Special care must be taken when deciding where to place measures of demographic covariates in identity-related experiments, as these cases pose unique challenges from how scholars traditionally approach experimental design. We argue in this letter that identity scholars, particularly those whose subjects identify as women or minorities, are often right to measure covariates of interest posttreatment.