Researchers investigating causal mechanisms in survey experiments often rely on nonrandomized quantities to isolate the indirect effect of treatment through these variables. Such an approach, however, requires a “selection-on-observables” assumption, which undermines the advantages of a randomized experiment. In this paper, we show what can be learned about casual mechanisms through experimental design alone. We propose a factorial design that provides or withholds information on mediating variables and allows for the identification of the overall average treatment effect and the controlled direct effect of treatment fixing a potential mediator. While this design cannot identify indirect effects on its own, it avoids making the selection-on-observable assumption of the standard mediation approach while providing evidence for a broader understanding of causal mechanisms that encompasses both indirect effects and interactions. We illustrate these approaches via two examples: one on evaluations of US Supreme Court nominees and the other on perceptions of the democratic peace.

Existing approaches to estimating ideal points offer no method for consistent estimation or inference without relying on strong parametric assumptions. In this paper, I introduce a nonparametric approach to ideal-point estimation and inference that goes beyond these limitations. I show that some inferences about the relative positions of two pairs of legislators can be made with minimal assumptions. This information can be combined across different possible choices of the pairs to provide estimates and perform hypothesis tests for all legislators without additional assumptions. I demonstrate the usefulness of these methods in two applications to Supreme Court data, one testing for ideological movement by a single justice and the other testing for multidimensional voting behavior in different decades.

In regression discontinuity (RD), a running variable (or “score”) crossing a cutoff determines a treatment that affects the mean-regression function. This paper generalizes this usual “one-score mean RD” in three ways: (i) considering multiple scores, (ii) allowing partial effects due to each score crossing its own cutoff, not just the full effect with all scores crossing all cutoffs, and (iii) accommodating quantile/mode regressions. This generalization is motivated by (i) many multiple-score RD cases, (ii) the full-effect identification needing the partial effects to be separated, and (iii) informative quantile/mode regression functions. We establish identification for multiple-score RD (MRD), and propose simple estimators that become “local difference in differences” in case of double scores. We also provide an empirical illustration where partial effects exist.

Computers hold the potential to draw legislative districts in a neutral way. Existing approaches to automated redistricting may introduce bias and encounter difficulties when drawing districts of large and even medium-sized jurisdictions. We present a new algorithm that can neutrally generate legislative districts without indications of bias that are contiguous, balanced and relatively compact. The algorithm does not show the kinds of bias found in prior algorithms and is an advance over previously published algorithms for redistricting because it is computationally more efficient. We use the new algorithm to draw 10,000 maps of congressional districts in Mississippi, Virginia, and Texas. We find that it is unlikely that the number of majority-minority districts we observe in the Mississippi, Virginia, and Texas congressional maps of these states would happen through a neutral redistricting process.

In the study of social processes, the presence of unobserved heterogeneity is a regular concern. It should be particularly worrisome for the statistical analysis of networks, given the complex dependencies that shape network formation combined with the restrictive assumptions of related models. In this paper, we demonstrate the importance of explicitly accounting for unobserved heterogeneity in exponential random graph models (ERGM) with a Monte Carlo analysis and two applications that have played an important role in the networks literature. Overall, these analyses show that failing to account for unobserved heterogeneity can have a significant impact on inferences about network formation. The proposed frailty extension to the ERGM (FERGM) generally outperforms the ERGM in these cases, and does so by relatively large margins. Moreover, our novel multilevel estimation strategy has the advantage of avoiding the problem of degeneration that plagues the standard MCMC-MLE approach.

This article examines how the presentation of information during a laboratory experiment can alter a study’s findings. We compare four possible ways to present information about hypothetical candidates in a laboratory experiment. First, we manipulate whether subjects experience a low-information or a high-information campaign. Second, we manipulate whether the information is presented statically or dynamically. We find that the design of a study can produce very different conclusions. Using candidate’s gender as our manipulation, we find significant effects on a variety of candidate evaluation measures in low-information conditions, but almost no significant effects in high-information conditions. We also find that subjects in high-information settings tend to seek out more information in dynamic environments than static, though their ultimate candidate evaluations do not differ. Implications and recommendations for future avenues of study are discussed.

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

Survey experiments often manipulate the description of attributes in a hypothetical scenario, with the goal of learning about those attributes’ real-world effects. Such inferences rely on an underappreciated assumption: experimental conditions must be information equivalent (IE) with respect to background features of the scenario. IE is often violated because subjects, when presented with information about one attribute, update their beliefs about others too. Labeling a country “a democracy,” for example, affects subjects’ beliefs about the country’s geographic location. When IE is violated, the effect of the manipulation need not correspond to the quantity of interest (the effect of beliefs about the focal attribute). We formally define the IE assumption, relating it to the exclusion restriction in instrumental-variable analysis. We show how to predict IE violations ex ante and diagnose them ex post with placebo tests. We evaluate three strategies for achieving IE. Abstract encouragement is ineffective. Specifying background details reduces imbalance on the specified details and highly correlated details, but not others. Embedding a natural experiment in the scenario can reduce imbalance on all background beliefs, but raises other issues. We illustrate with four survey experiments, focusing on an extension of a prominent study of the democratic peace.

Does treatment mode matter in studies of the effects of candidate race or ethnicity on voting decisions? The assumption implicit in most such work is that such treatment mode differences are either small and/or theoretically well understood, so that the choice of how to signal the race of a candidate is largely one of convenience. But this assumption remains untested. Using a nationally representative sample of white voting-age citizens and a modified conjoint design, we evaluate whether signaling candidate ethnicity with ethnic labels and names results in different effects than signaling candidate ethnicity with ethnically identifiable photos and names. Our results provide strong evidence that treatment-mode effects are substantively large and statistically significant. Further, these treatment-mode effects are not consistent with extant theoretical accounts. These results highlight the need for additional theoretical and empirical work on race/ethnicity treatment-mode effects.

Despite the popularity of unsupervised techniques for political science text-as-data research, the importance and implications of preprocessing decisions in this domain have received scant systematic attention. Yet, as we show, such decisions have profound effects on the results of real models for real data. We argue that substantive theory is typically too vague to be of use for feature selection, and that the supervised literature is not necessarily a helpful source of advice. To aid researchers working in unsupervised settings, we introduce a statistical procedure and software that examines the sensitivity of findings under alternate preprocessing regimes. This approach complements a researcher’s substantive understanding of a problem by providing a characterization of the variability changes in preprocessing choices may induce when analyzing a particular dataset. In making scholars aware of the degree to which their results are likely to be sensitive to their preprocessing decisions, it aids replication efforts.

Automated text analysis allows researchers to analyze large quantities of text. Yet, comparative researchers are presented with a big challenge: across countries people speak different languages. To address this issue, some analysts have suggested using Google Translate to convert all texts into English before starting the analysis (Lucas et al. 2015). But in doing so, do we get lost in translation? This paper evaluates the usefulness of machine translation for bag-of-words models—such as topic models. We use the europarl dataset and compare term-document matrices (TDMs) as well as topic model results from gold standard translated text and machine-translated text. We evaluate results at both the document and the corpus level. We first find TDMs for both text corpora to be highly similar, with minor differences across languages. What is more, we find considerable overlap in the set of features generated from human-translated and machine-translated texts. With regard to LDA topic models, we find topical prevalence and topical content to be highly similar with again only small differences across languages. We conclude that Google Translate is a useful tool for comparative researchers when using bag-of-words text models.

Network analysis has typically examined the formation of whole networks while neglecting variation within or across networks. These approaches neglect the particular roles actors may adopt within networks. While cross-sectional approaches for inferring latent roles exist, there is a paucity of approaches for considering roles in longitudinal networks. This paper explores the conceptual dynamics of temporally observed roles while deriving and introducing a novel statistical tool, the ego-TERGM, capable of uncovering these latent dynamics. Estimated through an Expectation–Maximization algorithm, the ego-TERGM is quick and accurate in classifying roles within a broader temporal network. An application to the Kapferer strike network illustrates the model’s utility.

The item count technique (ICT-MLE) regression model for survey list experiments depends on assumptions about responses at the extremes (choosing no or all items on the list). Existing list experiment best practices aim to minimize strategic misrepresentation in ways that virtually guarantee that a tiny number of respondents appear in the extrema. Under such conditions both the “no liars” identification assumption and the computational strategy used to estimate the ICT-MLE become difficult to sustain. I report the results of Monte Carlo experiments examining the sensitivity of the ICT-MLE and simple difference-in-means estimators to survey design choices and small amounts of non-strategic respondent error. I show that, compared to the difference in means, the performance of the ICT-MLE depends on list design. Both estimators are sensitive to measurement error, but the problems are more severe for the ICT-MLE as a direct consequence of the no liars assumption. These problems become extreme as the number of treatment-group respondents choosing all the items on the list decreases. I document that such problems can arise in real-world applications, provide guidance for applied work, and suggest directions for further research.

Representative democracy entails the aggregation of multiple policy issues by parties into competing bundles of policies, or “manifestos,” which are then evaluated holistically by voters in elections. This aggregation process obscures the multidimensional policy preferences underlying a voter’s single choice of party or candidate. We address this problem through a conjoint experiment based on the actual party manifestos in Japan’s 2014 House of Representatives election. By juxtaposing sets of issue positions as hypothetical manifestos and asking respondents to choose one, our study identifies the effects of specific positions on the overall assessment of manifestos, heterogeneity in preferences among subgroups of respondents, and the popularity ranking of manifestos. Our analysis uncovers important discrepancies between voter preferences and the portrayal of the election results by politicians and the media as providing a policy mandate to the Liberal Democratic Party, underscoring the potential danger of inferring public opinion from election outcomes alone.

Accumulated evidence identifies discernible gender gaps across many dimensions of professional academic careers including salaries, publication rates, journal placement, career progress, and academic service. Recent work in political science also reveals gender gaps in citations, with articles written by men citing work by other male scholars more often than work by female scholars. This study estimates the gender gap in citations across political science subfields and across methodological subfields within political science, sociology, and economics. The research design captures variance across research areas in terms of the underlying distribution of female scholars. We expect that subfields within political science and social science disciplines with more women will have smaller gender citation gaps, a reduction of the “Matthew effect” where men’s research is viewed as the most central and important in a field. However, gender citation gaps may persist if a “Matilda effect” occurs whereby women’s research is viewed as less important or their ideas are attributed to male scholars, even as a field becomes more diverse. Analysing all articles published from 2007–2016 in several journals, we find that female scholars are significantly more likely than mixed gender or male author teams to cite research by their female peers, but that these citation rates vary depending on the overall distribution of women in their field. More gender diverse subfields and disciplines produce smaller gender citation gaps, consistent with a reduction in the “Matthew effect”. However, we also observe undercitation of work by women, even in journals that publish mostly female authors. While improvements in gender diversity in academia increase the visibility and impact of scholarly work by women, implicit biases in citation practices in the social sciences persist.

Data sets quantifying phenomena of social-scientific interest often use multiple experts to code latent concepts. While it remains standard practice to report the average score across experts, experts likely vary in both their expertise and their interpretation of question scales. As a result, the mean may be an inaccurate statistic. Item-response theory (IRT) models provide an intuitive method for taking these forms of expert disagreement into account when aggregating ordinal ratings produced by experts, but they have rarely been applied to cross-national expert-coded panel data. We investigate the utility of IRT models for aggregating expert-coded data by comparing the performance of various IRT models to the standard practice of reporting average expert codes, using both data from the V-Dem data set and ecologically motivated simulated data. We find that IRT approaches outperform simple averages when experts vary in reliability and exhibit differential item functioning (DIF). IRT models are also generally robust even in the absence of simulated DIF or varying expert reliability. Our findings suggest that producers of cross-national data sets should adopt IRT techniques to aggregate expert-coded data measuring latent concepts.

Measuring the causal impact of state behavior on outcomes is one of the biggest methodological challenges in the field of political science, for two reasons: behavior is generally endogenous, and the threat of unobserved variables that confound the relationship between behavior and outcomes is pervasive. Matching methods, widely considered to be the state of the art in causal inference in political science, are generally ill-suited to inference in the presence of unobserved confounders. Heckman-style multiple-equation models offer a solution to this problem; however, they rely on functional-form assumptions that can produce substantial bias in estimates of average treatment effects. We describe a category of models, flexible joint likelihood models, that account for both features of the data while avoiding reliance on rigid functional-form assumptions. We then assess these models’ performance in a series of neutral simulations, in which they produce substantial (55% to ${>}$90%) reduction in bias relative to competing models. Finally, we demonstrate their utility in a reanalysis of Simmons’ (2000) classic study of the impact of Article VIII commitment on compliance with the IMF’s currency-restriction regime.

We introduce a model that extends the standard vote choice model to encompass text. In our model, votes and speech are generated from a common set of underlying preference parameters. We estimate the parameters with a sparse Gaussian copula factor model that estimates the number of latent dimensions, is robust to outliers, and accounts for zero inflation in the data. To illustrate its workings, we apply our estimator to roll call votes and floor speech from recent sessions of the US Senate. We uncover two stable dimensions: one ideological and the other reflecting to Senators’ leadership roles. We then show how the method can leverage common speech in order to impute missing data, recovering reliable preference estimates for rank-and-file Senators given only leadership votes.