Matching is a conceptually straightforward method to make groups of units comparable on observed characteristics. The method is, however, limited to settings where the study design is simple and the sample is moderately sized. We illustrate these limitations by asking what the causal effects would have been if a large-scale voter mobilization experiment that took place in Michigan for the 2006 election were scaled up to the full population of registered voters. Matching could help us answer this question, but no existing matching method can accommodate the six treatment arms and the 6,762,701 observations involved in the study. To offer a solution for this and similar empirical problems, we introduce a generalization of the full matching method that can be used with any number of treatment conditions and complex compositional constraints. The associated algorithm produces near-optimal matchings; the worst-case maximum within-group dissimilarity is guaranteed to be no more than four times greater than the optimal solution, and simulation results indicate that it comes considerably closer to the optimal solution on average. The algorithm’s ability to balance the treatment groups does not sacrifice speed, and it uses little memory, terminating in linearithmic time using linear space. This enables investigators to construct well-performing matchings within minutes even in complex studies with samples of several million units.

A key challenge facing many large, in-person public opinion surveys is ensuring that enumerators follow fieldwork protocols. Implementing “quality control” processes can improve data quality and help ensure the representativeness of the final sample. Yet while public opinion researchers have demonstrated the utility of quality control procedures such as audio capture and geo-tracking, there is little research assessing the relative merits of such tools. In this paper, we present new evidence on this question using data from the 2016/17 wave of the AmericasBarometer study. Results from a large classification task demonstrate that a small set of automated and human-coded variables, available across popular survey platforms, can recover the final sample of interviews that results when a full suite of quality control procedures is implemented. Taken as a whole, our results indicate that implementing and automating just a few of the many quality control procedures available can streamline survey researchers’ quality control processes while substantially improving the quality of their data.

Questions about people’s perceptions of politicians or other political actors are of central interest in a wide variety of research areas. But measuring these perceptions is difficult in part because respondents may use survey response scales in different ways. In a classic article, Aldrich and McKelvey (1977) introduce a method adjusting for such differential item functioning by assuming that all respondents perceive political stimuli identically. I propose a modeling approach built on the Aldrich and McKelvey framework but incorporating anchoring vignettes. This approach allows for scale use adjustments without assuming that all respondents perceive a given politician identically. I apply this model to data on Americans’ perceptions of parties, elected officials, and other political actors, showing that, contrary to previous arguments, most variation in ideology ratings is due not to differing scale use, but to differences in underlying perceptions. Specifically, while perceptions of Republican politicians and the Republican party show no significant differences by respondent partisanship, Democratic and Republican respondents differ strongly in their perceptions of the ideology of Democratic political actors as well as the Supreme Court.

Recent accounts of American politics focus heavily on urban–rural gaps in political behavior. Rural politics research is growing but may be stymied by difficulties defining and measuring which Americans qualify as “rural.” We discuss theoretical and empirical challenges to studying rurality. Much existing research has been inattentive to conceptualization and measurement of rural geography. We focus on improving estimation of different notions of rurality and provide a new dataset on urban–rural measurement of U.S. state legislative districts. We scrutinize construct validity and measurement in two studies of rural politics. First, we replicate Flavin and Franko (2020, Political Behavior, 845–864) to demonstrate empirical results may be sensitive to measurement of rural residents. Second, we use Mummolo and Nall’s (2017, The Journal of Politics, 45–59) survey data to show rural self-identification is not well-captured with objective, place-based classifications, suggesting a rethinking of theoretical and empirical accounts of rural identity. We conclude with strategies for operationalizing rurality using readily available tools.

When experimental subjects can interact with each other, the outcome of one individual may be affected by the treatment status of others. In many social science experiments, such spillover effects may occur through multiple networks, for example, through both online and offline face-to-face networks in a Twitter experiment. Thus, to understand how people use different networks, it is essential to estimate the spillover effect in each specific network separately. However, the unbiased estimation of these network-specific spillover effects requires an often-violated assumption that researchers observe all relevant networks. We show that, unlike conventional omitted variable bias, bias due to unobserved networks remains even when treatment assignment is randomized and when unobserved networks and a network of interest are independently generated. We then develop parametric and nonparametric sensitivity analysis methods, with which researchers can assess the potential influence of unobserved networks on causal findings. We illustrate the proposed methods with a simulation study based on a real-world Twitter network and an empirical application based on a network field experiment in China.

Political districts may be drawn to favor one group or political party over another, or gerrymandered. A number of measurements have been suggested as ways to detect and prevent such behavior. These measures give concrete axes along which districts and districting plans can be compared. However, measurement values are affected by both noise and the compounding effects of seemingly innocuous implementation decisions. Such issues will arise for any measure. As a case study demonstrating the effect, we show that commonly used measures of geometric compactness for district boundaries are affected by several factors irrelevant to fairness or compliance with civil rights law. We further show that an adversary could manipulate measurements to affect the assessment of a given plan. This instability complicates using these measurements as legislative or judicial standards to counteract unfair redistricting practices. This paper accompanies the release of packages in C++, Python, and R that correctly, efficiently, and reproducibly calculate a variety of compactness scores.

We extend classical ideal point estimation to allow voters to have different preferences when voting in different domains—for example, when voting on agricultural policy than when voting on defense policy. Our scaling procedure results in estimated ideal points on a common scale. As a result, we are able to directly compare a member’s revealed preferences across different domains of voting (different sets of motions) to assess if, for example, a member votes more conservatively on agriculture motions than on defense. In doing so, we are able to assess the extent to which voting behavior of an individual voter is consistent with a uni-dimensional spatial model—if a member has the same preferences in all domains. The key novelty is to estimate rather than assume the identity of “stayers”—voters whose revealed preference is constant across votes. Our approach offers methodology for investigating the relationship between the basic space and issue space in legislative voting (Poole 2007). There are several methodological advantages to our approach. First, our model allows for testing sharp hypotheses. Second, the methodology developed can be understood as a kind of partial-pooling model for item response theory scaling, resulting in less uncertainty of estimates. Related, our estimation method provides a principled and unified approach to the issue of “granularity” (i.e., the level of aggregation) in the analysis of roll-call data (Crespin and Rohde 2010; Roberts et al. 2016). We illustrate the model by estimating U.S. House of Representatives members’ revealed preferences in different policy domains, and identify several other potential applications of the model including: studying the relationship between committee and floor voting behavior; and investigating constituency influence and representation.

Automated text analysis methods have made possible the classification of large corpora of text by measures such as topic and tone. Here, we provide a guide to help researchers navigate the consequential decisions they need to make before any measure can be produced from the text. We consider, both theoretically and empirically, the effects of such choices using as a running example efforts to measure the tone of New York Times coverage of the economy. We show that two reasonable approaches to corpus selection yield radically different corpora and we advocate for the use of keyword searches rather than predefined subject categories provided by news archives. We demonstrate the benefits of coding using article segments instead of sentences as units of analysis. We show that, given a fixed number of codings, it is better to increase the number of unique documents coded rather than the number of coders for each document. Finally, we find that supervised machine learning algorithms outperform dictionaries on a number of criteria. Overall, we intend this guide to serve as a reminder to analysts that thoughtfulness and human validation are key to text-as-data methods, particularly in an age when it is all too easy to computationally classify texts without attending to the methodological choices therein.

We propose a new methodology for inferring political actors’ latent memberships in communities of collective activity that drive their observable interactions. Unlike existing methods, the proposed Bipartite Link Community Model (biLCM) (1) applies to two groups of actors, (2) takes into account that actors may be members of more than one community, and (3) allows a pair of actors to interact in more than one way. We apply this method to characterize legislative communities of special interest groups and politicians in the 113th U.S. Congress. Previous empirical studies of interest group politics have been limited by the difficulty of observing the ties between interest groups and politicians directly. We therefore first construct an original dataset that connects the politicians who sponsor congressional bills with the interest groups that lobby on those bills based on more than two million textual descriptions of lobbying activities. We then use the biLCM to make quantitative measurements of actors’ community memberships ranging from narrow targeted interactions according to industry interests and jurisdictional committee membership to broad multifaceted connections across multiple policy domains.

A new theoretical method for examining gerrymandering is presented based on lattice models of voters, in which districts are constructed by partitioning the lattice. We propose three novel algorithms for constructing equal-population, connected districts which favor the gerrymanderer and incorporate the spatial distribution of voters. Due to the probabilistic population fluctuations inherent to our voter models, Monte Carlo techniques can be applied to study the impact of gerrymandering. We use the method developed here to compare our different gerrymandering algorithms, show approaches which ignore spatial data lead to (legally prohibited) disconnected districts, and examine the effectiveness of isoperimetric quotient tests.

Bureaucratic discretion and executive delegation are central topics in political economy and political science. The previous empirical literature has measured discretion and delegation by manually coding large bodies of legislation. Drawing from computational linguistics, we provide an automated procedure for measuring discretion and delegation in legal texts to facilitate large-scale empirical analysis. The method uses information in syntactic parse trees to identify legally relevant provisions, as well as agents and delegated actions. We undertake two applications. First, we produce a measure of bureaucratic discretion by looking at the level of legislative detail for US states and find that this measure increases after reforms giving agencies more independence. This effect is consistent with an agency cost model, where a more independent bureaucracy requires more specific instructions (less discretion) to avoid bureaucratic drift. Second, we construct measures of delegation to governors in state legislation. Consistent with previous estimates using non-text metrics, we find that executive delegation increases under unified government.

In text, images, merged surveys, voter files, and elsewhere, data sets are often missing important covariates, either because they are latent features of observations (such as sentiment in text) or because they are not collected (such as race in voter files). One promising approach for coping with this missing data is to find the true values of the missing covariates for a subset of the observations and then train a machine learning algorithm to predict the values of those covariates for the rest. However, plugging in these predictions without regard for prediction error renders regression analyses biased, inconsistent, and overconfident. We characterize the severity of the problem posed by prediction error, describe a procedure to avoid these inconsistencies under comparatively general assumptions, and demonstrate the performance of our estimators through simulations and a study of hostile political dialogue on the Internet. We provide software implementing our approach.

Given the increasing quantity and impressive placement of work on Bayesian process tracing, this approach has quickly become a frontier of qualitative research methods. Moreover, it has dominated the process-tracing modules at the Institute for Qualitative and Multi-Method Research (IQMR) and the American Political Science Association (APSA) meetings for over five years, rendering its impact even greater. Proponents of qualitative Bayesianism make a series of strong claims about its contributions and scope of inferential validity. Four claims stand out: (1) it enables causal inference from iterative research, (2) the sequence in which we evaluate evidence is irrelevant to inference, (3) it enables scholars to fully engage rival explanations, and (4) it prevents ad hoc hypothesizing and confirmation bias. Notwithstanding the stakes of these claims and breadth of traction this method has received, no one has systematically evaluated the promises, trade-offs, and limitations that accompany Bayesian process tracing. This article evaluates the extent to which the method lives up to the mission. Despite offering a useful framework for conducting iterative research, the current state of the method introduces more bias than it corrects for on numerous dimensions. The article concludes with an examination of the opportunity costs of learning Bayesian process tracing and a set of recommendations about how to push the field forward.

Distinguishing substantively meaningful spillover effects from correlated residuals is of great importance in cross-sectional studies. Both forms of spatial dependence not only hold different implications for the choice of an unbiased estimator but also for the validity of inferences. To guide model specification, different empirical strategies involve the estimation of an unrestricted spatial Durbin model and subsequently use the Wald test to scrutinize the nonlinear restriction of common factors implied by pure error dependence. However, the Wald test’s sensitivity to algebraically equivalent formulations of the null hypothesis receives scant attention in the context of cross-sectional analyses. This article shows analytically that the noninvariance of the Wald test to such reparameterizations stems from the application of a Taylor series expansion to approximate the restriction’s sampling distribution. While asymptotically valid, Monte Carlo simulations reveal that alternative formulations of the common factor restriction frequently produce conflicting conclusions in finite samples. An empirical example illustrates the substantive implications of this problem. Consequently, researchers should either base inferences on bootstrap critical values for the Wald statistic or use the likelihood ratio test which is invariant to such reparameterizations when deciding on the model specification that adequately reflects the spatial process generating the data.

Various theories in political science point to temporal heterogeneity in relationships of interest. Yet, empirical research typically ignores such heterogeneity or employs fairly crude measures to evaluate it. Advances in models for change point detection offer opportunities to study temporal heterogeneity more carefully. We customize a recent such method for political science purposes, for instance so that it accommodates panel data, and provide an accompanying R-package. We evaluate the methodology, and how it behaves when different assumptions about the number and abrupt nature of change points are violated, by using simulated data. Importantly, the methodology allows us to evaluate changes to different quantities of interest (for various estimators). It also allows us to provide comprehensive estimates concerning uncertainty in the timing and size of changes. We illustrate the utility of this flexible change point methodology on two types of regression models (Probit and OLS) in two empirical applications. We first re-investigate the proposition by Albertus (2017) that labor-dependent agriculture had a more pronounced negative effect on democratic survival before the “third wave of democratization.” Next, we utilize data extending from the French revolution to the present, from V-Dem, to examine the time-variant nature of the income–democracy relationship.

We develop a general approach to measuring electoral competitiveness for parties and governments, which is distinct from existing approaches in two ways. First, it allows us to estimate the actual probability of re-electing the incumbent into office, which lies closer to the theoretical concept of interest than most widely used proxies. Second, it incorporates both pre-electoral competitiveness—that is, the uncertainty about the outcome of the upcoming election—and post-electoral competitiveness—that is, the uncertainty concerning who will form the government given a certain election result. The approach can be applied to, and compared across, a multitude of institutional settings and is particularly advantageous in analyses of multiparty democracies. To demonstrate its full potential, we first apply the approach on 1,700 local government elections in Sweden. Three advantages over existing approaches are documented: Our election probability measure shows substantial variation over the election cycle, it can be accurately measured for a single party as well as a government, and it is more capable of predicting re-election into office than any previous measure of electoral competitiveness. A second application on 400 national elections in 34 democracies shows that the approach also works well in a more challenging cross-national setting.

How often do articles depend on suppression effects for their findings? How often do they disclose this fact? By suppression effects, we mean control-variable-induced increases in estimated effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that authors have a strong explanation for them, especially when the statistical significance of the key finding depends on them. In a reanalysis of observational studies from a leading journal, we find that over 30% of articles depend on suppression effects for statistical significance. Although increases in key effect estimates from including control variables are of course potentially justifiable, none of the articles justify or disclose them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.

Conjoint experiments are popular, but there is a paucity of research on respondents’ underlying decision-making processes. We leverage eye-tracking methodology and a series of conjoint experiments, administered to university students and local community members, to examine how respondents process information in conjoint surveys. There are two main findings. First, attribute importance measures inferred from the stated choice data are correlated with attribute importance measures based on eye movement. This validation test supports the interpretation of common conjoint metrics, such as average marginal component effects (AMCEs), as measures of attribute importance. Second, when we experimentally increase the number of attributes and profiles in the conjoint table, respondents view a larger absolute number of cells but a smaller fraction of the total cells displayed. Moving from two to three profiles, respondents search more within-profile, rather than within-attribute, to build summary evaluations. However, respondents’ stated choices remain fairly stable regardless of the number of attributes and profiles in the conjoint table. Together, these patterns speak to the robustness of conjoint experiments and are consistent with a bounded rationality mechanism. Respondents adapt to complexity by selectively incorporating relevant new information to focus on important attributes, while ignoring less relevant information to reduce cognitive processing costs.

We introduce a method for scaling two datasets from different sources. The proposed method estimates a latent factor common to both datasets as well as an idiosyncratic factor unique to each. In addition, it offers a flexible modeling strategy that permits the scaled locations to be a function of covariates, and efficient implementation allows for inference through resampling. A simulation study shows that our proposed method improves over existing alternatives in capturing the variation common to both datasets, as well as the latent factors specific to each. We apply our proposed method to vote and speech data from the 112th U.S. Senate. We recover a shared subspace that aligns with a standard ideological dimension running from liberals to conservatives, while recovering the words most associated with each senator’s location. In addition, we estimate a word-specific subspace that ranges from national security to budget concerns, and a vote-specific subspace with Tea Party senators on one extreme and senior committee leaders on the other.

Regression discontinuity (RD) designs are increasingly common in political science. They have many advantages, including a known and observable treatment assignment mechanism. The literature has emphasized the need for “falsification tests” and ways to assess the validity of the design. When implementing RD designs, researchers typically rely on two falsification tests, based on empirically testable implications of the identifying assumptions, to argue the design is credible. These tests, one for continuity in the regression function for a pretreatment covariate, and one for continuity in the density of the forcing variable, use a null of no difference in the parameter of interest at the discontinuity. Common practice can, incorrectly, conflate a failure to reject evidence of a flawed design with evidence that the design is credible. The well-known equivalence testing approach addresses these problems, but how to implement equivalence tests in the RD framework is not straightforward. This paper develops two equivalence tests tailored for RD designs that allow researchers to provide statistical evidence that the design is credible. Simulation studies show the superior performance of equivalence-based tests over tests-of-difference, as used in current practice. The tests are applied to the close elections RD data presented in Eggers et al. (2015b).