2 Neil and Winship on benchmark tests

Neil and Winship (Reference Neil and Winship2019) illustrate the shortcomings of benchmark tests using a hypothetical population with 100,000 black individuals and 100,000 white individuals, in which the criminality rate is higher among blacks (15,000 vs. 10,000). Police stop 10,000 black individuals and 5,000 white ones, making the stop rate for blacks twice as high. Although one might suggest this reflects the higher criminality rate in the black population, that rate is only 50% higher than the rate in the white population, and thus it cannot account for the difference. This reasoning is captured by the criminal-based benchmark test, which statistically adjusts for criminality by comparing stop rates as a proportion of criminality rates in each population. Numerically: ${{10,000} \over {15,000}}:{{5,000} \over {10,000}}$ =1.33:1. Because this benchmark adjusts for the different criminality rates—revealing that even after adjustment, the black stop rate is higher—it plausibly provides evidence of discrimination. Yet Neil and Winship show that as one specifies further details about this population, there may be either no discrimination or a level of discrimination different from that suggested by this benchmark statistic.

Here’s an illustration of why benchmark statistics may mislead (Neil and Winship Reference Neil and Winship2019, 79). Imagine police only stop people in public spaces and that the distribution of public-space users consists of 40,000 blacks and 20,000 whites. Assuming police stop all individuals at a rate of .25, regardless of race, the number of stops will match those in the hypothetical population. But, by stipulation, police are not using race in choosing whom to stop. Neil and Winship analyze discrimination using the “similarly situated” criterion, which entails that when police do not differentiate among individuals who are otherwise similar except for race, they are not discriminating. ^{Footnote 1} Accordingly, a benchmark statistic that might suggest discrimination no longer does so after specifying additional information.

Neil and Winship’s (Reference Neil and Winship2019) analysis of this scenario reflects their general strategy for criticizing benchmarks. After pointing out that the criminal-based benchmark fails to reveal discrimination if black and white individuals are observed by police at different rates, they claim that users of the benchmark falsely presuppose that individuals of different races are observed at the same rates. More generally, they present scenarios in which the degree of discrimination diverges from that of the benchmark, then fault those using the benchmark for assuming the scenario does not obtain. Although they are correct that benchmarks are only justified given substantive assumptions, their treatment of specific benchmarks as corresponding to particular assumptions suggests that the assumptions in question are statistical. Moreover, they deny that establishing discrimination requires causal inference (2019, 76). In what follows, I use Simpson’s paradox to illustrate why statistical assumptions do not suffice.

3 The causal analysis of Simpson’s paradox

Simpson’s paradox refers to cases in which the probabilistic relationship between two variables in a population differs from that in every subpopulation, where subpopulations are derived by conditioning on values of a variable. For example, in winter 2020, the fatality rate among COVID-19 cases was higher in Italy than in China. But within every age group, the fatality rate was higher in China than in Italy (von Kügelgen et al. Reference von Kügelgen, Gresele and Schölkopf2021). Put differently, although learning that someone was infected in Italy as opposed to China supports their being more likely to die, once one learns the person’s age, this relationship reverses (no matter the age). Such cases are consistent with probability theory. They are paradoxical in the sense of being perplexing rather than impossible.

Simpson’s paradox is just one challenge Neil and Winship raise for benchmark tests, but its proper analysis matters for all of them. Because benchmark tests compare proportions within populations partitioned using a set of variables, understanding how such proportions change as one conditions on additional variables is essential to their interpretation. In cases of Simpson’s paradox, the relationships in the population do not match those in the subpopulations. I now present a standard causal analysis of the paradox, highlighting common misunderstandings that also appear in Neil and Winship.

Table 1 compares the success rates of a medical treatment in different populations. In the whole population, the success rate is the same in both the treatment and control groups, so treatment ( $X$ ) and success ( $Y$ ) are uncorrelated. However, the treatment raises the probability of success in both the male ( $M$ ) and female ( $\neg M$ ) subpopulations (modeled as exhaustive). To see what is going on, first note that the population has more females than males. Second, although the treatment raises the probability of success in both subpopulations, men are more likely to recover than females, even without the treatment.

Table 1. The type of association at the population level (positive, negative, independent) changes at the level of subpopulations. Numbers taken from Simpson’s (Reference Simpson1951) original example

	Full Population, $N$ = 52		Success	Men (M), $N$ = 20		Success	Women ( $\neg $ M), $N$ = 32		Success
	Success (Y)	Failure ( $\neg $ Y)	Rate	Success	Failure	Rate	Success	Failure	Rate
Treatment (X)	20	20	50%	8	5	$ \approx $ 61%	12	15	$ \approx $ 44%
Control ( $\neg $ X)	6	6	50%	4	3	$ \approx $ 57%	2	3	$ \approx $ 40%

Figure 1 causally represents the example. The arrows indicate that gender and treatment are causes of success (the quantitative relationship is unspecified). The dashed line between gender and treatment indicates a correlation, which is presumed not to result from treatment causing gender.

Figure 1. Causal graph for Simpson’s (1951) example.

Determining whether the treatment causes success involves disentangling two ways that the former may be evidentially relevant to the latter. First, those receiving the treatment may have a higher success rate because the treatment causes success. Second, it may be merely that learning someone received the treatment provides evidence of their gender (due to the correlation) and that gender predicts one’s chance of recovery independent of whether they receive the treatment. Experiments in which one intervenes to make the treatment uncorrelated with any potential influence of success (e.g., gender) help disentangle causal from merely evidential relevance. Formally, this distinction is marked by the operator ${\rm{do}}()$ , where $P(Y|{\rm{do}}\left( X \right))$ indicates the probability distribution of $Y$ that would result from intervening on $X$ (this may differ from the “observational” distribution $P(Y|X)$ ).

The key concept for bridging causal and statistical assumptions is identifiability. Identifiability relates (i) a probability distribution, (ii) a causal graph, and (iii) a causal quantity (e.g., the magnitude of $X$ ’s effect on $Y$ ). A causal quantity is identifiable if and only if (iff), given a graph, one can uniquely determine its value from the distribution. To illustrate, the effect of treatment on success would not be identifiable if these variables had an unmeasured common cause, because even given the probability distribution, one could not determine the extent to which any correlation between treatment and success results from the causal relationship as opposed to the common cause.

In contrast, in figure 1, the effect of treatment ( $X$ ) on success ( $Y$ ) is identified conditional on gender ( $M$ ), using the following formula:

(1) $$P(Y|{\rm{do}}\left( X \right)) = \mathop \sum \limits_M P(Y|X,M){\rm{\;}}P\left( M \right).$$

This equation derives the effect of $X$ on $Y$ in the population from a weighted average of the conditional probabilities in the male and female subpopulations. Its significance is that although the noncausal conditional probability $P(Y|X)$ may differ arbitrarily from $P(Y|X,M)$ and $P(Y|X,\neg M)$ , the causal conditional probability $P(Y|{\rm{do}}\left( X \right))$ averages over conditional probabilities in the subpopulations (which here also correspond to the subpopulation-specific effects). It is therefore impossible for $X$ to causally raise $Y$ ’s probability in the population, but not in any subpopulations.

Equation (1) applies only when $X$ does not cause $M$ . For a different case in which $X$ does cause $M$ —perhaps $M$ is a blood chemical via which the treatment is effective—one should not condition on $M$ when identifying the effect of $X$ on $Y$ . So whether one should condition on $M$ —and correspondingly, whether to consult the population or the $M$ -partitioned subpopulations—depends on the causal model. Note that the causal model in figure 1 is statistically indistinguishable from one in which $X$ causes $M$ : any data generated by one model could have been generated by the other. This highlights the importance of causal information in determining whether to consider populations or subpopulations.

As we will see, although one should not condition on intermediate variables (“mediators”) in evaluating the total effect of $X$ on $Y$ along all paths, one does condition on mediators (in particular ways) when evaluating $X$ ’s influence on $Y$ along particular paths.

After presenting examples involving Simpson’s paradox, Neil and Winship explain:

This comment reveals two common misconceptions about the paradox (Sprenger and Weinberger Reference Sprenger, Weinberger and Edward2021, §3.2). First, the paradox does not show that aggregation over heterogeneous populations is problematic. Aggregation is not a problem when it results in an average effect, and the solution to the paradox is not always to disaggregate because one should not condition on mediators. Second, their comment regarding statistics that are “inconsistent with reality” suggests a concern about objectivity. But if one relies on conditional probabilities identifying genuine effects, Simpson’s paradox does not threaten the objectivity of causation. The subpopulation effects derived from conditioning can differ from the population effect, but this is because the subpopulations have different distributions of background factors, and the effects are therefore objectively different.

Finally, note the claim that the paradox can “bias” benchmark tests. To talk about a measurement being biased, one needs to specify which quantity one aims to measure. Neil and Winship (Reference Neil and Winship2019) do not do so. They write as if there is some comparison of stop rates across the populations that would establish discrimination, and benchmarks simply provide the wrong comparison. Figuring out which rates are the relevant ones is treated as a statistical problem. But Simpson’s paradox reveals that comparisons of rates or proportions across populations are not, in general, invariant to conditioning on new variables, absent further assumptions. I will now show how causal models represent those assumptions in discrimination contexts.

4 Using causal models to interpret statistics

Causal models should not be conceived as a substitute for normative theorizing about which criteria police may use. Nevertheless, given an account specifying legitimate search criteria, linking discrimination claims to effects in particular models enables one to define one’s measurement target, as I now illustrate.

Let’s begin with the scenario in which whether someone is stopped only depends on whether they are in public or in private. Panel A of figure 2 presents a plausible model in which race is a correlated non-cause of public-space usage. If we link discrimination here to whether race causes being stopped, the model indicates that this effect is identified only when one conditions upon public-space usage.

Figure 2. Two possible models for the public-space scenario.

Now suppose race does cause public-space usage. For example, perhaps race influenced where individuals could get mortgages, and black families ended up in urban neighborhoods where public-space use is more common. Public-space usage is then a mediator between race and being stopped, as in panel B of figure 2. There are two possible uses of this model for evaluating discrimination. One possibility is that because race causes public-space usage, deciding to stop people in public spaces is discriminatory. The correct measure of discrimination is then the total effect of race on being stopped, and one should not condition upon the mediator (i.e., absent common causes, $P({\rm{stopped}}|{\rm{do}}\left( {{\rm{race}}} \right)) = P({\rm{stopped}}|{\rm{race}})$ ). Alternatively, if the racial distribution of public spaces did not influence policies regarding where to stop people, one might link discrimination to the influence of race on being stopped along the direct path. This is the natural direct effect (Pearl Reference Pearl, Breese and Koller2001), which is the change in the probability of being stopped that would result if one were of a different race, but still had the same public-space usage. For instance, would a black individual stopped in public have been less likely to be stopped if they were white but still in public? If so, there is a natural direct effect, providing evidence of discrimination.

Which of these models is appropriate is a subtle matter. Even if the correlation between race and public-space usage results from a common cause (e.g., parental race), using this correlation in choosing where to patrol could itself be discriminatory. If so, the model in panel B of figure 2 seems to better account for the influence of race on being stopped via public-space usage, even though panel A of figure 2 gives the data-generating process. This tension between understanding causal models as representing data-generating processes and as modeling decision making calls for its own article.

Here, the key point is that linking discrimination hypotheses to quantities in a causal model enables one to specify which variables must (and must not) be conditioned upon. Moreover, causal quantities come with a theoretical guarantee against Simpson’s paradox. If one believes that race causally promotes being stopped in a population, but it lowers the probability of being stopped in all subpopulations, then either the subpopulation-specific probabilistic relationships do not identify causal effects, or one erred in positing an effect in the population. Although conditioning can certainly yield subpopulation-specific effects differing from the population average, it cannot make a genuine effect disappear.

I have emphasized confounding due to common causes, but conditioning on a common effect of two variables can also hinder identifiability. This phenomenon of endogenous selection bias (Elwert and Winship Reference Elwert and Winship2014) is especially relevant to discrimination contexts because sampling from a population that is homogeneous with respect to a variable (e.g., public-space usage, being stopped by police [Knox et al. Reference Knox, Lowe and Mummolo2020]) amounts to conditioning on it, and can thus bias an effect. ^{Footnote 2} Causal assumptions are thus essential for evaluating the processes influencing one’s statistical sample (Knox and Mummolo Reference Knox and Mummolo2020).

Because Neil and Winship (Reference Neil and Winship2019) lack a framework for specifying which quantities measure discrimination, they cannot spell out what makes a benchmark “biased.” They are correct that benchmarks are only justified given additional assumptions about the scenario, but their criticism of particular benchmarks as employing the “wrong denominator” (2019, 79) in deciding what to condition upon suggests that the assumptions can be read from the mathematical derivation of the statistical quantities. One might suppose that Neil and Winship do give an account of the measurement target via the “similarly situated” criterion, which says that there is discrimination when individuals are treated differently despite being similar in all respects except for race. Statistically, this corresponds to grouping people based on a list of non-race variables, then determining whether, within a particular homogeneous group, race influences the probability of being stopped. But which non-race variables must be included? Without additional assumptions, anything short of a maximal attribute set leaves open the possibility that probabilistic relationships may disappear upon further partitioning. Thus, the similarly situated criterion cannot be of practical use if interpreted statistically; it further requires the assumptions supplied by causal models.

5 Evidence and assumptions

Pollock (Reference Pollock1987) distinguishes between two ways evidence can undermine one’s belief in some proposition $P$ . Suppose that $P$ is that it will rain tomorrow, which I believe based on a colleague’s testimony. A rebutting defeater for $P$ is evidence of its falsity. For example, perhaps my weather app predicts no chance of rain. In contrast, an undercutting defeater is evidence that my original evidence was unreliable. For example, suppose I learn that my colleague was conducting a study in which she randomly told people it would rain. I have no new evidence against $P$ —I cannot infer that it won’t rain—it’s just that my original justification is undermined.

Now imagine one observes that police stop minorities at the same rate as nonminorities within a population and infers that there is no discrimination. One then notices that within both the criminal and non-criminal subpopulations, minorities are stopped at a higher rate and thus concludes that there is discrimination. One might be inclined to describe this reasoning as follows. The first observation provided prima facie evidence against discrimination, which was then rebutted by the further information. I submit that we should reject this description. Given Simpson’s paradox, statistics alone provide no reason to assume that the probabilistic relationships among variables in a population will be preserved upon partitioning. The subpopulation information does not rebut one’s initial belief but rather reveals that one was never justified in holding it. Statistics only provide evidence regarding discrimination when coupled with further assumptions, such as those embedded in causal models.

The question of what counts as evidence for discrimination is not an idle concern. Within the American legal system, a defendant claiming police discrimination must bring evidence to advance to the “discovery” stage, enabling a more thorough investigation. Following the U.S. Supreme Court decision United States v. Armstrong (1996), this evidence must involve a “credible showing of different treatment of similarly situated persons” (470) of another race. Siegler and Admussen (Reference Siegler and Admussen2020) argue that this creates an insurmountable barrier to being granted discovery because police need not disclose their selection criteria, and without these criteria, one cannot determine who counts as “similarly situated.”

That the similarly situated criterion cannot be applied without knowing the police’s selection criteria reinforces my point that it cannot be interpreted purely statistically. What matters is not whether those who are and are not stopped are similar in all respects, but rather whether, among those satisfying the purportedly legitimate criteria employed by police, race makes any further difference. Yet requiring defendants to present a well-established causal model to obtain discovery would be too high a bar, because only through discovery can one obtain the required evidence. Siegler and Admussen (Reference Siegler and Admussen2020) therefore defend a statistical basis for granting discovery:

Although I endorse Siegler and Admussen’s aim of lowering the evidential standard required for obtaining discovery, a purely statistical standard remains problematic. Disparities by themselves do not reveal discrimination, and one should avoid making inferences from them absent background assumptions about the modeled scenario. Rather than abandoning causal considerations, it would be better to allow the defense to propose a causal model favoring their claim. The prosecution might then be allowed to submit their own model specifying some of the selection criteria, and, if they abstain, the defense would be entitled to their preferred model. Even if, however, the prosecution submits their own model, this by itself is an improvement, because currently, they have no incentive to disclose their selection criteria. This, of course, is only a sketch of a procedure. The main takeaway is that because statistical evidence alone cannot differentiate between discriminatory and nondiscriminatory explanations, one should build causal assumptions into every evidentiary stage of the process.