The previous chapter introduced the Neyman causal model and described that model as a sensible starting point for many natural experiments. Parameters such as the average causal effect are defined in terms of this causal model. Yet, estimators of those parameters, such as the observed difference of means in treatment and control groups, also depend on the model of the chance—i.e., stochastic—process that gives rise to observed data. The Neyman urn model is one such model: it says that units are sampled at random from the study group and placed into treatment and control groups.

Because this sampling process is stochastic, estimators such as the difference of means will vary across different realizations of the data-generating process. Suppose Nature could run a given natural experiment over and over again, drawing units at random from an urn and assigning them to the treatment and control group. The data would likely turn out somewhat differently each time, due to chance error. Some units that went into the treatment group in one realization of the natural experiment would go the second time into control, and vice versa. Thus, the observed difference of means would differ across each hypothetical replication of the natural experiment. The spread or distribution of all the hypothetical differences of means is called the sampling distribution of the difference of means. A natural measure of the size of this spread is the standard deviation of this sampling distribution—the standard error. The standard error tells us how much the observed difference of means is likely to vary from the mean of the sampling distribution of the estimator, in any given replication.