2 Approximate Balancing with Hierarchical Regularization

Hainmueller (Reference Hainmueller2012) proves that the global solution for each unit i’s weight exists and is given by $w_{i}^{ebal}=\frac {q_{i}\exp {(-G_{i}^{\prime }Z)}}{\sum _{D_{i} = 0}q_{i}\exp {(-G_{i}^{\prime }Z)}}$ , where $Z = \{\lambda _{1}, \lambda _{2} \dots , \lambda _{J}\}^{\prime }$ is a set of Lagrange multipliers for the balance and normalizing constraints and $D_{i} = \{0, 1\}$ is the binary treatment indicator. Using the Lagrangian multipliers and the solution weights $w_{i}^{ebal}$ , the constrained optimization problem can be rewritten as the following dual problem:

(1) $$ \begin{align} \min_{Z} L^{d} = \log \left(\sum_{i\in\mathcal{C}}q_{i}\exp{\left(-\sum_{j=1}^{J}\lambda_{j}G_{ij}\right)}\right) + \sum_{j=1}^{J}\lambda_{j}m_{j}. \end{align} $$

After obtaining $w_{i}^{ebal}$ , one can use a difference in means (DIM) approach to estimate the ATT:

$$ \begin{align*}\hat\tau_{ebal} = \frac{1}{n_{1}} \sum_{D_{i} = 1} Y_{i} - \sum_{D_{i} = 0} w_{i}^{ebal}Y_{i},\end{align*} $$

in which $n_{1}$ is the number of treated units. Zhao and Percival (Reference Zhao and Percival2017) show that Problem (1) is an M-estimator for the propensity score with a logistic link using G as predictors, and the solution weights $w_{i}^{ebal}$ belong to a class of inverse probability weights. They also show that under strong ignorability and positivity, $\hat \tau _{ebal}$ is a doubly robust estimator for the ATT: when either the untreated potential outcome $Y_{i}(0)$ or treatment assignment is linear in G, $\hat \tau _{ebal}$ is consistent (see the Supplementary Material (SM) for details).

In practice, the linearity assumption may be unrealistic. To make this assumption more plausible, we can conduct a series expansion of G, for example, by including higher-order terms and various kinds of interactions, obtaining $X\in \mathbb {R}^{T}$ , $(T \gg J)$ . However, exact balancing on high-dimensional X is often infeasible; even when it is, the large number of balancing constraints may cause the solution weights to be heavily concentrated on a few control units, resulting in high variance of the ATT estimates. hbal addresses this problem by modifying the objective function in (1), that is, adding an $\ell ^{2}$ penalty with a hierarchical structure to the Lagrangian multipliers Z:

(2) $$ \begin{align} \min_{Z^{+}} L^{d} =\log \left(\sum_{D_{i} = 0}q_{i}\exp{\left(-\sum_{t=1}^{T}\lambda_{t}X_{it}\right)}\right) + \sum_{t=1}^{T}\lambda_{t}m_{t} + \sum_{k=1}^{K} \alpha_{k}r_{k}, \end{align} $$

where $Z^{+} = \{\lambda _{1} \dots \lambda _{T}\}^{\prime }$ is a vector of Lagrangian multipliers corresponding to T moment conditions. $\sum _{k=1}^{K} \alpha _{k}r_{k}$ is the newly added penalty term, in which $\alpha _{k}$ is a scalar tuning parameter that adjusts the strength of penalty for the kth group, for $k = 1, 2, \dots , K$ ; $r_{k} = \sum _{t \in P_{k}} \lambda _{t}^{2}$ is the squared $\ell ^{2}$ norm of the Lagrangian multipliers ( $\lambda _{t}$ ) for moment conditions in the kth group, in which $P_{k}$ is the set of their indices. We choose $\ell ^{2}$ penalty over $\ell ^{1}$ penalty mainly because the former is twice differentiable, making computation much more efficient. This grouped structure allows differential strengths of regularization to be applied to different groups of balance constraints and prioritizes feature groups that have heavy influence on the overall covariate balance between the treatment and control groups. For example, it is possible that two-way interactions are more important to the overall balance in an application than the squared terms of the covariates (see the SM for a performance comparison of hierarchical and nonhierarchical regularization). This optimization problem gives $w_{i}^{hbal}=\frac {q_{i}\exp {(-X^{\prime }Z^{+})}}{\sum _{D_{i} = 0}q_{i}\exp {(-X^{\prime }Z^{+})}}$ ( $i\in \mathcal {C}$ ) and

$$ \begin{align*}\hat\tau_{hbal} = \frac{1}{n_{1}} \sum_{D_{i} = 1} Y_{i} - \sum_{D_{i} = 0} w_{i}^{hbal}Y_{i}.\end{align*} $$

2.2 Combining hbal with an Outcome Model

When the number of control units is small relative to the number of moment conditions, hbal only achieves approximate balance. To remove bias caused by the remaining imbalance, we suggest combining hbal with an outcome model that includes the same set of covariates in the moment conditions, which we label as hbal+. Because hbal gives higher weights to units that have similar propensity scores to those of the treated units, this strategy leads to efficiency gain for a regression-based double selection approach. When combined with an outcome model, an ATT estimator is given by

$$ \begin{align*}\hat{\tau}_{hbal+} = \frac{1}{n_{1}}\sum_{D_{i}=1}(Y_{i}-\hat{g}_{0}(G_{i})) - \sum_{D_{i}=0}w^{hbal}(Y_{i}-\hat{g}_{0}(G_{i})),\end{align*} $$

where $\hat {g}_{0}(G_{i}) = X_{i}^{\prime }\hat \beta $ is based on a linear regression on the expanded features. Zhao and Percival (Reference Zhao and Percival2017) show that $\hat {\tau }_{hbal+}$ is consistent for the ATT when either $g_{0}$ is linear in $X_{i}$ or $w^{hbal}$ converges to the logit of the true propensity scores. With nonzero tuning parameters, hbal achieves exact balancing on G and only approximate balance on $X\backslash G$ . Hence, if the true propensity scores depend on $X\backslash G$ , $w^{hbal}$ does not converge to the logit of the true propensity scores, in which case a correctly specified outcome model $g_{0}$ ensures the consistency of $\hat {\tau }_{hbal+}$ .

3 Monte Carlo Evidence

To evaluate the performance of hbal, we conduct Monte Carlo simulations to compare hbal and hbal+ with five commonly used matching and weighting methods, including inverse propensity score weighting (PSW; e.g., Hirano, Imbens, and Ridder Reference Hirano, Imbens and Ridder2003), covariate balancing propensity score (CBPS; Imai and Ratkovic Reference Imai and Ratkovic2014), coarsened exact matching (CEM; Iacus, King, and Porro Reference Iacus, King and Porro2012), ebal (Hainmueller Reference Hainmueller2012), and kernel balancing (Hazlett Reference Hazlett2020). To illustrate the advantage of hierarchical regularization, we also report the results from using ebal to balance on the serially expanded covariate set (ebal*). The naive DIM (Raw) estimator is also included as a benchmark.

3.2 Results

Figure 1 presents the simulation result with sample size $N=900$ and control to treatment ratio 5:1. We report additional results with varying outcome designs, sample sizes, and control to treatment ratios in the SM. The comparative performance of hbal (or hbal+) is similar across different simulation setups.

Figure 1 Monte Carlo evidence. The top-left, top-right, and bottom-left panels correspond to results from outcome designs 1, 2, and 3, respectively. All three designs share the same treatment assignment mechanism, that is, $D=\mathbf {1}\{f(G)-2 +\epsilon> 0\}$ . The bottom-right panel compares the speed between kbal and hbal.

For the linear outcome design 1, most methods substantially reduce bias as compared to the naive DIM estimator. One exception is ebal*, whose poor performance is caused by nonconvergence of the ebal algorithm with many moment constraints. In comparison, hbal’s ability to discriminate balance among covariate moments leads to superior performance in both bias and variance reduction. As the outcome-generating process becomes more complex, methods that rely only on G to estimate propensity scores or weights perform poorly. Compared to ebal, ebal*, and kbal, hbal and hbal+ yield estimates with substantially less bias and smaller variance in designs 2 and 3. These results demonstrate that, when the data-generating process is not too far off from a polynomial expansion of the covariates, hbal’s hierarchical structure is able to tailor the strengths of regularization to the importance of balance constraints, thus reducing bias while maintaining a relatively small variance of the estimates. In the SM, we provide additional evidence that hierarchical regularization leads to higher correlation between the solution weights and the true propensity scores.

Figure 2 The effect of vacancy period on presidential ideological vote. The solid circle and triangle represent the ATT estimates from ebal+ using the original seven covariates and hbal+ using covariates after series expansion, respectively, with 95% confidence intervals. The gray bars represent the 95% confidence intervals of 500 ebal+ estimates using various combination of the covariates and their higher-order terms.

Moreover, hbal is also much more computationally efficient and scalable than kbal. The bottom-right panel of Figure 1 shows the average running time across $500$ samples for varying sample sizes. Across sample sizes, hbal finds the solution weights using a fraction of kbal’s time and hbal’s advantage of scalability becomes more evident as the sample size increases.

4 Promotion Prospect and Circuit Court Judge Behavior

To illustrate how hbal works in empirical settings, we replicate a recent article by Black and Owens (Reference Black and Owens2016), who study the effect of promotion prospect to the Supreme Court on the behavior of circuit court judges. Using CEM, the authors show that judges who are on the president’s shortlist to fill Supreme Court vacancies are more likely to vote in line with the president’s position during the vacancy period as compared to the nonvacancy period; they find no such effect among noncontending judges who stand little chance to be nominated to the Supreme Court. We focus on whether circuit court judges ruled in line with the president as the outcome of interest. The binary treatment variable is vacancy period (vs. nonvacancy period). To address potential confounding, the authors use CEM to match cases on seven covariates that might influence a judge’s behavior and the treatment, such as the judge’s Judicial Common Space score, the judge’s ideological alignment with the president, and whether the case decision was reversed by the circuit court.

In Figure 2, we compare the results from mean balancing on the level terms of the covariates using ebal+ (shown in solid circle) and from balancing on a set of serially expanded covariate using hbal+ (shown in solid triangle). To assess whether the ebal+ estimate is sensitive to different model specifications, we also include an additional 500 models in which random higher moments of the covariates are included (shown in gray). For both methods, we use the solution weights to estimate a weighted linear regression and report 95% confidence intervals based on standard errors clustered at the individual judge level. For contending judges, estimates from both methods indicate judges are more likely to rule in line with the president during vacancy periods than nonvacancy periods, although the estimate from ebal+ using the level terms only is not statistically significant at the $5\%$ level. Because hbal+’s specification includes higher-order terms that can explain additional variation in the outcome and treatment assignment, we obtain a more precise and likely more reliable estimate than ebal+’s. For noncontending judges, ebal+’s estimate suggests that noncontending judges tend to be more likely to rule in line with the president during a vacancy period, while hbal+’s estimate shows no significant difference between the vacancy and nonvacancy periods. In short, hbal+’s results are broadly in line with Black and Owens’ (Reference Black and Owens2016) original findings, whereas the estimates from ebal+ for both contending and non-contending judges vary widely depending on specifications, ranging from negative to positive effects.