2 Multiplicative Interaction Models

Consider the classical linear multiplicative interaction model that is often assumed in empirical work and is given by the following regression equation:

(1) $$\begin{eqnarray}Y=\unicode[STIX]{x1D707}+\unicode[STIX]{x1D702}X+\unicode[STIX]{x1D6FC}D+\unicode[STIX]{x1D6FD}(D\cdot X)+Z\unicode[STIX]{x1D6FE}+\unicode[STIX]{x1D716}.\end{eqnarray}$$

In this model $Y$ is the outcome variable, $D$ is the key independent variable of interest or “treatment,” $X$ is the moderator—a variable that affects the direction and/or strength of the treatment effect,Footnote ⁶   $D\cdot X$ is the interaction term between $D$ and $X$ , $Z$ is a vector of control variables, and $\unicode[STIX]{x1D707}$ and $\unicode[STIX]{x1D716}$ represent the constant and error terms, respectively.Footnote ⁷

We focus on the cases where the treatment variable $D$ is either binary or continuous and the moderator $X$ is continuous. When $D$ and $X$ are both binary or discrete with few unique values one should employ a fully saturated model that dummies out the treatment and the moderator and includes all interaction terms to obtain the treatment effect at each level of $X$ . Moreover, in the following discussion we focus on the interaction effect components of the model ( $D$ , $X$ , and $D\cdot X$ ). When covariates $Z$ are included in the model, we maintain the typical assumption that the model is correctly specified with respect to these covariates.

The coefficients of Model (1) are consistently estimated under the usual linear regression assumptions which imply that the functional form is correctly specified and that $\mathbb{E}[\unicode[STIX]{x1D716}\mid D,X,Z]=0$ . In the multiplicative interaction model this implies the LIE assumption which says that the marginal effect of the treatment $D$ on the outcome $Y$ is

(2) $$\begin{eqnarray}\mathit{ME}_{D}=\frac{\unicode[STIX]{x2202}Y}{\unicode[STIX]{x2202}D}=\unicode[STIX]{x1D6FC}+\unicode[STIX]{x1D6FD}X,\end{eqnarray}$$

which is a linear function of the moderator $X$ . This LIE assumption implies that the effect of $D$ on $Y$ can only linearly change with $X$ , so if $X$ increases by one unit, the effect of $D$ on $Y$ changes by $\unicode[STIX]{x1D6FD}$ and this change in the effect is constant across the whole range of $X$ . This is a strong assumption, because we often have little theoretical or empirical reason to believe that the heterogeneity in the effect of $D$ on $Y$ takes such a linear form. Instead, it might well be that the effect of $D$ on $Y$ is nonlinear or nonmonotonic. For example, the effect might be small for low values of $X$ , large at medium values of $X$ , and then small again for high values of $X$ .

The LIE assumption in Equation (2) means that the relative effect of treatment $D=d_{1}$ vs.  $D=d_{2}$ can be expressed by the difference between two linear functions in $X$ :

(3) $$\begin{eqnarray}\displaystyle \mathit{Eff}(d_{1},d_{2}) & = & \displaystyle Y(D=d_{1}\mid X,Z)-Y(D=d_{2}\mid X,Z)\nonumber\\ \displaystyle & = & \displaystyle (\unicode[STIX]{x1D707}+\unicode[STIX]{x1D6FC}d_{1}+\unicode[STIX]{x1D702}X+\unicode[STIX]{x1D6FD}d_{1}X)-(\unicode[STIX]{x1D707}+\unicode[STIX]{x1D6FC}d_{2}+\unicode[STIX]{x1D702}X+\unicode[STIX]{x1D6FD}d_{2}X)\nonumber\\ \displaystyle & = & \displaystyle \unicode[STIX]{x1D6FC}(d_{1}-d_{2})+\unicode[STIX]{x1D6FD}(d_{1}-d_{2})X.\end{eqnarray}$$

This decomposition makes clear that under the LIE assumption, the effect of $D$ on $Y$ is the difference between two linear functions, $\unicode[STIX]{x1D707}+\unicode[STIX]{x1D6FC}d_{1}+(\unicode[STIX]{x1D702}+\unicode[STIX]{x1D6FD}d_{1})X$ and $\unicode[STIX]{x1D707}+\unicode[STIX]{x1D6FC}d_{2}+(\unicode[STIX]{x1D702}+\unicode[STIX]{x1D6FD}d_{2})X$ , and therefore the LIE assumption will be most likely to hold if both functions are linear for all modeled contrasts of $d_{1}$ vs.  $d_{2}$ .Footnote ⁸ $^{,}$ Footnote ⁹

This illustrates how attempts to estimate interaction effects with multiplicative interaction models are susceptible to misspecification bias because the LIE assumption will fail if one or both functions are misspecified due to nonlinearities, nonmonotonicities, or a skewed distribution of $X$ and/or $D$ , resulting in bad influence points, etc. As our empirical survey shows below, in practice this LIE assumption often fails because at least one of the two functions is not linear.Footnote ¹⁰

The decomposition in Equation (3) also highlights the issue of common support. Since the conditional effect of $D$ on $Y$ is the difference between two linear functions, it is important that the two functions share common support over $X$ . In other words, at any given value of the moderator $X=x_{0}$ , there should be (1) a sufficient number of data points in the neighborhood of $X=x_{0}$ and (2) those data points should exhibit variation in the treatment, $D$ . If, for example, in the neighborhood of $X=x_{0}$ all data points are treated units ( $D=1$ ), we have a lack of common support and, since there are no control units ( $D=0$ ) in the same region at all, the estimated conditional effect will be driven by interpolation or extrapolation and thus model dependent.Footnote ¹¹

Multiplicative interaction models are susceptible to the lack of common support problem because if the goal is to estimate the conditional effect of $D$ across the range of $X$ then this requires common support across the entire joint distribution of $D$ and $X$ . Otherwise, estimation of the conditional marginal effect will rely on interpolation or extrapolation of at least one of the functions to an area where there is no or only very few observations. It is well known that such interpolation or extrapolation purely based on an assumed functional form results in fragile and model dependent estimates. Slight changes in the assumed functional form or data can lead to very different answers (King and Zeng Reference King and Zeng2006). In our empirical survey below we show that such interpolation or extrapolation is common in applied work using multiplicative interaction models.

In sum, there are two problems with multiplicative interaction models. The LIE assumption states that the interaction effect is linear, but if this assumption fails, the conditional marginal-effect estimates are inconsistent and biased. In addition, the common support condition suggests that we need sufficient data on $X$ and $D$ because otherwise the estimates will be model dependent. Both problems are currently overlooked because they are not detected by scholars following the current best practice guidelines. In the next section we develop simple diagnostic tools and estimation strategies that allow scholars to diagnose these problems and estimate conditional marginal effects while relaxing the LIE assumption.

3 Diagnostics

Before introducing the diagnostic tools, we present simulated data samples to highlight three scenarios: (1) linear marginal effect with a dichotomous treatment, (2) linear marginal effect with a continuous treatment, and (3) nonlinear marginal effect with a dichotomous treatment.

The data generating process (DGP) for both samples that contain a linear marginal effect is as follows:

$$\begin{eqnarray}Y_{i}=5-4X_{i}-9D_{i}+3D_{i}X_{i}+\unicode[STIX]{x1D716}_{i},\quad i=1,2,\ldots ,200.\end{eqnarray}$$

$Y_{i}$ is the outcome for unit $i$ , the moderator is $X_{i}\stackrel{\text{i.i.d.}}{{\sim}}{\mathcal{N}}(3,1)$ , and the error term is $\unicode[STIX]{x1D716}_{i}\stackrel{\text{i.i.d.}}{{\sim}}{\mathcal{N}}(0,4)$ . Both samples share the same sets of $X_{i}$ and $\unicode[STIX]{x1D716}_{i}$ , but in the first sample, the treatment indicator is $D_{i}\stackrel{\text{i.i.d.}}{{\sim}}Bernoulli(0.5)$ , while in the second one it is $D_{i}\stackrel{\text{i.i.d.}}{{\sim}}{\mathcal{N}}(3,1)$ . The marginal effect of $D$ on $Y$ therefore is $ME_{D}=-9+3X$ .

The DGP for the sample with a nonlinear marginal effect is:

$$\begin{eqnarray}Y_{i}=2.5-X_{i}^{2}-5D_{i}+2D_{i}X_{i}^{2}+\unicode[STIX]{x1D701}_{i},\quad i=1,2,\ldots ,200.\end{eqnarray}$$

$Y_{i}$ is the outcome, the moderator is $X_{i}\stackrel{\text{i.i.d.}}{{\sim}}{\mathcal{U}}(-3,3)$ , the treatment indicator is $D_{i}\stackrel{\text{i.i.d.}}{{\sim}}\mathit{Bernoulli}(0.5)$ , and the error term is $\unicode[STIX]{x1D701}_{i}\stackrel{\text{i.i.d.}}{{\sim}}{\mathcal{N}}(0,4)$ . The marginal effect of $D$ on $Y$ therefore is $ME_{D}=-5+2X^{2}$ . For simplicity, we do not include any control variables. Note that all three samples have 200 observations.

We now present a simple visual diagnostic to help researchers to detect potential problems with the LIE assumption and the lack of common support. The diagnostic that we recommend is a scatterplot of raw data. This diagnostic is simple to implement and powerful in the sense that it readily reveals the main problems associated with the LIE assumption and lack of common support.

If the treatment $D$ is binary, we recommend plotting the outcome $Y$ against the moderator $X$ separately for the sample of treatment group observations ( $D=1$ ) and the sample of control group observations ( $D=0$ ). In each sample we recommend superimposing a linear regression line as well as LOESS fits in each group (Cleveland and Devlin Reference Cleveland and Devlin1988).Footnote ¹² The upper panel of Figure 1 presents examples of such a plot for the simulated data with the binary treatment in cases where the marginal effect is (a) linear and (b) nonlinear.

Figure 1. Linear Interaction Diagnostic (LID) plots: simulated samples. Note: The above plots show the relationships among the treatment $D$ , the outcome $Y$ , and the moderator $X$ using the raw data: (a) when $D$ is binary and the true marginal effect is linear; (b) when $D$ is binary and the true marginal effect is nonlinear (quadratic); and (c) when $D$ is continuous and the true marginal effect is linear.

The first important issue to check is whether the relationship between $Y$ and $X$ is reasonably linear in both groups. For this we can simply check if the linear regression lines (blue) and the LOESS fits (red) diverge considerably across the range of $X$ values. In Figure 1(a), where the true DGP contains a linear marginal effect, the two lines are very close to each other in both groups indicating that both conditional expectation functions are well approximated with a linear fit as required by the LIE assumption. However, as Figure 1(b) shows, LOESS (i.e. locally weighted regression) and ordinary least squares (OLS) will diverge considerably when the true marginal effect is nonlinear, thus alerting the researcher to a possible misspecification error.

We call these plots the Linear Interaction Diagnostic (LID) plots. In addition to shedding light on the validity of the LIE assumption, they provide other important insights as well. In Figure 1(a), the slope of $Y$ on $X$ in the treatment group is apparently larger (less negative) than that of the control group ( $\hat{\unicode[STIX]{x1D702}}+\hat{\unicode[STIX]{x1D6FD}}>\hat{\unicode[STIX]{x1D702}}$ ), suggesting a possible positive interaction effect of $D$ and $X$ on $Y$ . The LOESS fit in Figure 1(b) also gives evidence that the relationship between $X$ and $Y$ differs between the two groups, (in fact, the functions are near mirror opposites), a result that is masked by the OLS fit.

A final important issue to look out for is whether there is sufficient common support in the data. For this we can simply compare the distribution of $X$ in both groups and examine the range of $X$ values for which there are a sufficient number of data points for the estimation of marginal effects. The box plots near the center of the figures display quantiles of the moderator at each level of the treatment. The dot in the center denotes the median, the end points of the thick bars denote the 25th and 75th percentiles, and the end points of the thin bars denote the 5th and 95th percentiles. In Figure 1(a), we see that both groups share a common support of $X$ for the range between about 1.5 to 4.5—whereas support exists across the entire range of $X$ in Figure 1(b)—as we would expect given the simulation parameters.Footnote ¹³

If the treatment and moderator are continuous, then visualizing the conditional relationship of $Y$ and $D$ across levels of $X$ is more complicated, but in our experience a simple binning approach is sufficient to detect most problems in typical political science data. Accordingly, we recommend that researchers split the sample into three roughly equal sized groups based on the moderator: low $X$ (first tercile), medium $X$ (second tercile), and high $X$ (third tercile). For each of the three groups we then plot $Y$ against $D$ while again overlaying both the linear and LOESS fits.

Panel (c) of Figure 1 presents an LID plot for the simulated data with the continuous treatment and linear marginal effect. The plot reveals that the conditional expectation function of $Y$ given $D$ is well approximated by a linear model in all three samples of observations with low, medium, or high values on the moderator $X$ .

There is also clear evidence of an interaction as the slope of the line which captures the relationship between $D$ on $Y$ is negative at low levels of $X$ , flat at medium levels of $X$ , and positive at high levels of $X$ . In this case of a continuous treatment and continuous moderator it is also useful to generate the LID plot in both directions to examine the conditional relationships of $D\mid X$ and $X\mid D$ as the standard linear interaction model assumes linearity in both directions. Moreover, it can be useful to visualize interactions using a three-dimensional surface plot generated by a generalized additive model (GAM, Hastie and Tibshirani Reference Hastie and Tibshirani1986).Footnote ¹⁴

4 Estimation Strategies

In this section we develop two simple estimation strategies to estimate the conditional marginal effect of $D$ on $Y$ across values of the moderator $X$ . These approaches have the advantage that they remain in the regression framework familiar to applied researchers and at the same time relax the LIE assumption and flexibly allow for heterogeneity in how the conditional marginal effect changes across values of $X$ . When the marginal effect is indeed linear in $X$ , these strategies are less efficient than the linear interaction model, however, they offer protection against excessive model dependency and the lack of common support—a classic case of the bias-variance trade-off.

4.1 Binning estimator

The first estimation approach is a binning estimator. Simply put, we break a continuous moderator into several bins represented by dummy variables and interact these dummy variables with the treatment indicator, with some adjustment to improve interpretability.Footnote ¹⁵ There are three steps to implement the estimator. First, we discretize the moderator variable $X$ into three bins (respectively corresponding to the three terciles) as before and create a dummy variable for each bin. More formally, we define three dummy variables that indicate the interval $X$ falls into:

$$\begin{eqnarray}G_{1}=\left\{\begin{array}{@{}ll@{}}1\quad & X<\unicode[STIX]{x1D6FF}_{1/3}\\ 0\quad & \text{otherwise}\\ \quad \end{array}\right.,\quad G_{2}=\left\{\begin{array}{@{}ll@{}}1\quad & X\in [\unicode[STIX]{x1D6FF}_{1/3},\unicode[STIX]{x1D6FF}_{2/3})\\ 0\quad & \text{otherwise}\end{array}\right.,\quad G_{3}=\left\{\begin{array}{@{}ll@{}}1\quad & X\geqslant \unicode[STIX]{x1D6FF}_{2/3}\\ 0\quad & \text{otherwise}\end{array}\right.,\end{eqnarray}$$

in which $\unicode[STIX]{x1D6FF}_{1/3}$ and $\unicode[STIX]{x1D6FF}_{2/3}$ are respectively the first and second terciles of $X$ . We can choose other numbers in the support of $X$ to create the bins but the advantage of using terciles is that we obtain estimates of the effect at typical low, medium, and high values of $X$ . While three bins tend to work well in practice for typical political science data that we encountered, the researcher can create more than three bins in order to get a finer resolution of the effect heterogeneity. Increasing the number of bins requires a sufficiently large number of observations.

Second, we pick an evaluation point within each bin, $x_{1}$ , $x_{2}$ , and $x_{3}$ , where we want to estimate the conditional marginal effect of $D$ on $Y$ . Typically, we choose $x_{1}$ , $x_{2}$ , and $x_{3}$ to be the median of $X$ in each bin, but researchers are free to choose other numbers within the bins (for example, the means).

Third, we estimate a model that includes interactions between the bin dummies $G$ and the treatment indicator $D$ , the bin dummies and the moderator $X$ minus the evaluation points we pick ( $x_{1}$ , $x_{2}$ , and $x_{3}$ ), as well as the triple interactions. The last two terms are to capture the changing effect of $D$ on $Y$ within each bin defined by $G$ . Formally, we estimate the following model:

(4) $$\begin{eqnarray}Y=\mathop{\sum }_{j=1}^{3}\{\unicode[STIX]{x1D707}_{j}+\unicode[STIX]{x1D6FC}_{j}D+\unicode[STIX]{x1D702}_{j}(X-x_{j})+\unicode[STIX]{x1D6FD}_{j}(X-x_{j})D\}G_{j}+Z\unicode[STIX]{x1D6FE}+\unicode[STIX]{x1D716}\end{eqnarray}$$

in which $\unicode[STIX]{x1D707}_{j}$ , $\unicode[STIX]{x1D6FC}_{j}$ , $\unicode[STIX]{x1D702}_{j}$ , and $\unicode[STIX]{x1D6FD}_{j}$ ( $j=1,2,3$ ) are unknown coefficients.

The binning estimator has several key advantages over the standard multiplicative interaction model given in Model (1). First, the binning estimator is much more flexible as it jointly fits the interaction components of the standard model to each bin separately, thereby relaxing the LIE assumption.Footnote ¹⁶ Since $(X-x_{j})$ equals zero at each evaluation point $x_{j}$ , the conditional marginal effect of $D$ on $Y$ at the chosen evaluation points within each bin, $x_{1}$ , $x_{2}$ , and $x_{3}$ , is simply given by $\unicode[STIX]{x1D6FC}_{1}$ , $\unicode[STIX]{x1D6FC}_{2}$ , and $\unicode[STIX]{x1D6FC}_{3}$ , respectively. Here, the conditional marginal effects can vary freely across the three bins and therefore can take on any nonlinear or nonmonotonic pattern that might describe the heterogeneity in the effect of $D$ on $Y$ across low, medium, or high levels of $X$ .Footnote ¹⁷

Second, since the bins are constructed based on the support of $X$ , the binning ensures that the conditional marginal effects are estimated at typical values of the moderator and do not rely on excessive extrapolation or interpolation.Footnote ¹⁸

Third, the binning estimator is easy to implement using any regression software and the standard errors for the conditional marginal effects are directly estimated by the regression so there is no need to compute linear combinations of coefficients to compute the conditional marginal effects.

Fourth, the binning estimator provides a generalization that nests the standard multiplicative interaction model as a special case. It can therefore serve as a formal test on the validity of global LIE assumption imposed by the standard model. In particular, if the standard multiplicative interaction Model (1) is the true model, we have the following relationships:

$$\begin{eqnarray}\displaystyle & \displaystyle \unicode[STIX]{x1D707}=\unicode[STIX]{x1D707}_{j}-\unicode[STIX]{x1D702}_{j}x_{j}\quad j=1,2,3; & \displaystyle \nonumber\\ \displaystyle & \displaystyle \unicode[STIX]{x1D702}=\unicode[STIX]{x1D702}_{j}\quad j=1,2,3; & \displaystyle \nonumber\\ \displaystyle & \displaystyle \unicode[STIX]{x1D6FC}=\unicode[STIX]{x1D6FC}_{j}-\unicode[STIX]{x1D6FD}_{j}x_{j}\quad j=1,2,3; & \displaystyle \nonumber\\ \displaystyle & \displaystyle \unicode[STIX]{x1D6FD}=\unicode[STIX]{x1D6FD}_{j}\quad j=1,2,3. & \displaystyle \nonumber\end{eqnarray}$$

The marginal effect of $D$ at $X=x_{j}$ (  $j=1,2,3$ ), therefore, is:

$$\begin{eqnarray}ME(x_{j})=\unicode[STIX]{x1D6FC}_{j}=\unicode[STIX]{x1D6FC}+\unicode[STIX]{x1D6FD}_{j}x_{j}=\unicode[STIX]{x1D6FC}+\unicode[STIX]{x1D6FD}x_{j}.\end{eqnarray}$$

In the Appendix we formally show that when Model (1) is correct we have

$$\begin{eqnarray}\hat{\unicode[STIX]{x1D6FC}}_{j}-(\hat{\unicode[STIX]{x1D6FC}}+\hat{\unicode[STIX]{x1D6FD}}x_{j})\overset{p}{\rightarrow }0,\quad j=1,2,3,\end{eqnarray}$$

in which $\hat{\unicode[STIX]{x1D6FC}}$ and $\hat{\unicode[STIX]{x1D6FD}}$ are estimated from Model (1) and $\hat{\unicode[STIX]{x1D6FC}}_{j}~(j=1,2,3)$ are estimated using Model (4). As mentioned above, we face a bias-variance trade-off. In the special case when the standard multiplicative interaction model is correct and therefore the global LIE assumption holds, then—as the sample size grows—the marginal-effect estimates from the binning estimator converge in probability on the unbiased marginal-effect estimates from the standard multiplicative interaction model given by $ME(X)=\hat{\unicode[STIX]{x1D6FC}}+\hat{\unicode[STIX]{x1D6FD}}X$ . In this case, the standard estimator will be the most efficient estimator for the marginal effect at any given point in the range of the moderator and the estimates will be more precise than those from the binning estimator at the evaluation points simply because the linear model utilizes more information based on the modeling assumptions. However, when the linear interaction assumption does not hold, the standard estimator will be biased and inconsistent and researchers interested in minimizing bias are better off using the more flexible binning estimator that requires more degrees of freedom. Although the binning estimator may also have bias under this circumstance (the bias will disappear as the model becomes more and more flexible), disagreement between the binning estimates and estimates from the linear interaction model gives an indication the LIE assumption is invalid.

To illustrate the performance from the binning estimator we apply it to our simulated datasets that cover the cases of a binary treatment with linear and nonlinear marginal effects. The results are shown in Figure 2. To clarify the correspondence between the binning estimator and the standard multiplicative interaction model we superimpose the three estimates of the conditional marginal effects of $D$ on $Y$ , $\hat{\unicode[STIX]{x1D6FC}}_{1}$ , $\hat{\unicode[STIX]{x1D6FC}}_{2}$ and $\hat{\unicode[STIX]{x1D6FC}}_{3}$ , and their 95% confidence intervals from the binning estimator in their appropriate places (i.e., at $X=x_{j}$ in bin $j$ ) on the marginal-effect plot generated from the standard multiplicative interaction model as recommended by Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006).

In the case of a binary treatment, we also recommend to display at the bottom of the figure a stacked histogram that shows the distribution of the moderator $X$ . In this histogram the total height of the stacked bars refers to the distribution of the moderator in the pooled sample and the red and white shaded bars refer to the distribution of the moderator in the treatment and control groups, respectively. Adding such a histogram makes it easy to judge the degree to which there is common support in the data. In the case of a continuous treatment, we recommend a histogram at the bottom that simply shows the distribution of $X$ in the entire sample.Footnote ¹⁹

Figure 2(a) was generated using the DGP where the standard multiplicative interaction model is the correct model and therefore the LIE assumption holds. Hence, as Figure 2(a) shows, the conditional effect estimates from the binning estimator and the standard multiplicative interaction model are similar in both datasets. Even with a small sample size (i.e., $N=200$ ), the three estimates from the binning estimator, labeled L, M, and H, sit almost right on the estimated linear marginal-effect line from the true standard multiplicative interaction model. Note that the estimates from the binning estimator are only slightly less precise than those from the true multiplicative interaction model, which demonstrates that there is at best a modest cost in terms of decreased efficiency from using this more flexible estimator. We also see from the histogram that the three estimates from the binning estimator are computed at typical low, medium, and high values of $X$ with sufficient common support which is what we expect given the binning based on terciles.

Contrast these results with those in Figure 2(b), which were generated using our simulated data in which the true marginal effect of $D$ is nonlinear. In this case, the standard linear model indicates a slightly negative, but overall very weak, interaction effect, whereas the binning estimates reveal that the effect of $D$ is actually strongly conditioned by $X$ : $D$ exerts a positive effect in the low range of $X$ , a negative effect in the midrange of $X$ , and a positive effect again in the high range of $X$ . In the event of such a nonlinear effect, the standard linear model delivers the wrong conclusion. When the estimates from the binning estimator are far off the line or when they are non-monotonic, we have evidence that the LIE assumption does not hold.

Figure 2. Conditional marginal effects from binning estimator: simulated samples. Note: The above plots show the estimated marginal effects using both the conventional linear interaction model and the binning estimator: (a) when the true marginal effect is linear; (b) when the true marginal effect is nonlinear (quadratic). In both cases, the treatment variable $D$ is dichotomous.

4.2 Kernel estimator

The second estimation strategy is a kernel smoothing estimator of the marginal effect, which is an application of semiparametric smooth varying-coefficient models (Li and Racine Reference Li and Racine2010). This approach provides a generalization that allows researchers to flexibly estimate the functional form of the marginal effect of $D$ on $Y$ across the values of $X$ by estimating a series of local effects with a kernel reweighting scheme. While the kernel estimator requires more computation and its output is less easily summarized than that of the binning estimator, it is also fully automated (e.g., researchers do not need to select a number of bins) and characterizes the marginal effect across the full range of the moderator, rather than at just a few evaluation points.

Formally, the kernel smoothing method is based on the following semiparametric model:

(5) $$\begin{eqnarray}Y=f(X)+g(X)D+\unicode[STIX]{x1D6FE}(X)Z+\unicode[STIX]{x1D716},\end{eqnarray}$$

in which $f(\cdot )$ , $g(\cdot )$ , and $\unicode[STIX]{x1D6FE}(\cdot )$ are smooth functions of $X$ , and $g(\cdot )$ captures the marginal effect of $D$ on $Y$ . It is easy to see that this kernel regression nests the standard interaction model given in Model (1) as a special case when $f(X)=\unicode[STIX]{x1D707}+\unicode[STIX]{x1D702}X$ , $g(X)=\unicode[STIX]{x1D6FC}+\unicode[STIX]{x1D6FD}X$ and $\unicode[STIX]{x1D6FE}(X)=\unicode[STIX]{x1D6FE}$ . However, in the kernel regression the conditional effect of $D$ on $Y$ need not to be linear as required by the LIE assumption, but can vary freely across the range of $X$ . In addition, if covariates $Z$ are included in the model, the coefficients of those covariates are also allowed to vary freely across the range of $X$ resulting in a very flexible estimator that also helps to guard against misspecification bias with respect to the covariates.

We use a kernel based method to estimate Model (5). Specially, for each given $x_{0}$ in the support of $X$ , $\hat{f}(x_{0})$ , ${\hat{g}}(x_{0})$ , and $\hat{\unicode[STIX]{x1D6FE}}(x_{0})$ are estimated by minimizing the following weighted least-squares objective function:

$$\begin{eqnarray}\displaystyle & \displaystyle \left(\hat{\unicode[STIX]{x1D707}}(x_{0}),\hat{\unicode[STIX]{x1D6FC}}(x_{0}),\hat{\unicode[STIX]{x1D702}}(x_{0}),\hat{\unicode[STIX]{x1D6FD}}(x_{0}),\hat{\unicode[STIX]{x1D6FE}}(x_{0})\right)=\underset{\tilde{\unicode[STIX]{x1D707}},\tilde{\unicode[STIX]{x1D6FC}},\tilde{\unicode[STIX]{x1D702}},\tilde{\unicode[STIX]{x1D6FD}},\tilde{\unicode[STIX]{x1D6FE}}}{\text{argmin}}~L(\tilde{\unicode[STIX]{x1D707}},\tilde{\unicode[STIX]{x1D6FC}},\tilde{\unicode[STIX]{x1D702}},\tilde{\unicode[STIX]{x1D6FD}},\tilde{\unicode[STIX]{x1D6FE}}) & \displaystyle \nonumber\\ \displaystyle & \displaystyle L=\mathop{\sum }_{i}^{N}\left\{\left[Y_{i}-\tilde{\unicode[STIX]{x1D707}}-\tilde{\unicode[STIX]{x1D6FC}}D_{i}-\tilde{\unicode[STIX]{x1D702}}(X_{i}-x_{0})-\tilde{\unicode[STIX]{x1D6FD}}D_{i}(X_{i}-x_{0})-\tilde{\unicode[STIX]{x1D6FE}}Z_{i}\right]^{2}K\left(\frac{X_{i}-x_{0}}{h}\right)\right\}, & \displaystyle \nonumber\end{eqnarray}$$

in which $K(\cdot )$ is a Gaussian kernel, $h$ is a bandwidth parameter that we automatically select via least-squares cross-validation, $\hat{f}(x_{0})=\hat{\unicode[STIX]{x1D707}}(x_{0})$ , and ${\hat{g}}(x_{0})=\hat{\unicode[STIX]{x1D6FC}}(x_{0})$ . The two terms $\unicode[STIX]{x1D702}(X-x_{0})$ and $\unicode[STIX]{x1D6FD}D(X-x_{0})$ are included to capture the influence of the first partial derivative of $Y$ with respect to $X$ at each evaluation point of $X$ , a common practice that reduces bias of the kernel estimator on the boundary of the support of $X$ (e.g., Fan, Heckman, and Wand Reference Fan, Heckman and Wand1995). As a result, we obtain three smooth functions $\hat{f}(\cdot )$ , ${\hat{g}}(\cdot )$ , and $\hat{\unicode[STIX]{x1D6FE}}(\cdot )$ , in which ${\hat{g}}(\cdot )$ represents the estimated marginal effect of $D$ on $Y$ with respect to $X$ .Footnote ²⁰ We implement this estimation procedure in both R and STATA and compute standard errors and confidence intervals using a bootstrap.

Figure 3. Kernel smoothed estimates: simulated samples. Note: The above plots show the estimated marginal effects using the kernel estimator: (a) when the true marginal effect is linear; (b) when the true marginal effect is nonlinear (quadratic). In both cases, the treatment variable $D$ is dichotomous. The dotted line denotes the “true” marginal effect, while the solid line denotes the marginal-effect estimates.

Figure 3 shows the results of our kernel estimator applied to the two simulated samples in which the true DGP contains either a linear or nonlinear marginal effect (the bandwidths are selected using a standard 5-fold cross-validation procedure). As in Figure 2, the x-axis is the moderator $X$ and the y-axis is the estimated effect of $D$ on $Y$ . The confidence intervals are generated using 1,000 iterations of a nonparametric bootstrap where we resample the data with replacement. We again add our recommended (stacked) histograms at the bottom to judge the common support based on the distribution of the moderator.

Figure 3 shows that the kernel estimator is able to accurately uncover both linear and nonlinear marginal effects. Figure 3(a) shows a strong linear interaction where the conditional marginal effect of $D$ on $Y$ grows constantly and monotonically with $X$ . The marginal-effect estimates from the kernel estimator are close to those from the true multiplicative interaction model (red dashed line).Footnote ²¹ In addition, the kernel estimates in Figure 3(b) are a close approximation of the true quadratic marginal effect (red dashed line). In short, by utilizing a more flexible estimator, we are able to closely approximate the marginal effect whether the LIE assumption holds or not.

Also note that toward the boundaries in Figure 3(a), where there is limited common support on $X$ , the conditional marginal-effect estimates are increasingly imprecisely estimated as expected given that even in this simulated data there is less data to estimate the marginal effects at these points. The fact that the confidence intervals grow wider at those points is desirable because it makes clear the increasing lack of common support.

5 Data

We now apply our diagnostic and estimation strategies to published papers that used classical linear interaction models and claimed an interaction effect. To broadly assess the practical validity of the assumptions of the multiplicative interaction model, we canvassed studies published in five top political science journals, The American Political Science Review (APSR), The American Journal of Political Science (AJPS), The Journal of Politics (JOP), International Organization (IO) and Comparative Political Studies (CPS). Sampling occurred in two stages. First, for all five journals, we used Google Scholar to identify every study which cited Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006), roughly 170 articles. Within these studies, we subset to cases which: used plain OLS or linear fixed-effect models; had a substantive claim tied to an interaction effect; and interacted at least one continuous variable. We excluded methods and review articles, as well as triple interactions because those models impose even more demanding assumptions.

Second, we conducted additional searches to identify all studies published in the APSR and AJPS which included the terms “regression” and “interaction” published since Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006), roughly 550 articles. We then subset to articles which did not cite Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006). In order to identify studies within this second sample which featured interaction models prominently, we selected articles which included a marginal-effect plot of the sort recommended by Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006) and then applied the same sampling filters as above. In the end, these two sampling strategies produced roughly 40 studies that met our sampling criteria.

After identifying these studies, we then sought out replication materials by emailing the authors and searching through the dataverses of the journals. (Again, we thank all authors who generously provided their replication data.) We excluded an additional 18 studies due to a lack of replication materials or an inability to replicate published findings, leaving a total of 22 studies from which we replicated 46 interaction effects. For studies that included multiple interaction effects, we focused on the most important ones which we identified as either: (1) those for which the authors generated a marginal-effect plot of the sort Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006) recommends, or, (2) if no such plots were included, those which were most relied upon for substantive claims. We excluded interaction effects where the marginal effect was statistically insignificant across the entire range of the moderator and/or where the authors did not claim to detect an interaction effect.Footnote ²²

While we cannot guarantee that we did not miss a relevant article, we are confident that our literature review has identified a large portion of recent high-profile political science studies employing this modeling strategy and claiming an interaction effect.Footnote ²³ The articles cover a broad range of topics and are drawn from all empirical subfields of political science. Roughly 37% of the interaction effects are from the APSR, 20% are from the AJPS, 22% are from CPS, 15% are from IO, and 7% are from JOP, respectively.

There are at least three reasons why the conclusions from our sample might provide a lower bound for the estimated share of published studies where the assumptions of the standard multiplicative interaction model do not hold. The first one is that we only focus on top journals. Second, for three journals we focus exclusively on the studies that cite Brambor, Clark, and Golder (Reference Brambor, Clark and Golder2006) and therefore presumably took special care to employ and interpret these models correctly. Third, we restrict our sample to the subset of potentially more reliable studies where the authors made replication data available and where we were able to successfully replicate the results.

It is important to emphasize that our replications and the conclusions that we draw from them are limited to reanalyzing the main models that underlie the interaction effect plots and tables presented in the original studies. Given the methodological focus of our article, we do not consider any additional evidence that the authors might have presented in their original studies to corroborate their substantive claims. Readers should keep this caveat in mind and consult the original studies and replication data to judge the credibility of the original claims in light of our replication results. Moreover, we emphasize that our results should not be interpreted as accusing any scholars of malpractice or incompetence. We remind readers that the authors of the studies that we replicate below were employing the accepted best practices at the time of publication, but following these existing guidelines for interaction models did not alert them to the problems that we describe below.