Creating Effect Size Based on Partial Correlation Coefficient

Elasticities are the popular measures of effect size in meta-analysis. However, they are largely unsuitable when primary studies do not provide sufficient information to calculate elasticities or when they use a variety of noncomparable scales and different functional forms. Elasticities become difficult to interpret in this situation. When elasticities are unsuitable in meta-analysis, partial correlation coefficient r _i becomes the appropriate measure of effect size (Oczkowski and Doucouliagos Reference Oczkowski and Doucouliagos2014). Partial correlation coefficient r _i has a natural interpretation across studies because it is independent of the metrics used to measure the explanatory and dependent variables in the primary studies (Stanley and Doucouliagos Reference Stanley and Doucouliagos2015). Hence, the partial correlation coefficient offers a convenient way of deriving comparable estimates from a large number of studies (Doucouliagos Reference Doucouliagos2011). This standardized measure of effect size can be computed as shown in equation 1 using the measure of statistical significance (t-test statistics) of the estimate and its degree of freedom (df) obtained from the primary study:

(1)$$r_i = t_i/\sqrt {t_i^2 + {\rm d}{\rm f}_i} $$

The standard error of the partial correlation coefficient, se_ri, is given by:

(2)$${\rm s}{\rm e}_{ri} = \sqrt {(1 - r_i^2 )/{\rm d}{\rm f}_i} $$

According to Cohen (Reference Cohen1988), an absolute value of r _i below 0.10, between 0.25 and 0.4, and above 0.4 represent small, medium, and large effect sizes, respectively. However, Doucouliagos (Reference Doucouliagos2011) suggests that r _i with absolute values of 0.07, 0.17, and 0.33 should be considered small, medium, and large effect sizes, respectively.

Although the focus of the study is to synthesize the literature on the impact of adopting agricultural innovations on production and welfare, the use of r _i in the present study cannot be a substitute measure of real economic impact across the selected case studies. In this context, r _i is used to measure the relationship (i.e., direction) between the adoption of agricultural innovations and indicators of farm production/household welfare measures and the extent of such relationships across the selected case studies. The sign of r _i represents the direction, while the size of r _i denotes the magnitude/extent of the relationship.

Funnel Asymmetry Test and Precision Effect Test

The funnel asymmetry test (FAT) and the precision effect test (PET) of MRA are employed to examine the existence of publication bias among the literature under review, and to deduce the relationship and magnitude of such relationship after correcting for publication bias, respectively. According to Card and Krueger, (Reference Card and Krueger1995), publication bias may arise from: (1) a predisposition of the reviewers and editors to accept papers that are consistent with the conventional view, (2) researchers’ selection of models that are in line with the conventional view, and (3) a general predisposition to treat statistically significant results favorably. Publication bias can, however, make uncovering the true effect size of interest difficult because the effect sizes reported in the literature under consideration will be skewed (Stanley Reference Stanley2005).

Both tests, proposed by Egger et al. (Reference Egger, Smith, Sceider and Minder1997), are jointly known as the FAT-PET MRA within the framework of univariate meta-regression and are defined by:

(3)$$r_i = {\rm \beta} _0 + {\rm \alpha} _0{\rm s}{\rm e}_i + {\rm \mu} _i$$

where r _i represents partial correlation coefficient (PCC); se_i is the standard error of the partial correlation coefficient (PCC); β₀ and α₀ are parameters of interest; and μ_i is the error term of the regression. Publication bias exists if α₀ is statistically significant, irrespective of its sign. A genuine empirical effect that is free of publication bias is determined by testing whether or not β₀ = 0.

Multivariate MRA to Explain Sources of Heterogeneity in the Effect Size

To identify the sources of differences/variations in the computed partial correlation coefficient r _i, we employ an MRA defined below:

(4)$$r_i = {\rm \beta }_0 + {\rm \alpha }_0{\rm s}{\rm e}_i + \mathop \sum \limits_{k = 1}^k {\rm \lambda }_kX_{ki} + \varepsilon _i$$

where r _i, α₀, and se_i are as earlier defined;. X _k is a vector of study attributes hypothesized to explain r _i; α₀, β₀ and λ_k are the parameters to be estimated;. ε_i is a normally distributed error term.

Guided by economic theory and meta-analysis literature, the study attributes X _k detailed description presented in Table 1 and summary statistics in Table B of the Appendix) in equation 4 include publication outlets (i.e., journals, conference papers,working papers), types of data (i.e., cross-sectional data vs. panel data) survey design (i.e., experimental method vs. nonexperimental method), types of products (i.e., crop vs. non-crop), and econometric method (i.e., studies using matching, endogenous switching regression, DID, instrumental regressions vs. ordinary least square regression with no control for selection bias). Other variables include types of innovations adopted by farmers in the primary studies selected (i.e., studies focusing on mechanization, pest management, conservation agriculture, integrated farming system, and high yield variety), types of matching (i.e., kernel, nearest neighbor, radius caliper), checking for model consistency (whether or not the authors checked for model consistency in the primary studies), and regions where the primary studies were conducted (i.e., West Africa, East Africa, Southern Africa, Central-Africa).

Table 1. Moderator variables for the meta-regression analysis from the selected studies

MRA Model Estimation

Estimation of the parameters of equations 3 and 4 follows the approach of Stanley (Reference Stanley2008) and Stanley and Doucouliagos (Reference Stanley and Doucouliagos2015), which are based on the weighted least square (WLS) regression model, with the weight equal to the inverse of the variance of the dependent variable. They argue that weighted regression simultaneously reduces publication bias and discount small sample studies. In addition, the variance of the weighted average of the estimates is also minimized. As noted by Ogundari and Abdulai (Reference Ogundari and Abdulai2013), the WLS model, unlike unweighted OLS regression, corrects for outliers and measurement errors by giving them less weight. It thus ensures that they do not confound results with inflation of the variance of the regression.

We employ WLS to estimate the parameters of equations 3 and 4 using robust and cluster-adjusted standard errors to address the research questions raised earlier in the study (see Costa-Font and Hernandez-Quevedo, Reference Costa-Font and Hernandez-Quevedo2015, Bocker and Finger, Reference Bocker and Finger2017). Both the robust and cluster-adjusted standard errors are essential to improve efficiency of the estimated parameters. However, the use of cluster-adjusted standard error (with study ID as the cluster indicator) is to further minimize or control the influence that multiple estimates from the same primary study may have on the efficiency of the estimated parameters of the meta-regression.

In addition to the WLS model, the study also employs mixed effect ML and fixed effect models with weight being the inverse of the variance of the dependent variable. This methodology is increasingly popular in MRAs in recent time (see: Galindo et al. Reference Galindo, Samaniego, Alatorre, Carbonell and Reyes2015, Laroche, Reference Laroche2016, Tokunaga and Iwasaki Reference Tokunaga and Iwasaki2017).Footnote ⁵

The Size and Distribution of the Partial Correlation Coefficient

Table 2 presents the partial correlation coefficient of the relationship between adoption of agricultural innovations and farm production/household welfare defined by r _i. We considered average estimates of r _i from simple arithmetic mean that did not account for heterogeneity in r _i. On the other hand, we employed fixed effect average and random effect average that account for heterogeneity in r _i. Figure 1 presents the distribution of r _i for the whole sample and across the potential outcomes.

Figure 1. Distribution of partial correlation coefficient (PCC) of the impact of agricultural technology adoption for the whole sample and on production and welfare measures

Table 2. Average values of partial correlation coefficient (PCC) by potential outcomes

PCC is denoted by r _i and measures the relationship between adoption of agricultural innovations and the outcome measures.

Table 2 shows a simple average, fixed effect average, and that of random effect average of r _i as 0.182, 0.099, and 0.180, respectively, for studies that focus on the relationship between agricultural innovation and farm production. This suggests a medium size/magnitude according to Doucouliagos's (Reference Doucouliagos2011) new guideline for interpretation of r _i, earlier stated in this paper. Results from the table show absolute values of 0.151, 0.114, and 0.145 for simple average, fixed effect average, and random effect average estimates, respectively, for studies that focus on the relationship between agricultural innovation and household welfare. These results also suggest a medium size effect. Also using Cohen's (Reference Cohen1988) rule on r _i, all the estimates in Table 2 are small in magnitude. The implication of these results is that irrespective of the guidelines used in interpreting the estimates, the relationship between agricultural innovations and farm production/welfare presented in Table 2 still appears weak. In Section :RQ1: Does Adoption of Agricultural Innovations Have Any Impact on Farm Production and Household Welfare in SSA?” below, we further discuss whether or not r _i obtained after correcting for publication bias is significantly different from zero.

Publication Bias

We use the funnel plot and FAT-PET specified in equation 3 to examine the possibility of publication bias in the study. The funnel plot visually assesses the degree of publication bias, while the FAT-PET quantitatively assesses publication bias using a meta-regression approach. As mentioned earlier, publication bias can make uncovering the true effect size difficult because it skews the effect sizes reported in the primary studies.

Therefore, in line with previous literature on meta-analysis, we first examined the funnel plot (Figure 2) for any evidence of publication bias. The left-hand figure in Figure 2 depicts the funnel plot for production measures, while the right-hand figure represents the funnel plot for household welfare measures. The vertical line from both figures denotes the average value of r _i. The two dashed lines represent the boundaries of conventional statistical significance at the 5 percent level, such that estimates outside the boundaries are statistically significantly different from the underlying effect. When outlying estimates form more than 5 percent of the data, it suggests the existence of publication bias.

Figure 2. Funnel plots of relationship between agricultural innovations adoption on production (LEFT) and welfare (RIGHT) measures from the sampled study. Note: Partial correlation coefficient (PCC)

Because the estimates appear to be heavier on both sides of the boundaries in Figure 2, we infer that publication bias does exist among the sampled studies. This shows that most sampled studies preferred reporting statistically significant estimates of the impact of adopting agricultural innovation on farm production and household welfare measures considered in the primary studies.

To further confirm the presence of selection bias, we considered the FAT-PET model of equation 3 presented in Tables 3 and 4. Because the FAT represented by α₀ in equation 3 shows a statistically significant positive estimate in both tables, results suggest that publication bias exists in the selected studies employed in our analysis. These results reveal that reviewers and editors are biased towards studies that report a significant impact of adopting agricultural innovations on farm production and household welfare measures. The implication of the result is that the distribution of the reported effect size is likely to be skewed. Thus, the estimated effect size is likely larger than actuality.

Table 3. Test for publication bias and precision effect (FAT-PET) model using WLS model.

Note: Dependent variable is partial correlation coefficient r _i. *,** and *** stand for significant estimate at the level of 10%, 5%, and 1% respectively.

Table 4. Test for publication bias and precision effect (FAT-PET) model using mixed effects and fixed effects model

Note: Dependent variable is partial correlation coefficient r _i. *, ** and *** stand for significant estimate at the level of 10%, 5%, and 1% respectively.

RQ1: Does Adoption of Agricultural Innovations Have any Impact on Farm Production and Household Welfare in SSA?

The estimates of PET represented by β₀ in equation 3 and reported in Tables 3 and 4 show that adoption of agricultural innovation does have a positive and significant relationship with indicators of farm production and household welfare measures employed in the study. More importantly, the size of the coefficients of PET ranges from 0.053–0.259 across the estimates reported in the tables. According to Table 3, these results suggest that the estimated magnitude of PET for both the production and welfare measures are small. With the exception of welfare measure for the fixed effects model in Table 4, the estimated magnitudes of PET are of a medium size, which also suggests a weak relationship. These findings strongly align with the earlier results reported in the Section “The size and Distribution of the Partial Correlation Coefficient.” On this basis, we conclude that the weak relationship observed between adoption of agricultural innovation and the outcome variables considered in this study could probably explain why adoption of agricultural innovations has limited impact in SSA.

Lack of efforts to develop and disseminate an integrated package of new and useful technologies, which include improved management practices and natural resource management practices, have been identified as a reason for the failure of the agricultural innovations’ adoption to exhibit a significantly higher impact in SSA (Otsuka and Muraoka Reference Otsuka and Muraoka2015). Also, there is increasing concern among Development Economists as to whether or not institutional constraints such as credit markets, poor transportation networks, weak extension services, absence of private input retailers, are responsible for the poor performance of green revolution in SSA (The New York Times 2015). Nonetheless, it is possible that outright lack of development efforts or combinations of the aforementioned observations could help explain the weak relationship between adoption of agricultural innovation and farm production/household welfare measures.

RQ2: Which of the Study-Specific Attributes Explain the Variations in the Reported Estimates of the Impact of Adopting Agricultural Innovation on Farm Production and Household Welfare Measures Reported in the Sampled Studies?

The answer to this research question requires identification of the study attributes that significantly explain r _i using MRA. Table 5 shows the multivariate MRA estimates based on WLS model, while Table 6 provides the results of the mixed effects ML and fixed effects regression models. The estimated parameters from both models are almost identical when farm production measures are considered as the outcome variable of interest. However, differences exist in some of the variable estimates when household welfare measures are considered as the outcome of interest. We also observed that a number of study attributes employed in our MRA have consistent results in terms of sign and level of significance across the estimated models. These variables include the coefficients of standard error, cross-section data, experimental method, matching technique, ESR, instrumental variable regression approach, and the set of regional dummies.

Table 5. Meta-regression analysis using WLS Model

Note: Dependent variable is partial correlation coefficient (r) measure of the impact of agricultural technology adoption from selected case studies; All results are weighted by inverse variance of standard error of PCC and cluster with study ID. *, **, and *** stand for significant estimate at the level of 10%, 5% and 1% respectively.

Table 6. Meta-regression analysis using mixed effect ML model and fixed effect model

Note: Dependent variable is partial correlation coefficient (r) measure of the impact of agricultural technology adoption from selected case studies; A=all results are weighted by inverse variance of standard error of PCC and cluster with study ID. *, ** and *** stand for significant estimate at the level of 10%, 5%, and 1% respectively.