Introduction
A growing body of evidence indicates that alterations in immune-inflammatory pathways play important roles in the pathophysiology of depression (Maes, Reference Maes1995; Miller & Raison, Reference Miller and Raison2016). Compared with healthy controls, patients with depression show elevated blood levels of inflammatory cytokines, such as tumor necrosis factor-α (TNF-α), interleukin 1-β (IL-1β), and IL-6 (Howren, Lamkin, & Suls, Reference Howren, Lamkin and Suls2009; Liu, Ho, & Mak, Reference Liu, Ho and Mak2012). In addition, C-reactive protein (CRP), an acute-phase reactant, has also been found to be elevated in depression (Wium-Andersen, Orsted, Nielsen, & Nordestgaard, Reference Wium-Andersen, Orsted, Nielsen and Nordestgaard2013).
However, given the number of primary studies reporting conflicting results on the associations between inflammatory biomarkers and depression, meta-analyses are typically employed as a state-of-the-art empirical summary, integrating data across multiple primary studies and providing a more reliable answer to a research question than a single study (Berlin & Golub, Reference Berlin and Golub2014). Nonetheless, whilst several meta-analyses have attempted to clarify the associations between depression and inflammatory biomarkers, they have still reported inconsistent findings (D'Acunto, Nageye, Zhang, Masi, & Cortese, Reference D'Acunto, Nageye, Zhang, Masi and Cortese2019; Dowlati et al., Reference Dowlati, Herrmann, Swardfager, Liu, Sham, Reim and Lanctot2010; Goldsmith, Rapaport, & Miller, Reference Goldsmith, Rapaport and Miller2016; Howren et al., Reference Howren, Lamkin and Suls2009; Kohler et al., Reference Kohler, Freitas, Maes, de Andrade, Liu, Fernandes and Carvalho2017; Liu et al., Reference Liu, Ho and Mak2012; Ng et al., Reference Ng, Tam, Zhang, Ho, Husain, McIntyre and Ho2018; Osimo, Baxter, Lewis, Jones, & Khandaker, Reference Osimo, Baxter, Lewis, Jones and Khandaker2019; Perrin, Horowitz, Roelofs, Zunszain, & Pariante, Reference Perrin, Horowitz, Roelofs, Zunszain and Pariante2019). Although elevated peripheral levels of TNF-α were associated with depression in some meta-analyses (Dowlati et al., Reference Dowlati, Herrmann, Swardfager, Liu, Sham, Reim and Lanctot2010; Goldsmith et al., Reference Goldsmith, Rapaport and Miller2016; Kohler et al., Reference Kohler, Freitas, Maes, de Andrade, Liu, Fernandes and Carvalho2017; Liu et al., Reference Liu, Ho and Mak2012; Perrin et al., Reference Perrin, Horowitz, Roelofs, Zunszain and Pariante2019), others reported no association between them (D'Acunto et al., Reference D'Acunto, Nageye, Zhang, Masi and Cortese2019; Ng et al., Reference Ng, Tam, Zhang, Ho, Husain, McIntyre and Ho2018). Likewise, IL-1β was associated with depression in several meta-analyses (Howren et al., Reference Howren, Lamkin and Suls2009; Ng et al., Reference Ng, Tam, Zhang, Ho, Husain, McIntyre and Ho2018), but not in others (D'Acunto et al., Reference D'Acunto, Nageye, Zhang, Masi and Cortese2019; Dowlati et al., Reference Dowlati, Herrmann, Swardfager, Liu, Sham, Reim and Lanctot2010; Goldsmith et al., Reference Goldsmith, Rapaport and Miller2016; Kohler et al., Reference Kohler, Freitas, Maes, de Andrade, Liu, Fernandes and Carvalho2017; Liu et al., Reference Liu, Ho and Mak2012; Perrin et al., Reference Perrin, Horowitz, Roelofs, Zunszain and Pariante2019).
Errors in data extraction can be one potential explanation for why different meta-analyses reach different conclusions for the same research question. Investigators who have attempted to replicate published meta-analyses found that 59–100% contain errors (Ford, Guyatt, Talley, & Moayyedi, Reference Ford, Guyatt, Talley and Moayyedi2010; Gotzsche, Hrobjartsson, Maric, & Tendal, Reference Gotzsche, Hrobjartsson, Maric and Tendal2007; Jones, Remmington, Williamson, Ashby, & Smyth, Reference Jones, Remmington, Williamson, Ashby and Smyth2005). Such errors in data extraction can lead to overestimating or nullifying the significance of the results. For example, when performing a meta-analysis, it is sometimes necessary to standardize measurements on a uniform scale, such as standardized mean difference (SMD), before pooling across primary studies. During this process, sample sizes may be incorrectly exported (Park, Eisenhut, van der Vliet, & Shin, Reference Park, Eisenhut, van der Vliet and Shin2017), and standard errors (s.e.s) may be mistaken for standard deviations (s.d.s), which can substantially inflate point estimates and heterogeneity (Gotzsche et al., Reference Gotzsche, Hrobjartsson, Maric and Tendal2007). Inaccuracy in calculation (Messori, Scroccaro, & Martini, Reference Messori, Scroccaro and Martini1993) and data analysis with incomplete information (Buscemi, Hartling, Vandermeer, Tjosvold, & Klassen, Reference Buscemi, Hartling, Vandermeer, Tjosvold and Klassen2006) can also occur during data extraction.
To address this, we selected four peripheral inflammatory biomarkers, TNF-α, IL-1β, IL-6, and CRP, which have been extensively investigated in relation to depression. Then, we examined errors in meta-analyses of the association between depression and the four peripheral inflammatory biomarkers. We employed an umbrella review of meta-analyses to evaluate the presence, frequency, and nature of errors in data extraction and their impact on the results. Furthermore, we corrected the errors and then re-estimated the meta-analytical associations between depression and these peripheral inflammatory biomarkers. Finally, we collected all primary studies included in the meta-analyses and calculated the updated total pooled effect sizes (ESs) of the association between depression and the inflammatory biomarkers. With the updated ESs, we aimed to evaluate the association of the immune-inflammatory pathway with depression.
Methods
The protocol for this study was registered in PROSPERO (CRD42019133888). We adhered to the Preferred Reported Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines (Moher, Liberati, Tetzlaff, & Altman, Reference Moher, Liberati, Tetzlaff and Altman2009) and the Meta-analysis of Observational Studies in Epidemiology (MOOSE) guidelines (Stroup et al., Reference Stroup, Berlin, Morton, Olkin, Williamson, Rennie and Thacker2000) (online Supplementary Appendix 1).
Search strategy and selection criteria
Four investigators (SL, KMP, SJP, and WJK) searched PubMed/MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published between database inception and 14 January 2020 using the search terms (CRP OR IL-1beta OR IL-6 OR TNF-alpha) AND depress* AND meta using the [All Fields] search tag for all terms. The searching process was initially performed until 11 February 2019 and then repeated until 14 January 2020 to update the newly published meta-analyses. The full names and abbreviations of all four peripheral inflammatory biomarkers were employed in the search strategy. We chose eligible articles by consecutively screening their titles and abstracts, followed by their full texts (Fig. 1). Disagreements were resolved via discussion among the authors SL, KHL, EL, and JIS.
Fig. 1. Flow chart of literature search and screening.
We included meta-analyses of observational cross-sectional studies examining the association between unipolar or bipolar depression (Dargél, Godin, Kapczinski, Kupfer, & Leboyer, Reference Dargél, Godin, Kapczinski, Kupfer and Leboyer2015; Fernandes et al., Reference Fernandes, Steiner, Molendijk, Dodd, Nardin, Goncalves and Berk2016; Goldsmith et al., Reference Goldsmith, Rapaport and Miller2016; Rowland et al., Reference Rowland, Perry, Upthegrove, Barnes, Chatterjee, Gallacher and Marwaha2018) and levels of TNF-α, IL-1β, IL-6, or CRP in circulating blood (plasma/serum). ES metrics as outcome were limited to SMD [Cohen's d], Hedges' g, mean difference, or odds ratio, which are obtained from comparisons of the depressive and normal group. Some meta-analyses included primary studies for any depressive disorder and others included only studies on major depressive disorders. Our definition of depression followed that of each original meta-analysis. The international prospective register of systematic reviews (PROSPERO) registration status was evaluated.
We screened articles written in English or at least those with titles and abstracts in English and only included meta-analyses that reported ESs for individual primary studies or the data necessary for their calculation. Hereinafter, we have used the term ‘overlapping meta-analyses’ to indicate meta-analyses of the same association between depression and an inflammatory biomarker, and ‘overlapping studies’ to indicate primary studies that were included in more than one meta-analysis.
Data extraction
From each meta-analysis, four investigators (SL, KMP, SJP, and WJK) extracted the first author, publication date, literature search date, inflammatory biomarker of interest, model of analysis (e.g. fixed effect or random effects), sample sizes, maximally adjusted individual study estimates and corresponding 95% confidence intervals (CIs), and ES metrics presented for results [e.g. SMD (Cohen's d), Hedges' g, mean difference, or odds ratio]. From the primary studies included in the meta-analyses, we extracted sample sizes and means ± s.d.s, means ± s.e.s, or medians and interquartile ranges (IQRs) for each inflammatory biomarker. If data were presented in terms of a median and IQR, the calculation method for their conversion to mean ± s.d. was investigated. If a study compared two or more subgroups of depression to the same control, we combined the subgroups to create a single pair-wise comparison. A few primary studies that reported stimulated levels of inflammatory biomarkers from in vitro assays were found in one meta-analysis (Perrin et al., Reference Perrin, Horowitz, Roelofs, Zunszain and Pariante2019). In this case, extracted data were used for the evaluation of the detection of errors and recalculation of ESs in the meta-analysis, but not in the calculation of total pooled results from all primary studies. If data were presented only in graphs, they were extracted with GetData Graph Digitizer (version 2.26) (GetData Graph Digitizer, 2013). If there were any discrepancies in extracted data among individual raters, the most appropriate data were selected through consensus-building discussion.
Data analysis
We recalculated ESs and 95% CIs of primary studies included in the meta-analyses and reanalyzed each meta-analysis accordingly. Pooled ESs, 95% CIs, and p values were recalculated using Comprehensive Meta-Analysis (version 3.3.070, Biostat, Englewood, NJ, USA). The level of statistical significance was set at p < 0.05.
To evaluate discrepancies between initial and re-estimated results, we applied 0.1 as a cut-off point in accordance with previous studies that also evaluated data extraction errors and disagreements of results in meta-analyses (Gotzsche et al., Reference Gotzsche, Hrobjartsson, Maric and Tendal2007; Tendal et al., Reference Tendal, Higgins, Juni, Hrobjartsson, Trelle, Nuesch and Gotzsche2009). Meanwhile, we followed the ES metrics of the original meta-analyses (i.e. SMD, Hedges' g, mean difference, or odds ratio) in recalculation. If our recalculated results for an ES or its CI differed from those of a primary study reported in the meta-analysis by 0.1 or more, this was regarded as a change in results. Then, we thoroughly investigated the reason of the difference. If an error was identified during the investigation, this was classified as either ‘incorrect sample sizes’, ‘incorrectly used s.d.’, ‘incorrect participant inclusion’, ‘calculation error’, or ‘analysis with insufficient data’, as indicated below. We defined a case in which sample sizes in a primary study were wrongly extracted as incorrect sample sizes. If an extracted s.d. from a primary study was incorrect, the case was regarded as an incorrectly used s.d. When a primary study that did not meet the inclusion criteria of each meta-analysis was included, the case was defined as incorrect participant inclusion. Calculation error was defined as a case in which the reported ES was inaccurate despite no errors in reported primary study data for calculating ES. If not enough information with which to calculate SMD was provided in a primary study, the case was classified as an analysis with insufficient data.
We conducted data analysis as follows. At first, we evaluated the presence and type of errors in all primary studies in each meta-analysis. Next, we re-evaluated errors in only overlapping studies included in more than one meta-analysis. If data were extracted directly from previous meta-analyses, not from primary studies, and an error in the previous meta-analyses was noted, there was a chance of error duplication from the previous meta-analyses. Thus, we recalculated pooled ESs of the meta-analyses after correcting all errors. If there was a case in which an initial pooled ES was different by more than 0.1 from our recalculated value, this was presented as a ‘change in result’. Lastly, we gathered all primary studies included in the meta-analyses and calculated total pooled ESs and the 95% CIs of associations between depression and the four peripheral inflammatory biomarkers using a random-effects model and SMD as an ES metric. To assess heterogeneity among primary study ESs, the I 2 index was calculated. We assessed the presence of publication bias using funnel plots and Egger's tests. Data in primary studies presented with s.e.s were converted to s.d.s, and calculation methods for converting medians and IQRs to means and s.d.s were applied in recalculation, if necessary (Luo, Wan, Liu, & Tong, Reference Luo, Wan, Liu and Tong2018; Wan, Wang, Liu, & Tong, Reference Wan, Wang, Liu and Tong2014). If ESs were presented with metrics other than SMD, we recalculated SMD with the information provided in the primary studies.
Results
Database
A total of 517 potentially eligible articles were retrieved by the literature search (Fig. 1). During the screening process, 502 articles were excluded, with 15 articles included for analyses (Table 1) (Bizik, Reference Bizik2010; D'Acunto et al., Reference D'Acunto, Nageye, Zhang, Masi and Cortese2019; Dargél et al., Reference Dargél, Godin, Kapczinski, Kupfer and Leboyer2015; Dowlati et al., Reference Dowlati, Herrmann, Swardfager, Liu, Sham, Reim and Lanctot2010; Ellul, Boyer, Groc, Leboyer, & Fond, Reference Ellul, Boyer, Groc, Leboyer and Fond2016; Fernandes et al., Reference Fernandes, Steiner, Molendijk, Dodd, Nardin, Goncalves and Berk2016; Goldsmith et al., Reference Goldsmith, Rapaport and Miller2016; Howren et al., Reference Howren, Lamkin and Suls2009; Kohler et al., Reference Kohler, Freitas, Maes, de Andrade, Liu, Fernandes and Carvalho2017; Liu et al., Reference Liu, Ho and Mak2012; Munkholm, Vinberg, & Vedel Kessing, Reference Munkholm, Vinberg and Vedel Kessing2013; Ng et al., Reference Ng, Tam, Zhang, Ho, Husain, McIntyre and Ho2018; Osimo et al., Reference Osimo, Baxter, Lewis, Jones and Khandaker2019; Perrin et al., Reference Perrin, Horowitz, Roelofs, Zunszain and Pariante2019; Rowland et al., Reference Rowland, Perry, Upthegrove, Barnes, Chatterjee, Gallacher and Marwaha2018). The publication years of the meta-analyses ranged from 2010 to 2019. All meta-analyses exhibited significant heterogeneity among primary studies. Only two (13.3%) recently published meta-analyses were registered in PROSPERO (D'Acunto et al., Reference D'Acunto, Nageye, Zhang, Masi and Cortese2019; Perrin et al., Reference Perrin, Horowitz, Roelofs, Zunszain and Pariante2019).
Table 1. Literature search, analysis, and reporting of overlapping meta-analyses of the association between depression and peripheral inflammatory biomarkers
PROSPERO, international prospective register for systematic review protocols; SMD, standardized mean difference; MD, mean difference; OR, odds ratio; TNF-α, tumor necrosis factor-α; IL-1β, interleukin 1-β; IL-6, interleukin 6; CRP, C-reactive protein; N/A, not applicable.
Errors detected across meta-analyses
Errors detected in meta-analyses of the association between depression and the four peripheral inflammatory biomarkers are detailed in Table 2. The number of primary studies included in the meta-analyses ranged from 3 to 61. Except for the meta-analysis performed by Rowland et al. (Reference Rowland, Perry, Upthegrove, Barnes, Chatterjee, Gallacher and Marwaha2018), which investigated the association of bipolar depression with TNF-α, IL-6, and CRP (Rowland et al., Reference Rowland, Perry, Upthegrove, Barnes, Chatterjee, Gallacher and Marwaha2018), all meta-analyses (93.3%) had at least one type of an error. Overall, among the 521 primary studies included in the meta-analyses, errors were identified for 118 (22.6%) studies. The types of errors included incorrect sample sizes (16.9%), incorrectly used s.d. (29.7%), incorrect participant inclusion (5.9%), calculation error (28.0%), or analysis with insufficient data (19.5%). Among the different types of errors, incorrectly used s.d. was considered to be the most frequent.
Table 2. Results from overlapping meta-analyses of the association between depression and peripheral inflammatory biomarkers
SMD, standardized mean difference; DD, depressive disorder, BD, bipolar depression; MDD, major depressive disorder; TNF-α, tumor necrosis factor-α; IL-1β, interleukin 1-β; IL-6, interleukin 6; CRP, C-reactive protein; N/A, not applicable.
a Analysis with insufficient data indicates that sufficient information with which to calculate SMD was not provided in a primary study.
b Effect sizes in some single population studies were calculated using correlation coefficients. In such cases, the number of all participants in the study was not included in this column.
Errors detected across overlapping primary studies
A total of 305 overlapping primary studies were included in the meta-analyses (Table 3). Overall, 61 of the 305 primary studies (20.0%) were associated with incorrect data extraction. ‘Twelve from 79 overlapping studies of TNF-α (15.2%) were found with data extraction errors. In the overlapping studies of IL-1β, IL-6, and CRP, 20.5%, 17.5%, and 44.4% of each study showed errors, respectively.’ The types of errors consisted of incorrect sample sizes (24.6%), incorrectly used s.d. (44.2%), incorrect participant inclusion (8.2%), calculation error (19.7%), or analysis with insufficient data (3.3%). Again, incorrectly used s.d. was the most frequent error among overlapping studies.
Table 3. Summary of characteristics of overlapping primary studies of peripheral inflammatory biomarkers with errors
s.d., standard deviation; TNF-α, tumor necrosis factor-α; IL-1β, interleukin 1-β; IL-6, interleukin 6; CRP, C-reactive protein.
‘Analysis with insufficient data’ indicates that sufficient information with which to calculate SMD was not provided in a primary study.
Re-estimation of meta-analytical findings
Table 4 shows a comparison of originally calculated pooled ESs and their CIs with our recalculated values. Eleven (29.7%) of the 37 pooled ESs from overlapping meta-analyses changed by more than 0.1 after recalculation, ranging from 0.11 to 1.15, and those changes were categorized as a ‘change in results’. In 11 pooled ESs for TNF-α, two (18.2%) recalculated pooled ESs were classified as a change in results. In studies of IL-1β, the error rates were higher compared to studies of TNF-α. Five (62.5%) out of eight recalculated pooled ESs for IL-1β showed a change in results. During recalculation and comparison of pooled ESs of studies of IL-1β, two pooled ESs by Ellul et al. (Reference Ellul, Boyer, Groc, Leboyer and Fond2016) were not included because their meta-analysis did not specify a method to classify high- and low-quality studies. Thus, it was not possible to separate and recalculate the pooled ESs of the high- and low-quality studies in the same way. Therefore, we had to recalculate a pooled ES by integrating all studies in the meta-analysis. Although the recalculated pooled ES differed from a non-significant result of low-quality studies, we were unable to determine a change in results because those ESs were derived from non-comparable data. In 12 and six pooled ESs for IL-6 and CRP, three (25.0%) and one (16.7%) recalculated pooled ESs were changes in results, respectively.
Table 4. Comparison of results from overlapping meta-analyses with recalculated effect sizes and confidence intervals
BD, bipolar depression; MDD, major depressive disorder; SMD, standardized mean difference; MD, mean difference; ES, effect size; s.d., standard deviation; CI, confidence interval; PI, participant inclusion; ID, insufficient data; TNF-α, tumor necrosis factor-α; IL-1β, interleukin 1-β; IL-6, interleukin 6; CRP, C-reactive protein.
a If our recalculation of the pooled ES for each meta-analysis differed from that of the original meta-analysis by 0.1 or more, this is denoted as ‘changed’.
b When the MD was converted to SMD, the difference in pooled SMD between the original meta-analysis and our recalculation was 0.1 or more.
c The first row of reported ESs presents the results of high-quality studies, and the second row presents the result of low-quality studies. The recalculated ES results from all primary studies.
We also included all primary studies for each inflammatory biomarker and calculated total pooled SMDs of the associations with depression. The number of primary studies for the four peripheral inflammatory biomarkers ranged from 39 to 112. We found that elevated peripheral levels of three inflammatory biomarkers (TNF-α, IL-6, and CRP) were significantly associated with depression. Total pooled ESs with 95% CIs for TNF-α, IL-6, and CRP were 0.49 (95% CI 0.34–0.65), 0.46 (95% CI 0.38–0.54), and 0.27 (95% CI 0.21–0.33), respectively. IL-1β was not associated with depression. Significant heterogeneity was found for all four biomarkers, with I 2 values ranging from 85.1% to 88.2% (online Supplementary Figs S1–S4). Funnel plots and Egger's tests showed publication bias among studies of TNF-α, IL-6, and CRP (all p < 0.001), but not among studies of IL-1β (p = 0.257) (online Supplementary Figs S5–S8).
Discussion
We found a considerable number of errors in 14 (93.3%) of the 15 overlapping meta-analyses investigating the association between depression and four peripheral inflammatory biomarkers. Of the 521 primary studies included in the overlapping meta-analyses, errors were identified for 118 (22.6%) studies. The most common errors were incorrect use of s.d.s (29.7%), followed by calculation errors (28.0%), analysis with insufficient data (19.5%), incorrect sample sizes (16.9%), and incorrect participant inclusion (5.9%). From the 305 overlapping studies that were included in more than one meta-analysis, errors were found in 61 (20.0%) of them. Again the most common errors were incorrectly used s.d.s (44.2%), followed by incorrect sample sizes (24.6%), calculation errors (19.7%), incorrect participant inclusion (8.2%), and analysis with insufficient data (3.1%). After correcting these errors and repeating the analyses, 11 (29.7%) of 37 pooled ESs from the meta-analyses changed the magnitude of the ES by more than 0.1, ranging from 0.11 to 1.15. The updated meta-analyses showed that elevated levels of peripheral TNF-α, IL-6, and CRP, but not IL-1β, were associated with depression.
Incorrectly identifying sample sizes is a potential meta-analytical problem. Although this type of error was more prominent among overlapping studies of CRP, it was also noted in studies of the other three inflammatory biomarkers. As the data extraction process is usually performed manually, it may increase the risk of errors. In future, machine learning may be applied to search for and screen studies to include in a meta-analysis and further improve the meta-analytic research (Xiong et al., Reference Xiong, Liu, Tse, Gong, Gladding, Smaill and Zhao2018).
An incorrectly used s.d. was the most common data extraction error in our umbrella review, consistent with previous reports of s.e.s mistaken for s.d.s (Gotzsche et al., Reference Gotzsche, Hrobjartsson, Maric and Tendal2007; Tendal et al., Reference Tendal, Higgins, Juni, Hrobjartsson, Trelle, Nuesch and Gotzsche2009). This type of error can inflate point estimates and artificially reduce CIs substantially (Gotzsche et al., Reference Gotzsche, Hrobjartsson, Maric and Tendal2007), impacting pooled ESs and estimated heterogeneity. Therefore, it can change the clinical meaningfulness of meta-analytical results. Some primary studies did not even indicate precisely whether their results were presented with s.e.s or s.d.s. In the review of Jones et al. (Reference Jones, Remmington, Williamson, Ashby and Smyth2005), 34 systematic reviews conducted by the Cochrane Cystic Fibrosis and Genetic Disorders Group were evaluated for data-handling and reporting errors (Jones et al., Reference Jones, Remmington, Williamson, Ashby and Smyth2005). As a result, errors were found in 20 reviews, of which four (20.0%) out of the 20 reviews were related with incorrectly used s.d.s. Thirty-five primary studies in 11 meta-analyses included in our study were found to have incorrectly used s.d.s. Accordingly, this type of error may be more frequent in meta-analytical studies in psychiatry research than in other medical disciplines.
Inaccuracies in participant inclusion and calculation were also noticed. Some studies that were not related to depression and inflammatory biomarkers or that did not meet the inclusion criteria of a meta-analysis were found to be erroneously included in the meta-analytic results (Koenig et al., Reference Koenig, Cohen, George, Hays, Larson and Blazer1997; Komulainen et al., Reference Komulainen, Lakka, Kivipelto, Hassinen, Penttila, Helkala and Rauramaa2007). Ford et al. (Reference Ford, Guyatt, Talley and Moayyedi2010) reported in their review that five (62.5%) out of eight meta-analyses of pharmacological interventions for irritable bowel syndrome had included studies that were ineligible according to the predefined eligibility criteria (Ford et al., Reference Ford, Guyatt, Talley and Moayyedi2010). In our study of 15 meta-analyses, only seven primary studies included in three meta-analyses were related to this type of error, and the error was relatively less frequent than that of the study conducted by Ford et al.
In some cases, even though no error was found in the reported primary study data, inaccurate ESs were identified. This can lead us to presume that there were calculation errors and a possible discrepancy in data used for actual calculation and reported data (Soneji, Reference Soneji2018). In the review of Gotzsche et al. (Reference Gotzsche, Hrobjartsson, Maric and Tendal2007), 27 meta-analyses published in 2004 that had used SMD were included for the evaluation of errors. The authors randomly selected two trials from each meta-analysis and found that 10 (37%) of the 27 meta-analyses had at least one error. In the 10 meta-analyses with errors, one (8.3%) out of 12 trials was related to calculation errors. Although it would be difficult to compare the calculation error rate of our results to that of the review directly, we can presume that considerable calculation errors may also influence results in psychiatry.
In some cases, primary studies did not report sufficient information about their analyses. Therefore, with the absence of essential data for meta-analysis, it was not possible to extract data from some primary studies (Steptoe, Kunz-Ebrecht, & Owen, Reference Steptoe, Kunz-Ebrecht and Owen2003; Suarez, Reference Suarez2004). In line with this, non-statistically significant effects (NSUEs), which are frequently unreported, should be addressed. Some studies with non-significant group differences sometimes did not present any statistics that are necessary to be converted into ES. Recent statistical approaches (e.g. MetaNSUE) have been developed to overcome this problem (Albajes-Eizagirre, Solanes, & Radua, Reference Albajes-Eizagirre, Solanes and Radua2019; Radua et al., Reference Radua, Schmidt, Borgwardt, Heinz, Schlagenhauf, McGuire and Fusar-Poli2015).
Like many previous meta-analyses (Dowlati et al., Reference Dowlati, Herrmann, Swardfager, Liu, Sham, Reim and Lanctot2010; Fernandes et al., Reference Fernandes, Steiner, Molendijk, Dodd, Nardin, Goncalves and Berk2016; Goldsmith et al., Reference Goldsmith, Rapaport and Miller2016; Kohler et al., Reference Kohler, Freitas, Maes, de Andrade, Liu, Fernandes and Carvalho2017; Liu et al., Reference Liu, Ho and Mak2012; Osimo et al., Reference Osimo, Baxter, Lewis, Jones and Khandaker2019), total pooled ESs of the four peripheral inflammatory biomarkers in our study showed that elevated peripheral levels of TNF-α, IL-6, and CRP, but not IL-1β, are associated with depression. We also found significant heterogeneity among primary studies of all four biomarkers, presumably reflecting diversity in the characteristics of individual studies and the important roles of biological, clinical, and technical confounders.
Some limitations of our umbrella review and meta-analysis should be acknowledged. The severity and duration of depression, medication status, and other confounding factors, such as body mass index, can affect the association between depression and inflammatory biomarkers (Beurel, Toups, & Nemeroff, Reference Beurel, Toups and Nemeroff2020; Köhler-Forsberg et al., Reference Köhler-Forsberg, Buttenschøn, Tansey, Maier, Hauser, Dernovsek and Mors2017). However, such factors were not identically adjusted in primary studies included in the overlapping meta-analyses. The significant heterogeneity among primary studies also reveals that the total pooled ESs of the four biomarkers should be interpreted and applied carefully to individual levels. We included unipolar and bipolar depression and summarized those data together. However, as differences between unipolar and bipolar depression have been reported (Brunoni et al., Reference Brunoni, Supasitthumrong, Teixeira, Vieira, Gattaz, Benseñor and Maes2020; Goya-Maldonado et al., Reference Goya-Maldonado, Brodmann, Keil, Trost, Dechent and Gruber2016), this should be taken into consideration before generalizing our results. In addition, data were extracted from graphs in 51 primary studies. Data extracted from graphs may be less accurate than data extracted from numbers, and incorrectly used s.d. and calculation errors can be related to inaccuracies in data extraction from graphs. However, only three cases of incorrectly used s.d. and nine cases of calculation error were noticed in all extracted data from graphs, and our results were not primarily affected by it. Although any discrepancies during study selection and data extraction were resolved via discussion among the raters, the methods of reporting interrater reliability were not used in our study (Belur, Tompson, Thornton, & Simon, Reference Belur, Tompson, Thornton and Simon2018). Lastly, although we spent much time and effort checking for the presence of errors in previous meta-analyses, the possibility of errors in our umbrella review itself cannot be excluded.
Although the statistical calculations in meta-analyses are ostensibly simple, data extraction and analysis are particularly prone to errors. Our study findings show that data extraction in meta-analyses can lead to significant errors. We noted that the errors could change the statistical significance of the association between depression and inflammatory biomarkers. Because errors in data extraction may influence ES and inflate heterogeneity among studies, efforts to reduce data extraction errors are important in studies of the association between depression and peripheral inflammatory biomarkers, for which high heterogeneity and conflicting results have continuously been reported.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291721003767
Acknowledgments
This paper presents independent research. The views expressed in this publication are those of the authors and not necessarily those of the acknowledged institutions.
Author contributors
SL, KHL, EL, and JIS designed the study. SL, KMP, SJP, and WJK performed the literature search and screening; extracted, analyzed, and interpreted the data; and made the figures and tables. Any discrepancies were resolved via discussion among SL, KHL, EL, and JIS. SL, EL, and JIS drafted the manuscript. JL, AK, LS, MS, BS, AK (Koyanagi), LJ, AS, TT, ED, HO, ARB, AFC, JR, SKA, KN, and PFP were involved in critically revising the manuscript for important intellectual content. All authors approved the final version of the manuscript for publication. EL and JIS contributed as joint corresponding authors.
Financial support
This review was supported by the Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Science, ICT & Future Planning, Republic of Korea (2017R1A2B3008214). The funders had no role in the study design, data collection, and analysis, decision to publish, or preparation of the manuscript. All authors had full access to all of the study data, and the corresponding authors had the final responsibility for the decision to submit for publication.
Conflict of interests
None.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
