Methodology underlying systematic reviews and meta-analyses

As Reference Smith, Cipriani and GeddesSmith et al (2016) highlight, we have seen a significant rise in the number of published papers in the field of medicine, particularly reviews and meta-analyses (Reference BohlinBohlin 2012). About 10% of all published articles are reviews and 1% are meta-analyses. Systematic reviews and, in particular, meta-analyses are often highly cited. For example, meta-analyses account for as many as 20% of the most important papers published in any given year and notated by Thomson Reuters Essential Science Indicators (ESI) as ‘hot papers’ (Box 1). It is vital therefore that systematic reviews and meta-analyses are accurate and unbiased. Smith et al clearly describe the process of how to conduct a systematic review, focusing on key stages: formulation of a valid question; systematic identification of all the relevant studies; and critical appraisal of each study. They also provide a very nice summary of strengths and weaknesses of meta-analysis, focusing on three major factors: quality of the data-set, comparability of the underlying studies and bias. In particular, the power of any given meta-analysis is very much limited by the quality of the underlying primary studies. This is effectively the rate-limiting step, but one that is easily overlooked by readers of such studies. However, one factor that is difficult to gauge remains the enemy of the systematic review: namely, confirmation bias (Reference LittleLittle 2008). Although Smith et al address bias, they should perhaps have emphasised a little more strongly the problem of confirmation bias and steps to avoid it. Confirmation bias is the effect whereby authors attempt to fit and interpret findings to their a priori beliefs. We are all subject to this influence, which is why some responsibility for neutrality lies with co-authors, editors and peer reviewers. An excellent example of confirmation bias in mental health is the argument for and against depression screening (Reference Goodyear-Smith, van Driel and ArrollGoodyear-Smith 2012).

Influential systematic reviews and meta-analyses in mental health

To complement Smith et al's excellent summary of the methodology of systematic reviews and meta-analyses, I thought it might be useful to the reader to list some highly cited examples of recent decades. To that end, I took a ‘snapshot’ of citations in August 2015. Two older systematic reviews, one on the Beck Depression Inventory (Reference Beck, Steer and GarbinBeck 1988) and the other on Alzheimer's disease (Reference SelkoeSelkoe 2001), are among the top 20 most cited papers of all time in the field of mental health. More recent systematic reviews gaining a great deal of influence are on suicide prevention strategies (Reference Mann, Apter and BertoloteMann 2005) and the cannabis and psychosis debate (Reference Moore, Zammit and Lingford-HughesMoore 2007). However, it is typically meta-analyses that have most impact in psychiatry. Table 1 shows the top 10 most influential systematic reviews and meta-analyses in mental health when judged by total citations using the Web of Science database citation count.

TABLE 1 Top ten most influential meta-analyses in mental health (based on Web of Science database citation count to January 2015)

Two highly cited papers in the table looked at selective reporting in antidepressant trials: Reference Turner, Matthews and LinardatosTurner et al (2008) and Reference Kirsch, Deacon and Huedo-MedinaKirsch et al (2008). Turner et al examined 74 studies registered with the US Food and Drug Administration (FDA) and found that, of the 31% that were not published, only one had a positive result. Kirsch et al examined both published and unpublished data from 35 trials and found that drug–placebo differences in antidepressant efficacy increased as a function of baseline illness severity, but were relatively small even for patients with severe depression, thereby fuelling the debate about the merits of antidepressants in mild depression.

Another highly cited paper Table 1 is on the genetics of severe mental illness (Reference Lewis, Levinson and WiseLewis 2003). Ten years later a paper by Ripke and colleagues on the same topic was to be designated an ESI hot paper, i.e. one destined to be important because of its high initial citation rate (Box 1). This was a meta-analysis of genome-wide association studies of schizophrenia (8832 cases and 12 067 controls) that, from replication of single nucleotide polymorphisms (SNPs), identified 22 loci associated at genome-wide significance (Reference Ripke, O'Dushlaine and ChambertRipke 2013).

Meta-analyses on the efficiency (and side-effects) of antipsychotics are also highly cited. Reference Leucht, Corves and ArbterLeucht et al (2009) initially looked at 150 double-blind short-term studies, with 21 533 participants, and found that second-generation antipsychotics differ in many properties and are not a homogeneous class (Table 1). A follow-up meta-analysis of 212 trials that ranked efficacy v. side-effects of 15 antipsychotics became an ESI hot paper on publication (Reference Leucht, Cipriani and SpineliLeucht 2013). Another ESI hot paper of 2013 on a related topic was one on which I was a co-author (Reference Mitchell, Vancampfort and SweersMitchell 2013). This reported that across 126 studies (25 692 participants)the overall rate of metabolic syndrome in patients with schizophrenia was 32.5%, but in drug-naive patients who were in their first episode the rate was not appreciably higher than that in the general population.

An interesting property of well-conducted research is the ability to refute false positives, no matter how much they are discussed. One good example, included in Table 1, relates to the serotonin transporter gene, widely purported in the 1990s to be a risk for depression. In 2009, Merikangas's group (Reference Risch, Herrell and LehnerRisch 2009) found that, across 14 studies (including 10 with individual patient data), stressful life events were linked with depression (odds ratio OR = 1.41), but the serotonin transporter gene was not.