Priscilla Coleman’s recent meta-analysis ^{Reference Coleman1} ignores guidelines for proper scientific conduct of meta-analyses of observational data. Her results violate at least three major principles of meta-analysis: she fails to assess the underlying validity of included studies; she fails to examine statistical heterogeneity; and she illogically combines estimates for distinct outcomes. Furthermore, she accuses previous reviews of lacking ‘reasonable justification’ for declining to quantitatively summarise effects, when declining to do so actually reflected sound epidemiological judgement.

Coleman contends that ‘Through a process of systematically combining the quantitative results from numerous studies addressing the same basic question… far more reliable results are produced than from particular studies that are limited in size and scope’. However, expert consensus suggests that ‘the likelihood that the treatment effect reported in a systematic review approximates the truth depends on the validity of the included studies…’. ^{Reference Liberati, Altman, Tetzlaff, Mulrow, Gotzsche and Ioannidis2} Coleman fails to assess the validity of included studies and erroneously asserts that ‘as a methodology wherein studies are weighted based on objective scientific criteria, meta-analysis offers a logical, more objective alternative to qualitative reviews…’. In fact, studies in meta-analyses are typically weighted by sample size, which is not always related to study quality, ^{Reference Egger, Smith, Schneider, Egger, Smith and Altman3} and decisions on which studies to include and how to include them remain subjective. If poor-quality studies are included, as occurred in Coleman’s review, a poor-quality quantitative estimate will be generated. Coleman combines statistically heterogeneous results, and illogically combines effect estimates for outcomes that vary substantially (i.e. marijuana use and suicide), thus generating a summary estimate void of meaning or utility.

Meta-analysis of observational data can be useful when carefully conducted. However, it is essential that a summary estimate be accompanied by a qualitative description of risk of bias in included studies (which Coleman’s review lacked) since ‘potential biases in the original studies, relative to biases in RCTs, make the calculation of a single summary estimate of effect of exposure potentially misleading’. ^{Reference Stroup, Berlin, Morton, Olkin, Williamson and Rennie4}

Coleman ignores other essential requirements of a high-quality statistical meta-analysis. ^{Reference Liberati, Altman, Tetzlaff, Mulrow, Gotzsche and Ioannidis2} She makes no attempt to present a replicable search strategy or article selection diagram. She attempts to justify excluding articles prior to 1995 by noting that study methodology has improved, but fails to adequately justify selected cut-off dates. Ultimately, she includes multiple methodologically weak studies, and excludes at least two older but methodologically stronger studies. She authored her review alone, despite Cochrane and PRISMA recommendations to involve multiple reviewers to reduce the possibility of investigator bias or error. ^{Reference Liberati, Altman, Tetzlaff, Mulrow, Gotzsche and Ioannidis2,Reference Higgins and Green5}

Coleman makes disingenuous accusations about previous reviews. For example, she claims that our 2008 systematic review ^{Reference Charles, Polis, Sridhara and Blum6} ‘overlooked’ ten articles which met inclusion criteria, and ‘lacked sufficient methodologically based selection criteria’. This unfounded attack is puzzling, particularly since in 2008, we directly emailed to Coleman the reasons (consistent with our methodologically based selection criteria detailed on p. 437) for excluding seven of these ten articles. The remaining three (not previously enquired about) also fail to meet inclusion criteria: two had a follow-up period of less than 90 days and the other compared medical v. surgical termination.

Coleman continues to ignore the scientific importance of accounting for pregnancy intention in this body of literature. If women who abort (many of which are unintended pregnancies) are compared against women who deliver (many of which are intended pregnancies), effects of unintended pregnancy are difficult to disentangle from effects of abortion. Circumstances surrounding an intentional v. an unintentional conception or pregnancy may be related to mental health outcomes. Most aborted pregnancies in the USA were unintended. ^{Reference Finer and Henshaw7} Coleman wrongly assumes that since nearly half of pregnancies in the USA are unintended, most births are too, failing to acknowledge that almost half of unintended pregnancies end in abortion. ^{Reference Finer and Zolna8} Thus, her assertion that ‘the majority of women in the control groups in studies comparing abortion with term pregnancy actually delivered unintended pregnancies even if the variable was not directly assessed’ has no empirical grounding. Similarly, her assertion that a ‘no pregnancy’ group may be a ‘cleaner’ comparison group ignores the fact that the ‘no pregnancy’ group would not have experienced unintended pregnancy.

The scientific validity and rigour of Priscilla Coleman’s work has been questioned before. ^{Reference Steinberg and Finer9} However, we are surprised and disappointed that the multiple egregious scientific errors in her review went undetected by the editorial or peer-review process of the British Journal of Psychiatry.