Introduction
Reporting bias has been demonstrated in many medical fields (McGauran et al., Reference McGauran, Wieseler, Kreis, Schüler, Kölsch and Kaiser2010; Hart et al., Reference Hart, Lundh and Bero2012; Dwan et al., Reference Dwan, Gamble, Williamson and Kirkham2013). An important form of reporting bias is study publication bias, which occurs when trials with positive results are more likely to be published than those with negative results (Higgins and Green, Reference Higgins and Green2011). In studies on reporting bias, trials that are published exclusively in pooled-trials publications, which pool data from multiple trials, are usually regarded as unpublished (Turner et al., Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008, Reference Turner, Knoepflmacher and Shapley2012) or incompletely published (Rising et al., Reference Rising, Bacchetti and Bero2008). However, some pharmaceutical companies have argued that these trials have actually been published (Lilly press release 2008).
Although pooled-trials publications can provide new information, they may be particularly susceptible to bias, for example, because it is often unclear how trials were selected for inclusion (Thaler et al., Reference Thaler, Morgan, Van Noord, Jonas, McDonagh, Peterson, Glechner and Gartlehner2013). The research question of pooled-trials publications also often differs from the original research question. For example, they may focus on differential efficacy in patient subgroups, leading to substantial redundancy and the suggestion that many of these articles represent ‘salami publications’ (Spielmans et al., Reference Spielmans, Biehn and Sawrey2010, Reference Spielmans, Olson and Keicher2017). Additionally, negative trials, in contrast to positive trials, may be published exclusively in pooled-trials publications. A study examining trials for five antidepressants approved between 1989 and 1994 found that positive trials were usually reported in stand-alone publications, while negative trials were frequently ‘bundled’ into pooled-trials publications (Melander et al., Reference Melander, Ahlqvist-Rastad, Meijer and Beermann2003). Consequently, pooled-trials publications may actually further bias the published literature and the apparent risk–benefit profile of treatments, rather than helping to provide transparent access to trial results. However, that study (Melander et al., Reference Melander, Ahlqvist-Rastad, Meijer and Beermann2003) did not assess the research questions of pooled-trials publications, whether they were likely to reach positive conclusions, and whether individual trial results were also presented.
A previous meta-analysis found that 31% of antidepressant trials for major depressive disorder remained unpublished (Turner et al., Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008). However, pooled-trials publications were excluded from that study. In the present study, we use that meta-analysis and data from four novel antidepressants approved subsequently to investigate whether the practice of pooling trials constitutes a specific form of reporting bias. We assessed whether unpublished trials were actually published in pooled-trials publications and determined how frequently negative trials were published exclusively in pooled-trials publications compared with positive trials. Secondly, we evaluated whether the research question of pooled-trials publications corresponded to the original primary research question of the included trials and whether these publications reported individual trial results for this primary outcome. Finally, we assessed how often pooled-trials publications reached positive conclusions.
Methods
FDA-registered trials
Information on phase 2/3 clinical trial programs for 16 second-generation antidepressants (bupropion sustained release, citalopram, desvenlafaxine, duloxetine, escitalopram, fluoxetine, levomilnacipran, mirtazapine, nefazodone, paroxetine immediate release, paroxetine controlled release, sertraline, venlafaxine immediate release, venlafaxine extended release, vilazodone, and vortioxetine) was extracted from an earlier publication (Turner et al., Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008) or from Food and Drug Administration (FDA) reviews using the same methodology. These reviews can also be accessed online (Turner, Reference Turner2013a; OHSU Digital Commons, n.d.). Because pharmaceutical companies must preregister trials they intend to conduct in support of US marketing approval, FDA reviews can be used as a registry and results database (Turner, Reference Turner2004).
These programs included 105 trials investigating the short-term treatment of depression. Consistent with Turner et al. (Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008), we extracted the FDA's regulatory decision [i.e. whether the primary endpoint(s) were judged to be positive]; 54 trials were considered positive and 51 trials not-positive in the current study. Detailed information on these trials is provided by Turner et al. (Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008). We included only FDA-registered trials because this enabled us to assemble a complete cohort of premarketing trials; although some pharmaceutical companies have detailed trial registries (e.g. GlaxoSmithKline), it is generally difficult or impossible to obtain information on older, unpublished trials that are not FDA-registered.
Trials published as stand-alone publications
We retrieved the references of 50 journal articles reporting the results of FDA-registered antidepressant trials from Turner et al. (Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008). One article presented the pooled results of two identically designed trials of paroxetine controlled release (Golden et al., Reference Golden, Nemeroff, McSorley, Pitts and Dubé2002); we regarded this article as a pooled-trials publication and the included trials were considered unpublished in stand-alone form. Additionally, we found a matching stand-alone publication (Miller et al., Reference Miller, Naylor, Murtagh and Winslow1989) for one trial of paroxetine (UK-06) considered unpublished by Turner et al., and we found 23 stand-alone articles reporting the results of novel antidepressant trials.
Trials published in pooled-trials publications only
We assessed whether trials not published in stand-alone form were published in pooled-trials publications. Pooled-trials publications were defined as publications in which the individual patient data of two or more trials were analyzed. This included publications described as individual patient data meta-analyses, but it did not include meta-analyses based on aggregate data. A systematic literature search was conducted in PubMed, EMBASE, and the Cochrane Central Register of Controlled Trials, restricted to articles in English, until 15 August 2017. The search strategy included the drug, depression-related terms, and ‘placebo’. Terms were customized to each database; for example, the search string for searching PubMed for citalopram publications was: citalopram [Title] AND depress* [Title/abstract] AND placebo. After identifying pooled-trials publications, trial matches were identified using the drug name, active comparator (when applicable), dosage groups, sample sizes, trial duration, and names of investigators. We only included pooled-trials publications for which included trials could be matched to FDA-registered trials. From each publication, we extracted the primary endpoint, which was categorized as ‘primary efficacy’, i.e. the pooled-trials publication endpoint was the same as the original trial's primary endpoint (efficacy of the drug compared with placebo), or ‘not primary efficacy’. The second category consisted of secondary efficacy outcomes (e.g. anxiety, efficacy compared with an active comparator), predictors of efficacy (e.g. efficacy in subgroups, baseline severity), and other efficacy or safety outcomes. We also extracted whether publications reported individual trial results for the original primary outcome of the included trials. Additionally, AR and YV classified each pooled-trials publication as positive, neutral, or negative, based on the abstract (96% inter-rater agreement). Differences were resolved by consensus. Publications were considered positive when the abstract claimed that the antidepressant was more effective than placebo or an active comparator; equally effective as an active comparator; safer, better tolerated, or equal in safety/tolerability to placebo or an active comparator; or simply ‘safe’ or ‘well-tolerated’. Publications were considered neutral when the publication was primarily methodological in orientation or otherwise did not address the antidepressant in question.
Statistical analysis
We examined whether not-positive trials were more likely to be published exclusively in pooled-trials publications than positive trials. Because of small cell sizes, p values were obtained with Fisher's exact test, using Stata (version 13.1).
Results
Pooled-trials publications
As shown in the flow diagram (Fig. 1), of 105 FDA-registered antidepressant trials, 32 were not published in stand-alone publications. Of these, 23 (71.9%) were included in 107 pooled-trials publications (Table 1).
Fig. 1. Flowchart of trial inclusion and pooled-trials publication inclusion.
Table 1. Number of pooled-trials publications and their research questions for all trials not published in stand-alone form
CR, controlled release; FDA, Food and Drug Administration; IR, immediate release; XR, extended release.
Of the 51 not-positive trials, 21 (41.1%) were exclusively published in pooled-trials publications, compared with two (3.7%) positive trials (Fig. 2). Consequently, all positive trials were published in some form (either stand-alone or pooled), as were 82.4% of not-positive trials. Compared with positive trials, not-positive trials were more likely to be published exclusively in pooled-trials publications (Fisher's exact p < 0.001).
Fig. 2. Publication status of positive and not-positive FDA trials.
Characteristics of pooled-trials publications
Thirteen of 107 (12.1%) pooled-trials publications, including 10 (43%) of 23 trials published exclusively in pooled-trials publications, had the same research question as the included trial's original primary research question (drug efficacy compared with placebo) (Table 1). Only four (3.7%) publications presented individual efficacy data for the primary research question, reporting efficacy results for five (21.7%) trials. Other publications reported on secondary efficacy outcomes (27 publications), predictors of efficacy (26 publications), other efficacy data (13 publications), or safety outcomes (28 publications) (Table 1).
Only five pooled-trials publications (4.7%) reported a negative conclusion (online Supplementary Table S1). One publication examined the general safety profile of duloxetine, two examined suicidality during paroxetine treatment, one examined discontinuation symptoms after stopping desvenlafaxine, and one examined the effect of desvenlafaxine on blood pressure. All five concluded that the drug was associated with more adverse events than placebo. Fourteen (13.1%) publications had neutral conclusions (predictors of efficacy = 5; other efficacy = 7; safety = 2) while the remaining 88 (82.2%) were positively framed.
Discussion
Main findings
To our knowledge, this study is the first to show that pooled-trials publication bias constitutes a specific form of reporting bias, which results in the publication of many positively framed articles supporting use of a drug, while the original, negative efficacy results of included trials remain obscured. Although 32 of 74 antidepressant trials were not published in stand-alone articles, only nine were completely unpublished, while the other 23 were included in pooled-trials publications. Trials lacking positive results were often exclusively published in pooled-trials publications (41.1%), whereas positive trials were usually published as stand-alone articles (96.3%). Importantly, only 12% of all pooled-trials publications [including 10 (43%) of 23 trials] examined the original primary research question (efficacy compared with placebo), and individual trial results for this original primary question were provided for only 22% of trials published exclusively in pooled-trials publications. Finally, only 5% of pooled-trials publications had a negative conclusion, all of which concerned safety. Therefore, although these trials have technically been published, the negative efficacy results are obscured, thus distorting the drugs’ apparent risk–benefit profile.
For some drugs, particularly duloxetine, the number of pooled-trials publications was very high. Trials HMAT-A and HMAQ-B were included in 44 and 36 publications, respectively. Our study thus replicates a prior report on the ‘salami slicing’ of duloxetine trials (Spielmans et al., Reference Spielmans, Biehn and Sawrey2010). We also found many pooled-trials publications for venlafaxine (18 publications for immediate- and extended-release combined) and escitalopram (16 publications). Several of these publications seemed redundant. For instance, three pooled-trials publications compared the efficacy of escitalopram to citalopram; three examined the effects of age and gender on the efficacy of venlafaxine; and eight compared the efficacy of venlafaxine to selective serotonin reuptake inhibitors.
It is noteworthy that duloxetine (approved in 2004) and escitalopram (approved in 2002) were the two newest antidepressants in Turner et al.’s (Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008) meta-analysis, although venlafaxine ER was approved somewhat earlier in 1997. This suggests that the practice of pooling trials in many separate publications is relatively new, perhaps developing concurrently with physicians’ growing skepticism of advertising and sales representatives and greater trust in peer-reviewed publications (Spielmans and Parry, Reference Spielmans and Parry2010). Consistent with this, a previous study examining antidepressants approved between 1989 and 1994 identified at most six pooled-trials publications for a single drug (Melander et al., Reference Melander, Ahlqvist-Rastad, Meijer and Beermann2003). Among novel antidepressants, we also found 12 pooled-trials publications for desvenlafaxine, but none for vilazodone. The latter may be because the unpublished vilazodone trials used a variety of dosages and sometimes used flexible dosages that included the approved dosage (20–40 mg), but also higher (e.g. 20–100 mg) or lower (e.g. 10–20 mg) doses.
Others have also noted that antidepressant meta-analyses (including pooled-trials publications) are massively produced, frequently have some kind of industry involvement, and rarely include negative statements (Ebrahim et al., Reference Ebrahim, Bance, Athale, Malachowski and Ioannidis2015). In our study, 97 (91%) of 107 pooled-trials publications had at least one industry-employed author and, as noted, only five publications had negative conclusions. In all cases, this was because the antidepressant was associated with more adverse events than placebo, a finding that is not unexpected.
Although we cannot be certain of the intent behind these pooled-trials publications, the fact that they usually included industry employees as authors raises the question as to whether commercial interests may have played some role. In light of the growing concern that the medical literature may function as a marketing tool for pharmaceutical companies (Smith, Reference Smith2005; Sismondo, Reference Sismondo2009; The PLoS Medicine Editors, 2009; Spielmans and Parry, Reference Spielmans and Parry2010; Vedula et al., Reference Vedula, Goldman, Rona, Greene and Dickersin2012), pooled-trials publications may provide an easy and inexpensive way to keep a drug ‘in the spotlight’. Regardless of intent, the overall effect of these pooled-trials publications is to flood the evidence base with articles that encourage use of a drug, while discouraging results from the original trials remain invisible.
Many pooled-trials publications examined safety and tolerability. Because adverse events may occur infrequently, pooling trials can be necessary to achieve sufficient statistical power. The link between antidepressants and suicidality, for instance, was convincingly established only by pooling trials (Hammad et al., Reference Hammad, Laughren and Racoosin2006; Stone et al., Reference Stone, Laughren, Jones, Levenson, Holland, Hughes, Hammad, Temple and Rochester2009). However, pooling can also mislead. For instance, bupropion SR was only approved at dosages of 300–400 mg/day, but a pooled-trials publication assessing its safety pooled all doses (50–400 mg/day) (Settle et al., Reference Settle, Stahl, Batey, Johnston and Ascher1999). Since adverse events are often dose-dependent, this is likely to paint an overly optimistic picture. Furthermore, there is ongoing concern that meta-analyses of harm outcomes may be particularly threatened by selective outcome reporting (Saini et al., Reference Saini, Loke, Gamble, Altman, Williamson and Kirkham2014), a concern that is further increased by the possibility of selective trial inclusion in pooled-trials publications (Thaler et al., Reference Thaler, Morgan, Van Noord, Jonas, McDonagh, Peterson, Glechner and Gartlehner2013).
Limitations
Because some publications provided too little information for matching, we may have missed some relevant pooled-trials publications; however, individual trial results are never included in these publications. We specifically excluded 22 pooled-trials publications because it was unclear which trials were included. A second limitation is that we did not count pooled-trials publications that reported individual trial results for the original primary outcome but focused on a secondary research question, as it seemed unlikely that these results would be found by researchers or clinicians interested in the primary outcome. However, there were only five such publications, including three additional trials. Furthermore, most included trials were conducted prior to requirements for trial registration, and in the current study, more recent negative trials were more frequently published as stand-alone articles. However, older antidepressants are still in common use, so the results of these older trials continue to be relevant to clinicians. More subtle biases, such as spin or pooling trials, may also become more important as non-publication of a full trial becomes less common. We also did not conduct a meta-analysis, but focused on the apparent risk–benefit profile, following previous work in which the effect of reporting biases on apparent efficacy was larger than the effects on the meta-analytic effect size. For depression, for instance, the effect size of antidepressants was inflated from 0.31 to 0.41; while only 51% of trials were positive, 94% of published articles were positive (Turner et al., Reference Turner, Matthews, Linardatos, Tell and Rosenthal2008). Finally, we assessed the presence of pooled-trials publication bias in a narrow field. However, such bias would be expected in other fields of medicine, because reporting bias has been found throughout psychiatry (Turner, Reference Turner2013b; Le Noury et al., Reference Le Noury, Nardo, Healy, Jureidini, Raven, Tufanaru and Abi-Jaoude2015; Roest et al., Reference Roest, de Jonge, Williams, de Vries, Schoevers and Turner2015), medicine (McGauran et al., Reference McGauran, Wieseler, Kreis, Schüler, Kölsch and Kaiser2010; Hart et al., Reference Hart, Lundh and Bero2012; Dwan et al., Reference Dwan, Gamble, Williamson and Kirkham2013), and science in general (Fanelli, Reference Fanelli2012).
Conclusions
Meta-analyses on reporting bias have been criticized by some for excluding pooled-trials publications, but our study shows that these publications are biased toward positive conclusions. As these publications rarely include individual trial results, they appear to serve primarily to heighten the (positive) visibility of a drug, rather than to transparently report negative results. Therefore, inclusion of pooled-trials publications in meta-analyses could lead to biased results, and caution is warranted. Pooled-trials publications may, however, also provide new information, for instance on secondary outcomes (Thaler et al., Reference Thaler, Morgan, Van Noord, Jonas, McDonagh, Peterson, Glechner and Gartlehner2013). Ideally, individual patient data of all trials should be accessible so that biased reporting can no longer hamper the efforts of systematic reviewers. To mitigate the potential for bias, journal editors could also require pooled-trials publications to present individual trial results or reference stand-alone articles for all included trials. Additionally, editors, peer reviewers, and readers should be aware of the potential for bias and redundancy (Spielmans et al., Reference Spielmans, Biehn and Sawrey2010) and perhaps ask whether pooled-trials publications enhance or merely distort and bloat the evidence base.
In summary, the practice of pooling trials distorts the apparent risk–benefit profile of antidepressants by flooding the literature with publications that highlight positive results and obscure negative results. Together with study publication bias, selective outcome reporting, and spin, pooled-trials publication bias is a form of reporting bias that should be taken into account in future research.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291718002805
Author contributions
AR, ET, and PJ conceived of the initial idea for the study. All authors contributed to the final design of the study. YV and AR extracted and analyzed the data and wrote a first draft of the manuscript, which was critically revised by all authors. All authors gave final approval of the version to be published.
Financial support
This study was supported by a grant from the Dutch Brain Foundation to Peter de Jonge, Grant number: KS2011(1)-120. The funder had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript
Conflict of interest
None.
Ethical standards
Ethics approval was not required for this work.
Open access
