Identification and selection of studies

We used a database of 1036 papers on the psychological treatment of depression, which includes studies on combined treatments and comparisons with pharmacotherapies. This database has been described in detail elsewhere, ^{Reference Cuijpers, van Straten, Warmerdam and Andersson14} and has been used in a series of earlier meta-analyses (www.evidencebasedpsychotherapies.org). It was developed through a comprehensive literature search (from 1966 to January 2009) in which we examined 9011 abstracts in PubMed (1629 abstracts), PsycINFO (n = 2439), Embase (n = 2606) and the Cochrane Central Register of Controlled Trials (n = 2337). These abstracts were identified by combining terms indicative of psychological treatment and depression (both MeSH terms and text words). For this database we also collected the primary studies from 42 meta-analyses of psychological treatment for depression (www.evidencebasedpsychotherapies.org). For the study reported here we examined the full texts of these 1036 papers. The abstracts from the electronic bibliographic databases were screened by the first author (P.C.) and the 1036 retrieved reports were examined independently by two reviewers for possible inclusion. When the two reviewers disagreed, they discussed the differences with the third reviewer until agreement was reached.

We included published studies in which the effects of a psychological treatment on adults with a diagnosed depressive disorder or scoring above a cut-off point on a depression measurement instrument were compared with a control condition in a randomised controlled trial. Depressive disorders could be diagnosed with one of the versions of the DSM, the ICD, Research Diagnostic Criteria or another diagnostic system. ‘Psychological treatments’ were defined as interventions in which verbal communication between a therapist and a client was the core element, or in which a psychological treatment was written down in book format (bibliotherapy) which the client worked through more or less independently, but with some kind of personal support from a therapist (by telephone, email or otherwise).

In earlier meta-analyses of psychotherapies for adult depression, few indications were found that different types of psychotherapy have differential effects on depression. Few indications for significant differences are found in multivariate meta-regression analyses, ^{Reference Cuijpers, van Straten, Warmerdam and Smits1} as well as in meta-analytic research in which direct comparisons of different types of psychotherapy are examined. ^{Reference Wampold, Mondin, Moody, Stich, Benson and Hyun-nie15,Reference Cuijpers, van Straten, Andersson and van Oppen16} However, we decided to examine the effects of the full sample of psychotherapies, as well as the sample of studies examining cognitive–behavioural therapies. We defined cognitive–behavioural therapy as a psychological treatment in which the therapist focuses on the impact a patient's present dysfunctional thoughts have on current behaviour and future functioning. ^{Reference Cuijpers, van Straten, Andersson and van Oppen16} The therapy is aimed at evaluating, challenging and modifying the patient's dysfunctional beliefs (cognitive restructuring). The definitions of other psychotherapies we distinguished have been given elsewhere. ^{Reference Cuijpers, van Straten, Andersson and van Oppen16}

We excluded studies on children and adolescents (below 18 years of age). Studies in which the psychological intervention could not be distinguished from other elements of the intervention were also excluded (managed care interventions and disease management programmes), as were studies in which a standardised effect size could not be calculated (mostly because no test was performed in which the difference between experimental and control groups was examined), studies in which patients were not randomly assigned to conditions, and studies of in-patients. We also excluded unpublished dissertations, studies aimed at maintenance treatments and relapse prevention, and studies that also included participants with anxiety disorders. Comorbid general medical or psychiatric disorder was not used as an exclusion criterion. No language restriction was applied.

Data extraction

Studies were coded according to patient characteristics, intervention characteristics and general characteristics of the study. We coded the following characteristics: type of recruitment (through the community, from clinical samples, other recruitment method); diagnosis of depression (diagnosed mood disorder, other definition of depression – usually a high score on a self-rating instrument); target group (adults in general, older adults, student populations, women with postpartum depression, general medical patients with depression, other target groups); type of psychotherapy (cognitive–behavioural therapy according to the manual by Beck et al, ^{Reference Beck, Rush, Shaw and Emery17} other cognitive–behavioural therapy in which cognitive restructuring is the core element, interpersonal psychotherapy, problem-solving therapy, non-directive supportive therapy, behavioural activation treatment, other psychotherapy; full definitions of these treatments are given elsewhere); ^{Reference Cuijpers, van Straten, Andersson and van Oppen16} treatment format (individual, group, guided self-help); type of control group (waiting list, care as usual, pill placebo, other control group); and data analyses (intention to treat, completers only).

All studies were coded by two independent assessors. When they disagreed, they discussed the differences with a third reviewer until agreement was reached. A study of the quality of the included studies has been reported elsewhere. ^{Reference Cuijpers, van Straten, Bohlmeijer, Hollon and Andersson18}

Statistical analysis

We first calculated effect sizes (d) for each study by subtracting (at post-test) the average score of the control group from the average score of the experimental group and dividing the result by the pooled standard deviations of the experimental and control group. An effect size of 0.5 thus indicates that the mean of the experimental group is half a standard deviation larger than the mean of the control group. Effect sizes of 0.80 can be assumed to be large, effect sizes of 0.50 are moderate and effect sizes of 0.20 are small. ^{Reference Cohen19} In the calculations of effect sizes, only standardised instruments were used that explicitly measured depression (online Table DS1). If more than one depression measure was used, the mean of the effect sizes was calculated, such that each study (or contrast group) contributed only one effect size to the meta-analysis. When means and standard deviations were not reported we used other statistics (t, P) to calculate effect sizes.

To calculate the pooled mean effect size we used the computer program Comprehensive Meta-analysis version 2.2.021 for Windows, developed for support in meta-analysis (www.meta-analysis.com). We conducted all analyses using the random effects model. ^{Reference Higgins and Green20} Subgroup analyses were conducted according to the Comprehensive Meta-analysis procedures. In the subgroup analyses we used mixed effects analyses that pooled studies within subgroups with the random effects model but tested for significant differences between subgroups with the fixed effects model.

In order to assess the heterogeneity of effect sizes we calculated the I ² statistic, which is an indicator of heterogeneity in percentages. A value of 0% indicates no observed heterogeneity; larger values show increasing heterogeneity, with 25% as low, 50% as moderate and 75% as high heterogeneity. ^{Reference Higgins, Thompson, Deeks and Altman21} We also calculated the Q statistic, but only report whether this was significant or not.

Indicators of publication bias in the full sample

The overall effect size of the 175 comparisons between psychotherapy and a control condition was 0.67 (95% CI 0.60–0.75), with high heterogeneity (I ² = 70.27). Adjustment for publication bias according to Duval & Tweedie's trim and fill procedure resulted in a mean effect size of 0.42 (95% CI 0.33–0.51) with 51 studies missing. Both Begg & Mazumbar's test and Egger's test resulted in highly significant indicators of publication bias (P<0.001). The results of these analyses are summarised in Table 1. The funnel plot of the effect sizes of the studies (with the standard error on the vertical axis and the effect size at the horizontal axis), first without and then with the imputed studies, clearly shows that smaller studies with lower effect sizes are missing (Fig. 2) and this is confirmed after the imputation of missing studies.

Fig. 2 Funnel plots. (a) All psychotherapy studies, without imputed studies; (b) studies of cognitive–behavioural therapy (CBT) only, without imputed studies; (c) all psychotherapy studies, with imputed studies (black circles); (d) CBT studies only, with imputed studies. Imputation according to Duval & Tweedie trim and fill procedure.

Table 1 Mean effect sizes (d) for psychological treatments of adult depression from published studies: tests for publication bias in the overall sample

		Unadjusted values		Adjusted valuesa			Egger's test
	N c	d (95% CI)	I 2	d (95% CI)	N imp a	Taub	I c (95% CI)
All comparisons	175	0.67 (0.60-0.75)	70.27***	0.42 (0.33-0.51)	51	0.37***	1.58 (1.06-2.11) ***
Outliers excludedd	153	0.51 (0.45-0.57)	43.98***	0.39 (0.32-0.45)	38	0.26***	1.01 (0.58-1.44) ***
Multiple comparisons excluded (lowest retained)	117	0.57 (0.49-0.66)	67.98***	0.39 (0.29-0.49)	28
Multiple comparisons excluded (highest retained)	117	0.70 (0.60-0.80)	74.87***	0.45 (0.33-0.56)	33	0.37***	1.63 (0.92-2.34) ***
BDI only	113	0.76 (0.65-0.87)	66.81***	0.45 (0.33-0.56)	38	0.42***	2.39 (1.78-2.99) ***
HRSD only	67	0.83 (0.70-0.96)	65.61***	0.54 (0.40-0.69)	21	0.47***	1.68 (0.99-2.36) ***

Because the overall results of our analyses may have been influenced by possible outliers, we conducted a new meta-analysis in which all effect sizes from 1.5 or larger were removed. As can be seen in Table 1, the mean effect size dropped somewhat (d = 0.51), but all indicators of publication bias remained highly significant (P<0.001), and Duval & Tweedie's trim and fill procedure estimated the adjusted effect size to be considerably smaller (d = 0.39), with a total of 38 missed studies.

In order to examine the influence of multiple comparisons from one study, we conducted another meta-analysis in which we included only one comparison per study (online Table DS2). From the 46 studies with multiple comparisons we included only the comparison with the largest effect size (i.e. the largest difference between the psychotherapy and control groups). We then conducted another analysis in which only the comparison with the smallest effect size was included. As can be seen in Table 1, the resulting mean effect sizes differed somewhat from the overall analyses, and heterogeneity dropped a little, but all indicators of publication bias remained highly significant (P<0.001).

When we limited the analyses to the effect sizes found for the BDI, somewhat larger effect sizes were found, but again all indicators for publication bias remained highly significant (P<0.001). The same was true when we limited the analyses to the effect sizes found for the HRSD.

Publication bias in subgroups of studies

It is possible that the publication bias we found differs for subtypes of studies. We therefore conducted a series of analyses in which we first selected a subgroup of studies based on a specific characteristic, and then examined the indicators of publication bias within this subgroup (online Table DS2). Significant indicators of publication bias were found for most subgroups of studies. No indication of publication bias was found for studies examining interpersonal psychotherapy, studies examining psychotherapy for women with postpartum depression, and studies in which patients were not recruited through the community or from clinical samples (usually from general medical patient groups or systematic screening).

Publication bias in studies of cognitive–behavioural therapy

Because more than half of the comparisons examined cognitive–behavioural therapy, we decided to investigate publication bias in these studies in more detail. The results of these analyses are presented in Fig. 2 and in online Table DS3. The main analyses pointed at a considerable and significant risk of publication bias among studies examining cognitive–behavioural therapy, and this remained high in all sensitivity analyses. The overall effect size of the 89 comparisons between cognitive–behavioural therapy and a control condition was 0.69, and after adjustment for publication bias this was reduced to 0.49 with 26 studies missing. Both Begg & Mazumbar's test and Egger's test were highly significant (P<0.001).

In the subgroup analyses we merged several subgroups because of the small number of studies per group (the target group had two categories: adults and ‘more specific target group’; and placebo and other control groups were merged into ‘other control group’). We also conducted a subgroup analysis in which we examined cognitive–behavioural therapy according to the manual by Beck et al, ^{Reference Beck, Rush, Shaw and Emery17} and other types of cognitive–behavioural therapy: in the majority of subgroups, all indicators pointed at significant publication bias (online Table DS3).

Limitations

This meta-analytic review has several limitations. The most important is that our tests for publication bias do not provide direct evidence of such bias. These procedures only test whether the funnel plot is symmetrical and whether small negative studies are missing. This cannot be considered as direct evidence, and there may in principle be other reasons why these studies are missing. It may be possible, for example, that early pilot studies of new treatments result in large effect sizes because the developer of such a new treatment is the best expert, realising larger effect sizes than later studies by other groups who test the effects of this treatment in routine care. Alternatively, it may be possible that such early pilot projects attract a specific type of patient, who is willing to undergo a new treatment. If these patients were more receptive to change or to treatment in general, that would result in larger effect sizes of smaller pilot studies. However, funnel plot asymmetry cannot be considered to be proof of bias in a meta-analysis, ^{Reference Sterne, Gavaghan and Egger31} and no statistical imputation method can recover the hidden and missing truth. With the highly significant indicators of funnel plot asymmetry, it is very likely that meta-analyses of these studies overestimate the true effect size of psychotherapy for adult depression.

We also used a rather narrow definition of publication bias in this study. A broader definition of publication bias might indicate not only selective publication of studies but also selective reporting of outcomes. ^{Reference Chan, Hrobjartsson, Haahr, Gotzsche and Altman32,Reference Furukawa, Watanabe, Omori, Montori and Guyatt33} Such selective reporting of outcomes might also cause asymmetry of the funnel plot. A further limitation is that many of the included studies were small and although they were all randomised controlled trials they may not meet standard quality criteria for clinical trials. On the other hand the number of included studies was high, enabling us to control for several basic characteristics of the populations, interventions and study designs.

The statistical tests we used to assess the asymmetry of the funnel plot have several weaknesses. ^{Reference Higgins and Green20,Reference Borenstein, Hedges, Higgins and Rothstein34} A problem with Duval & Tweedie's trim and fill procedure is that it depends strongly on the assumptions of the model for why studies are missing, and the algorithm for detecting asymmetry can be influenced by one or two aberrant studies. ^{Reference Borenstein, Hedges, Higgins and Rothstein34} Begg & Mazumdar's test and Egger's test may also yield a different picture depending on the index used in the analyses, ^{Reference Tang and Liu24} and they tend to have low power. Furthermore, they only make sense if there is a reasonable amount of dispersion in the sample sizes and a reasonable number of studies. In our study, however, we included a large number of studies and we found strong indications of publication bias. In this situation, these weaknesses do not suggest that the main conclusion – that there is considerable risk of publication bias – should be doubted.

Implications

Research on psychotherapy for adult depression does not seem to be any freer from publication bias than research on medication treatment, although it is likely that the sources of this bias differ between the two types of studies. There have long been concerns that the large pharmaceutical companies that fund much of the drug research have an economic incentive to make the most positive case possible for the medications that they sell and that this incentive may influence the investigators whose research they support. Although psychological treatments are not supported by large pharmaceutical companies with strong economic interests in larger effect sizes, there appears to be a considerable risk of publication bias in studies of psychotherapy as well. Researchers of psychological treatments do have personal interests in publication of (larger) effects, as these are more likely to lead to tenure and lucrative workshop fees. Pharmaceutical companies have clear financial reasons to inflate research findings, and psychological investigators have both personal and professional reasons for doing the same. Results of a psychotherapy study might not be published because of authors' failure to submit the results of negative studies, journal editors preferring large significant outcomes over small non-significant effects, and negative reports by reviewers. ^{Reference Turner, Matthews, Linardatos, Tell and Rosenthal9} A recent study examining psychotherapy for depression among children and adolescents also found strong indications of publication bias, suggesting that this problem is not restricted to psychotherapy for adult depression alone. ^{Reference Watanabe, Hunot, Omori, Churchill and Furukawa35} Furthermore, there is a growing concern not only that research findings in the field of depression treatment are biased, but that most current published research findings are false and that claimed research findings may be simply accurate measures of prevailing bias. ^{Reference Ioannidis36}

We strongly encourage psychotherapy researchers to register their studies in trial registries, as this will facilitate later investigations into publication bias. We also believe that the investigation of publication bias is a concern not only for psychotherapy research but for psychology in general. The ‘file drawer’ effect is probably present in all areas of psychological research, ^{Reference Rosenthal37} and may have implications for how research findings are interpreted. Meta-analysts are therefore encouraged to quantify systematically the number of studies likely to have been missed in their review, and to conduct sensitivity analyses as exemplified in this study to obtain an impression of the impact of publication bias on the overall estimate.

Not only are the effects of antidepressant medication overestimated because of publication bias, but the same seems to be true for psychotherapy for adult depression. The two most important first-line treatments of adult depression appear not to be as effective as is often assumed and it may be that the presumably active ingredients of treatment account for a smaller proportion of the outcomes observed than is widely believed. It may be time to search for new and more effective treatments of depression or to isolate the active ingredients of those that already exist, and to examine in more depth which treatments are most effective for which patients.