Depression is prevalent in primary care, but current management is suboptimal (Reference Simon and Von KorffSimon & Von Korff, 1995). There is increasing evidence of the effectiveness of ‘collaborative care’ (Reference Gilbody, Whitty and GrimshawGilbody et al, 2003), a multifaceted organisational intervention involving new staff and ways of working (Reference Von Korff and GoldbergVon Korff & Goldberg, 2001). However, collaborative care interventions vary in content and intensity, and it is unclear which aspects are crucial determinants of effectiveness (the ‘active ingredients’). Most of the current collaborative care literature derives from the USA, and designing collaborative care interventions for use in other settings requires an understanding of these ‘active ingredients’.
Collaborative care is an example of a complex intervention, involving a number of separate mechanisms, where the ‘active ingredient’ is difficult to specify (Reference Campbell, Fitzpatrick and HainesCampbell et al, 2000). If different collaborative care interventions vary in their inclusion of ‘active ingredients’, then this should lead to significant variation in outcomes. Such variation in outcomes in a meta-analysis is described as statistical heterogeneity. Meta-regression is a method used to explore statistical heterogeneity in meta-analysis (Reference Sutton, Abrams and JonesSutton et al, 1998; Reference Thompson and HigginsThompson & Higgins, 2002).
A phased approach to the development of complex interventions has been proposed (Reference Campbell, Fitzpatrick and HainesCampbell et al, 2000). Modelling of complex interventions, where the ‘active ingredients’ are explored, is a critical step in the phased model prior to further trials. However, there are relatively few examples of the phased model in the literature (Reference Bradley, Kinmonth and MantBradley et al, 1999; Reference Campbell, Fitzpatrick and HainesCampbell et al, 2000; Medical Research Council, 2000; Reference LoebLoeb, 2002) and a lack of consensus as to the optimal modelling methods.
The authors are developing and testing a collaborative care intervention in the UK using the phased approach, and used meta-regression to examine the relationship between the content of collaborative care interventions and outcomes, to identify ‘active ingredients’ and thus assist in the design of a UK collaborative care model for the care of depression.
METHOD
Data sources
We based our meta-regression on a systematic review. A published systematic review of organisational interventions in primary care mental health completed by S.G. was used as the initial source of studies (Reference Gilbody, Whitty and GrimshawGilbody et al, 2003); this review included collaborative care as well as other types of organisational interventions used to improve the management of depression. Searches included Medline, EMBASE, CINAHL, PsycINFO, the Cochrane Library and the Database of Abstracts of Reviews of Effectiveness (DARE), and were run from the inception date of each database to 2003. We updated the comprehensive search strategies from this review (without language restriction) to June 2004 to find recent studies, and then conducted a second update to November 2005 (Fig. 1). Details of the exact search methods and a table of excluded studies are available from the authors upon request. From this comprehensive database, we then identified the subset of collaborative care studies.
Fig. 1 QUOROM (Quality of Reporting Meta-analyses) flow diagram.
Inclusion criteria
The population of interest was patients with depressive symptoms or diagnosed depressive disorders in primary care settings. Primary care was defined as the provision of medical care by professionals who provide first contact and ongoing care to patients, regardless of the patient's age, gender or presenting problem.
Although we have published a broad typology of models of quality improvement which includes collaborative care (Reference Bower and GilbodyBower & Gilbody, 2005), developing precise inclusion criteria for such complex interventions is more problematic, because by definition it is not clear a priori which mechanisms have to be in place in order to define an intervention as ‘collaborative care’. Therefore, any definition is potentially arbitrary, reflected by published reviews of collaborative care that disagree on which studies and interventions are included and excluded (Reference Von Korff and GoldbergVon Korff & Goldberg, 2001; Reference Gilbody, Whitty and GrimshawGilbody et al, 2003; Reference Bijl, van Marwijk and de HaanBijl et al, 2004).
The purpose of the study was to examine the relationship between variation in the interventions within collaborative care studies, and outcomes. Therefore, we used a broad definition, and defined ‘collaborative care’ as a multifaceted organisational intervention, which could include a number of components:
-
(a) the introduction of a new role (case manager) into primary care, to assist in the management of patients with depression through structured and systematic delivery of interventions;
-
(b) the introduction of mechanisms to foster closer liaison between primary care clinicians and mental health specialists (including case managers) around individual patient care;
-
(c) the introduction of mechanisms to collect and share information on the progress of individual patients.
We excluded educational and training interventions and the provision of brief psychological therapy where these were the sole intervention and were not supported by other enhancements of care outlined above. The full list of studies is given in Table 1.
Table 1 Studies included in the review
| Study | Reference | Setting | Unit of randomisation | Sample size n | Patient population | Antidepressant use data? | Depressive symptoms data? |
|---|---|---|---|---|---|---|---|
| Adler 2004 | Adler et al (Reference Adler, Bungay and Wilson2004) | USA | Patient | 533 | Adults with major depression or dysthymia | Yes | Yes |
| Akerblad 2003 | Bungay et al (Reference Bungay, Adler and Rogers2004) | Sweden | General practitioner | 1031 | Adults with major depression and an indication for antidepressants | Yes | Yes |
| Araya 2003 | Araya et al (Reference Araya, Rojas and Fritsch2003) | Chile | Patient | 240 | Adult women with major depression | Yes | Yes |
| Blanchard 1995 | Blanchard et al (Reference Blanchard, Waterreus and Mann1995) | UK | Patient | 96 | Elderly people with depression warranting clinical intervention | Yes | Yes |
| Brook 2003 | Brook et al (Reference Brook, van Hout and Nieuwenhuyse2003a ,Reference Brook, van Hout and Nieuwenhuyse b ) | The Netherlands | Patient | 147 | Adults with depressive complaints, prescribed new antidepressant | No | Yes |
| Bruce 2004 | Coyne et al (Reference Coyne, Brown and Datto2001); Bruce et al (Reference Bruce, Ten Have and Reynolds2004) | USA | Practice | 598 | Elderly people with major depression, dysthymia and minor depression | Yes | Yes |
| Callahan 1994 | Callahan et al (Reference Callahan, Hendrie and Dittus1994) | USA | Practice | 175 | Elderly people with newly diagnosed depression | Yes | Yes |
| Capoccia 2004 | Boudreau et al (Reference Boudreau, Capoccia and Sullivan2002); Capoccia et al (Reference Capoccia, Boudreau and Blough2004) | USA | Patient | 74 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
| Coleman 1999 | Coleman et al (Reference Coleman, Grothaus and Sandhu1999) | USA | Practice | 169 | Frail elderly people | No | Yes |
| Datto 2003 | Datto et al (Reference Datto, Thompson and Horowitz2003) | USA | Practice | 61 | Adults with depressive symptoms | No | Yes |
| Dietrich 2004 | Dietrich et al (Reference Dietrich, Oxman and Williams2004a ,Reference Dietrich, Oxman and Williams b ) | USA | Practice | 405 | Adults with major depression and dysthymia, starting/changing treatment | Yes | Yes |
| Finley 1999 | Finley et al (Reference Finley, Rens and Gess1999, Reference Finley, Rens and Pont2003) | USA | Patient | 125 | Adults with current major depression, prescribed a new antidepressant | Yes | Yes |
| Hunkeler 2000 | Hunkeler et al (Reference Hunkeler, Meresman and Hargreaves2000) | USA | Patient | 302 | Adults with major depression or dysthymia, prescribed a new antidepressant | Yes | Yes |
| Katon 1995 | Katon et al (Reference Katon, Von Korff and Lin1995); Von Korff et al (Reference Von Korff, Katon and Bush1998) | USA | Patient | 217 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
| Katon 1996 | Katon et al (Reference Katon, Robinson and Von Korff1996); Von Korff et al (Reference Von Korff, Katon and Bush1998) | USA | Patient | 153 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
| Katon 1999 | Katon et al (Reference Katon, Von Korff and Lin1999); Simon et al (Reference Simon, Katon and Von Korff2001a ) | USA | Patient | 228 | Adults on antidepressants, at high risk of persistent depression, recurrent depression or dysthymia | Yes | Yes |
| Katon 2001 | Katon et al (Reference Katon, Rutter and Ludman2001a ), Simon et al (Reference Simon, Von Korff and Ludman2002) | USA | Patient | 386 | Adults, prescribed a new antidepressant, at high risk of relapse | Yes | Yes |
| Katon 2004 | Katon et al (Reference Katon, Von Korff and Lin2003, Reference Katon, Von Korff and Lin2004) | USA | Patient | 329 | Adults with diabetes with depressive symptoms | No | Yes |
| Katzelnick 2000 | Katzelnick et al (Reference Katzelnick, Simon and Pearson2000); Simon et al (Reference Simon, Manning and Katzelnick2001b ) | USA | Practice | 407 | Adults, high users of services, with depressive symptoms | Yes | Yes |
| Mann 1998 | Mann et al (Reference Mann, Blizard and Murray1998) | UK | Patient | 419 | Adults with depression | Yes | Yes |
| Oslin 2003 | Oslin et al (Reference Oslin, Sayers and Ross2003) | USA | Physician | 97 | Adults with depression or dysthymia, at risk drinking | No | Yes |
| Peveler 1999 | Peveler et al (Reference Peveler, George and Kinmonth1999) | UK | Patient | 160 | Diagnosis of depression, prescribed a new antidepressant | Yes | Yes |
| Rickles 2003 | Rickles (Reference Rickles2003) | USA | Patient | 63 | Prescribed a new antidepressant | No | Yes |
| Rost 2001a | Rost et al (Reference Rost, Nutting and Smith2000, Reference Rost, Nutting and Smith2001); Pyne et al (Reference Pyne, Smith and Fortney2003) | USA | Practice | 243 | Adults with major depression, prescribed a new antidepressant, recently treated | Yes | Yes |
| Rost 2001b | As above | USA | Practice | 189 | Adults with major depression, prescribed a new antidepressant, beginning new episode | Yes | Yes |
| Simon 2000 | Simon et al (Reference Simon, Von Korff and Rutter2000) | USA | Patient | 392 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
| Simon 2004a | Simon et al (Reference Simon, Ludman and Tutty2004) | USA | Patient | 402 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
| Simon 2004b | As above | USA | Patient | 393 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
| Swindle 2003 | Swindle et al (Reference Swindle, Rao and Helmy2003) | USA | Firm | 268 | Adults with major depression, dysthymia or partially remitted major depression | Yes | Yes |
| Unutzer 2002 | Unutzer et al (Reference Unutzer, Katon and Williams2001a , Reference Unutzer, Katon and Callahan2003) | USA | Patient | 1801 | Elderly people with major depression, dysthymia, or both | Yes | Yes |
| Wells 2000a | Wells et al (Reference Wells, Sherbourne and Schoenbaum2000); Sherbourne et al (Reference Sherbourne, Wells and Duan2001); Schoenbaum et al (Reference Schoenbaum, Unutzer and Sherbourne2001); Unutzer et al (Reference Unutzer, Rubenstein and Katon2001b ); Wells et al (Reference Wells, Sherbourne and Schoenbaum2004) | USA | Practice | 867 | Adults with major depression or dysthymia | Yes | Yes |
| Wells 2000b | As above | USA | Practice | 932 | Adults with major depression or dysthymia | Yes | Yes |
| Whooley 2000 | Whooley et al (Reference Whooley, Stone and Soghikian2000) | USA | Practice | 331 | Elderly people with depressive symptoms | Yes | Yes |
| Wilkinson 1993 | Wilkinson et al (Reference Wilkinson, Allen and Marshall1993) | UK | Patient | 61 | Adults with depression, prescribed a new antidepressant | Yes | Yes |
Data extraction
Content of collaborative care
We initially tested published coding schemes relating to quality improvement (Reference Weingarten, Henning and BadamgaravWeingarten et al, 2002; Reference Bero, Eccles and GrilliBero et al, 2006), but these lacked the detail to capture the specific issues of relevance to collaborative care. Therefore, a basic coding frame was developed on the basis of the ‘prototypical’ collaborative care model, described in terms of the three roles potentially involved in patient care: primary care provider, mental health specialist and case manager (Reference Katon, Von Korff and LinKaton et al, 2001b ). Variables were created relating to the professional background of each worker and additional intervention-specific training. These codes were then supplemented by variables describing the potential interprofessional relationships (e.g. specialist supervision of the case manager, and case manager feedback of information to the primary care provider). Because professional–patient contact within collaborative care is focused on the case manager–patient relationship, we added variables relating to the intensity and nature of that contact. Finally, we added three variables related to the characteristics of the patients and study setting (see Appendix).
After piloting the data extraction among the team, data from each study were extracted by two different members of the research team working independently. There was no formal measurement of reliability, but disagreements were few and were resolved by discussion. Owing to inconsistent reporting of data in the published papers we were only able to extract comprehensive data on 8 of the original 27 variables (see Table 2).
Table 2 Intervention content variables (n=34)
| Characteristic | n |
|---|---|
| Setting | |
| USA | 27 |
| Non-USA | 7 |
| Recruitment method | |
| Systematic identification | 22 |
| Referral by clinicians | 12 |
| Patient population | |
| Patients with depression | 16 |
| Patients with depression specifically | 18 |
| willing to take antidepressants | |
| Primary care physician training | |
| Training provided | 15 |
| No training provided | 19 |
| Case manager background | |
| Mental health | 17 |
| Non-mental health | 17 |
| Case management sessions | |
| 4 or fewer | 13 |
| 5-7 | 11 |
| 8+ | 10 |
| Case manager supervision | |
| Regular/planned | 24 |
| Other arrangements | 10 |
| Case management content | |
| Medication management alone | 21 |
| Medication management plus | 13 |
| psychological therapy |
Concealment of allocation is the quality attribute with the best evidence for an association with outcomes (Schultz & Grimes, 2002), and we extracted data on concealment to test whether outcomes were related to study quality.
Intervention outcomes
Collaborative care interventions often seek to improve adherence to antidepressant medication, and the first outcome measure was changes in measures of antidepressant use. Most studies reported data in dichotomous form, e.g. the proportion of patients taking antidepressants or meeting standardised guidelines for antidepressant use.
The second outcome measure was reduction in depressive symptoms. A wide variety of outcomes were reported at a number of different time points. Because the meta-regression required as large a sample of studies as possible for reliable analysis (Reference Thompson and HigginsThompson & Higgins, 2002), we restricted the analysis to short-term outcomes (approximately 6 months after randomisation), as these outcomes were by far the most frequently reported. Where alternative measures of depressive outcomes were reported within the same study, the data extracted were chosen on the basis of an a priori decision rule which extracted any identified primary outcome first, and then prioritised observer-rated scales over self-report measures where available.
We extracted all measures of anti-depressant use as dichotomous outcomes, analysed using odds ratios. Measures of depressive symptoms included a mix of dichotomous and continuous outcomes. We translated continuous measures to a standardised effect size, i.e. the mean of the intervention group minus the mean of the control group, divided by the pooled standard deviation. We translated outcomes reported as dichotomous variables to standardised effect sizes using the logit transformation (Reference Lipsey and WilsonLipsey & Wilson, 2001). In 5 of 62 (8%) comparisons, missing data (e.g. standard deviations) were imputed from other relevant studies, in line with accepted practice (Reference Furukawa, Barbui and CiprianiFurukawa et al, 2006).
Previous reviews have identified that unit of analysis errors are common in the evaluation of collaborative care (Reference Gilbody, Whitty and GrimshawGilbody et al, 2003), making studies more susceptible to type 1 errors. We identified all studies using cluster randomisation and where necessary adjusted the precision of these studies in the meta-analysis using methods recommended by the Effective Practice and Organisation of Care (EPOC) group of the Cochrane Collaboration (Reference Bero, Eccles and GrilliBero et al, 2006) and assuming an intraclass correlation of 0.02. The effects of adjustment for clustering were examined in a sensitivity analysis using intraclass correlations of 0.00 and 0.05 (Reference Donner and KlarDonner & Klar, 2002).
Analysis
Analyses were conducted in Stata version 8 for Windows, using the metan and metareg macros. The initial meta-analyses used random effects modelling (Reference Sutton, Abrams and JonesSutton et al, 1998) to provide an overall pooled measure of effect of collaborative care on the two outcomes. However, the main focus of the analysis was on heterogeneity. Heterogeneity was measured using the I 2 statistic, which estimates the percentage of total variation across studies that can be attributed to heterogeneity rather than chance. As a guide, I 2 values of 25% may be considered low, 50% moderate and 75% high (Reference Higgins, Thompson and DeeksHiggins et al, 2003).
The main analysis used random effects meta-regression, which provided estimates of the relationships between eight intervention content variables and the two outcomes. The permutation test was used to calculate P values (using 1000 Monte Carlo simulations) and to reduce the chance of spurious false-positive findings (Reference Higgins and ThompsonHiggins & Thompson, 2004). The amount of heterogeneity explained by the intervention content variables was examined by reductions in the I 2 statistic. Initial univariate analyses (using a criterion of significance of P<0.10) were followed by estimation of a multivariate model. The multivariate model was not based on any automated selection procedure, but involved examination of a number of candidate models involving different combinations of variables. The final model was chosen on the basis of the greatest reduction in heterogeneity. A secondary meta-regression provided an estimate of the relationships between the two outcomes (i.e. whether antidepressant use predicted depressive symptoms).
RESULTS
We identified 28 published studies of collaborative care interventions with outcome data on antidepressant use and 34 studies with outcome data on depressive symptoms (Table 1). Intervention content variables are summarised in Table 2.
Meta-analysis
We found a positive effect of collaborative care on antidepressant use (odds ratio 1.92, 95% CI 1.54–2.39; Fig. 2) and depressive outcomes (standardised mean difference 0.24, 95% CI 0.17–0.32; Fig. 3). The I 2 estimates of inconsistency were 80% and 54% respectively.
Fig. 2 Meta-analysis of antidepressant use. Note: the Wells (2000) and Simon (2004) studies involved two intervention groups compared against a single control; to avoid double-counting the controls, the sample size and event rate in the control were divided by 2. The Rost 2001 study data are only available analysed in two subgroups, rather than as an overall analysis; in our analysis these subgroups were treated as separate comparisons.
Fig. 3 Meta-analysis of depressive symptoms (see note for Fig. 2).
Meta-regression
Analyses of the effects of intervention content variables are shown in Tables 3 and 4. There was insufficient variability in quality of allocation concealment, as most studies were rated as ‘not clear’, and this variable was not used as a covariate in the final analysis.
Table 3 Univariate analysis of associations between intervention content variables and antidepressant use
| Variable | Category 1 | Category 2 | Log odds ratio regression coefficient (95% CI) | P | I 2 (%) |
|---|---|---|---|---|---|
| Study setting | Outside USA | USA | 0.076 (-0.558 to 0.710) | 0.804 | 80.2 |
| Patient sample | Patients with depression | Patients with depression willing to take antidepressants | -0.123 (-0.631 to 0.385) | 0.647 | 80.1 |
| Recruitment method | Referral | Systematic identification | 0.345 (-0.167 to 0.858) | 0.183 | 78.5 |
| Primary care physician training | No training provided | Training provided | 0.328 (-0.163 to 0.818) | 0.194 | 79.5 |
| Case manager background | Non-mental health professional | Mental health professional | 0.220 (-0.280 to 0.721) | 0.393 | 78.9 |
| Content of case management | Medication management | Medication management plus psychotherapeutic processes | -0.104 (-0.617 to 0.409) | 0.683 | 80.5 |
| Supervision of case manager | None or variable | Regular and planned | 0.039 (-0.549 to 0.627) | 0.906 | 80.4 |
| Case management sessions1 | -0.053 (-0.126 to 0.020) | 0.151 | 79.6 |
Table 4 Univariate analysis of associations between intervention content variables and depressive symptoms
| Variable | Category 1 | Category 2 | Effect size regression coefficient (95% CI) | P | I 2 (%) |
|---|---|---|---|---|---|
| Study setting | Outside USA | USA | 0.007 (-0.193 to 0.206) | 0.930 | 54.4 |
| Patient sample | Patients with depression | Patients with depression willing to take antidepressants | -0.087 (-0.243 to 0.070) | 0.285 | 52.1 |
| Recruitment method | Referral | Systematic identification | 0.146 (-0.014 to 0.306) | 0.061 | 47.8 |
| Primary care physician training | No training provided | Training provided | 0.093 (-0.065 to 0.252) | 0.237 | 54.9 |
| Case manager background | Non-mental health professional | Mental health professional | 0.187 (0.046 to 0.327) | 0.004 | 42.7 |
| Content of case management | Medication management | Medication management plus psychotherapeutic processes | 0.093 (-0.064 to 0.250) | 0.206 | 50.7 |
| Supervision of case manager | None or variable | Regular and planned | 0.169 (0.002 to 0.337) | 0.033 | 49.3 |
| Case management sessions1 | 0.015 (-0.008 to 0.039) | 0.174 | 50.9 |
None of the intervention content variables was significantly associated with anti-depressant use, and no multivariate model was estimated. Three intervention content variables predicted improvement in depressive symptoms: recruitment by systematic identification (P=0.061), case managers having a specific mental health background (P=0.004) and provision of regular supervision for case managers (P=0.033), which reduced the overall heterogeneity I 2 from 54% to 48% and 43 to 49% respectively.
In multivariate analysis, four intervention content variables produced the most robust meta-regression in relation to depressive symptom outcomes. The analysis indicated that non-US studies (P=0.038), recruiting through systematic identification of patients (P=0.081) and using case managers having a specific mental health background (P=0.027) who received regular supervision (P=0.055) were more effective. The combination of these four covariates reduced the overall heterogeneity to 36% (low to moderate between study heterogeneity). The inclusion of ‘setting’ (which was not statistically significant in the univariate analyses) reflects the fact that the multivariate analysis accounts for both the relationships between each intervention content variable and the outcome, and the relationships between intervention content variables (Reference Tabachnick and FidellTabachnick & Fidell, 2001).
The meta-regression of the relationships between antidepressant use and depressive symptoms showed a positive association (β coefficient 0.20, 95% CI 0.02–0.38, P=0.028; Fig. 4).
Fig. 4 Relationship between antidepressant use outcomes and depressive symptoms outcomes.
The results of these analyses were not substantively influenced by the sensitivity analysis using estimates of intracluster correlations of 0.00 and 0.05.
DISCUSSION
Overall, the analysis showed an interesting pattern of results. No variable predicted variation in relation to our first outcome, antidepressant use. However, the study did identify several predictors of the second outcome, depressive symptoms. Furthermore, antidepressant use did predict depressive symptom outcomes, which suggests that effects of collaborative care on the latter may be mediated through changes in the former. However, it is not clear whether this association would remain significant if the antidepressant use variables were analysed alongside the other intervention content variables. Clearly the causal pathways between intervention content variables, intermediate outcomes (such as antidepressant use) and final outcomes (such as depressive symptoms) are potentially complex, and analytic techniques such as path analysis might be useful in examining these relationships further.
If the associations between intervention content variables and depressive symptom outcomes are robust, they have interesting implications for the design of collaborative care interventions. For example, the use of case managers with a mental health background and regular specialist supervision both predict outcomes, which suggests that expertise is important. This may have implications for the involvement of the new paraprofessional graduate workers in collaborative care models (Reference Whitty and GilbodyWhitty & Gilbody, 2005).
Clearly the meta-regression cannot determine the process by which expertise has its influence. This may relate to specific technical skills, such as knowledge of anti-depressants or the effective use of psychotherapeutic techniques, or may reflect non-specific skills, such as the ability to engage with patients or to work effectively in collaboration with other professionals. Exploration of this issue might benefit from qualitative research on the nature of patient–professional and interprofessional contact in collaborative care, and the influence of context and organisational variables (Reference Weaver, Tyrer and RitchieWeaver et al, 2003).
However, models of care which require personnel with significant expertise are likely to be more difficult to implement in some contexts, which may limit their usefulness, reflecting the potential tension between ‘efficacy’ as demonstrated in trials and ‘effectiveness’ in routine contexts. Also, models using expert personnel may be more costly, which raises issues about trade-offs between effectiveness and cost that need to be considered when designing collaborative care interventions.
Limitations of the systematic review
As a complex intervention, collaborative care defies simple definition. Our decisions about inclusion and exclusion were informed by our previous conceptual work (Reference Bower and GilbodyBower & Gilbody, 2005), but we took a liberal approach to inclusion precisely because the study focused on the degree to which variability in collaborative care models influenced outcomes. Clearly the inclusion or exclusion of particular studies may have important implications, and thus our findings should be considered exploratory rather than definitive. It should also be noted that most studies were conducted in the USA and the results may not generalise to other contexts. Setting was a significant predictor in the multivariate analysis.
The validity of the coding scheme used to extract data on the interventions has not been confirmed. As noted previously, there were problems of inconsistent reporting and missing data in the published studies. A significant proportion of intervention content variables could not be included as they were not reported consistently, and it is unlikely that it would have been possible to extract data on many additional issues. However, it remains possible that other variables might be more effective predictors than those included in our analyses.
The difficulties encountered in deriving a full description of the interventions echoes traditional problems with poor reporting in randomised trials. There may be a case for adopting a more standardised approach to the reporting of the content of complex interventions (equivalent to CONSORT (Consolidated Standards of Reporting Trials; Reference Moher, Schutz and AltmanMoher et al, 2001) and QUOROM (Quality of Reporting Metaanalyses; Reference Moher, Cook and EastwoodMoher et al, 1999) in order to overcome these problems. The proliferation of web-based journal archives for the presentation of data outside the word limits of the paper-based journals provides an appropriate platform. However, determining the appropriate content and structure of such standardised reports would be challenging, given the potential range of processes involved in complex interventions.
Limitations of the meta-regression technique
The technique of meta-regression has several limitations (Reference Thompson and HigginsThompson & Higgins, 2002). The analysis represents an observational association only, because meta-regression across trials does not have the benefits of randomisation. Equally, statistical power to detect useful associations using meta-regression is limited by (among other things) the number of available studies (Reference Lambert, Sutton and AbramsLambert et al, 2002). Outliers may have a large influence, particularly in the context of a limited sample size. The multivariate model described earlier was found to be sensitive to the particular variables included in the analysis. It should also be noted that the analysis will not be able to detect ‘active ingredients’ that are necessary but do not vary between interventions. Furthermore, it is possible that with certain variables, such as the number of case management sessions, the relationship with average numbers of sessions across trials may not be the same as the relationship within trials. Only individual patient data analysis could overcome this ‘ecological fallacy’ (Reference Thompson and HigginsThompson & Higgins, 2002).
Finally, the analyses were not controlled for quality criteria. The a priori quality criterion (concealment of allocation) showed little variation, as the majority of studies failed to report this adequately. However, it is not clear whether inadequate reporting of concealment always reflects inadequate methods (Reference Soares, Daniels and KumarSoares et al, 2004; Reference Pildal, Chan and HrobjartssonPildal et al, 2005).
Alternatives to meta-regression in the analysis of complex interventions
The controversy over fidelity to assertive community treatment and outcomes (Reference Fiander, Burns and McHugoFiander et al, 2003) indicates that the identification and measurement of ‘active ingredients’ in mental health interventions has important implications for both research and service provision (Reference Marshall and CreedMarshall & Creed, 2000). It is therefore critical to consider the optimal methods of identifying ‘active ingredients’. Our study has shown that the use of meta-regression is feasible but has limitations. The key issue is how well meta-regression compares with the available alternatives, which include clinical expertise, qualitative work, theoretical models and ‘dismantling’ or ‘factorial’ trials.
Clinical expertise is a potentially useful source of hypotheses, and rigorous qualitative work is ideally suited to capture the complexity of care processes, and is especially useful at exploring the perspectives of stakeholders and illuminating context (Reference Weaver, Tyrer and RitchieWeaver et al, 2003; Reference Marshall, Lockwood and LewisMarshall et al, 2004). However, it is unclear whether patients and professionals can reliably identify ‘active ingredients’. Acknowledgement of the limitations of clinical expertise in identifying causal mechanisms is fundamental to evidence-based medicine, and patients will presumably face many of the same challenges as professionals. Insights from theoretical models are another useful source, but few theoretical models within mental health services research are so well validated that they provide a comprehensive description of ‘active ingredients’, and complex mental health issues such as depression will have many competing theories. Although theory is a necessary aspect of the development of a complex intervention, it will rarely be sufficient.
Dismantling and factorial studies test different combinations of ingredients within a randomised comparison. Relevant examples exist in the collaborative care literature. For example, a recent study compared outcomes in patients randomised to a depression care programme (including systematic follow-up) and systematic follow-up alone. There was no difference in outcomes, suggesting that systematic follow-up is critical (Reference Vergouwen, Bakker and BurgerVergouwen et al, 2005). The advantage of such designs is that randomisation is preserved, allowing causal inference. However, the use of such costly designs to identify ‘active ingredients’ may not always be the optimal use of limited research resources.
Clearly comparisons of the different methods are required, and the intervention development currently being conducted by the authors also includes qualitative work which can be compared with the findings of the meta-regression. It is likely that complex interventions will increasingly be required to improve patient care within mental health, and the evaluation of such interventions raises particular challenges. Although there are potential problems with the application of meta-regression, we conclude that the technique has potential in developing useful insights into the active ingredients in complex interventions in mental health, and thus assist in the design and evaluation of future interventions.
Appendix Initial intervention content variables
| Variable | Description |
|---|---|
| Setting | What was the geographical location of the study? |
| Patient population | Did the population include all patients with depression, or was it restricted to patients who had depression and were currently taking, or willing to take, antidepressants? |
| Screening | Were patients referred by their primary care providers, or systematically identified (e.g. through screening)? |
| Primary care providers | |
| PCP professional group | What was the professional background of the primary care providers? |
| PCP training and education | What training and education did the primary care providers receive? |
| PCP EBM guidelines? | Did the primary care providers receive an evidence-based guideline? |
| PCP T+E time | How much time was involved in the training of the primary care providers? |
| PCP T+E materials | What other materials were used in the training of the primary care providers? |
| Case managers | |
| CM professional group | What was the professional background of the case managers? |
| CM training and education | What training and education did the case managers receive? |
| CM T+E time | How much time was involved in the training of the case managers? |
| CM T+E materials | What other materials were used in the training of the case managers? |
| CM session number planned | How many case management sessions were planned? |
| CM session frequency planned | How often were case management sessions designed to be delivered? |
| CM session duration planned | What was the planned duration of case management sessions? |
| CM total time planned | What was the total planned time for the case management? |
| CM session number delivered | How many case management sessions were delivered? |
| CM session frequency delivered | How often were case management sessions delivered? |
| CM session duration delivered | What was the actual duration of case management sessions? |
| CM total time delivered | What was the actual total time for the case management? |
| CM intervention content | What was the content of the case management sessions? |
| CM intervention patient materials | What patient materials were used in the case management session? |
| CM liaison with PCP | How did the case manager liaise with the primary care provider? |
| Specialist care | |
| Specialist professional group | What was the professional background of the specialist? |
| Specialist training and education | What training and education did the specialist receive? |
| Specialist liaison with PCP | How did the specialist liaise with the primary care provider? |
| Specialist liaison with CM | How did the specialist liaise with the case manager? |
Acknowledgements
The study was implemented in the context of research led by D.R. (funded by the Medical Research Council) in which P.B. and S.G. are grant holders. P.B. and J.F. are funded through the Department of Health, but the views expressed in this paper are those of the authors alone. P.B. thanks John Cape for useful input into the paper.
eLetters
No eLetters have been published for this article.