Depressive illness remains one of the most important world health problems, with an estimated 264 million people of all ages currently affected.Reference Liu, He, Yang, Feng, Zhao and Lyu1 The Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study confirmed that despite existing antidepressant and psychotherapeutic treatments, a large proportion of patients have a chronic (25%) or recurrent (75%) illness course.Reference Rush, Kilner, Fava, Wisniewski, Warden and Nierenberg2 Depressive illness is usually resistant to treatment, as most people do not reach remission with their first treatment trial, indicated by treatment effectiveness studies reporting that only 11–30% of patients achieve remission, even after 8–12 months of treatment.Reference Rush, Kilner, Fava, Wisniewski, Warden and Nierenberg2 Unfortunately, for those who do achieve remission, symptom relapse occurs in 10–45%.Reference Rush, Kilner, Fava, Wisniewski, Warden and Nierenberg2 Further research is a priority to identify more effective treatment strategies.
For people unresponsive to antidepressants and psychotherapy, two large meta-analyses have concluded that electroconvulsive therapy (ECT) is the most effective treatment for moderate to severe depressive illness,Reference Pagnin, de Queiroz, Pini and Cassano3,4 resulting in a good return to health-related quality of life and function.Reference Rosenquist, Brenes, Arnold, Kimball and McCall5 These meta-analytic findings for the effectiveness of ECT are confirmed by published reports from organisations that monitor ECT in the UK: the Electroconvulsive Therapy Accreditation Network (ECTAS)Reference Buley, Copland, Hodge and Chaplin6 and the Scottish ECT Audit Network (SEAN).Reference Fergusson, Cullen, Freeman and Hendry7 However, although there is consensus on efficacy, there is less consensus on sensitive and specific predictors of ECT responseReference Kellner, Popeo, Pasculli, Briggs and Gamss8 to inform clinical decision-making.
Since most studies are small and heterogeneous, meta-analytic approaches have been used. A recent systematic review and meta-analysis of 34 studies reporting on 3276 people with a depressive disorder treated with ECT concluded that, for major depression, older age and psychotic features were significant predictors of both ECT remission and response, pre-ECT severity of depression predicted response but not remission, and data on melancholic symptoms were inconclusive.Reference van Diermen, van den Ameele, Kamperman, Sabbe, Vermeulen and Schrijvers9
However, research studies usually have inclusion and exclusion criteria, so may not reflect routine clinical practice. Consequently, routinely collected clinical audit data, acquired over many years using standardised methods at a national level, may provide the best available information on ECT clinical practice. In Scotland, all centres providing ECT have to be registered with SEAN, providing clinical audit information, with descriptive data published annually in reports by Public Health Scotland.10 This data-set provides a unique opportunity to examine possible clinical associations with ECT outcome and explore possible predictive modelling.
Aims
Using 10 years of national Scottish ECT data collected by SEAN, we tested two hypotheses. First, that that there were significant group-level associations between post-ECT clinical outcomes and clinical variables available before starting a course of ECT treatment. Second, that it was possible to develop a new method for predicting illness remission for individual patients using machine learning.
Method
SEAN data 2009–2018
Anonymised SEAN audit data for the period 2009–2018 were provided by the Scottish National Audit Programme (SNAP) of Public Health Scotland. Release of the data was approved by the SEAN steering group (SEAN reference: IR2022-00182) for all ECT treatment episodes in Scotland for 2009–2018, comprising 4474 individual treatment records (some patients will have had more than one treatment). National audit data are anonymised by SEAN staff before release to researchers, meaning that approval by the National Research Ethics Service was not required, confirmed by use of the NHS Human Research Authority online decision tools (www.hra-decisiontools.org.uk/) and following discussion with the local senior research and development facilitator.
SEAN data stratification and coding
Data were stratified by primary diagnosis coded according to ICD-10.11 The available SEAN data comprised depressive disorder or recurrent depressive disorder (n = 3307), bipolar disorder (n = 605), postnatal depression (n = 51), schizophrenia and related primary psychoses (n = 270), neurotic or related disorders (n = 58), personality disorders (n = 10), organic disorders (n = 8) and no specific diagnosis recorded (n = 165). Our primary interest was determining whether it was possible to predict response to ECT in the acute unipolar depressive illness group with moderate to severe illness, so SEAN data were first filtered to include only cases of depressive disorder or recurrent depressive disorder for which entry and exit Montgomery–Åsberg Depression Rating Scale (MADRS) scores were recorded, reducing the total number of data records (n = 2230). SEAN data codes are detailed in Table 1.
Table 1 Scottish ECT Accreditation Network (SEAN) data-set codes and descriptions
ECT, electroconvulsive therapy; CGI, Clinical Global Impression scale; CGI-I, Clinical Global Impression-Improvement scale; MADRS, Montgomery–Åsberg Depression Rating Scale.
This group included patients who were receiving continuation ECT treatment and typically had low entry Clinical Global Impression-Severity (CGI-S) and depression severity scores. As our primary interest was specifically in the efficacy of ECT in acute unipolar depressive episodes, rather than maintenance ECT, we further filtered the data to include only moderate to severe depressive episodes with or without psychotic symptoms. Depression severity was graded by the entry (baseline pre-ECT) MADRS score.Reference Montgomery and Asberg12 By convention, MADRS scores ≤8 indicate absence of depressive symptoms (scores ≤10 indicate remission in many outcome studies), scores of 9–17 indicate mild depression, 18–34 moderate depression, and 35–60 severe depression.Reference Muller, Szegedi, Wetzel and Benkert13 Cases with a MADRS entry score of ≤17 (mild or absent symptoms) were excluded from analyses, resulting in a final sample size of 2074 (n = 1015 with single-episode depression; n = 1059 with recurrent-episode depression) for acute episodes of illness.
Individual MADRS item scoring included scores recorded pre-ECT course (MEn1–MEn10) and post-ECT course (MEx1–MEx10). The 10 MADRS items coded (with a score of 0–6) were: (1) apparent sadness, (2) reported sadness, (3) inner tension, (4) reduced sleep, (5) reduced appetite, (6) concentration difficulties, (7) lassitude, (8) inability to feel, (9) pessimistic thoughts and (10) suicidal thoughts. Although ethnicity was recorded in 3186 cases in the original depressive disorder group, in 3105 cases (97.5%) it was coded as ‘Caucasian’, so ethnicity was not included as a covariable in the analyses. We had specifically requested SEAN to include a clinical rating of early improvement, in the form of a Clinical Global Impression-Improvement (CGI-I) score recorded by the responsible consultant after two ECT treatments (CGI-I2), which to our knowledge has not been reported in previous studies, but here was available for 1538 patients.
Statistical analyses
Individual patient-focused decision-making has always been central to psychiatry and other medical specialties. Clinical research recognises this priority, focusing increasingly on ‘personalised medicine’. Importantly though, group-level association measures (e.g. linear regression, logistic regression, correlation, odds ratios) are insufficient to establish a claim for individual patient predictions.Reference Poldrack, Huckins and Varoquaux14–Reference Varga, Niss, Estampador, Collin and Moseley16 Unfortunately, the term ‘prediction’ is often incorrectly used in studies when only association statistics have been reported in psychiatryReference Poldrack, Huckins and Varoquaux14,Reference Steele and Paulus15 and other medical specialties.Reference Varga, Niss, Estampador, Collin and Moseley16 We report both types of analysis. Descriptive and association analyses used JASP (version 0.19)17 and machine learning prediction analyses used NeuroMiner (version 1.2)Reference Koutsouleris, Vetter and Wiegand18 both running on Linux Mint 21.2 Cinnamon operating system.
Group level association analyses
The aim was to determine whether SEAN-recorded clinical indications for ECT and other clinical characteristics of patients (‘pre-ECT variables’) were associated, at a group level, with measures of response and remission (‘post-ECT variables’) (variables are defined in Table 1). We tested for associations with each pre-ECT variable separately because combinations of ‘independent’ pre-ECT variables can interact if they are correlated. The advantage of this approach is that it facilitates comparison with previous reports. Tests of associations with the post-ECT continuous variable MADRSexit used simple linear regression, and for the post-ECT dichotomous variables MADRSRemission and MADRSResponder, simple logistic regression was used. Significance was defined as P < 0.05. No correction for multiple testing was done because these were a priori planned tests of pre-ECT clinical variables collected by SEAN, selected because they have long been considered relevant to ECT, as summarised in Table 1.
Individual patient prediction analyses
Here the aim was to determine whether it was possible to generate a predictive model of remission for individual patients, as persisting symptoms are strongly associated with relapse.Reference Paykel, Ramana, Cooper, Hayhurst, Kerr and Barocka19 In contrast to the association analyses, all pre-ECT variables were used as potential predictor variables, with the aim of identifying the most accurate predictive model for remission at an individual patient level, despite the already high efficacy of ECT for selected patients. Missing data can adversely affect machine learningReference Wu20 so only complete patient data-sets were used. Two predictive model analyses were explored. The first (model 1) used only clinical variables available before starting a course of ECT. The second (model 2) included the CGI-I2 score, as this is available early, at the end of the first week of a course of ECT.
For both predictive analyses, machine learning with cross-validation (within-study replication) was used, resulting in quantification of individual patient predictive accuracy, sensitivity, specificity, receiver operating curve (ROC) and area under the curve (AUC). We followed recommended best practice for reporting evidence for prediction in psychiatry:Reference Poldrack, Huckins and Varoquaux14 in-sample fits are not reported as evidence, cross-validation encompassed all data operations, prediction analyses did not include groups smaller than a few hundred samples, the coefficient of determination was calculated using the sum of squares, and k-fold cross-validation was used rather than leave-one-out cross-validation.
Results
Descriptive statistics
The data were for 2074 treatment episodes. The mean age of patients at the time of treatment was 59.44 years (s.d. = 15.67) with a range of 15–96 years. The male to female ratio was 1:1.94, the average MADRS entry (pre-ECT baseline) score was 38.8 (s.d. = 8.6; median 39.0, mode 42.0, range 18–60). The post-ECT MADRS mean score was 13.6 (s.d. = 11.7; median 10.0, mode 0). The difference between the mean and median or mode indicates a strong right skew to the post-ECT MADRS scores, with most patients having an exit MADRS score in the 0–5 range (Table 2).
Table 2 Change in individual item and total MADRS scoring at entry and exit of ECT episode (n = 2074)
MADRS, Montgomery–Åsberg Depression Rating Scale; ECT, electroconvulsive therapy; d.f., degrees of freedom; Sig, significance.
Most patients (68%) were treated informally, that is fully consenting and not subject to the Scottish Mental Health Act or Adults with Incapacity legislation. The overall incidence of depression with psychotic symptoms was 34%, and the incidence of recurrent depression was 51%. For the two sub-types of depressive illness, the incidence of psychotic symptoms was 30% in recurrent depression and 38% in single-episode depression. Where indications for ECT had been stated, 8% of treatment episodes were performed as an emergency, 28% for severe distress, 24% for severe psychomotor retardation or agitation, 26% for clinically concerning suicidal ideation, 23% for psychotic symptoms and 62% for medication resistance, mostly to antidepressants (61%), but also antipsychotics (24%) and mood stabilisers (13%). These indications overlap as some patients had more than one indication for treatment.
Data were not available on electrode placement, although it is common practice in Scotland to treat with bilateral rather than unilateral electrode placement. The average number of ECT treatments per patient was 9.1 (s.d. = 5.3; mean 8.0, mode 12). The mean dose per treatment was 310.9 mC (s.d. = 191.9; median 262.6, mode 200.0 mC). Again, the difference between mean and median and mode indicates a right-skewed distribution. The total treatment series cumulative dose was 2915.3 mC (s.d. = 2889.0; median 2141.5, mode 550 mC). This again reflected a right-skewed distribution, with most patients receiving 12 treatments with a total treatment series dose of 550 mC.
The total response rate for ECT was 73% (≥50% reduction in MADRS score from pre-ECT baseline) and the remission rate was 51% (final MADRS score ≤10). Changes in scoring of individual MADRS items are summarised in Table 2.
As the remission rate was close to 50%, MADRSRemission was used as the post-ECT outcome variable for individual patient predictions, as this allowed minimal exclusion of data in the larger group. Clinically, achieving remission is important for minimising the risk of relapse.Reference Paykel, Ramana, Cooper, Hayhurst, Kerr and Barocka19
Group-level association analyses
Using simple linear regression, the final MADRS score (MADRSexit) was significantly negatively associated (Table 3) with age (Age_years_at_episode), total change in MADRS score (MADRSchange), presence of psychotic symptoms (PsychoticSx) and two of the reported indications for ECT: psychotic symptoms (IndPsycot) and previous good response to ECT (IndGoodResp). The final MADRS score (MADRSexit) was significantly positively associated with consent status (Informal), MADRS score before ECT (MADRSentry), Clinical Global Impression-Improvement score after two treatments (CGI-I2), total number of treatments (TreatmentsTotal) and total ECT dose over the full episode (EpisodeDoseTotal). For individual items on the MADRS, significant positive associations were found for reported sadness (MEn2), inner tension (MEn3), reduced sleep (MEn4) and suicidal thoughts (MEn10). Significant positive associations were also found for medication resistance (IndMedResis; IndMedResis1 [antidepressants]; IndMedResis2 [antipsychotics]; IndMedResis3 [mood stabilisers]) and risk of suicide (IndSuicide). This means that better outcome (lower MADRSexit) was associated with older age, presence of psychotic symptoms, sadness, inner tension, reduced sleep, suicidal thoughts, previous good ECT response, being a detained patient, clear improvement after two treatments and lower total ECT dose.
Table 3 MADRSexita associations using simple linear regression
MADRSexit, total score on the Montgomery–Åsberg Depression Rating Scale on exit from the course of electroconvulsive therapy.
a. Low MADRSexit scores correspond to low levels of depressive symptoms. For definitions of other individual variables see Table 1.
* Significant at P < 0.05.
For the post-ECT dichotomous variables MADRSResponder and MADRSRemission (Table 4; Supplementary Table 4a available at https://doi.org/10.1192/bjp.2024.126), simple logistic regression found significant associations between MADRSResponder and measures of Clinical Global Impression (CGIEntry, CGI-I2), ECT treatment (TreatmentsTotal, EpisodeDoseTotal), age (Age_years_at_episode), consent status (Informal, Capacity_Consent), presence of psychotic symptoms (PsychoticSx), specific indications for ECT (urgent necessity [IndEmergency]; severe distress [IndDistressed]; psychomotor retardation [IndSevereRetard]; psychotic symptoms [IndPsycot]; medication resistance (IndMedResis, IndMedResis1 [antidepressants], IndMedResis3 [mood stabilisers]); and previous good response to ECT [IndGoodResp]), the entry MADRS scores (MADRSEntry), all individual items on the MADRS entry assessment with the exception of reduced sleep and suicidal ideation (i.e. not MEn4 or MEn10), and the change in MADRS score following treatment (MADRSChange).
Table 4 Response and remission associations using simple logistic regression for comparisona
a. For definitions of the individual variables see Table 1. For a more detailed version of these data see the Supplementary Material.
b. An odds ratio OR ≥ 1.0 indicates a positive association.
* Significant at P < 0.05.
MADRSRemission was significantly associated with the Clinical Global Impression-Improvement score after two treatments (CGI-I2), ECT treatment (TreatmentsTotal, EpisodeDoseTotal), age (Age_years_at_episode), consent status (Informal, Capacity_Consent), presence of psychotic symptoms (PsychoticSx), history of recurrent depression (RecurrentDD), specific indications for ECT (severe distress [IndDistressed]; risk of suicide [IndSuicide]; psychomotor retardation [IndSevereRetard]; psychotic symptoms [IndPsycot]; medication resistance (IndMedResis, IndMedResis1 [antidepressants], IndMedResis3 [mood stabilisers]); and previous good response to ECT [IndGoodResp]), the entry MADRS scores (MADRSEntry), specific items on the MADRS entry assessment (reported sadness [MEn2]; reduced sleep [MEn4]; and suicidal thoughts [MEn10]) and the change in MADRS score following treatment (MADRSChange).
Individual patient prediction analyses for remission (Fig. 1)
For machine learning prediction we used the SVM algorithm with a linear kernel within NeuroMiner. No feature reduction method was used, and when this was explored, no benefit was found. To get an accurate representation of the true accuracy of our model, we used a 10-fold nested cross-validation (10-fold inner and 10-fold outer) with the hyperparameter tuning done in an automated fashion (as already coded within NeuroMiner) within the inner folds to avoid any potentiation overfitting and we used a fixed random seed value of 654.
Fig. 1 Mean classifier scores for support vector machine (SVM) modelling and receiver operator curve (ROC) analysis from model 1 ((a) and (b)) and model 2 ((c) and (d)) for individual patient prediction of remission.
The variables we used for prediction were not related to final outcomes measures – they were pre-ECT variables and degree of improvement after two ECT treatments (Table 1). For model 1 these were CGIEntry, MADRSEntry, Age_years_at_episode, Female, Capacity_Consent, Informal, PsychoticSx, RecurrentDD, IndEmergency, IndDistressed, IndSevereRetard, IndSuicide, IndPsycot, IndMedResis, IndMedResis1, IndMedResis2, IndMedResis3, IndPatPref and IndGoodResp. The individual patient predictive accuracy for model 1 was 60.4%, sensitivity 58.5%, specificity 62.2% and AUC 0.63 (95% CI 0.61–0.66). For model 2 the pre-ECT variables used were CGIEntry, CGII2, MADRSEntry, Age_years_at_episode, Female, Capacity_Consent, Informal, PsychoticSx, RecurrentDD, IndEmergency, IndDistressed, IndSevereRetard, IndSuicide, IndPsycot, IndMedResis, IndMedResis1, IndMedResis2, IndMedResis3, IndPatPref and IndGoodResp. The individual patient predictive accuracy for model 2 was 61%, sensitivity 56.3%, specificity 65.3% and AUC 0.65 (95% CI 0.62–0.68). For this more accurate model the positive predictive value (PPV) was 60.9% and the negative predictive value (NPV) was 61.2% (for additional details see the Supplementary Material).
Discussion
Using 10 years of SEAN ECT data we tested two hypotheses. First, that that there were significant group-level associations between post-ECT clinical outcomes and clinical variables available before starting a course of ECT. A range of group-level associations were found. Second, that it was possible to develop a new method for predicting illness remission for individual patients using machine learning. We found it possible to predict individual patient remission with ECT to an accuracy of 61% for moderate to severe acute unipolar depression, despite ECT being an already very effective treatment for such illness.
Group-level associations with final outcome
Our sample of patients was selected as having moderate to severe symptoms of depression, meaning an average MADRS baseline (pre-ECT) score of 38.8. We found that the final (post-ECT course) MADRS score was associated with the total MADRS baseline score and baseline MADRS item scores for sadness, inner tension, reduced sleep and suicidal thoughts. Response (≥50% reduction in baseline MADRS score) was significantly associated with baseline MADRS scores for all items except reduced sleep and suicidal thoughts. Remission (final MADRS score ≤10) was associated with baseline MADRS scores for sadness, reduced sleep and suicidal thoughts. Other baseline items, such as being detained under the Mental Health Act, presence of psychotic symptoms, severe distress and psychomotor retardation, were significantly associated with ECT efficacy – i.e. response and remission. This is broadly consistent with a recent meta-analysisReference van Diermen, van den Ameele, Kamperman, Sabbe, Vermeulen and Schrijvers9 which reported that ECT was more effective in patients with depression with psychotic features, in those with more severe depression and in older patients. Older age has long been recognised as being associated with better response to ECTReference Kellner, Popeo, Pasculli, Briggs and Gamss8,Reference Waite, Tor, Mohan, Davidson, Hussain and Dong21 and with remission.Reference van Diermen, van den Ameele, Kamperman, Sabbe, Vermeulen and Schrijvers9
A recent report utilising the Global ECT-MRI Research Collaboration (GEMRIC) data-setReference Blanken, Oudega, Hoogendoorn, Sonnenberg, Rhebergen and Klumpers22 found that, in 500 people with major depressive disorder, remission rates with ECT were independent of gender. Van Diermen et alReference van Diermen, van den Ameele, Kamperman, Sabbe, Vermeulen and Schrijvers9 did not specifically examine gender, since a previous meta-analysis by Haq et alReference Haq, Sitzmann, Goldman, Maixner and Mickey23 found that gender (along with bipolar diagnosis, age at onset and number of previous episodes) did not significant predict the efficacy of ECT. However, there is significant heterogeneity in the pooled data of these meta-analyses. Our study is more homogeneous and specifically examined only moderate to severe unipolar depressive episodes. In this population, even though approximately two-thirds of our 2074 patients were female, we confirmed that ECT outcomes are independent of gender.
We found that the presence of antidepressant treatment resistance was associated with poorer ECT response, similar to a recent large Swedish National Quality Register for ECT study (n = 4244) that compared outcomes for patients meeting criteria for treatment resistant (TRD) or non-treatment resistant depression (non-TRD) and found them to be lower for the TRD group compared with the non-TRD group.Reference Nygren, Reutfors, Brandt, Boden, Nordenskjold and Tiger24 Previous meta-analyses have also found the absence of medication failure to be associated with better ECT outcomes.Reference Haq, Sitzmann, Goldman, Maixner and Mickey23,Reference Heijnen, Birkenhager, Wierdsma and van den Broek25 Patients who do not respond to ECT may have subtle genetic differences compared with those who respond.Reference Clements, Karlsson, Lu, Jureus, Ruck and Andersson26,Reference Foo, Streit, Frank, Witt and Treutlein27
There are few studies that have specifically looked at ECT outcomes based on single-episode versus recurrent depressive disorder. Our data support a history of recurrent depression being associated with remission but not response. This is supported by a meta-analysisReference Haq, Sitzmann, Goldman, Maixner and Mickey23 in which the number of previous depressive episodes was not significantly associated with response, as well as studies examining recurrence of depression in individuals in remission who discontinued ECT, which reported shorter time to relapse associated with the number of previous depressive episodesReference Yang, Lin and Chen28 and an association between risk of relapse and greater number of previous ECT courses.Reference Lambrichts, Vansteelandt, Crauwels, Obbels, Pilato and Denduyver29 Taken together, these studies suggest that individuals with recurrent depression constitute a cohort who are likely to respond very well to ECT and may require repeated courses or maintenance ECT.
Many national guidelines, including those of the Royal College of Psychiatrists in the UK,30 recommend ECT as a first-line treatment in people with depression who require emergency treatment owing to the risk to physical health and/or high suicide risk, because of ECT's efficacy and speed of action.Reference Husain, Rush, Fink, Knapp, Petrides and Rummans31 The findings from linked registry data of the Swedish National Quality Register for ECT and the Swedish National Inpatient Register (n = 5525 individuals with depression treated with or without ECT) support ECT as being significantly associated with a decreased risk of suicide in in-patients who are severely depressed, especially those who are older than 45 years and those with a psychotic subtype.Reference Ronnqvist, Nilsson and Nordenskjold32 A large Canadian retrospective cohort study of records of people admitted to psychiatric hospital for depression (n = 67 327) reported that ECT was associated with a significantly reduced risk of death by suicide in the year after discharge.Reference Kaster, Blumberger, Gomes, Sutradhar, Wijeysundera and Vigod33 In our analyses suicidal thoughts on individual MADRS symptom scoring (MEn10) and suicide as a recorded indication for ECT (IndSuicide) were not associated with treatment response and were negatively associated with remission of symptoms; however, ECT improved (Table 2) all baseline MADRS symptoms. Urgent necessity (IndEmergency) was associated with response but not remission, and severe distress as an indication for ECT (IndDistressed) was associated with both response and remission.
We found that patient preference was not a predictor of either response or remission, consistent with previous reports,Reference Brodaty, Berle, Hickie and Mason34,Reference Iltis, Fortier, Ontjes and McCall35 which implies that patient expectation is not a significant factor (placebo effect) affecting outcome. We found that patients detained under the Mental Health Act were more likely to be MADRS responders or remitters, perhaps because detention was associated with illness severity, severe distress or potential risk of dying.Reference Salagre, Rohde, Ishtiak-Ahmed, Gasse and Ostergaard36
Early response associations and predictions
Studies have shown that early improvement after sixReference Lin, Chen, Yang and Lane37–Reference Martinez-Amoros, Goldberg, Galvez, de Arriba-Arnau, Soria and Menchon39 or threeReference Zheng, He, Gu, Lao, Wang and Mai40,Reference Tsuchiyama, Nagayama, Yamada, Isogawa, Katsuragi and Kiyota41 ECT sessions was associated with final outcome. A 15% reduction in MADRS score after two sessions of ECT was reported to predict final remission with modest sensitivity (51%) and better specificity (79%).Reference Birkenhager, Roos and Kamperman42 The SEAN data-set does not include ratings for depression symptoms throughout the course of ECT but does include the Clinical Global Impression-Improvement rating after the second treatment and at the end of treatment. Previous relatively small studies utilising CGI-I scores in adolescentsReference Maoz, Nitzan, Goldwyn, Krieger and Bloch43 and adultsReference Chen, Lin, Yang and Chen44 did not report clear associations with outcomes, although later CGI scores were associated with good final outcomes. We found that the CGI-I2 score (rated after just two treatments) was significantly correlated with final (post-ECT course) MADRS scores.
Individual patient predictions of outcome
There are very few studies that have tried to create a clinically applicable prediction model for ECT efficacy. De Vreede et alReference de Vreede, Burger and van Vliet45 used multivariable analyses to derive a simple index of four independent predictors (age >65 years, psychotic depression, refractory to antidepressant medication, and personality disorder) which predicted good response (reduction in Hamilton Rating Scale for Depression score of ≥50% compared with baseline) with an AUC of 0.76. However, accuracy of individual patient predictions was not provided nor was cross-validation used. Based on the response prediction literature, as well as clinical experience, Kellner et alReference Kellner, Popeo, Pasculli, Briggs and Gamss8 proposed a 3-item appropriateness scale for ECT that included depression severity, heritability and the episodic nature of depression. However, this model has not been validated in a patient sample. Using an adapted Maudsley Staging Method (MSM), van Diermen et alReference van Diermen, Hebbrecht, Schrijvers, Sabbe, Fransen and Birkenhager46 found that depressive episode duration alone was the best predictor of remission (as defined by a score on the 17-item Hamilton Rating Scale for Depression of ≤7) after ECT (AUC = 0.72). However, cross-validation was not used and hence results are likely to be overestimates.
Several studies have applied machine learning models to the prediction of ECT outcomes, usually using neuroimaging data.Reference Jiang, Abbott, Jiang, Du, Espinoza and Narr47–Reference Wang, Wei, Yuan, Jiang, Xu and Zhou54 A recent study used Global ECT-MRI Research Collaboration (GEMRIC) data to predict remission using a combination of grey matter volume and functional connectivity measures in a training set of 189 people with depression, resulting in 0.70–0.73 AUC cross-validation accuracyReference Bruin, Oltedal, Bartsch, Abbott, Argyelan and Barbour55 for predicting response. Nakajima et alReference Nakajima, Takamiya, Uchida, Kudo, Nishida and Minami56 reported a model based on clinical information, finding that shorter duration of the current illness episode, lower baseline depression severity, higher dose of antidepressant medications before ECT and lower body mass index predicted remission following ECT with 71% accuracy (sensitivity of 86%, and specificity of 46%). Although cross-validation was used, their study was limited by remission being evaluated retrospectively using the c-CGI (a 4-point clinical note CGI-Improvement scale), and clinical heterogeneity, as both bipolar and unipolar depression were included. Our study addressed some of these limitations by its larger large sample size, clinical homogeneity and use of prospective MADRS and CGI scores, finding a cross-validated accuracy of 61% for predicting remission.
It is worth noting that the reason for including machine learning was to demonstrate that it is possible to make predictions for individual patient ECT outcomes from the available data, i.e. that the baseline data collected by SEAN contain sufficient information to allow this. It was not to suggest that this should be introduced into clinical practice. Prediction methods cannot be introduced into clinical practice without prospective clinical trials (as with new drugs), given the potential for harm
Strengths
Retrospective, observational analysis of routinely collected clinical data, such as our use of the SEAN data-set, can achieve much larger sample sizes than would be feasible in a randomised controlled trial (RCT) and may better reflect clinical practice. Analyses also have sufficient statistical power to draw more meaningful conclusions regarding associations and allow utilisation of predictive modelling with cross-validation.
Limitations
Data acquired across multiple geographical locations and over a 10-year period may include unknown heterogeneity that the use of standardised assessment instruments, such as ICD-10 coding of diagnosis, MADRS for clinical severity and CGI for severity and improvement measures, will minimise to some extent. However, this may be a strength, as SEAN data reflect actual clinical practice.
The data provided by SEAN is for episodes of treatment, meaning we do not know how often individual patients appear in the data-set, which might inflate the results if patients having a previously good response are more likely to have multiple treatments. However, our reported remission rate of 51% is very similar to other studies, such as van Diermen et al's recent meta-analysisReference van Diermen, van den Ameele, Kamperman, Sabbe, Vermeulen and Schrijvers9 in which the remission rate was 57.8% for patients with depression and psychotic symptoms and 50.9% for those without psychotic symptoms. Analyses could only be done using the pre-specified SEAN data fields, and other clinical data not included in our SEAN data request may be relevant for response and remission, for example electrode position,Reference Semkovska, Landau, Dunne, Kolshus, Kavanagh and Jelovac57 anaesthetic agentsReference Lihua, Su, Ke and Ziemann-Gimmel58,Reference Galvez, McGuirk and Loo59 or the number of ECT treatments in a course.Reference van Diermen, van den Ameele, Kamperman, Sabbe, Vermeulen and Schrijvers9 Although these factors have the potential to influence the efficacy of ECT they were not individual patient baseline (pre-ECT) factors.
Conclusions and clinical implications
Using a decade of national SEAN data we report a number of group-level associations between baseline patient characteristics and clinical outcomes, including response and remission, consistent with previous studies. ECT response was associated with older age, psychotic symptoms, necessity for urgent intervention, severe distress, psychomotor retardation, previous good response, lack of medication resistance and lack of capacity to consent. Remission had the same associations, except for urgent necessity and in addition a history of recurrent depression and low suicide risk. ECT is a very effective treatment for moderate to severe depressive illness and is associated with a response rate of 73% and a remission rate of 51%. Despite the high efficacy of ECT as a treatment, we were able to predict individual patient remission using routinely collected SEAN baseline clinical data with an accuracy of 60% (model 1), rising to 61% (model 2) with the inclusion of an early measure of treatment response obtained after the first two treatments (CGI-I2). The use of a predictive tool could help inform the shared treatment decision-making process and prevent both the unnecessary use of ECT when it is unlikely to be of clinical benefit and the unnecessary delay in commencing ECT when it is likely to be most effective.
Supplementary material
Supplementary material is available online at https://doi.org/10.1192/bjp.2024.126.
Data availability
The data that support the findings of this study are available on request from the corresponding author D.M.S., following appropriate permissions being obtained from the Scottish National Audit Programme of Public Health Scotland and the Scottish ECT Audit Network.
Acknowledgements
We thank the Scottish ECT Accreditation Network (SEAN) Steering Group for approving our use of the SEAN data-set. In addition, we thank data analyst David Murphy at Public Health Scotland for his work in extracting anonymised data.
Author contributions
D.M.S. and J.D.S. formulated the research questions, designed the study, analysed the data and wrote the article. S.S. contributed to the statistical analysis and machine learning prediction analyses, which used NeuroMiner.
Funding
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Declaration of interest
None.
eLetters
No eLetters have been published for this article.