Introduction
Anxiety disorders are among the most common mental health problems in children and adolescents and have a range of adverse consequences across several developmental, familial, social, and school domains (Asselmann, Wittchen, Lieb, & Beesdo-Baum, Reference Asselmann, Wittchen, Lieb and Beesdo-Baum2018; Sicouri, Perkes, & Hudson, Reference Sicouri, Perkes and Hudson2022). At present, cognitive behavioral therapy (CBT) is the first line intervention recommended for childhood anxiety disorders, with on average 5 out of 10 children demonstrating remission at post-treatment (James, Reardon, Soler, James, & Creswell, Reference James, Reardon, Soler, James and Creswell2020). It has become crucial to identify risk factors that may indicate which children are more or less likely to respond to treatment, that is, identify factors that predict outcome regardless of the treatment received (Kraemer, Wilson, Fairburn, & Agras, Reference Kraemer, Wilson, Fairburn and Agras2002). Identifying individual risk is considered an important step towards personalized intervention and may assist clinicians when making treatment decisions (Bertie & Hudson, Reference Bertie and Hudson2021; Ozomaro, Wahlestedt, & Nemeroff, Reference Ozomaro, Wahlestedt and Nemeroff2013).
A problem for the field is the lack of consistency in which predictors (potential risk factors) that have been examined across the multiple trials that have been conducted over the last decades. Existing evidence suggests that a diagnosis of social anxiety disorder, comorbid depression, and parent psychopathology are the most robust baseline predictors of poorer treatment outcomes (Hudson et al., Reference Hudson, Keers, Roberts, Coleman, Breen, Arendt and Eley2015). A recent review and meta-analysis of predictors of child anxiety and depression concluded that higher anxiety symptom severity predicted negative CBT outcomes (Kunas, Lautenbacher, Lueken, & Hilbert, Reference Kunas, Lautenbacher, Lueken and Hilbert2021). However, questions remain regarding the potential effects of several potential additional predictors, ranging from simple demographics (e.g. gender), pretreatment clinical factors (e.g. comorbid externalizing disorders), and treatment factors (e.g. format) on anxiety treatment outcomes. The inconsistent pattern of results may implicate limitations inherent to searching for a single predictor to identify which children are at risk of poor treatment outcomes and the potential utility of a combination of factors instead (Wehry, Beesdo-Baum, Hennelly, Connolly, & Strawn, Reference Wehry, Beesdo-Baum, Hennelly, Connolly and Strawn2015).
In addition to focusing on a variety of divergent risk factors, investigations into the diagnostic complexity involved in predicting likelihood of response or remission following CBT have been hindered mainly by small samples, varied methodology, and traditional statistical methods (Beam & Kohane, Reference Beam and Kohane2018). This has led to increased incentive for the use of predictive analytics, such as machine learning methods in mental health research (Bertie & Hudson, Reference Bertie and Hudson2021), which confers several advantages relative to the linear models typically employed (DeRubeis, Reference DeRubeis2019). For instance, machine learning can integrate larger sets of multidimensional psychological data to build multivariate models that increase predictive ability, can capture complex and non-linear relationships in the data, handle large datasets more effectively, and are better suited for personalized predictions (Coutanche & Hallion, Reference Coutanche, Hallion, Wright and Hallquist2020; LeCun, Bengio, & Hinton, Reference LeCun, Bengio and Hinton2015; Scott, Reference Scott2021). A systematic review of 16 studies that predicted adult anxiety outcomes found increased use of several machine learning methods with predictive accuracy ranging from 73% to 93% (Pintelas, Kotsilieris, Livieris, & Pintelas, Reference Pintelas, Kotsilieris, Livieris and Pintelas2018).
The present study aimed to build on previous predictor work that identified more robust but single predictors of treatment outcome. Using a large, archived dataset from multiple child anxiety treatment trials, the goal of the current exploratory work was to develop machine learning models that predicted remission from anxiety disorders to demonstrate its feasibility for use by clinicians when assessing children and planning treatment for childhood anxiety based on the combination of factors and estimate the likely effectiveness of such a potential tool in clinical practice. The present study took advantage of the large, combined sample size and the inclusion of multiple potential predictors to examine the role and combination of impactful factors. This study used a ‘data-driven’ approach (Cheng & Phillips, Reference Cheng and Phillips2014), which relies on all available predictor data for analysis. Therefore, the models included demographic (i.e. age, gender) and clinical predictors (i.e. severity, type of anxiety disorder, and comorbidity). Treatment variables (i.e. modality, therapist experience) were also included in the model. Given the exploratory nature of this study, the combination of specific factors that would predict remission is not hypothesized.
Transparency and openness
This study involved an analysis of existing data rather than new data collection. We report on how we determined the sample under investigation, eligibility criteria, and all measures in this secondary analysis of existing data. Data were analyzed using Python 3.9. This study's design and its analyses were not preregistered. The code and all used packages are available here: https://github.com/juancq/remission-cbt. Further site and treatment information, as well as details relevant to the assessment of risk of bias of the developed prognostic prediction models (Wolffet al., Reference Wolff, Moons, Riley, Whiting, Westwood and Collins2019) are included in online Supplementary material.
Methods
Sample
The sample comprised data from 2114 children collected from 11 global sites: Sydney, Australia (n = 1199); Reading, UK (n = 342); Aarhus, Denmark (n = 120); Bergen, Norway (n = 156); Bochum, Germany (n = 82); Basel, Switzerland (n = 87); Groningen, the Netherlands (n = 41); Amsterdam, the Netherlands (n = 7); Oxford, United Kingdom (n = 28); Cambridge, UK (n = 3); and Miami, Florida (n = 49). Participants met inclusion criteria if (a) they were assigned a primary diagnosis of a DSM-IV anxiety disorder (APA, 2003), assigned at the individual site after a semi-structured diagnostic interview, and (b) received a course of manualized individual or group CBT for anxiety disorders.
Measures
Anxiety outcomes
The two primary outcomes were remission of (1) all anxiety disorders and (2) the primary anxiety disorder. These remission outcomes served as the target variables that the machine learning models predicted. Remission was defined as clinician-assessed sub-clinical anxiety status at any point during the follow-up period, for both the primary anxiety disorder and all anxiety disorder diagnoses Anxiety status was measured by the Anxiety Disorders Interview Schedule Child and Parent Version (ADIS-IV-C/P) (Silverman & Albano, Reference Silverman and Albano1996), a semi-structured clinical interview administered to both parents and children that assessed baseline severity and diagnosis based on a composite report. Impairment was operationalized as clinician-rated severity ratings (CSR) and was rated per disorder on a scale of 0–8 and according to DSM criteria, with a diagnosis made when a CSR score of 4 or more was assigned. All sites assigned diagnoses according to the ADIS-IV-C/P except for two (Bochum and Basel), where a diagnostically comparable measure, the Kinder-DIPS for DSM-IV-TR (Diagnostisches Interview bei psychischen Störungen im Kindes-und Jugendalter or Diagnostic interview for mental disorders for children and adolescents) (Schneider, Unnewehr, & Margraf, Reference Schneider, Unnewehr and Margraf2009) was used. Assessments were completed at pre-treatment, immediately post-treatment, and at least once more at three months, six months, or one-year post-treatment (one-year follow-up). Assessments were mainly completed by graduate assistants or clinical psychologists who were trained in the administration of the relevant instruments. An assessment of remission at any point during follow-up was coded as a binary variable (1 representing remission at any point, 0 representing no remission during follow-up), for each of the target outcome models.
Demographic and clinical variables
The models contained several parent-reported demographic and clinical baseline factors including age, gender, and ethnicity for the child, and parental age and family structure. Comorbid mood (depression and dysthymia) and externalizing disorders (oppositional defiant disorder, conduct disorder, or attention-deficit/hyperactivity disorder (ADHD)) were assessed using ADIS-IV-C/P or Kinder/DIPS-C/P at ten sites except for Bergen. In Bergen, the Development and Well-being Assessment (DAWBA: Goodman, Ford, Richards, Gatward, & Meltzer, Reference Goodman, Ford, Richards, Gatward and Meltzer2000) was used. Other self-report measures for children and their parents were included which have all shown satisfactory reliability and validity. These included the Spence Children's Anxiety Scale (SCAS: Spence, Reference Spence1998), child and parent versions (SCAS-C, SCAS-P), which assessed child anxiety symptoms. The Short Mood and Feelings Questionnaire (SMFQ: Messer, Angold, Costello, & Loeber, Reference Messer, Angold, Costello and Loeber1995), child and parent versions, were used to assess child's depressive symptoms. The Strengths and Difficulties Questionnaire (SDQ, Goodman, Reference Goodman1997), addressed the child's externalizing problems and functional impairment. Finally, eight sites used the Depression Anxiety Stress Scale (DASS: Lovibond and Lovibond, Reference Lovibond and Lovibond1995), to assess parental self-rated depression, anxiety, and stress.
Treatment variables
All participants received CBT categorized as follows: individual CBT (with or without a parent; n = 1075, 51%) for individual children or family dyads, or group-based CBT (49%) for children (n = 263) or families (n = 776). The present study relied on archived data and as such, treatment variables of intensity, parental involvement, and therapist experience were coded according to trial specific information mostly recorded at the general study level. Treatment intensity was coded as low (i.e. guided self-help with therapist intervention), medium (i.e. standard face-to-face intervention), and high (i.e. increased number of sessions or medication intervention). In the present sample parental involvement was coded as low involvement, active involvement with high contingency management (CM) or strategies for transfer of control (TC), and active involvement with low CM/TC (Manassis et al., Reference Manassis, Changgun Le, Bennett, Zhao, Mendlowitz, Duda and Wood2014). Therapist experience was coded as low (i.e. therapy delivered by trainees or students), medium (therapy delivered by novice therapists), and high (experienced therapists). This variable was not coded based on individual therapists but rather the level of experience of therapists delivering the allocated treatment condition. Although treatment delivery formats varied across studies, treatment variables were coded according to the study protocols which were comparable for core elements of CBT (i.e. coping skills, cognitive restructuring, and exposure), and have been reported in prior subgroup studies (Coleman et al., Reference Coleman, Lester, Keers, Roberts, Curtis, Arendt and Eley2016; Jennifer L Hudson et al., Reference Hudson, Keers, Roberts, Coleman, Breen, Arendt and Eley2015; McKinnon et al., Reference McKinnon, Keers, Coleman, Lester, Roberts, Arendt and Hudson2018).
Machine learning methodology
Our machine learning approach followed established practice (Dwyer, Falkai, & Koutsouleris, Reference Dwyer, Falkai and Koutsouleris2018) including feature selection, training a model on a subset of data, evaluating the model on unseen data, and selecting metrics to assess predictive performance and generalizability. Three machine learning algorithms were used to predict remission of all anxiety disorders and the primary anxiety disorder (Fig. 1) from the demographic, clinical, and treatment variables: logistic regression (LR), a standard baseline model for binary classification tasks; gradient boosted trees (LGBM), an efficient algorithm for structured data; and neural oblivious decision ensembles (NODE), a deep learning model that performs well on structured data (Popov, Morozov, & Babenko, Reference Popov, Morozov and Babenko2019). Using multiple models allows for a robust comparison of predictive performance across different algorithmic approaches, each with its own strength. Logistic regression used the implementation from the scikit-learn library for Python (Pedregosa et al., Reference Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel and Dubourg2011), gradient boosted trees used the LightGBM library for Python (Ke et al., Reference Ke, Meng, Finley, Wang, Chen, Ma and Liu2017), and NODE used the Pytorch Tabular implementation (Joseph, Reference Joseph2021). As such, three machine learning models were trained for each of the two target outcomes, wherein both outcomes were binary. The total number of variables used for each model was 34. Missing values for continuous variables were imputed using a multi-variate imputer that estimated each variable from all the other variables. Missing values for categorical variables were imputed with the most-frequent value.
Figure 1. Machine learning prediction model development and explanation.
Models were evaluated using a train-validation-test split, with this process repeated 30 times (10-fold cross-validation, repeated three times). Metrics from the cross-validation are presented as mean and standard deviation over the 30 runs. The metrics included area under the receiving operating characteristic curve (ROC), area under the precision-recall curve (PR-ROC), and expected calibration error (ECE) (Géron, Reference Géron2022). ROC and PR-ROC measure the discriminative performance of a predictive model at various classification thresholds. The ROC curve is generated by plotting the sensitivity against the false positive rate. The PR-ROC curve is generated by plotting precision (positive predictive value) against sensitivity, a useful metric when the outcomes of interest in the dataset are imbalanced (i.e. number of remissions much lower than non-remissions). Calibration of a model refers to the reliability of the probability predictions generated by a model, as probabilities of experiencing an outcome are more clinically informative than a binary outcome of remission/non-remission. LR and LGBM were calibrated using a validation set with post-hoc calibration using the isotonic method. NODE probabilities were not recalibrated with post-hoc calibration. Calibration was assessed using ECE and reliability diagrams. Hyperparameters for all models were tuned on the validation set of each fold.
Model interpretation and utility
We used SHapley Additive exPlanations (SHAP) to identify the contribution of each variable to the remission predictions made by the trained machine learning models (Lundberg & Lee, Reference Lundberg and Lee2017). The results are visualized with SHAP summary plots showing the magnitude and direction of variable attributions to the prediction of anxiety outcomes. Each dot represents an individual in the dataset and the density represents multiple dots at the same position. The position of the dot on the x-axis represents the impact of a particular variable value on the model predicting remission, that is, negative values on the x-axis contribute to a lower likelihood of remission prediction and positive values on the x-axis contribute to a higher likelihood of remission prediction. The variables are ordered by mean impact on the predictive ability of the model and not by individual variable significance. The colors represent the variable values, with red indicating high and blue indicating low. Explanation was restricted to the 20 variables with the highest mean impact for readability purposes. Results will be discussed by describing variables in terms of demographic, clinical, or treatment variables having an impact on model prediction.
Results
Sample characteristics
Participants were 5–18 years of age (M = 10, s.d. = 2.4), balanced by gender (female = 52%, male = 48%), and overall contained more children than adolescents (5–12 years = 84%). The demographic characteristics of the sample are presented in Table 1, and the treatment details are presented in Table 2.
Table 1. Total sample demographic and clinical characteristics
Note: Sample characteristics based on raw data. ADIS-IV-C/P, Anxiety Disorder Interview Schedule Child and Parent Version; KINDER-DIPS, Diagnostic interview for mental disorders for children and adolescents; DAWBA, Development and Well-being Assessment; SCAS, Spence child anxiety scale; SMFQ, Short mood and feelings questionnaire; SDQ, Strengths and difficulties questionnaire. DASS, Depression, anxiety, and stress scale. ‘Other disorders’ included agoraphobia, panic disorder (with and without agoraphobia), obsessive-compulsive disorder, post-traumatic stress disorder, selective mutism, or unspecified anxiety disorder.
Table 2. Treatment characteristics
Note: CBT, Cognitive Behavioral Therapy; CM/TC, Contingency management and transfer of control.
Predicting remission
Model performance
Table 3 shows the performance of the prognostic models predicting all anxiety disorder remission and remission of the primary anxiety disorder. The NODE model achieved the highest discriminative performance, with a balanced precision and recall demonstrated by the high PR-ROC. The performance of LR and LGBM was similar, but NODE also had the lowest calibration error.
Table 3. Cross validation and standard deviation model results by anxiety outcome
Note: ROC, receiving operating characteristic curve; PR-ROC, area under the precision recall curve; ECE, expected calibration error; LR, logistic regression; LGBM, logistic gradient boosted model; NODE, neural oblivious decision ensembles.
Figure 2 shows the calibration achieved by the NODE model for both primary and all anxiety disorder remission predictions. The model probabilities predicting remission from all anxiety disorders are well calibrated. For primary anxiety disorder remission, the model is not as well calibrated because smaller predictions under-predict the actual likelihood of remission, whereas the larger predictions show good calibration (probabilities > = 0.4 show good calibration, but probabilities < 0.4 are not as trustworthy).
Figure 2. Calibration achieved by the NODE model for both primary and all anxiety disorder remission model prediction.
Remission from all anxiety disorders
Post-hoc explanations of the NODE model predicting remission of all anxiety disorders are presented in Fig. 3. The SHAP summary plot indicates the factors contributing to the model predicting remission from all anxiety disorders. Considering the direction and magnitude of each feature contribution, child age was one demographic factor that contributed to the model predicting that older ages was related to a lower likelihood of remission. Clinical factors contributing to a lower likelihood of remission included greater number of anxiety disorders, the presence of a mood disorder, higher physical injury anxiety (mother and child report), higher internalizing symptoms (mother report), higher externalizing symptoms (mother and child report), as well as higher maternal anxiety, depression, and stress. Among the treatment factors, receiving group CBT delivery, treatment with low treatment intensity, and treatment with low parental involvement also contributed to the model predicting lower likelihood of remission. In contrast, the absence of a social anxiety diagnoses and receiving therapy from a therapist with less experience contributed to the model predicting higher probability of remission from all anxiety disorders.
Figure 3. Summary plot describing the relationship between the value of the pre-treatment factors and the impact on the predictive model for remission of all anxiety disorders (outcome 1). The top twenty factors were displayed.
Remission from the primary anxiety disorder
Post-hoc explanations of the NODE model predicting remission of the primary anxiety disorder are presented in Fig. 4. Similar to the previous model, older child age contributed to the model predicting lower likelihood of remission. Clinical factors contributing to a lower likelihood of remission in this model included the presence of both a mood disorder and an externalizing disorder, as well as higher social anxiety (mother report), higher physical injury anxiety (mother and child report), higher internalizing symptoms (mother report), and higher externalizing symptoms (mother and child report). Among the treatment factors in this model, group CBT delivery and low treatment intensity contributed to the model predicting lower likelihood of remission. Like the previous model, the absence of a social anxiety diagnoses and receiving therapy from a therapist with less experience contributed to the model predicting higher probability of remission from the primary anxiety disorder.
Figure 4. Summary plot describing the relationship between the value of the pre-treatment factors and the impact on the remission of primary anxiety disorders (outcome 2). The top twenty factors were displayed.
Discussion
Key findings – model
In this exploratory study, we applied machine learning techniques to develop prognostic models for predicting childhood anxiety disorder remission after receiving CBT. The NODE deep learning model had the best discriminative performance and the best calibration amongst the three evaluated algorithms. It was anticipated that in the present study, an informative model would have an ROC of 0.5 for both target outcomes and a PR-ROC of 0.55 for all anxiety disorder remission and 0.69 for primary anxiety disorder remission. The results of ROC being less than 0.70 reflects the complexity of predicting remission after receiving CBT, and is often mischaracterized as representing bad discrimination (Perlis, Reference Perlis2011). Indicative of the range of ROC metrics in machine learning studies, a systematic review of prediction models in child and adolescent mental health reported that the ROC of included studies ranged from 0.57 to 0.99 (Senior, Fanshawe, Fazel, & Fazel, Reference Senior, Fanshawe, Fazel and Fazel2021). The ROC of 0.70 and PR-ROC of close to 0.70 demonstrate that the predictions of the models are useful and suggests that additional variables beyond those currently captured in the dataset are necessary to improve prediction performance. Future research should therefore explore the use of different clinical or biological variables for predicting remission.
Key findings – predictors
There was significant alignment between the two predictive models regarding the contributing factors present, with the models sharing 17 of the top 20 contributing factors. Of the demographic factors, older age was associated with model predictions of a lower probability of remission of all anxiety disorders and the primary anxiety disorder. This finding is inconsistent with a previous individual patient data meta-analysis (IPDMA) that reported treatment benefits were comparable between adolescents and younger children (Bennett et al., Reference Bennett, Manassis, Walter, Cheung, Wilansky-Traynor, Diaz-Granados and Wood2013). However, the IPDMA was based on a smaller sample (n = 1171) and our analysis has substantially increased power to detect effects not only because of the larger sample size but also because we examined age as a continuous variable rather than categorical. However, the present study sample contained a small percentage of adolescents (n = 316, 15%), the effect of age on treatment outcome predictions requires further investigation into whether delivering interventions during childhood is more effective than during adolescence.
Regarding clinical diagnostic factors, a diagnosis of Social Anxiety Disorder anywhere in the child's profile had the greatest average impact on both model predictions. The absence of a diagnosis of Social Anxiety Disorder contributed to the models predicting a greater likelihood of remission in this sample. This finding aligns with earlier evidence that Social Anxiety Disorder is a consistent and robust predictor of poorer treatment outcome (Hudson et al., Reference Hudson, Keers, Roberts, Coleman, Breen, Arendt and Eley2015). In addition, a recent systematic review and meta-analysis showed that children with a primary disorder of Social Anxiety had a significantly lower post-CBT recovery rate than children diagnosed with other primary anxiety disorders (Evans, Clark, & Leigh, Reference Evans, Clark and Leigh2021).
As expected, a greater number of diagnosed anxiety disorders predicted a greater likelihood that children would not remit all their anxiety disorders following treatment, in only the model for remission from all anxiety disorders. There is considerable evidence supporting the notion that targeting a primary anxiety disorder will have a positive effect on other comorbid anxiety disorders (Rapee et al., Reference Rapee, Lyneham, Hudson, Kangas, Wuthrich and Schniering2013). Despite this, when child experiences multiple fears and worries, treatment involving a fixed number of sessions reduces the available time allocated to each set of fears or worries. Although generalization across worries is expected to occur, it is likely that children with many feared situations may struggle to tackle all of these fears during treatment. Children with a greater number of anxiety disorders may require additional treatment sessions to ensure that each set of fears and worries has received adequate attention.
The presence of a mood disorder predicted a lower likelihood of remission in both models. This finding is consistent with studies examining treatment outcome based on remission or end state functioning with larger samples (Hudson et al., Reference Hudson, Keers, Roberts, Coleman, Breen, Arendt and Eley2015). Sample size and youth age are both important issues when examining the effects of depression comorbidity on treatment outcome, because children are much less likely to experience depression (Spoelma, Serafimovska, & Parker, Reference Spoelma, Serafimovska and Parker2023) than adolescents and the base rates of comorbid depression in clinical trials of anxiety (sometimes due to exclusion criteria) can be low (e.g.15.2%, see Rapee et al., Reference Rapee, Lyneham, Hudson, Kangas, Wuthrich and Schniering2013).
The presence of an externalizing disorder indicated a lower likelihood of remission, but only for the primary anxiety model. Studies have shown that children diagnosed with comorbid ADHD fared worse following CBT compared to children without such comorbidity (Gould, Porter, Lyneham, & Hudson, Reference Gould, Porter, Lyneham and Hudson2018; Halldorsdottir & Ollendick, Reference Halldorsdottir and Ollendick2016). However, in Halldorsdottir et al., a comorbid diagnosis of oppositional defiant disorder showed no such negative impact on treatment outcome. This finding may suggest that the effect of externalizing diagnoses on treatment outcome may depend on the specific disorder under investigation. Further, it may also indicate that the immediate effect of these comorbid diagnoses is related to the most interfering anxiety disorder and does not necessarily impact overall improvement following treatment.
Secondary anxiety measures (SCAS) also contributed to the prediction of remission. More specifically, mother report of child social anxiety contributed to the primary anxiety disorder remission model, whereas both mother and child rated fear of physical injury was a contributing factor in primary and all anxiety remission models. In all instances, these factors indicated that higher anxiety symptom scores predicted lower likelihood of remission. Internalizing symptomatology also contributed to remission prediction in both models, but only for mother ratings. Higher scores for depressive symptoms (SMFQ) were associated with a lower likelihood of remission. Similarly, both mother and child ratings of externalizing symptomatology contributed to remission prediction. In both models, measures of externalizing problems and functional impairment (SDQ) indicated that higher total scores had a strong negative impact on the probability of remission from anxiety disorders.
Regarding parental factors, higher mother self-reported depression, anxiety, and stress (DASS) scores contributed to model predictions of lower likelihood of remission from all anxiety disorders. This finding is supported by a recent systematic review and meta-analysis conclusion that parental psychopathology was a significant predictor of worse CBT outcomes for anxious youth (Kunas et al., Reference Kunas, Lautenbacher, Lueken and Hilbert2021). Father psychopathology was not included in our analysis due to the low number of father participants across the trials.
Several treatment factors contributed to remission prediction. In both prediction models, treatment modality made the models more likely to predict a lower likelihood of remission when youth received group child and family CBT, with the mean impact of group child CBT on the model output being greater than family CBT. Conversely in both models, receiving individual CBT contributed to predicting higher likelihood of remission. These predictions are consistent with findings from a meta-analysis showing youth who received group delivery showed worse outcomes than those who received individual therapy (Reynolds, Wilson, Austin, & Hooper, Reference Reynolds, Wilson, Austin and Hooper2012). In contrast, findings from a recent meta-analysis (Guo et al., Reference Guo, Su, Hu, Aalberg, Zhu, Teng and Zhou2021), concluded that therapy delivered in a group format did not significantly differ from individual delivery for children based on remission outcomes. However, the same study showed individual therapy was more effective than group therapy for adolescents. Despite model prediction differences in treatment modality, these findings do not reveal which children would respond to which treatment modality and future research should focus on identifying significant interactions that may indicate optimal treatment modality for certain subgroups of children.
Low treatment intensity predicted a lower likelihood of remission in both models. In the present sample, low intensity treatments also referred to delivery that required little therapist intervention and was mostly self-directed (i.e. online programs or bibliotherapy). This is consistent with remission rates following digital interventions being lower than traditional face to face therapies (Garrido et al., Reference Garrido, Millington, Cheers, Boydell, Schubert, Meade and Nguyen2019). Nevertheless, low intensity interventions have important value, particularly when shortages in the mental health workforce reduce access to evidence-based care (Grist, Croker, Denne, & Stallard, Reference Grist, Croker, Denne and Stallard2019).
With regards to therapist experience, across the two prediction models, lower therapist experience was associated with a higher likelihood of remission. Despite this, high level of therapist experience was not a top 20 contributing factor in either predictive model. This is a somewhat surprising finding. In previous explorations of therapist factors within school settings, external clinicians (that is, those with more experience treating mental disorders) deliver better outcomes for student mental health than school staff (Werner-Seidler et al., Reference Werner-Seidler, Spanos, Calear, Perry, Torok, O'Dea and Newby2021; Zhang, Wang, & Neitzel, Reference Zhang, Wang and Neitzel2023). In line with this, evidence from clinical settings also indicates greater therapist experience predicts improved CBT outcome (Podell et al., Reference Podell, Kendall, Gosch, Compton, March, Albano and Piacentini2013), yet this has not always been the case (Cecilione, McLeod, Southam-Gerow, Weisz, & Chorpita, Reference Cecilione, McLeod, Southam-Gerow, Weisz and Chorpita2021). There are a number of possible explanations for this seemingly surprising result. First, although the current analyses are conducted within a much larger sample size than previous studies, there were only three studies (two sites) that utilized therapists with low experience. It is also possible that student therapists with less experience receive more intensive supervision and have received more recent training. Also, the measurement of therapist experience in this study was coded retrospectively and based on the average therapist delivering therapy within the study condition, rather than on ratings of individual therapists allocated to each child. This effect warrants further investigation.
Finally, low parental involvement contributed to the model predicting lower likelihood of remission from all anxiety disorders but not for primary disorder remission. Earlier research suggests treatment that involve parents and used contingency management (CM) strategies for transfer of control (TC) showed better outcomes than other types or levels of involvement, especially in the longer term (Manassis et al., Reference Manassis, Changgun Le, Bennett, Zhao, Mendlowitz, Duda and Wood2014).
Strengths, limitations, and future directions
A major strength of the present study is the large, combined dataset used to facilitate the innovative machine learning approach to prediction modeling. Although the wide age range is a positive characteristic of the sample because it provides greater heterogeneity, the sample contained a relatively small percentage of adolescents. This restricts the generalization of findings primarily to children. Methodological strengths and limitations are also present. The explanations of the models included the most important variables used by the best performing model (NODE). This does not mean that all other variables had no impact. The SHAP plot post-hoc explanation serves as a proxy for understanding how the model makes predictions given the variables included in the model. It is important to bear in mind that the prognostic models in this framework exploit correlations between variables when calculating relative contribution by single factors and should not be interpreted as causal effects. We also explained the models based on the 20 factors with the highest mean impact on predicted ability of the model, although other factors not listed also impacted remission prediction. Finally, for settings where a fully transparent model is the highest priority for a clinical deployment, our results show that logistic regression achieves competitive results with the other algorithms. If obtaining the most accurate and calibrated algorithm is the top priority, then investing in building and training a deep learning model (such as the NODE algorithm) has the most potential. The definition of treatment success is an important consideration for understanding the differing effects across studies of treatment predictors. As the present study focused on models that predict remission from anxiety disorders, future research could investigate the impact of the top variables on outcomes that include treatment response such as improvement or change following therapy.
Clinical implications
The findings of these models have implications for clinical practice. Compared to earlier research, this study provides a performance evaluation of machine learning models that can predict remission after a standard course of CBT, and which could function as an application used by clinicians when assessing children. The explanation of the variables most impactful on the predictions made by the models can also assist clinicians to determine which children are less likely to respond to a standard course of CBT: older children, children whose mothers have elevated depression and anxiety, children diagnosed with SoAD, as well as a greater number of comorbid anxiety disorders, and the presence of a comorbid mood or externalizing disorder. Rather than determining ‘average’ effects, to increase clinical utility, understanding the combination of factors in determining idiographic patterns of outcome will be critical to improving the precision of individualized care for children and their families.
Although this takes us one step closer to improving personalization of mental health care for children undergoing treatment for anxiety disorders, limitations to the clinical utility of these findings should be noted. First, as models become more complex and incorporate a greater number of variables, it becomes increasingly difficult to interpret how they arrive at their predictions. Educating and training clinicians on how machine learning models generate predictions, the limitations of current interpretability methods, and understanding model calibration (a measure of uncertainty around a prediction) will be crucial for successful deployment in clinical practice (Hassija et al., Reference Hassija, Chamola, Mahapatra, Singal, Goel, Huang and Hussain2024). Second, implementation and integration into clinical workflows will require significant resources, including changes to standards of practice and ongoing monitoring and updates to the models to prevent performance degradation. To mitigate algorithmic bias and maintain relevance over time, models should be trained on continuously updated, representative data (Vollmer et al., Reference Vollmer, Mateen, Bohner, Király, Ghani, Jonsson and Myles2020). Overcoming these limitations will require ongoing evaluation, refinement, and close collaboration between data scientists, clinicians, and key stakeholders to ensure the models are both effective and practical in real-world settings.
Conclusion
We employed machine learning models using youth demographic, clinical, and treatment factors to predict the likelihood of remitting from a primary or all anxiety disorders. Twenty factors were reported based on their individual predictive contribution to the remission models, with striking overlap in 17 of those variables. These findings may be useful as they consolidate understanding regarding which factors contribute to lower likelihood of remission following treatment for childhood anxiety. Future research is needed to validate the usefulness of these predictive models in clinical practice, as well as their ability to predict more complex idiographic patterns of response.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291724002654.
Funding statement
This work was supported by Australian Research Council grant DP0878609 (J.L.H., Jenny Donald, PhD, R.M.R., T.C.E.); Australian National Health and Medical Research Council grants (1027556: R.M.R., J.L.H., H.J.L., Cathy Mihalopolous, BSc[Hons], PhD, V.W), (488505: H.J.L., J.L.H., R.M.R.), (382008: J.L.H. and R.M.R.), and (1103611 L.F.M., V.W); TrygFonden grant (7-10-1391: M.T. and Esben Hougaard, PhD); Edith og Godtfred Kirk Christiansens Fond grant (21- 5675: M.T.); Swiss National Science Foundation grant (105314-116517: S.S.); Western Norway Regional Health Authority grants (911253: Odd E. Havik, PhD) and (911366: E.H.); National Institute of Mental Health R01MH079943 (W.K.S.); UK MRC Clinical Fellowship (G0802821: R.M.-S.); UK National Institute for Health Research (NIHR) grants (PB-PG-0110-21190: C.C.) and (PB-PG-0107-12042: P.C.); UK MRC grants (09-800-17: P.C. and C.C.; G0802326: K.T., P.C., and C.C.; G1002011: P.W., C.C., and P.C.; and G0601874; C.C.). Grant 09-800-17 was managed by NIHR on behalf of the MRC-NIHR partnership. Combined study supported by UK MRC grant G0901874/1 (T.C.E.); Combined study presents independent research partly funded by the NIHR Biomedical Research Centre at South London and Maudsley, NHS Foundation Trust, and King's College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health.
Competing interests
The authors declare that there were no conflicts of interest with respect to the authorship or the publication of this article.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Supporting information
All supporting information is either included in this article or presented in the Online Supplementary Material.
Open access
