Introduction
Functional impairment is a major concern among those presenting to youth mental health services. It typically involves disruptions to education or work, social interactions, and daily functioning, which often indicate or exacerbate serious underlying mental health issues [Reference Scott, Fowler, McGorry, Birchwood, Killackey and Christensen1]. According to the 2019 Global Burden of Disease study, around one-fifth of all disease-related disability (from all causes) is attributable to mental disorders in children, adolescents, and young adults aged 5–24 [Reference Kieling, Buchweitz, Caye, Silvani, Ameis and Brunoni2]. A further concern is that functional impairment can have a profound impact on longer-term outcomes in adulthood due to the foundational importance of adolescence and young adulthood for social, emotional, and physical development [Reference McKnight and Kashdan3]. Thus, early recognition and prevention for those at risk of functional impairment in youth mental health settings is essential to guide effective care that aims to improve immediate and long-term outcomes.
Current primary youth mental health care often falls short of effectively addressing functional impairment. Available data from these settings show that many young people engaged in care experience poor functional outcomes [Reference Iorfino, Carpenter, Cross, Crouse, Davenport and Hermens4–Reference Hilferty, Cassells, Muir, Duncan, Christensen and Mitrou6], which can typically be characterized as persistently low or volatile [Reference Iorfino, Hermens, Shane, Zmicerevska, Nichles and Badcock7], and in some cases, only 37.8% experience clinically significant improvement [5]. This is critical given the bidirectional relationship between functioning and clinical symptoms. Indeed, the impact of functional impairment extends beyond the functional domain as it also predicts illness progression [Reference Iorfino, Scott, Carpenter, Cross, Hermens and Killedar8, Reference Fusar-Poli, Byrne, Valmaggia, Day, Tabraham and Johns9] and can have causal effects on suicidal thoughts and behaviors, self-harm, and psychosis-like experiences [Reference Iorfino, Varidel, Marchant, Cripps, Crouse and Prodan10]. Conversely, clinical symptoms such as depression and anxiety can directly impact young people’s participation in school and work. Effective functional recovery in youth requires interventions specifically designed to enhance engagement with education or employment, life skills, and social functioning [Reference Burns, Catty, Becker, Drake, Fioritti and Knapp11–Reference Fowler, Hodgekins, French, Marshall, Freemantle and McCrone15]. Without direct intervention targeting these areas, many young people may continue to experience significant challenges in their daily lives, even if their mental health symptoms have been addressed.
An effective approach to allocate specific interventions for functional impairment involves the use of prognostic stratification. This method involves stratifying young people based on the risk of functional impairment to identify those who would benefit most from early and targeted interventions focused on functioning [Reference Delgadillo and Lutz16, Reference Fusar-Poli, Hijazi, Stahl and Steyerberg17]. This stratified approach ensures that resources and interventions are efficiently directed toward those who need them most, potentially preventing or mitigating the progression of functional impairment. This approach also allows for a more tailored, proactive approach in mental health care, focusing not only on symptom reduction but also on enhancing overall functioning and quality of life from an early stage [Reference Colizzi, Lasalvia and Ruggeri18].
Prognostic models are commonly used in other areas of medicine (e.g., breast cancer and cardiovascular disease), and there are some promising examples in mental health for the onset and course of major mood, anxiety, and psychotic disorders [Reference Nichols, Ryan, Connor, Birchwood and Marshall19–Reference Iorfino, Ho, Carpenter, Cross, Davenport and Hermens30]. Though, few studies have focused on functional impairment as an outcome [Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer31], despite its importance as a consequential outcome of interest for those with lived experience, its widespread impact on other outcomes, and the need for specific intervention [Reference Iorfino, Carpenter, Cross, Davenport, Hermens and Guastella32]. This is particularly important in youth since current diagnostic approaches fail to capture the potential longer-term impact of illness even among those with subthreshold disorders [Reference O’Dea, Glozier, Purcell, McGorry, Scott and Feilds33, Reference Iorfino, McHugh, Richards, Skinner, Prodan and an34]. Unfortunately, the range of factors involved in functional impairment makes prediction and early intervention challenging which can lead to delayed access to effective interventions and highlights the need for prognostic models to identify those at risk earlier [Reference Iorfino, Carpenter, Cross, Crouse, Davenport and Hermens4].
The goal of this study was to develop a prognostic model to predict future functional impairment among young people presenting for youth mental health care. Given the detrimental impact of functional impairment for those presenting to mental health services there is a need for tools that can be used to guide indicated prevention and early intervention efforts. Specifically, we sought to combine basic demographic and clinical information commonly available in mental health services to predict functional outcomes over the subsequent 3 months. This time period was chosen since most change occurs early in the course of care, and to align with our focus on prevention and early intervention by detecting functional change before impairment becomes entrenched.
Methods
The study was approved by the University of Sydney Human Research Ethics Committee (2008/5453, 2012/1626) and participants (and/or guardians) gave written informed consent for the use of routinely collected clinical data for research purposes. The study is reported according to the STROBE reporting guidelines.
Participants
Participants were drawn from a cohort of 6743 individuals aged 12–30 years who presented to the Brain and Mind Centre’s youth mental health clinics in Sydney, Australia, and were recruited to a research register between June 2008 and July 2018 [Reference Carpenter, Iorfino, Cross, Nichles, Zmicerevska and Crouse35]. These clinics include primary care services (i.e., headspace [Reference McGorry, Tanti, Stokes, Hickie, Carnell and Littlefield36]) as well as more specialized mental health services. All participants received ongoing clinician-based case management involving treatment planning and coordination of psychological, social, and/or medical interventions for the duration of their care with the services. This may have involved contact with a psychiatrist, psychologist, occupational therapist, support worker, or hospitalization for those whose needs exceeded the capacity of the primary care services.
Eligibility criteria
By December 2019, phase two of data entry was completed, and so baseline data were available for 2901 participants. To be eligible for the current analysis, we selected individuals who had at least one follow-up visit to the clinic within the first 3 months after their initial visit (baseline). Application of these criteria reduced the sample to 718 individuals included for analyses.
Data collection
Research staff were trained through individual and group training sessions to extract key data from clinical and research files and code inputs according to a specifically designed clinical research proforma [Reference Carpenter, Iorfino, Cross, Nichles, Zmicerevska and Crouse35]. Clinical files included all available notes and records from standard clinical care (recorded by treating clinicians), and research files included various assessments as part of participation in substudies (which may include structured or unstructured clinical interviews and the use of symptom rating scales). Data were extracted from clinical files and code inputs according to the standardized proforma which records information at predetermined timepoints. The first available clinical assessment at the service was taken as the baseline timepoint for each participant and the date of this assessment was used to determine each of the follow-up timepoints. If there was no clinical information available for any given timepoint (i.e., the participant did not attend the service during that time) then that entry was left missing. All clinical notes from the preceding timepoints, up to and including the current timepoint were used to inform and complete the current proforma entry.
Assessments
The proforma recorded specific illness characteristics. More detailed descriptions about the proforma, including the interrater reliability, are reported in the supplement and cohort paper [Reference Carpenter, Iorfino, Cross, Nichles, Zmicerevska and Crouse35]. The measures used here include (Supplementary Materials): demographics, social and occupational functioning, using the Social and Occupational Functioning Assessment Scale (SOFAS), and “Not in Education, Employment or Training” (NEET) as a measure of participation with education or work.
Mental disorder diagnoses (DSM-5) were recorded for each participant and labeled as either primary, secondary, or tertiary based, however, these were collapsed for this study. At-risk mental states, including psychotic- and manic-like experiences and circadian disturbances, were also recorded.
A clinical stage was assigned to each young person alongside their diagnoses using the established criteria [Reference Hickie, Scott, Hermens, Naismith, Guastella and Kaur37]. Young people at stage 1a (“help-seeking”) typically have fewer clinical symptoms and milder impairment than those at stage 1b (“attenuated syndrome”). Stage 2+ (stages 2, 3, and 4) includes young people with full-threshold disorders that are major, discrete, and persistent. One of three illness subtypes was also coded for each participant, neurodevelopmental-psychosis (psychotic symptoms, cognitive impairment, and/or childhood neurodevelopmental disorder), hyperarousal-anxious depression (heightened stress sensitivity and/or depressive features), or circadian-bipolar spectrum (disrupted sleep and circadian rhythms, symptoms of atypical or bipolar spectrum). “N/A” was recorded if the clinical researchers determined no clear illness subtype.
Other clinical information included self-harm, suicidal thoughts and behaviors, physical health comorbidities (e.g., diabetes and respiratory illnesses), personal mental illness history, and treatment utilization.
Statistical analyses
Analyses were conducted using Python 3.9.12. A Bayesian growth mixture model [Reference Ram and Grimm38, Reference Kim, Vermunt, Bakk, Jaki and Van Horn39] was designed to predict whether an individual’s SOFAS score would improve, deteriorate, or stay the same during the first 3 months of care, and estimate the amount of change in SOFAS score, if any, over that same period. For these purposes, we adopted a latent class linear mixed effects model where individuals were assigned to one of three fixed classes: (i) constant, (ii) improve, and (iii) decline. Individual factors were used as predictors of class membership, which is jointly inferred with the rest of the model parameters. The inference is performed following a Bayesian approach via Markov chain Monte Carlo, where we use a no-U-turn sampler (NUTS) to estimate continuous model parameters within a Gibbs sampling scheme to sample the discrete random variables (i.e., the latent class for each individual) [Reference Hoffman and Gelman40]. To estimate our results, we ran the NUTS for 6000 iterations in 2 parallel chains. The first 4000 samples of each chain were discarded as burn-in, while the remaining 2000 samples of each chain were combined to form the model posterior samples, totaling 4000 posterior samples for each model. The resulting Gelman-Rubin R-hat statistic for each parameter’s posterior distribution was below 1.1 across all models, indicating convergence of the sampling algorithm. An overview of the model is presented in Figure 1 and further details can be found in the Supplementary Materials.
A total of 31 factors were available for these analyses. These included basic demographic and clinical factors commonly collected in mental health services when someone presents for care. We considered three different combinations of these factors for the model: (i) a model with all available factors; (ii) a model with no factors; and (iii) a model with a subset of eight factors that were deemed to be the optimal set for prediction (decided by expert knowledge and confirmatory analyses). The eight factors included in the model were NEET status, self-harm, psychotic-like experiences, physical health comorbidity, childhood-onset syndrome, illness type, clinical stage, and sleep–wake and/or circadian disturbances. Factor selection was evaluated using the Watanabe-Akaike information criterion (WAIC), a Bayesian estimate of the predictive performance of a model on unseen future data via an approximation based on the model’s posterior distribution given all available data [Reference Gelman, Carlin, Stern, Dunson, Vehtari and Rubin41, Reference Liu42]. In general, a lower WAIC value indicates a better out-of-sample predictive performance when comparing models.
We used the traditional interpretation of the c-statistic/area under the curve (AUC) to evaluate the performance of the model in two prediction tasks relevant to clinical practice. The first task consisted of predicting whether an individual’s SOFAS score would significantly drop over the course of 3 months, based solely on the information accessible at baseline. A predicted difference of 10 or more points in SOFAS was deemed significant [Reference Rickwood, Mazzer, Telford, Parker, Tanti and McGorry43]. To provide true labels for this assessment, the score at 3 months was computed based on the available data for an individual via linear extrapolation.
The second task consisted of predicting functional impairment at the next timepoint for individuals with initially good functioning (SOFAS above 70), again using only the information available at baseline. This task was designed to assist with indicated prevention and early intervention by identifying those individuals who are at risk of becoming functionally impaired.
To assess predictive performance on unseen data in both tasks, we used 5-fold cross-validation, where 80% of the data is used to estimate the model parameters posterior, and the remaining 20% is used as test data to assess the model’s predictions.
Results
Baseline characteristics for the 718 young people (mean age = 18.1 ± 3.3; 63% female) who met the criteria for this study are presented in Table 1.
Each row presents the count of individuals presenting with a certain covariate in the dataset, except for the case of age, which shows the mean and standard deviation of the participants’ age. Abbreviations: NEET, not in education, employment or training; No., number of participants; SD, standard deviation.
Overall model performance
The model using all factors had the highest WAIC (WAIC = 1942; AUC = 0.65) and the model with no factors had the lowest AUC (WAIC = 1802; AUC = 0.50), indicating that each of these models was the worst performing on at least one of the evaluation metrics. By contrast, the model using the subset of eight factors performed best with the lowest WAIC (1777) and best AUC with “acceptable” predictive accuracy for predicting whether a person’s functioning would significantly drop over the next 3 months (AUC = 0.70, 95% Credible Interval [CI], 0.56–0.81; Figure 2A).
Characteristics of change
The constant and decline clusters were comprised of individuals with similar initial SOFAS scores (constant: M = 64.1, SD = 9.0; decline: M = 64.7, SD = 7.9), though individuals in the decline cluster had a decrease of 8.8 SOFAS points on average (SD = 6.6) after 3 months. Individuals in the improve cluster started at an initially lower score (M = 58.6, SD = 8.3) and then progressed to an average increase of 7.8 SOFAS points (SD = 5.4) after 3 months. The time to a significant change in SOFAS score (10 points) was 5.0 months (95% CI, 1.7–11.0) for the improve cluster and 4.6 months (95% CI, 1.7–9.5) for the decline cluster.
The marginal probabilities for the whole sample indicate that in their first 3 months, most individuals had no change in score (58%), 16% deteriorated, and 26% improved. Being NEET shifted this marginal probability with an increase of 52% in the probability of improvement when present at baseline (Odds Ratio [OR], 1.52; 95% CI, 1.11–2.00). Conversely, a history of self-harm (OR, 1.49; 95% CI, 1.10–1.94) and having a physical health comorbidity (OR, 1.98; 95% CI, 1.43–2.69) led to a significant increase in the probability of deterioration (Supplementary Materials).
Typical cases for each cluster (i.e., individuals who maximize each cluster probability given their factors at baseline) are presented in Table 2. Typical cases across each cluster do not change very much between baseline and follow-up. The first noticeable difference between clusters is in the presence of medical and childhood history, with individuals in the decline cluster presenting with both at baseline. The typical case for each cluster also differed based on specific illness characteristics such as the type of mood disorder, presence of sleep–wake and/or circadian disturbances, clinical stage, and initial functioning.
Predicting impairment among those with good functioning (SOFAS above 70)
The model demonstrated “excellent” accuracy for predicting functional impairment among those who present with good functioning (AUC = 0.80, 95% CI, 0.67–0.89; Figure 2B). The model has high sensitivity (0.82, 95% CI, 0.71–0.97), while maintaining high specificity (0.74, 95% CI, 0.68–0.78), indicating good discriminative power. Table 3 presents odds ratios for the posterior probabilities of drops below 70 in the follow-up visit as a function of the factors and the SOFAS score at baseline. NEET status did not emerge as a predictor, which likely reflects how many of those with a SOFAS score above 70 are engaged in employment, education, or training. Consistent with the overall model results, physical health comorbidity and self-harm were associated with the greatest increase in the probability of future functional impairment among those who are functioning well, however psychotic-like experiences and circadian disturbances were also identified as key predictors of future impairment (Table 3).
The table presents odds ratios for the probability of the next SOFAS score being below 70 computed with respect to a baseline individual with no mood disorders or any comorbidities based on the model’s posterior predictions. The 95% credible interval for the predictions is also shown within brackets. Bold text indicates significance.
Discussion
This study describes the development and preliminary validation of a brief prognostic model for functional outcomes among young people presenting for youth mental health care. Overall model performance was “acceptable” indicating its potential utility for predicting functional outcomes over 3 months, and model performance was “excellent” for identifying those at risk of deterioration from good functioning to impairment. This work has clear implications for the future development of prognostic models. These models can inform individual-level decision-making about the type and intensity of indicated prevention and early intervention required to improve functional outcomes or prevent functional decline in youth mental health.
The goal of indicated prevention in youth mental health services is to provide those at risk for poor outcomes with the appropriate supports [Reference Cuijpers, Smit and Furukawa44]. The heterogeneity of presentations makes this type of prediction difficult. Here, we demonstrate how basic demographic and clinical information could be leveraged to identify those at risk and who could benefit from specific preventative approaches needed to prevent functional impairment (e.g., vocational support and school support).
Incorporating new predictive models into clinical services represents a transformative shift in youth mental health care. Early interventions guided by prognostic models hold promise for reducing the burden of mental health disorders in young people. By identifying those at risk before the onset of severe illness, preventive strategies can be deployed earlier, when people are more likely to be responsive to interventions [Reference McGorry and van Os45]. This shift aligns with a broader trend toward proactive, individualized care that addresses mental health challenges early in their course rather than after they have progressed significantly. While further validation and testing are required before this model can be recommended for clinical practice [Reference Senior, Fanshawe, Fazel and Fazel46], the present model demonstrates utility as a prognostic model to identify those most at risk of deteriorating mental health and function [Reference Meehan, Lewis, Fazel, Fusar-Poli, Steyerberg and Stahl47].
Critical factors associated with poor trajectories of functional impairment included a history of self-harm, physical health comorbidity, circadian rhythm disruptions, and psychotic-like experiences. This is consistent with previous evidence about the complicating role of complexity and comorbidity in diagnosis, treatment, and the course of illness [Reference Firth, Siddiqi, Koyanagi, Siskind, Rosenbaum and Galletly48]. While the relationship between these factors and impairment is likely bidirectional, their predictive utility among those functioning well points to some potential critical mechanisms.
Disrupted sleep and circadian rhythms can impair cognitive and mood regulation, affecting academic and social engagement [Reference Logan and McClung49–Reference Mehta51], and self-harm may result in social withdrawal and academic struggles [Reference Geulayov, Mansfield, Jindra, Hawton and Fazel52, Reference Wu, Chang, Huang, Liu and Stewart53]. Further, psychotic-like experiences, such as hallucinations or delusions, can significantly disrupt reality perception, leading to challenges in social interactions and daily activities [Reference Heinze, Lin, Nelson, Reniers, Upthegrove and Clarke54, Reference Chun, Cooper and Ellman55]. These factors collectively underscore the need for tailored, multifaceted treatment approaches to address these challenges and reiterate the importance of large-scale health service trials focused on developing service models that facilitate this level of personalized care [Reference Hickie, Iorfino, Rohleder, Song, Nichles and Zmicerevska56]. For example, behavioral interventions to improve the alignment between someone’s biological clocks with external environmental cues can foster a more regulated sleep–wake cycle [Reference Crouse, Carpenter, Song, Hockey, Naismith and Grunstein57]. This, in turn, can lead to improved mood, higher energy levels, and better cognitive functioning.
Prediction in mental health is inherently complex due to the variability in individual trajectories [Reference Varidel, Hickie, Prodan, Skinner, Marchant and Cripps58] and multiple causes of poor outcomes. This makes the communication of uncertainty vital in decision-support models, and ensures clinicians make informed, nuanced choices. For some individuals, the model’s predictions were highly concentrated around a particular trajectory which reflects the predictive power of some factors at baseline. However, for most people, predicted trajectories had higher degrees of uncertainty (see Figure 3 for examples). Acknowledging this uncertainty in prognostic models is crucial so health professionals can weigh the potential outcomes and risks appropriately, fostering a more transparent and collaborative decision-making process. Further, the use of digital technologies for measurement-based care may also help to reduce uncertainty and improve informed decision-making with greater involvement from young people [Reference Iorfino, Cross, Davenport, Carpenter, Scott and Shiran59]. These technologies can be leveraged to deliver patient-reported outcome measures regularly to provide a more detailed understanding of individuals’ trajectories across a range of domains [Reference Iorfino, Piper, Prodan, LaMonica, Davenport and Lee60].
We recognize that this study has limitations. The data may not be missing at random as individuals were not obliged to participate in follow-up, which could lead to biases in data available for these models as well as differences in characteristics of individuals who were retained versus those who dropped out. Similarly, the specific sample characteristics of those included in this study may limit the generalizability to other youth settings where the types of disorders and comorbidities differ. The demographic and clinical predictors used in this study are not exhaustive and rely on clinician input rather than standardized instruments, so there may be critical factors omitted and potential biases introduced by clinician reports. For example, relevant missing variables include factors such as neurocognition, social cognition, and level of educational attainment, among others (e.g., cultural status and socio-economic status). Importantly, however, the factors used here are basic clinical variables that should be easily generalized to other services, which means this work is more accessible to be validated in other samples and has the potential to be scaled more rapidly. Finally, the use of SOFAS as a measure of functional impairment, while common practice in many mental health settings, may be limited in its ability to fully capture and characterize the large and multifaceted construct of functioning. Future research should look to more detailed methods of characterizing functional impairment by using multiple perspectives or modalities, questionnaires, and repeat measurements.
The use of evidence-based clinical decision-support models holds promise to transform youth mental health care, assist in identifying those at heightened risk, and inform indicated prevention and early intervention. Our development of a brief, prognostic model for functional impairment is an example of this advance and should serve as a foundation for future model development. It offers a potential means to guide individual-level decision-support about the type and intensity of intervention required to improve functional outcomes. This type of prognostic stratification is integral to the evolution toward personalized psychiatry, enhancing the efficacy of health care resource allocation and ultimately improving real-world outcomes for young people.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1192/j.eurpsy.2024.1787.
Data availability statement
Data will be available upon request.
Acknowledgements
The authors thank all the young people who have participated in this study over the years. The authors thank Sophie Lloyd-Hurwitz for her assistance with the manuscript.
Financial support
This work was supported by the Medical Research Future Fund Applied Artificial Intelligence Research in Health Grant (MRFAI000097, “Explainable machine learning to improve youth mental health care”). M.V. was supported by philanthropic funding from The Johnston Fellowship and from other donor(s) who are families affected by mental illness who wish to remain anonymous. F.I. was supported by the Bill and Patricia Richie Foundation and an NHMRC EL1 Investigator Grant (GNT2018157). W.C. is supported by an Australian Government Research Training Program (RTP) Scholarship. J.C. is supported by an NMHRC Emerging Leadership Fellowship (GNT2008197). I.B.H. is supported by a NHMRC L3 Investigator Grant (GNT2016346). The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript.
Competing interest
I.B.H. is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC), University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to headspace. He is the Chief Scientific Advisor to, and a 3.2% equity shareholder in, Innow Well Pty Ltd, which supports the transformation of mental health services internationally through the use of innovative technologies.
E.S. is Principal Research Fellow at the Brain and Mind Centre, The University of Sydney. She is Discipline Leader of Adult Mental Health, School of Medicine, University of Notre Dame, and a Consultant Psychiatrist. She was the Medical Director, Young Adult Mental Health Unit, St Vincent’s Hospital Darlinghurst until January 2021. She has received honoraria for educational seminars related to the clinical management of depressive disorders supported by Servier, Janssen, and Eli-Lilly pharmaceuticals. She has participated in a national advisory board for the antidepressant compound Pristiq, manufactured by Pfizer. She was the National Coordinator of an antidepressant trial sponsored by Servier.
All other authors declare no financial or non-financial competing interests.
Comments
No Comments have been published for this article.