Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2024-12-22T07:13:58.814Z Has data issue: false hasContentIssue false

A prognostic model for predicting functional impairment in youth mental health services

Published online by Cambridge University Press:  19 December 2024

Frank Iorfino*
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia
Rafael Oliveira
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia Data61, CSIRO, Sydney, NSW Australia
Sally Cripps
Affiliation:
Human Technology Institute, University of Technology Sydney, Sydney, NSW Australia School of Mathematical and Physical Sciences, University of Technology Sydney, Sydney, NSW, Australia
Roman Marchant
Affiliation:
Human Technology Institute, University of Technology Sydney, Sydney, NSW Australia School of Mathematical and Physical Sciences, University of Technology Sydney, Sydney, NSW, Australia
Mathew Varidel
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia
William Capon
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia
Jacob J. Crouse
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia
Ante Prodan
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia Translational Health Research Institute, Western Sydney University, Sydney, NSW Australia School of Computer, Data and Mathematical Sciences, Western Sydney University, Sydney, NSW Australia
Elizabeth M. Scott
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia
Jan Scott
Affiliation:
Academic Psychiatry, Institute of Neuroscience, Newcastle University, Newcastle, UK
Ian B. Hickie
Affiliation:
Brain and Mind Centre, The University of Sydney, Camperdown, NSW Australia
*
Corresponding author: Frank Iorfino; Email: [email protected]

Abstract

Background

Functional impairment is a major concern among those presenting to youth mental health services and can have a profound impact on long-term outcomes. Early recognition and prevention for those at risk of functional impairment is essential to guide effective youth mental health care. Yet, identifying those at risk is challenging and impacts the appropriate allocation of indicated prevention and early intervention strategies.

Methods

We developed a prognostic model to predict a young person’s social and occupational functional impairment trajectory over 3 months. The sample included 718 young people (12–25 years) engaged in youth mental health care. A Bayesian random effects model was designed using demographic and clinical factors and model performance was evaluated on held-out test data via 5-fold cross-validation.

Results

Eight factors were identified as the optimal set for prediction: employment, education, or training status; self-harm; psychotic-like experiences; physical health comorbidity; childhood-onset syndrome; illness type; clinical stage; and circadian disturbances. The model had an acceptable area under the curve (AUC) of 0.70 (95% CI, 0.56–0.81) overall, indicating its utility for predicting functional impairment over 3 months. For those with good baseline functioning, it showed excellent performance (AUC = 0.80, 0.67–0.79) for identifying individuals at risk of deterioration.

Conclusions

We developed and validated a prognostic model for youth mental health services to predict functional impairment trajectories over a 3-month period. This model serves as a foundation for further tool development and demonstrates its potential to guide indicated prevention and early intervention for enhancing functional outcomes or preventing functional decline.

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - ND
This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives licence (http://creativecommons.org/licenses/by-nc-nd/4.0), which permits non-commercial re-use, distribution, and reproduction in any medium, provided that no alterations are made and the original article is properly cited. The written permission of Cambridge University Press must be obtained prior to any commercial use and/or adaptation of the article.
Copyright
© The Author(s), 2024. Published by Cambridge University Press on behalf of European Psychiatric Association

Introduction

Functional impairment is a major concern among those presenting to youth mental health services. It typically involves disruptions to education or work, social interactions, and daily functioning, which often indicate or exacerbate serious underlying mental health issues [Reference Scott, Fowler, McGorry, Birchwood, Killackey and Christensen1]. According to the 2019 Global Burden of Disease study, around one-fifth of all disease-related disability (from all causes) is attributable to mental disorders in children, adolescents, and young adults aged 5–24 [Reference Kieling, Buchweitz, Caye, Silvani, Ameis and Brunoni2]. A further concern is that functional impairment can have a profound impact on longer-term outcomes in adulthood due to the foundational importance of adolescence and young adulthood for social, emotional, and physical development [Reference McKnight and Kashdan3]. Thus, early recognition and prevention for those at risk of functional impairment in youth mental health settings is essential to guide effective care that aims to improve immediate and long-term outcomes.

Current primary youth mental health care often falls short of effectively addressing functional impairment. Available data from these settings show that many young people engaged in care experience poor functional outcomes [Reference Iorfino, Carpenter, Cross, Crouse, Davenport and Hermens4Reference Hilferty, Cassells, Muir, Duncan, Christensen and Mitrou6], which can typically be characterized as persistently low or volatile [Reference Iorfino, Hermens, Shane, Zmicerevska, Nichles and Badcock7], and in some cases, only 37.8% experience clinically significant improvement [5]. This is critical given the bidirectional relationship between functioning and clinical symptoms. Indeed, the impact of functional impairment extends beyond the functional domain as it also predicts illness progression [Reference Iorfino, Scott, Carpenter, Cross, Hermens and Killedar8, Reference Fusar-Poli, Byrne, Valmaggia, Day, Tabraham and Johns9] and can have causal effects on suicidal thoughts and behaviors, self-harm, and psychosis-like experiences [Reference Iorfino, Varidel, Marchant, Cripps, Crouse and Prodan10]. Conversely, clinical symptoms such as depression and anxiety can directly impact young people’s participation in school and work. Effective functional recovery in youth requires interventions specifically designed to enhance engagement with education or employment, life skills, and social functioning [Reference Burns, Catty, Becker, Drake, Fioritti and Knapp11Reference Fowler, Hodgekins, French, Marshall, Freemantle and McCrone15]. Without direct intervention targeting these areas, many young people may continue to experience significant challenges in their daily lives, even if their mental health symptoms have been addressed.

An effective approach to allocate specific interventions for functional impairment involves the use of prognostic stratification. This method involves stratifying young people based on the risk of functional impairment to identify those who would benefit most from early and targeted interventions focused on functioning [Reference Delgadillo and Lutz16, Reference Fusar-Poli, Hijazi, Stahl and Steyerberg17]. This stratified approach ensures that resources and interventions are efficiently directed toward those who need them most, potentially preventing or mitigating the progression of functional impairment. This approach also allows for a more tailored, proactive approach in mental health care, focusing not only on symptom reduction but also on enhancing overall functioning and quality of life from an early stage [Reference Colizzi, Lasalvia and Ruggeri18].

Prognostic models are commonly used in other areas of medicine (e.g., breast cancer and cardiovascular disease), and there are some promising examples in mental health for the onset and course of major mood, anxiety, and psychotic disorders [Reference Nichols, Ryan, Connor, Birchwood and Marshall19Reference Iorfino, Ho, Carpenter, Cross, Davenport and Hermens30]. Though, few studies have focused on functional impairment as an outcome [Reference Koutsouleris, Kambeitz-Ilankovic, Ruhrmann, Rosen, Ruef and Dwyer31], despite its importance as a consequential outcome of interest for those with lived experience, its widespread impact on other outcomes, and the need for specific intervention [Reference Iorfino, Carpenter, Cross, Davenport, Hermens and Guastella32]. This is particularly important in youth since current diagnostic approaches fail to capture the potential longer-term impact of illness even among those with subthreshold disorders [Reference O’Dea, Glozier, Purcell, McGorry, Scott and Feilds33, Reference Iorfino, McHugh, Richards, Skinner, Prodan and an34]. Unfortunately, the range of factors involved in functional impairment makes prediction and early intervention challenging which can lead to delayed access to effective interventions and highlights the need for prognostic models to identify those at risk earlier [Reference Iorfino, Carpenter, Cross, Crouse, Davenport and Hermens4].

The goal of this study was to develop a prognostic model to predict future functional impairment among young people presenting for youth mental health care. Given the detrimental impact of functional impairment for those presenting to mental health services there is a need for tools that can be used to guide indicated prevention and early intervention efforts. Specifically, we sought to combine basic demographic and clinical information commonly available in mental health services to predict functional outcomes over the subsequent 3 months. This time period was chosen since most change occurs early in the course of care, and to align with our focus on prevention and early intervention by detecting functional change before impairment becomes entrenched.

Methods

The study was approved by the University of Sydney Human Research Ethics Committee (2008/5453, 2012/1626) and participants (and/or guardians) gave written informed consent for the use of routinely collected clinical data for research purposes. The study is reported according to the STROBE reporting guidelines.

Participants

Participants were drawn from a cohort of 6743 individuals aged 12–30 years who presented to the Brain and Mind Centre’s youth mental health clinics in Sydney, Australia, and were recruited to a research register between June 2008 and July 2018 [Reference Carpenter, Iorfino, Cross, Nichles, Zmicerevska and Crouse35]. These clinics include primary care services (i.e., headspace [Reference McGorry, Tanti, Stokes, Hickie, Carnell and Littlefield36]) as well as more specialized mental health services. All participants received ongoing clinician-based case management involving treatment planning and coordination of psychological, social, and/or medical interventions for the duration of their care with the services. This may have involved contact with a psychiatrist, psychologist, occupational therapist, support worker, or hospitalization for those whose needs exceeded the capacity of the primary care services.

Eligibility criteria

By December 2019, phase two of data entry was completed, and so baseline data were available for 2901 participants. To be eligible for the current analysis, we selected individuals who had at least one follow-up visit to the clinic within the first 3 months after their initial visit (baseline). Application of these criteria reduced the sample to 718 individuals included for analyses.

Data collection

Research staff were trained through individual and group training sessions to extract key data from clinical and research files and code inputs according to a specifically designed clinical research proforma [Reference Carpenter, Iorfino, Cross, Nichles, Zmicerevska and Crouse35]. Clinical files included all available notes and records from standard clinical care (recorded by treating clinicians), and research files included various assessments as part of participation in substudies (which may include structured or unstructured clinical interviews and the use of symptom rating scales). Data were extracted from clinical files and code inputs according to the standardized proforma which records information at predetermined timepoints. The first available clinical assessment at the service was taken as the baseline timepoint for each participant and the date of this assessment was used to determine each of the follow-up timepoints. If there was no clinical information available for any given timepoint (i.e., the participant did not attend the service during that time) then that entry was left missing. All clinical notes from the preceding timepoints, up to and including the current timepoint were used to inform and complete the current proforma entry.

Assessments

The proforma recorded specific illness characteristics. More detailed descriptions about the proforma, including the interrater reliability, are reported in the supplement and cohort paper [Reference Carpenter, Iorfino, Cross, Nichles, Zmicerevska and Crouse35]. The measures used here include (Supplementary Materials): demographics, social and occupational functioning, using the Social and Occupational Functioning Assessment Scale (SOFAS), and “Not in Education, Employment or Training” (NEET) as a measure of participation with education or work.

Mental disorder diagnoses (DSM-5) were recorded for each participant and labeled as either primary, secondary, or tertiary based, however, these were collapsed for this study. At-risk mental states, including psychotic- and manic-like experiences and circadian disturbances, were also recorded.

A clinical stage was assigned to each young person alongside their diagnoses using the established criteria [Reference Hickie, Scott, Hermens, Naismith, Guastella and Kaur37]. Young people at stage 1a (“help-seeking”) typically have fewer clinical symptoms and milder impairment than those at stage 1b (“attenuated syndrome”). Stage 2+ (stages 2, 3, and 4) includes young people with full-threshold disorders that are major, discrete, and persistent. One of three illness subtypes was also coded for each participant, neurodevelopmental-psychosis (psychotic symptoms, cognitive impairment, and/or childhood neurodevelopmental disorder), hyperarousal-anxious depression (heightened stress sensitivity and/or depressive features), or circadian-bipolar spectrum (disrupted sleep and circadian rhythms, symptoms of atypical or bipolar spectrum). “N/A” was recorded if the clinical researchers determined no clear illness subtype.

Other clinical information included self-harm, suicidal thoughts and behaviors, physical health comorbidities (e.g., diabetes and respiratory illnesses), personal mental illness history, and treatment utilization.

Statistical analyses

Analyses were conducted using Python 3.9.12. A Bayesian growth mixture model [Reference Ram and Grimm38, Reference Kim, Vermunt, Bakk, Jaki and Van Horn39] was designed to predict whether an individual’s SOFAS score would improve, deteriorate, or stay the same during the first 3 months of care, and estimate the amount of change in SOFAS score, if any, over that same period. For these purposes, we adopted a latent class linear mixed effects model where individuals were assigned to one of three fixed classes: (i) constant, (ii) improve, and (iii) decline. Individual factors were used as predictors of class membership, which is jointly inferred with the rest of the model parameters. The inference is performed following a Bayesian approach via Markov chain Monte Carlo, where we use a no-U-turn sampler (NUTS) to estimate continuous model parameters within a Gibbs sampling scheme to sample the discrete random variables (i.e., the latent class for each individual) [Reference Hoffman and Gelman40]. To estimate our results, we ran the NUTS for 6000 iterations in 2 parallel chains. The first 4000 samples of each chain were discarded as burn-in, while the remaining 2000 samples of each chain were combined to form the model posterior samples, totaling 4000 posterior samples for each model. The resulting Gelman-Rubin R-hat statistic for each parameter’s posterior distribution was below 1.1 across all models, indicating convergence of the sampling algorithm. An overview of the model is presented in Figure 1 and further details can be found in the Supplementary Materials.

Figure 1. An overview of the predictive model. An individual’s clinical and demographic characteristics serve as inputs to the cluster assignment model, which predicts the probabilities of whether an individual’s score is going to remain the same, improve, or deteriorate. Combined with the cluster’s prediction, the individual’s initial score informs a trajectory prediction model which predicts an individual’s response over time. The magenta crosses indicate the actual scores given by clinicians. Note that the trajectory model is only informed of the individual’s initial score at baseline, and not the score at their second visit, which the model must predict.

A total of 31 factors were available for these analyses. These included basic demographic and clinical factors commonly collected in mental health services when someone presents for care. We considered three different combinations of these factors for the model: (i) a model with all available factors; (ii) a model with no factors; and (iii) a model with a subset of eight factors that were deemed to be the optimal set for prediction (decided by expert knowledge and confirmatory analyses). The eight factors included in the model were NEET status, self-harm, psychotic-like experiences, physical health comorbidity, childhood-onset syndrome, illness type, clinical stage, and sleep–wake and/or circadian disturbances. Factor selection was evaluated using the Watanabe-Akaike information criterion (WAIC), a Bayesian estimate of the predictive performance of a model on unseen future data via an approximation based on the model’s posterior distribution given all available data [Reference Gelman, Carlin, Stern, Dunson, Vehtari and Rubin41, Reference Liu42]. In general, a lower WAIC value indicates a better out-of-sample predictive performance when comparing models.

We used the traditional interpretation of the c-statistic/area under the curve (AUC) to evaluate the performance of the model in two prediction tasks relevant to clinical practice. The first task consisted of predicting whether an individual’s SOFAS score would significantly drop over the course of 3 months, based solely on the information accessible at baseline. A predicted difference of 10 or more points in SOFAS was deemed significant [Reference Rickwood, Mazzer, Telford, Parker, Tanti and McGorry43]. To provide true labels for this assessment, the score at 3 months was computed based on the available data for an individual via linear extrapolation.

The second task consisted of predicting functional impairment at the next timepoint for individuals with initially good functioning (SOFAS above 70), again using only the information available at baseline. This task was designed to assist with indicated prevention and early intervention by identifying those individuals who are at risk of becoming functionally impaired.

To assess predictive performance on unseen data in both tasks, we used 5-fold cross-validation, where 80% of the data is used to estimate the model parameters posterior, and the remaining 20% is used as test data to assess the model’s predictions.

Results

Baseline characteristics for the 718 young people (mean age = 18.1 ± 3.3; 63% female) who met the criteria for this study are presented in Table 1.

Table 1. Baseline characteristics of the sample selected for the analysis (N = 718)

Each row presents the count of individuals presenting with a certain covariate in the dataset, except for the case of age, which shows the mean and standard deviation of the participants’ age. Abbreviations: NEET, not in education, employment or training; No., number of participants; SD, standard deviation.

Overall model performance

The model using all factors had the highest WAIC (WAIC = 1942; AUC = 0.65) and the model with no factors had the lowest AUC (WAIC = 1802; AUC = 0.50), indicating that each of these models was the worst performing on at least one of the evaluation metrics. By contrast, the model using the subset of eight factors performed best with the lowest WAIC (1777) and best AUC with “acceptable” predictive accuracy for predicting whether a person’s functioning would significantly drop over the next 3 months (AUC = 0.70, 95% Credible Interval [CI], 0.56–0.81; Figure 2A).

Figure 2. Model performance metrics and AUC. Panel A presents results for the overall model predicting whether an individual’s SOFAS score would significantly drop by 10 points over the course of 3 months. Panel B presents results for the model predicting functional impairment at the next consultation for individuals with initially good functioning (SOFAS above 70). Abbreviations: NPV, negative predictive value; PPV, positive predictive value.

Characteristics of change

The constant and decline clusters were comprised of individuals with similar initial SOFAS scores (constant: M = 64.1, SD = 9.0; decline: M = 64.7, SD = 7.9), though individuals in the decline cluster had a decrease of 8.8 SOFAS points on average (SD = 6.6) after 3 months. Individuals in the improve cluster started at an initially lower score (M = 58.6, SD = 8.3) and then progressed to an average increase of 7.8 SOFAS points (SD = 5.4) after 3 months. The time to a significant change in SOFAS score (10 points) was 5.0 months (95% CI, 1.7–11.0) for the improve cluster and 4.6 months (95% CI, 1.7–9.5) for the decline cluster.

The marginal probabilities for the whole sample indicate that in their first 3 months, most individuals had no change in score (58%), 16% deteriorated, and 26% improved. Being NEET shifted this marginal probability with an increase of 52% in the probability of improvement when present at baseline (Odds Ratio [OR], 1.52; 95% CI, 1.11–2.00). Conversely, a history of self-harm (OR, 1.49; 95% CI, 1.10–1.94) and having a physical health comorbidity (OR, 1.98; 95% CI, 1.43–2.69) led to a significant increase in the probability of deterioration (Supplementary Materials).

Typical cases for each cluster (i.e., individuals who maximize each cluster probability given their factors at baseline) are presented in Table 2. Typical cases across each cluster do not change very much between baseline and follow-up. The first noticeable difference between clusters is in the presence of medical and childhood history, with individuals in the decline cluster presenting with both at baseline. The typical case for each cluster also differed based on specific illness characteristics such as the type of mood disorder, presence of sleep–wake and/or circadian disturbances, clinical stage, and initial functioning.

Table 2. Baseline characteristics of typical individuals in each cluster

Predicting impairment among those with good functioning (SOFAS above 70)

The model demonstrated “excellent” accuracy for predicting functional impairment among those who present with good functioning (AUC = 0.80, 95% CI, 0.67–0.89; Figure 2B). The model has high sensitivity (0.82, 95% CI, 0.71–0.97), while maintaining high specificity (0.74, 95% CI, 0.68–0.78), indicating good discriminative power. Table 3 presents odds ratios for the posterior probabilities of drops below 70 in the follow-up visit as a function of the factors and the SOFAS score at baseline. NEET status did not emerge as a predictor, which likely reflects how many of those with a SOFAS score above 70 are engaged in employment, education, or training. Consistent with the overall model results, physical health comorbidity and self-harm were associated with the greatest increase in the probability of future functional impairment among those who are functioning well, however psychotic-like experiences and circadian disturbances were also identified as key predictors of future impairment (Table 3).

Table 3. Factors that influence a drop in SOFAS score to below 70 at the next visit to the clinics given different initial score levels

The table presents odds ratios for the probability of the next SOFAS score being below 70 computed with respect to a baseline individual with no mood disorders or any comorbidities based on the model’s posterior predictions. The 95% credible interval for the predictions is also shown within brackets. Bold text indicates significance.

Discussion

This study describes the development and preliminary validation of a brief prognostic model for functional outcomes among young people presenting for youth mental health care. Overall model performance was “acceptable” indicating its potential utility for predicting functional outcomes over 3 months, and model performance was “excellent” for identifying those at risk of deterioration from good functioning to impairment. This work has clear implications for the future development of prognostic models. These models can inform individual-level decision-making about the type and intensity of indicated prevention and early intervention required to improve functional outcomes or prevent functional decline in youth mental health.

The goal of indicated prevention in youth mental health services is to provide those at risk for poor outcomes with the appropriate supports [Reference Cuijpers, Smit and Furukawa44]. The heterogeneity of presentations makes this type of prediction difficult. Here, we demonstrate how basic demographic and clinical information could be leveraged to identify those at risk and who could benefit from specific preventative approaches needed to prevent functional impairment (e.g., vocational support and school support).

Incorporating new predictive models into clinical services represents a transformative shift in youth mental health care. Early interventions guided by prognostic models hold promise for reducing the burden of mental health disorders in young people. By identifying those at risk before the onset of severe illness, preventive strategies can be deployed earlier, when people are more likely to be responsive to interventions [Reference McGorry and van Os45]. This shift aligns with a broader trend toward proactive, individualized care that addresses mental health challenges early in their course rather than after they have progressed significantly. While further validation and testing are required before this model can be recommended for clinical practice [Reference Senior, Fanshawe, Fazel and Fazel46], the present model demonstrates utility as a prognostic model to identify those most at risk of deteriorating mental health and function [Reference Meehan, Lewis, Fazel, Fusar-Poli, Steyerberg and Stahl47].

Critical factors associated with poor trajectories of functional impairment included a history of self-harm, physical health comorbidity, circadian rhythm disruptions, and psychotic-like experiences. This is consistent with previous evidence about the complicating role of complexity and comorbidity in diagnosis, treatment, and the course of illness [Reference Firth, Siddiqi, Koyanagi, Siskind, Rosenbaum and Galletly48]. While the relationship between these factors and impairment is likely bidirectional, their predictive utility among those functioning well points to some potential critical mechanisms.

Disrupted sleep and circadian rhythms can impair cognitive and mood regulation, affecting academic and social engagement [Reference Logan and McClung49Reference Mehta51], and self-harm may result in social withdrawal and academic struggles [Reference Geulayov, Mansfield, Jindra, Hawton and Fazel52, Reference Wu, Chang, Huang, Liu and Stewart53]. Further, psychotic-like experiences, such as hallucinations or delusions, can significantly disrupt reality perception, leading to challenges in social interactions and daily activities [Reference Heinze, Lin, Nelson, Reniers, Upthegrove and Clarke54, Reference Chun, Cooper and Ellman55]. These factors collectively underscore the need for tailored, multifaceted treatment approaches to address these challenges and reiterate the importance of large-scale health service trials focused on developing service models that facilitate this level of personalized care [Reference Hickie, Iorfino, Rohleder, Song, Nichles and Zmicerevska56]. For example, behavioral interventions to improve the alignment between someone’s biological clocks with external environmental cues can foster a more regulated sleep–wake cycle [Reference Crouse, Carpenter, Song, Hockey, Naismith and Grunstein57]. This, in turn, can lead to improved mood, higher energy levels, and better cognitive functioning.

Prediction in mental health is inherently complex due to the variability in individual trajectories [Reference Varidel, Hickie, Prodan, Skinner, Marchant and Cripps58] and multiple causes of poor outcomes. This makes the communication of uncertainty vital in decision-support models, and ensures clinicians make informed, nuanced choices. For some individuals, the model’s predictions were highly concentrated around a particular trajectory which reflects the predictive power of some factors at baseline. However, for most people, predicted trajectories had higher degrees of uncertainty (see Figure 3 for examples). Acknowledging this uncertainty in prognostic models is crucial so health professionals can weigh the potential outcomes and risks appropriately, fostering a more transparent and collaborative decision-making process. Further, the use of digital technologies for measurement-based care may also help to reduce uncertainty and improve informed decision-making with greater involvement from young people [Reference Iorfino, Cross, Davenport, Carpenter, Scott and Shiran59]. These technologies can be leveraged to deliver patient-reported outcome measures regularly to provide a more detailed understanding of individuals’ trajectories across a range of domains [Reference Iorfino, Piper, Prodan, LaMonica, Davenport and Lee60].

Figure 3. Model outputs for an individual who improved over the course of 3 months (Panel A), and for another individual who deteriorated over the course of 3 months (Panel B). The “cluster” graph shows the probability of each change cluster (“constant,” “up,” and “down”) for both individuals. The “predicted trajectories” graph shows the simulated trajectories based on the cluster model and the individuals initial score (~50 for the person in panel A and ~60 for the person in panel B). “Up” trajectories are shaded green, “constant” trajectories are shaded red, and “down” trajectories are shaded blue. The dotted black line and cross show the actual observed trajectory and SOFAS score for both individuals over the follow-up period.

We recognize that this study has limitations. The data may not be missing at random as individuals were not obliged to participate in follow-up, which could lead to biases in data available for these models as well as differences in characteristics of individuals who were retained versus those who dropped out. Similarly, the specific sample characteristics of those included in this study may limit the generalizability to other youth settings where the types of disorders and comorbidities differ. The demographic and clinical predictors used in this study are not exhaustive and rely on clinician input rather than standardized instruments, so there may be critical factors omitted and potential biases introduced by clinician reports. For example, relevant missing variables include factors such as neurocognition, social cognition, and level of educational attainment, among others (e.g., cultural status and socio-economic status). Importantly, however, the factors used here are basic clinical variables that should be easily generalized to other services, which means this work is more accessible to be validated in other samples and has the potential to be scaled more rapidly. Finally, the use of SOFAS as a measure of functional impairment, while common practice in many mental health settings, may be limited in its ability to fully capture and characterize the large and multifaceted construct of functioning. Future research should look to more detailed methods of characterizing functional impairment by using multiple perspectives or modalities, questionnaires, and repeat measurements.

The use of evidence-based clinical decision-support models holds promise to transform youth mental health care, assist in identifying those at heightened risk, and inform indicated prevention and early intervention. Our development of a brief, prognostic model for functional impairment is an example of this advance and should serve as a foundation for future model development. It offers a potential means to guide individual-level decision-support about the type and intensity of intervention required to improve functional outcomes. This type of prognostic stratification is integral to the evolution toward personalized psychiatry, enhancing the efficacy of health care resource allocation and ultimately improving real-world outcomes for young people.

Supplementary material

The supplementary material for this article can be found at http://doi.org/10.1192/j.eurpsy.2024.1787.

Data availability statement

Data will be available upon request.

Acknowledgements

The authors thank all the young people who have participated in this study over the years. The authors thank Sophie Lloyd-Hurwitz for her assistance with the manuscript.

Financial support

This work was supported by the Medical Research Future Fund Applied Artificial Intelligence Research in Health Grant (MRFAI000097, “Explainable machine learning to improve youth mental health care”). M.V. was supported by philanthropic funding from The Johnston Fellowship and from other donor(s) who are families affected by mental illness who wish to remain anonymous. F.I. was supported by the Bill and Patricia Richie Foundation and an NHMRC EL1 Investigator Grant (GNT2018157). W.C. is supported by an Australian Government Research Training Program (RTP) Scholarship. J.C. is supported by an NMHRC Emerging Leadership Fellowship (GNT2008197). I.B.H. is supported by a NHMRC L3 Investigator Grant (GNT2016346). The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript.

Competing interest

I.B.H. is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC), University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to headspace. He is the Chief Scientific Advisor to, and a 3.2% equity shareholder in, Innow Well Pty Ltd, which supports the transformation of mental health services internationally through the use of innovative technologies.

E.S. is Principal Research Fellow at the Brain and Mind Centre, The University of Sydney. She is Discipline Leader of Adult Mental Health, School of Medicine, University of Notre Dame, and a Consultant Psychiatrist. She was the Medical Director, Young Adult Mental Health Unit, St Vincent’s Hospital Darlinghurst until January 2021. She has received honoraria for educational seminars related to the clinical management of depressive disorders supported by Servier, Janssen, and Eli-Lilly pharmaceuticals. She has participated in a national advisory board for the antidepressant compound Pristiq, manufactured by Pfizer. She was the National Coordinator of an antidepressant trial sponsored by Servier.

All other authors declare no financial or non-financial competing interests.

References

Scott, J, Fowler, D, McGorry, P, Birchwood, M, Killackey, E, Christensen, H, et al. Adolescents and young adults who are not in employment, education, or training. BMJ. 2013;347:f5270.CrossRefGoogle ScholarPubMed
Kieling, C, Buchweitz, C, Caye, A, Silvani, J, Ameis, SH, Brunoni, AR, et al. Worldwide prevalence and disability from mental disorders across childhood and adolescence: evidence from the global burden of disease study. JAMA Psychiatr [Internet]. 2024 Jan 31 [cited 2024 Mar 4]. Available from: 10.1001/jamapsychiatry.2023.5051.CrossRefGoogle Scholar
McKnight, PE, Kashdan, TB. The importance of functional impairment to mental health outcomes: a case for reassessing our goals in depression treatment research. Clin Psychol Rev. 2009;29(3):243.CrossRefGoogle ScholarPubMed
Iorfino, F, Carpenter, JS, Cross, SP, Crouse, J, Davenport, TA, Hermens, DF, et al. Social and occupational outcomes for young people who attend early intervention mental health services: a longitudinal study. Med J Aust. 2022;216(2):8793.CrossRefGoogle ScholarPubMed
KPMG Consulting. Evaluation of the national headspace program. Canberra, Australia: Australia government department of health, 2022; p. 1347.Google Scholar
Hilferty, F, Cassells, R, Muir, K, Duncan, A, Christensen, D, Mitrou, F, et al. Is headspace making a difference to young people’s lives? Final report of the independent evaluation of the headspace program. Sydney: Social Policy Research Centre, UNSW; 2015.Google Scholar
Iorfino, F, Hermens, DF, Shane, PC, Zmicerevska, N, Nichles, A, Badcock, CA, et al. Delineating the trajectories of social and occupational functioning of young people attending early intervention mental health services in Australia: a longitudinal study. BMJ Open. 2018;8(3):e020678.CrossRefGoogle ScholarPubMed
Iorfino, F, Scott, EM, Carpenter, JS, Cross, SP, Hermens, DF, Killedar, M, et al. Clinical stage transitions in persons aged 12 to 25 years presenting to early intervention mental health services with anxiety, mood, and psychotic disorders. JAMA Psychiatr. 2019;76(11):1167–75.CrossRefGoogle ScholarPubMed
Fusar-Poli, P, Byrne, M, Valmaggia, L, Day, F, Tabraham, P, Johns, L, et al. Social dysfunction predicts two years clinical outcome in people at ultra high risk for psychosis. J Psychiatr Res. 2010;44(5):294301.CrossRefGoogle ScholarPubMed
Iorfino, F, Varidel, M, Marchant, R, Cripps, S, Crouse, J, Prodan, A, et al. The temporal dependencies between social, emotional and physical health factors in young people receiving mental healthcare: a dynamic Bayesian network analysis. Epidemiol Psychiatr Sci. 2023;32:e56.CrossRefGoogle Scholar
Burns, T, Catty, J, Becker, T, Drake, RE, Fioritti, A, Knapp, M, et al. The effectiveness of supported employment for people with severe mental illness: a randomised controlled trial. Lancet. 2007;370(9593):1146–52.CrossRefGoogle ScholarPubMed
Hickie, IB, Scott, EM, Cross, SP, Iorfino, F, Davenport, TA, Guastella, AJ, et al. Right care, first time: a highly personalised and measurement-based care model to manage youth mental health. Med J Aust. 2019;211(S9):S346.CrossRefGoogle Scholar
Simmons, MB, Chinnery, G, Whitson, S, Bostock, S, Braybrook, J, Hamilton, M, et al. Implementing a combined individual placement and support and vocational peer work program in integrated youth mental health settings. Early Interv Psychiatry. 2023;17(4):412–21.CrossRefGoogle ScholarPubMed
Brinchmann, B, Widding-Havneraas, T, Modini, M, Rinaldi, M, Moe, CF, McDaid, D, et al. A meta-regression of the impact of policy on the efficacy of individual placement and support. Acta Psychiatr Scand. 2020;141(3):206–20.CrossRefGoogle ScholarPubMed
Fowler, D, Hodgekins, J, French, P, Marshall, M, Freemantle, N, McCrone, P, et al. Social recovery therapy in combination with early intervention services for enhancement of social recovery in patients with first-episode psychosis (SUPEREDEN3): a single-blind, randomised controlled trial. Lancet Psychiatry. 2018;5(1):4150.CrossRefGoogle ScholarPubMed
Delgadillo, J, Lutz, W. A development pathway towards precision mental health care. JAMA Psychiatry. 2020;77(9):889–90.CrossRefGoogle ScholarPubMed
Fusar-Poli, P, Hijazi, Z, Stahl, D, Steyerberg, EW. The science of prognosis in psychiatry: a review. JAMA Psychiatry. 2018;75(12):1289–97.CrossRefGoogle ScholarPubMed
Colizzi, M, Lasalvia, A, Ruggeri, M. Prevention and early intervention in youth mental health: is it time for a multidisciplinary and trans-diagnostic model for care? Int J Ment Health Syst. 2020;14(1):23.CrossRefGoogle ScholarPubMed
Nichols, L, Ryan, R, Connor, C, Birchwood, M, Marshall, T. Derivation of a prediction model for a diagnosis of depression in young adults: a matched case-control study using electronic primary care records: depression in primary care. Early Interv Psychiatry. 2018;12(3):444–55.CrossRefGoogle Scholar
Rocha, TBM, Fisher, HL, Caye, A, Anselmi, L, Arseneault, L, Barros, FC, et al. Identifying adolescents at risk for depression: a prediction score performance in cohorts based in 3 different continents. J Am Acad Child Adolesc Psychiatry. 2021;60(2):262–73.CrossRefGoogle ScholarPubMed
Birmaher, B, Merranko, JA, Goldstein, TR, Gill, MK, Goldstein, BI, Hower, H, et al. A risk calculator to predict the individual risk of conversion from subthreshold bipolar symptoms to bipolar disorder I or II in youth. J Am Acad Child Adolesc Psychiatry. 2018;57(10):755763.e4.CrossRefGoogle ScholarPubMed
Birmaher, B, Merranko, JA, Gill, MK, Hafeman, D, Goldstein, T, Goldstein, B, et al. Predicting personalized risk of mood recurrences in youths and young adults with bipolar spectrum disorder. J Am Acad Child Adolesc Psychiatry. 2020;59(10):1156–64.CrossRefGoogle Scholar
Hafeman, DM, Merranko, J, Goldstein, TR, Axelson, D, Goldstein, BI, Monk, K, et al. Assessment of a person-level risk calculator to predict new-onset bipolar spectrum disorder in youth at familial risk. JAMA Psychiatry. 2017;74(8):841.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Ernala, SK, Rizvi, AF, Arenare, E, Van Meter, AR, De Choudhury, M, et al. Detecting relapse in youth with psychotic disorders utilizing patient-generated and patient-contributed digital data from Facebook. Npj Schizophr. 2019;5(1):17.CrossRefGoogle ScholarPubMed
Birnbaum, ML, Kulkarni, PP, Van Meter, A, Chen, V, Rizvi, AF, Arenare, E, et al. Utilizing machine learning on internet search activity to support the diagnostic process and relapse detection in young individuals with early psychosis: feasibility study. JMIR Ment Health. 2020;7(9):e19348.CrossRefGoogle Scholar
Cannon, TD, Cadenhead, K, Cornblatt, B, Woods, SW, Addington, J, Walker, E, et al. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch Gen Psychiatry. 2008;65(1):28.CrossRefGoogle ScholarPubMed
Ruhrmann, S, Schultze-Lutter, F, Salokangas, RKR, Heinimaa, M, Linszen, D, Dingemans, P, et al. Prediction of psychosis in adolescents and young adults at high risk: results from the prospective European prediction of psychosis study. Arch Gen Psychiatry. 2010;67(3):241.CrossRefGoogle Scholar
Ziermans, T, Wit, SD, Schothorst, P, Sprong, M, Engeland, HV, Kahn, R, et al. Neurocognitive and clinical predictors of long-term outcome in adolescents at ultra-high risk for psychosis: a 6-year follow-up. PLoS One. 2014;9(4):e93994.CrossRefGoogle ScholarPubMed
Ge, F, Li, Y, Yuan, M, Zhang, J, Zhang, W. Identifying predictors of probable posttraumatic stress disorder in children and adolescents with earthquake exposure: a longitudinal study using a machine learning approach. J Affect Disord. 2020;264:483–93.CrossRefGoogle ScholarPubMed
Iorfino, F, Ho, N, Carpenter, JS, Cross, SP, Davenport, TA, Hermens, DF, et al. Predicting self-harm within six months after initial presentation to youth mental health services: a machine learning study. PLoS One. 2020;15(12):e0243467.CrossRefGoogle ScholarPubMed
Koutsouleris, N, Kambeitz-Ilankovic, L, Ruhrmann, S, Rosen, M, Ruef, A, Dwyer, DB, et al. Prediction models of functional outcomes for individuals in the clinical high-risk state for psychosis or with recent-onset depression: a multimodal, multisite machine learning analysis. JAMA Psychiatry. 2018;75(11):1156–72.CrossRefGoogle ScholarPubMed
Iorfino, F, Carpenter, JS, Cross, SP, Davenport, TA, Hermens, DF, Guastella, AJ, et al. Multidimensional outcomes in youth mental health care: what matters and why? 2019;211 (Supplement 9):S4–11.Google Scholar
O’Dea, B, Glozier, N, Purcell, R, McGorry, PD, Scott, J, Feilds, KL, et al. A cross-sectional exploration of the clinical characteristics of disengaged (NEET) young people in primary mental healthcare. BMJ Open. 2014;4(12):e006378.CrossRefGoogle ScholarPubMed
Iorfino, F, McHugh, C, Richards, M, Skinner, A, Prodan, A, an, Occhipinti J, et al. Patterns of emergency department presentations for a youth mental health cohort: data-linkage cohort study. BJPsych Open. 2023;9(5):e170.CrossRefGoogle ScholarPubMed
Carpenter, JS, Iorfino, F, Cross, S, Nichles, A, Zmicerevska, N, Crouse, JJ, et al. Cohort profile: the brain and mind centre optymise cohort: tracking multidimensional outcomes in young people presenting for mental healthcare. BMJ Open. 2020;10(3):e030985.CrossRefGoogle Scholar
McGorry, PD, Tanti, C, Stokes, R, Hickie, IB, Carnell, K, Littlefield, LK, et al. Headspace: Australia’s National Youth Mental Health Foundation—where young minds come first. Med J Aust. 2007;187(S7):S68–70.CrossRefGoogle ScholarPubMed
Hickie, IB, Scott, EM, Hermens, DF, Naismith, SL, Guastella, AJ, Kaur, M, et al. Applying clinical staging to young people who present for mental health care. Early Interv Psychiatry. 2013;7(1):3143.CrossRefGoogle Scholar
Ram, N, Grimm, KJ. Methods and measures: growth mixture modeling: a method for identifying differences in longitudinal change among unobserved groups. Int J Behav Dev. 2009;33(6):565–76.CrossRefGoogle Scholar
Kim, M, Vermunt, J, Bakk, Z, Jaki, T, Van Horn, ML. Modeling predictors of latent classes in regression mixture models. Struct Equ Model Multidiscip J. 2016;23(4):601–14.CrossRefGoogle Scholar
Hoffman, M, Gelman, A. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J Mach Learn Res. 2011;15:15931623.Google Scholar
Gelman, A, Carlin, JB, Stern, HS, Dunson, DB, Vehtari, A, Rubin, DB. Introduction to multiparameter models. In: Bayesian data analysis. 3rd ed. US: CRC Press; 2020. p. 7998.Google Scholar
Liu, JS. Peskun’s theorem and a modified discrete-state Gibbs sampler. Biometrika. 1996;83(3):681–2.CrossRefGoogle Scholar
Rickwood, DJ, Mazzer, KR, Telford, NR, Parker, AG, Tanti, CJ, McGorry, PD. Changes in psychological distress and psychosocial functioning in young people visiting headspace centres for mental health problems. Med J Aust. 2015;202(10):537–42.CrossRefGoogle Scholar
Cuijpers, P, Smit, F, Furukawa, TA. Most at-risk individuals will not develop a mental disorder: the limited predictive strength of risk factors. World Psychiatry. 2021;20(2):224–5.CrossRefGoogle Scholar
McGorry, P, van Os, J. Redeeming diagnosis in psychiatry: timing versus specificity. Lancet. 2013;381(9863):343–5.CrossRefGoogle ScholarPubMed
Senior, M, Fanshawe, T, Fazel, M, Fazel, S. Prediction models for child and adolescent mental health: a systematic review of methodology and reporting in recent research. JCPP Adv. 2021;1(3):e12034.CrossRefGoogle ScholarPubMed
Meehan, AJ, Lewis, SJ, Fazel, S, Fusar-Poli, P, Steyerberg, EW, Stahl, D, et al. Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges. Mol Psychiatry. 2022;27(6):2700–8.CrossRefGoogle ScholarPubMed
Firth, J, Siddiqi, N, Koyanagi, A, Siskind, D, Rosenbaum, S, Galletly, C, et al. The Lancet Psychiatry Commission: a blueprint for protecting physical health in people with mental illness. Lancet Psychiatry. 2019;6(8):675712.CrossRefGoogle ScholarPubMed
Logan, RW, McClung, CA. Rhythms of life: circadian disruption and brain disorders across the lifespan. Nat Rev Neurosci. 2019;20(1):4965.CrossRefGoogle ScholarPubMed
Alhola, P, Polo-Kantola, P. Sleep deprivation: impact on cognitive performance. Neuropsychiatr Dis Treat. 2007;3(5):553–67.Google ScholarPubMed
Mehta, KJ. Effect of sleep and mood on academic performance—at interface of physiology, psychology, and education. Humanit Soc Sci Commun. 2022;9(1):113.CrossRefGoogle Scholar
Geulayov, G, Mansfield, K, Jindra, C, Hawton, K, Fazel, M. Loneliness and self-harm in adolescents during the first national COVID-19 lockdown: results from a survey of 10,000 secondary school pupils in England. Curr Psychol [Internet]. 2022 Sep 15 [cited 2024 Jan 30]. Available from: 10.1007/s12144-022-03651-5CrossRefGoogle Scholar
Wu, CY, Chang, CK, Huang, HC, Liu, SI, Stewart, R. The association between social relationships and self-harm: a case–control study in Taiwan. BMC Psychiatry. 2013;13:101.CrossRefGoogle ScholarPubMed
Heinze, K, Lin, A, Nelson, B, Reniers, RLEP, Upthegrove, R, Clarke, L, et al. The impact of psychotic experiences in the early stages of mental health problems in young people. BMC Psychiatry. 2018;18(1):214.CrossRefGoogle ScholarPubMed
Chun, CA, Cooper, S, Ellman, LM. Associations of psychotic-like experiences, related symptoms, and working memory with functioning. Eur Psychiatry. 63(1):e20.CrossRefGoogle Scholar
Hickie, IB, Iorfino, F, Rohleder, C, Song, YJC, Nichles, A, Zmicerevska, N, et al. EMPOWERED trial: protocol for a randomised control trial of digitally supported, highly personalised and measurement-based care to improve functional outcomes in young people with mood disorders. BMJ Open. 2023;13(10):e072082.CrossRefGoogle Scholar
Crouse, JJ, Carpenter, JS, Song, YJC, Hockey, SJ, Naismith, SL, Grunstein, RR, et al. Circadian rhythm sleep-wake disturbances and depression in young people: implications for prevention and early intervention. Lancet Psychiatry. 2021;8(9):813–23.CrossRefGoogle Scholar
Varidel, M, Hickie, IB, Prodan, A, Skinner, A, Marchant, R, Cripps, S, et al. Dynamic learning of individual-level suicidal ideation trajectories to enhance mental health care. NPJ Ment Health Res. 2024;3(1):26.CrossRefGoogle ScholarPubMed
Iorfino, F, Cross, SP, Davenport, T, Carpenter, JS, Scott, E, Shiran, S, et al. A digital platform designed for youth mental health services to deliver personalized and measurement-based care. Front Psychiatry. 2019;10:595.CrossRefGoogle ScholarPubMed
Iorfino, F, Piper, SE, Prodan, A, LaMonica, HM, Davenport, TA, Lee, GY, et al. Using digital technologies to facilitate care coordination between youth mental health services: a guide for implementation. Front Health Serv. 2021;1:745456.CrossRefGoogle ScholarPubMed
Figure 0

Figure 1. An overview of the predictive model. An individual’s clinical and demographic characteristics serve as inputs to the cluster assignment model, which predicts the probabilities of whether an individual’s score is going to remain the same, improve, or deteriorate. Combined with the cluster’s prediction, the individual’s initial score informs a trajectory prediction model which predicts an individual’s response over time. The magenta crosses indicate the actual scores given by clinicians. Note that the trajectory model is only informed of the individual’s initial score at baseline, and not the score at their second visit, which the model must predict.

Figure 1

Table 1. Baseline characteristics of the sample selected for the analysis (N = 718)

Figure 2

Figure 2. Model performance metrics and AUC. Panel A presents results for the overall model predicting whether an individual’s SOFAS score would significantly drop by 10 points over the course of 3 months. Panel B presents results for the model predicting functional impairment at the next consultation for individuals with initially good functioning (SOFAS above 70). Abbreviations: NPV, negative predictive value; PPV, positive predictive value.

Figure 3

Table 2. Baseline characteristics of typical individuals in each cluster

Figure 4

Table 3. Factors that influence a drop in SOFAS score to below 70 at the next visit to the clinics given different initial score levels

Figure 5

Figure 3. Model outputs for an individual who improved over the course of 3 months (Panel A), and for another individual who deteriorated over the course of 3 months (Panel B). The “cluster” graph shows the probability of each change cluster (“constant,” “up,” and “down”) for both individuals. The “predicted trajectories” graph shows the simulated trajectories based on the cluster model and the individuals initial score (~50 for the person in panel A and ~60 for the person in panel B). “Up” trajectories are shaded green, “constant” trajectories are shaded red, and “down” trajectories are shaded blue. The dotted black line and cross show the actual observed trajectory and SOFAS score for both individuals over the follow-up period.

Supplementary material: File

Iorfino et al. supplementary material

Iorfino et al. supplementary material
Download Iorfino et al. supplementary material(File)
File 188.3 KB
Submit a response

Comments

No Comments have been published for this article.