1. Introduction
Young adults seeking help at psychiatric services frequently suffer from adult attention-deficit/hyperactivity disorder (ADHD) or an at-risk mental state (ARMS) for psychosis. Although the latter is not yet accepted as an official diagnosis, it is increasingly identified and treated in clinical practice for the following reasons. First, it is now well established that patients who meet ARMS criteria not only have a largely increased risk of developing psychosis, but also suffer from psychopathological symptoms and impaired psychosocial functioning requiring clinical attention [Reference Ruhrmann, Schultze-Lutter and Klosterkötter1]. Second, although comorbidity with other psychiatric disorders, particularly with affective and anxiety disorders, is high [Reference Fusar-Poli, Nelson, Valmaggia, Yung and McGuire2], the specific psychopathology of ARMS patients is not adequately addressed by existing diagnostic categories [3, Reference Fusar-Poli, Carpenter, Woods and McGlashan4]. Consequently, the European Psychiatric Association (EPA) has recently issued evidence based recommendations for the early detection [Reference Schultze-Lutter, Michel, Schmidt, Schimmelmann, Maric and Salokangas5] and treatment [Reference Schmidt, Schultze-Lutter, Schimmelmann, Maric, Salokangas and Riecher-Rössler6] of these patients. Furthermore, attenuated psychosis syndrome, which was defined according to the most frequent ARMS criterion, has been placed in Section 3 of the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) [Reference American Psychiatric Association7] as a new disorder for further study.
The distinction between adult ADHD and ARMS for psychosis can be a challenging task for clinicians because these disorders have many overlapping signs and symptoms, such as difficulties concentrating, lack of attention and other cognitive deficits [Reference Hauser, Zhang, Sheridan, Burdick, Mogil and Kane8, Reference Balint, Czobor, Komlosi, Meszaros, Simon and Bitter9], disorganized behavior, performance problems in school or at work, and problems with relationships [Reference Fusar-Poli, Rocchetti, Sardella, Avila, Brandizzi and Caverzasi10, Reference Nijmeijer, Minderaa, Buitelaar, Mulligan, Hartman and Hoekstra11]. Additionally, both disorders can present with restlessness, nervousness, irritability, hypersensitivity, sudden lack of interest, initiative, energy and drive, low frustration tolerance, as well as poor resilience to stress [Reference Levy, Traicu, Iyer, Malla and Joober12, Reference Corbisiero, Riecher-Rossler, Buchli-Kammermann and Stieglitz13]. Furthermore, both disorders frequently occur in early adulthood and both are considered neurodevelopmental in origin [Reference Spencer, Biederman and Mick14, Reference Owen, O'Donovan, Thapar and Craddock15]. Moreover, ADHD in childhood has been found to be associated with an increased risk of schizophrenia in adulthood both in prospective follow-up [Reference Dalsgaard, Mortensen, Frydenberg, Maibing, Nordentoft and Thomsen16, Reference Shyu, Yuan, Lee, Yang, Yang and Lee17] and retrospective studies [Reference Rubino, Frank, Croce Nanni, Pozzi, Lanza di Scalea and Siracusano18, Reference Rho, Traicu, Lepage, Iyer, Malla and Joober19]. Likewise, individuals genetically at-risk for schizophrenia [Reference Keshavan, Montrose, Rajarethinam, Diwadkar, Prasad and Sweeney20, Reference Keshavan, Sujata, Mehra, Montrose and Sweeney21] and patients with psychosis [Reference Karatekin, White and Bingham22] have been found to more frequently demonstrate ADHD-like features than healthy controls. It has been suggested that the overlap between ADHD and psychotic disorders is due to shared genetic [Reference Hamshere, Stergiakouli, Langley, Martin, Holmans and Kent23] and environmental risk factors, particularly obstetric complications [Reference Peralta, de Jalon, Campos, Zandio, Sanchez-Torres and Cuesta24]. Another potential explanation is that – due to their similar clinical manifestation − ADHD is a frequent misdiagnosis of early signs of psychotic disorders [Reference Shyu, Yuan, Lee, Yang, Yang and Lee17].
Missclassifying patients with an ARMS for psychosis as ADHD patients can lead to inappropriate and even potentially harmful treatment of these patients. Specifically, since ADHD is most commonly treated with stimulant drugs which exert their pharmacological effects via increasing the levels of dopamine [Reference Arnsten25], and since increased levels of synaptic dopamine are implicated in the generation of psychotic symptoms [Reference Howes, Kambeitz, Kim, Stahl, Slifstein and Abi-Dargham26], treating ARMS patients as ADHD patients could potentially exacerbate (pre-)psychotic symptoms in these patients [Reference Kraemer, Uekermann, Wiltfang and Kisfor review, see Ref. 27].
A possible way to improve differential diagnosis between these two disorders is to take the neuropsychological performance profile into account. Although current meta-analyses indicate that − compared to healthy controls (HC) – both adult ADHD and ARMS for psychosis patients show impairments across a wide range of cognitive domains, the degree and pattern of impairments seem to differ. While adult ADHD patients were found to be most strongly impaired in sustained and selective attention, inhibition, and verbal learning and memory with effect sizes in the small to medium range [9, Reference Mowinckel, Pedersen, Eilertsen and Biele28, Reference Skodzik, Holling and Pedersen29]. ARMS patients showed the largest impairments in speed of processing, social cognition and verbal learning with, however, mostly small effect sizes [Reference Hauser, Zhang, Sheridan, Burdick, Mogil and Kane8].
However, to our knowledge, no study has directly tested cognitive performance differences between adult ADHD and ARMS patients. Furthermore, it is currently unknown whether potential differences on the group level could be exploited to facilitate diagnostic classification on the individual level by means of automated pattern recognition or machine learning methods. This is unfortunate because neuropsychological testing is already routinely conducted in early detection services for both disorders and has shown promise in the classification between ADHD and various other psychiatric disorders, including mood and anxiety disorders [Reference Fasmer, Mjeldheim, Forland, Hansen, Syrstad and Oedegaard30–Reference Holst and Thorell33], autism spectrum disorders [Reference Groom, Young, Hall, Gillott and Hollis34], and borderline personality disorders [Reference van Dijk, Schellekens, van den Broek, Kan, Verkes and Buitelaar35], and between ARMS for psychosis and depressive disorders [Reference Schulze, Zimmermann, Gschwandtner, Pflueger, Rapp and Studerus36].
Thus, the aim of this study was 1) to directly test neuropsychological performance differences between adult ADHD, ARMS for psychosis patients, and HC in the domains of sustained attention and impulsivity, verbal learning and memory, and problem solving abilities and 2) to estimate the classification accuracy of machine learning model predicting group membership from all neurocognitive performance measures combined.
Based on the above mentioned meta-analyses [Reference Hauser, Zhang, Sheridan, Burdick, Mogil and Kane8, Reference Balint, Czobor, Komlosi, Meszaros, Simon and Bitter9, Reference Mowinckel, Pedersen, Eilertsen and Biele28, Reference Skodzik, Holling and Pedersen29], we hypothesized that both patient groups would show worse cognitive performance in all tested domains relative to healthy controls and that adult ADHD patients would show larger deficits than ARMS patients in the domains of attention and impulsivity and verbal learning and memory, but not problem solving abilities. Furthermore, we expected that classification between adult ADHD and ARMS patients based on all neuropsychological performance measures combined can be achieved with moderate to high accuracy.
2. Methods
2.1. Study design
In this cross-sectional study, the cognitive performance of three groups was compared: 1) patients with adult ADHD, 2) patients with an ARMS, and 3) healthy controls (HC). All participants provided their written informed consent. The study was approved by the by the local ethics committee (Ethikkommission der Nordwest- und Zentralschweiz) and conformed to the Declaration of Helsinki.
2.2. Recruitment of patients with adult ADHD
Patients with adult ADHD were recruited via the ADHD Special Consultations Unit of the Outpatient Department of the University of Basel Psychiatric Hospital between 2010 and 2014. All referrals to this Unit underwent an extensive ADHD screening procedure conducted by two independent experts and including a clinical interview, a self-rating and an observer-rating scale. School certificates and/or reports from teachers on behavioral problems were also considered. The procedure conformed to general standards for clinical diagnostics and followed the recommendations for the diagnostics and management of ADHD [37]. The diagnosis was not made solely on the basis of rating scales, but it also took into account a full developmental and psychiatric history. Instruments applied in the diagnostic process were the Wender-Reimherr Adult Attention Deficit Disorder Rating Scale (WRAADDS) [Reference Wender38] (German version: [39, Reference Rosler, Retz, Retz-Junginger, Stieglitz, Kessler and Reimherr40]) and the Conners’ Adult ADHD Rating Scales (CAARS) [Reference Conners, Erhardt and Sparrow41]. Former symptoms in childhood were assessed systematically with the short version of the Wender Utah Rating Scale (WURS-k) [Reference Ward, Wender and Reimherr42] (German version: [Reference Retz-Junginger, Retz, Blocher, Weijers, Trott and Wender43]). To check for current symptoms, a combination of the results from the clinical interview and the rating scales was used. An ADHD diagnosis was given when an individual met at least six criteria of the dimension of inattention and/or six criteria of the dimension of hyperactivity/impulsivity according to DSM-IV. Only patients who met the diagnostic criteria for ADHD and who were at least 18 years old were included into this study. Exclusion criteria were an intelligence quotient (IQ) <85, schizophrenia or other psychotic disorders, a current or most recent episode of a manic disorder or current severe major depressive disorder, acute stress disorder, or substance intoxication or withdrawal.
2.3. Recruitment of patients with an ARMS for psychosis
ARMS patients were recruited and assessed between March 1, 2000 and December 31, 2016 as part of the Basel Früherkennung von Psychosen (FePsy) study, a prospective multilevel study, which aims to improve the early detection of psychosis. A detailed description of the study design can be found elsewhere [44, Reference Riecher-Rössler, Pflueger, Aston, Borgwardt, Brewer and Gschwandtner45]. In brief, individuals suspected to be in their early (prodromal) phase of psychosis were referred to our specialized early detection clinic at the Outpatient Department of the Psychiatric University Hospital Basel, Switzerland. All referrals to the clinic during the study period were screened with the Basel Screening Instrument for Psychosis (BSIP) [Reference Riecher-Rössler, Aston, Ventura, Merlo, Borgwardt and Gschwandtner46], which has been specifically designed to identify patients with an ARMS for psychosis or first episode of psychosis (FEP). Individuals were classified as being in an ARMS if they met one of the following risk criteria: (a) attenuated psychotic symptoms (APS) or brief limited intermittent psychotic symptoms (BLIPS) according to the PACE criteria [Reference Yung, Phillips, McGorry, McFarlane, Francey and Harrigan47], (b) familial aggregation of psychotic disorders in combination with at least two further risk factors similar to the PACE criteria, or (c) a minimal amount and combination of certain risk factors according to the BSIP [Reference Riecher-Rössler, Aston, Ventura, Merlo, Borgwardt and Gschwandtnerfor details, see Ref. 46]. The BSIP has been shown to have a good interrater reliability (κ = 0.67) for the assessment of the ARMS and a high predictive validity [Reference Riecher-Rössler, Aston, Ventura, Merlo, Borgwardt and Gschwandtner46]. Exclusion criteria were: age <18 years, insufficient knowledge of German, IQ below 85, previous episode of schizophrenic psychosis, psychosis clearly due to organic reasons or substance abuse, or psychotic symptomatology within a clearly diagnosed affective psychosis or borderline personality disorder. Patients who were treated with antipsychotics >3 weeks or who had exceeded a 2500 mg cumulative chlorpromazine equivalent dose were also excluded.
2.4. Recruitment of healthy controls (HC)
HC were also recruited and assessed as part of the FePsy study. They were recruited from trade schools, hospital staff and through advertisements. Subjects were excluded if they had a current or former psychiatric disorder or neurological disease, serious medical condition, substance abuse, or a family history of psychiatric disorder.
2.5. Neuropsychological assessment
Patients with adult ADHD were assessed at the ADHD Special Consultations Unit and patients with an ARMS for psychosis and HC were assessed at the early detection of psychosis clinic with neuropsychological test batteries that were overlapping in the following tests: 1) Continuous Performance Test (CPT) [Reference Knye, Roth, Westhus and Heine48], 2) California Verbal Learning Test (CVLT) [Reference Delis, Kramer, Kaplan and Thompkins49] and 3) the computer administered Tower of Hanoi (ToH) [Reference Gediga and Schöttke50]. A detailed description of these tests is provided in the online Supplementary material.
2.6. Statistical analysis
All data were analyzed using the R language and environment for statistical computing [Reference R Core Team51]. For testing differences between adult ADHD, ARMS and HC groups on sociodemographic variables, Pearson’s χ2-tests and analysis of variance (ANOVA) were used for categorical and continuous variables, respectively.
Group differences in neuropsychological performance measures were analyzed using generalized linear models (GLM). GLMs are flexible generalizations of the ordinary linear regression, which differ from them in two major aspects: First, the distribution of the response variable can be non-normal and/or non-continuous (e.g., it can be binary, ordered categorical, a count etc.). Second, the dependent variable values are predicted from a linear combination of predictor variables, which are connected to the dependent variable via a link function (e.g. logit, log, inverse etc.). Thus, GLMs can take advantage of the specific structure of each variable and respect natural boundaries and distributions of variables when making predictions [for applied examples of GLMs with neuropsychological data, see Refs. Reference Brown, Vinogradov, Kremen, Poole, Deicken and Penner52, Reference Freedman, Bao, Kremen, Vinogradov, McKeague and Brown53].
For the number of omissions and commissions in the CPT, we used negative binomial models with log-link function because these are count variables (i.e. only non-negative integer values are possible). We used negative binomial instead of standard Poisson regression for these measures because it allowed handling overdispersion [Reference Payne, Hardin, Egede, Ramakrishnan, Selassie and Gebregziabher54]. For CVLT performance measures, we used binomial models with logit link function, because each measure quantified the number of words that were remembered from one or several lists of words with fixed length. For the reaction time in the CPT and the number of moves and time to complete in the ToH, Gamma regression models with an inverse link function were a natural choice because these variables were non-negative and right-skewed.
To control for confounding effects, all GLMs included sex and age as covariates. We first tested for overall group differences using likelihood ratio tests and then performed all possible pairwise comparisons using the ghlt function in the R multcomp package [Reference Bretz, Hothorn and Westfall55]. P-values were adjusted for multiple testing using the Benjamini-Hochberg procedure [Reference Benjamini, Drai, Elmer, Kafkafi and Golani56] both across all pairwise group comparisons and neurocognitive variables.
To find out whether all neurocognitive performance measures combined can successfully discriminate between ARMS and ADHD patients on an individual level, we additionally analyzed our data with a supervised machine learning method. Specifically, a random forest model was trained and tested that predicted group membership based on the individual performance across all neurocognitive variables. We chose the random forest algorithm because it can automatically 1) accommodate non-linear relationships between predictor and outcome variables, 2) capture complex interactions between predictors and 3) deal with highly correlated predictors, outliers and missing data [Reference Strobl, Malley and Tutz57]. Furthermore, it has been shown to achieve excellent predictive accuracy over a wide of range predictive tasks even without hyperparameter tuning [Reference Fernandez-Delgado, Cernadas, Barro and Amorim58]. To eliminate the influence of age and sex, we first created two equally sized samples of ADHD and ARMS patients that were matched on age and sex using propensity score matching as implemented in the matching package for R [Reference Sekhon59]. Next, a random forest model for binary outcomes was trained and tested using 10-fold cross validation with 10 repetitions. The discriminative ability of the model was assessed with the cross-validated area under the receiver operating characteristic curve (AUC), as well as sensitivity and specificity. Model fitting and performance evaluation was conducted with the R packages randomForest [Reference Liaw and Wiener60] and mlr [Reference Bischl, Lang, Kotthoff, Schiffner, Richter and Studerus61], respectively. We additionally assessed the predictive importance of each variable by looking at how much the accuracy decreased when the variable was randomly permuted using the importance function in package randomForest [Reference Liaw and Wiener60].
3. Results
3.1. Sample description
During the above described study periods, 123 adult ADHD patients, 168 ARMS patients, and 109 HC fulfilled inclusion criteria and had completed at least one of the above described neuropsychological tests. Eleven HC and 23 ARMS patients had to be excluded from the current study due to completely missing neuropsychological assessment. The excluded patients did not differ from the included patients with regard to sociodemographic characteristics. The ADHD, ARMS and HC groups did not differ with regard to gender ratios. However, adult ADHD patients were significantly older than ARMS patients and HC (see Table 1 for socio-demographic sample characteristics).
Table 1 Socio-demographic sample characteristics.
HC = healthy controls; ARMS = at-risk mental state for psychosis; ADHD = attention-deficit/hyperactivity disorder.
3.2. Group differences in neuropsychological performance
Standardized mean differences in neuropsychological performance from HC for the ARMS and ADHD groups are displayed in Fig. 1. Means and SDs, as well as p-values for tests of overall and pairwise group differences are provided in Table 2.
Fig 1. Standardized mean differences of cognitive performance of ARMS and ADHD patients compared to healthy controls. Variables with a minus sign were reversed so that higher scores always represent better performance. Differences are adjusted for the influence of age and gender.
Table 2 Cognitive performance of ADHD and ARMS patients
CVLT = California Verbal Learning Task; CPT = Continuous Performance Task; ToH = Tower of Hanoi; HC = Healthy controls; ADHD = Attention-Deficit/Hyperactivity Disorder; ARMS = At-risk mental state for psychosis
All group comparisons are adjusted for the influence of sex and age and corrected for multiple testing using the Benjamini-Hochberg procedure
* p < 0.05.
** p < 0.01.
*** p < 0.001.
When adjusted for the influence age and sex and corrected for multiple comparisons, there were overall group differences in all tested neuropsychological variables, except for the number of moves in the 5 disc ToH. Pairwise group comparisons revealed that ADHD patients showed a significantly worse performance than HC in all CPT and CVLT measures, but not in any ToH measure. ARMS patients also performed worse than HC in all performance scores, except for List 1, Trial 1 recall and Long Delay False Alarms in the CVLT and number of moves in the 5 disc ToH. Finally, ADHD patients performed significantly worse than ARMS patients in all measures except the two ToH measures. Repeating the analyses without covariate adjustment did not change results except that ARMS patients additionally showed a significantly worse performance than HC in List 1, Trial 1 recall of the CVLT (p = 0.049). As can be seen in Fig. 1, cognitive impairments were mostly of medium effect sizes in adult ADHD patients and of small effect sizes in ARMS patients (see Fig. 1).
3.3. Discrimination between ARMS and ADHD patients using machine learning
The propensity score matching procedure selected two subgroups of ADHD and ARMS each consisting of 101 participants. The two subgroups were no longer significantly different with regard to age (ADHD: 29.0 ± 8.2 years, ARMS: 28.3 ± 7.9 years) and sex (ADHD: 34.7% Women; ARMS: 34.7% Women). The random forest algorithm was able to discriminate the two matched groups with relatively high accuracy based on all neurocognitive variables combined (cross-validated AUC = 0.82, balanced accuracy = 0.75, sensitivity = 0.73, and specificity = 0.77). The variable importance assessment revealed that CVLT long delay free recall, CPT reaction time, and CVLT recognition hits were the most important predictor variables for discriminating ADHD from ARMS patients (see Fig. 2).
Fig 2. Variable importance for predicting group membership (ARMS vs. ADHD) with random forest.
4. Discussion
To our knowledge, this is the first study aiming to discriminate adult ADHD from ARMS for psychosis patients based on neuropsychological performance. In line with our hypotheses, we could demonstrate that both patient groups show significantly worse cognitive performance than HC in the domains of sustained attention and impulsivity and verbal learning and memory and that adult ADHD patients show larger deficits in these domains than ARMS patients. Accordingly, we could demonstrate that a machine learning algorithm (i.e. random forest) can predict group membership (ADHD vs. ARMS) with relatively high accuracy solely based on the individual pattern of cognitive deficits.
Our findings of small to medium sized cognitive deficits in adult ADHD patients are largely consistent with the most recent meta-analyses on cognitive deficits in adult ADHD patients [9, Reference Mowinckel, Pedersen, Eilertsen and Biele28, Reference Skodzik, Holling and Pedersen29]. Furthermore, our results confirmed that deficits of attention in general and sustained attention in particular, as measured by the CPT, represent one of the core features of ADHD. However, in contrast to the most recent meta-analysis of the CPT [Reference Mowinckel, Pedersen, Eilertsen and Biele28], which reported the strongest effect sizes for the number of omissions and commissions and no statistically significant impairment with regard to reaction times, we observed significant impairments in all performance scores of the CPT, with reaction time being the most strongly impaired. This difference might be explained by the fact that the most commonly used CPT version, the Conner’s CPT [Reference Conners, Erhardt and Sparrow41], has 90% target trials with responses required for any letter except “X”, whereas the CPT version used in this study has only 25% target trials with responses required whenever an “O” is followed by an “X”. Our results also support a recent meta-analysis on memory performance in adult ADHD patients, which found that these patients show moderate impairments in verbal learning and memory resulting from deficits in memory acquisition, but not retrieval problems [Reference Skodzik, Holling and Pedersen29].
The cognitive performance deficits that we found in our sample of ARMS patients are also largely consistent with those reported in recent meta-analyses [8, Reference Bora, Lin, Wood, Yung, McGorry and Pantelis62], which have detected impairments of mostly small effects sizes in the domains of sustained attention, verbal learning and memory and executive functions.
In accordance with our hypothesis and reported effect sizes of meta-analyses on adult ADHD [Reference Balint, Czobor, Komlosi, Meszaros, Simon and Bitter9, Reference Mowinckel, Pedersen, Eilertsen and Biele28, Reference Skodzik, Holling and Pedersen29] and ARMS patients [Reference Hauser, Zhang, Sheridan, Burdick, Mogil and Kane8, Reference Bora, Lin, Wood, Yung, McGorry and Pantelis62], we could demonstrate that adult ADHD patients are significantly more impaired than ARMS patients in the domains sustained attention and impulsivity and verbal learning and memory, but not in problem solving abilities. Furthermore, we could show that these differences on the group level can be exploited to make meaningful prediction on the individual level. Specifically, by applying a machine learning algorithm to all neurocognitive variables combined we were able to discriminate ADHD from ARMS patients with a cross-validated sensitivity of 0.73 and specificity of 0.77. Thus, our results clearly support the notion that neuropsychological testing could facilitate the differential diagnosis between these two disorders. However, it should be noted that our results do not indicate that the pattern of cognitive deficits alone could predict the correct diagnosis with sufficient certainty, only that it could provide important hints when combined with other signs and symptoms.
4.1. Strengths and limitations
A limitation of this study is that adult ADHD and ARMS patients were recruited from two different psychiatric services, albeit in the same psychiatric university hospital and across a similar time period. Thus, differences in neuropsychological performance could only be tested in the three cognitive tasks that were overlapping across the two services. Consequently, other important cognitive domains, such as figural memory, verbal fluency, speed of processing, and social cognition could not be investigated. Another limitation is that ADHD patients were not assessed with the BSIP for being in an ARMS for psychosis. Hence, we cannot rule out the possibility that at least some patients would fulfill criteria for both disorders. Lastly, the generalization of our results is limited by the fact that women and patients diagnosed with the inattentive subtype of ADHD were underrepresented in our sample of adult ADHD patients.
Strengths of our study are the relatively large sample sizes and the fact that relatively few patients in both patient groups were treated with medication (see Table 1). Thus, only few ADHD patients were treated with methylphenidate, which has been shown to improve performance in the CPT and other cognitive tasks in adult ADHD patients [Reference Boonstra, Kooij, Oosterlaan, Sergeant and Buitelaar63]. Likewise, only few ARMS and adult ADHD patients were treated with antipsychotics, which have been shown to negatively affect cognition, particularly speed of processing [Reference Faber, Smid, Van Gool, Wiersma and Van Den Bosch64].
4.2 Conclusions
The results of this study suggest that neuropsychological testing, particularly of the domains of sustained attention and impulsivity and verbal learning and memory, can help to improve the differential diagnosis between adult ADHD and ARMS for psychosis. Future studies should replicate and extend our findings with larger samples, and more comprehensive neuropsychological test batteries.
Declarations of interest
None.
Acknowledgments
The authors thank all study participants and the referring specialists. The authors also would like to thank Claudine Pfister for her help with the preparation and submission of the manuscript.
Appendix A. Supplementary data
Supplementary data associated with this article can be found, in the online version, at https://doi.org/10.1016/j.eurpsy.2018.02.006.
Comments
No Comments have been published for this article.