Definitive diagnosis of Alzheimer’s disease (AD) is often unavailable, so clinical diagnoses with some degree of inaccuracy are often used in research instead. When researchers test methods that may improve clinical accuracy, the error in initial diagnosis can penalize predictions that are more accurate to true diagnoses but differ from clinical diagnoses. To address this challenge, the current study investigated the use of a simple bias adjustment for use in logistic regression that accounts for known inaccuracy in initial diagnoses.

A Bayesian logistic regression model was developed to predict unobserved/true diagnostic status given the sensitivity and specificity of an imperfect reference. This model considers cases as a mixture of true (with rate = sensitivity) and false positives (rate = 1 - specificity) while controls are mixtures of true (rate = specificity) and false negatives (rate = 1 - sensitivity). This bias adjustment was tested using Monte Carlo simulations over four conditions that varied the accuracy of clinical diagnoses. Conditions utilized 1000 iterations each generating a random dataset of n = 1000 based on a true logistic model with an intercept and three arbitrary predictors. Coefficients for parameters were randomly selected in each iteration and used to produce a set of two diagnoses: true diagnoses and observed diagnoses with imperfect accuracy. Sensitivity and specificity of the simulated clinical diagnosis varied with each of the four conditions (C): C1 = (0.77, 0.60), C2 = (0.87, 0.44), C3 = (0.71, 0.71), and C4 = (0.83, 0.55), which are derived from published values for clinical AD diagnoses against autopsy-confirmed pathology. Unadjusted and bias-adjusted logistic regressions were then fit to the simulated data to determine the models’ accuracy in estimating regression parameters and prediction of true diagnosis.

Under all conditions, the bias-adjusted logistic regression model outperformed its unadjusted counterpart. Root mean square error (the variability of estimated coefficients around their true parameter values) ranged from 0.23 to 0.79 for the unadjusted model versus 0.24 to 0.29 for the bias-adjusted model. The empirical coverage rate (the proportion of 95% credible intervals that include their true parameter) ranged from 0.00 to 0.47 for the unadjusted model versus 0.95 to 0.96 for the bias-adjusted model. Finally, the bias-adjusted model produced the best overall diagnostic accuracy with correct classification of true diagnostic values about 78% of the time versus 62-72% without adjustment.

Results of this simulation study, which used published AD sensitivity and specificity statistics, provide evidence that bias-adjustments to logistic regression models are needed when research involves diagnoses from an imperfect standard. Results showed that unadjusted methods rarely identified true effects with credible intervals for coefficients including the true value anywhere from never to less than half of the time. Additional simulations are needed to examine the bias-adjusted model’s performance under additional conditions. Future research is needed to extend the bias adjustment to multinomial logistic regressions and to scenarios where the rate of misdiagnosis is unknown. Such methods may be valuable for improving detection of other neurological disorders with greater diagnostic error as well.

Prior studies have presented demographic adjustment as beneficial because it helps equalize, across demographic groups, the percentage of participants (recruited from the general population without prior diagnosis) who fell beneath the test impairment cutoff (e.g., Smith, et al., 2008). This methodology ignores the possibility that group differences in those falling beneath an impairment cutoff could reflect cognitive impairment prevalence differences between demographic groups in the undiagnosed general population. Demographic group differences in cognitive test scores reflect a mixture of two categories of influences: measurement bias (item/test/examiner bias, language/cultural bias, stereotype threat, etc.) and factors which differentially increase the number of low scores in one group by increasing relative risk (RR) for cognitive impairment (biological aging processes, cognitive reserve, social determinants of health [SDoH], etc.). The current simulation study examined how the effect of demographic adjustment on the diagnostic accuracy of a hypothetical test (operationalized as the area under the curve [AUC] in an ROC analysis) varied as the mixture of influences which caused demographic differences in scores were varied.

215,040 samples were randomly generated. Each sample consisted of two demographic groups, with Group 0 always representing the lower scoring group. Across samples, Group 1's baseline risk of impairment and Group 0's relative risk were varied, and these determined the prevalence of cognitive impairment in the groups. Three facets of measurement bias were varied in the simulation: how much lower Group 0's average score was than Group 1's, the degree of non-homogeneity of variance between groups, and how much less reliable the measure was for Group 0. Additional parameters were included and varied to ensure the robustness of findings across a variety of situations. Samples reflected all possible combinations of all varied parameters. For each sample, a baseline AUC was calculated when impairment was regressed on the unadjusted test score. Then, test scores were adjusted for demographic group and difference in adjusted and unadjusted AUC was calculated. This adjusted/unadjusted AUC difference was then regressed on the simulation parameters to quantify their relative influence.

The more Group 0's average score was reduced by measurement bias, the more improvement in AUC was seen after adjustment (ß = 1.76). Trivial but significant main effects of variance non-homogeneity (ß = .09), increased relative risk (ß = -.08), and reduced reliability (ß = .02) were also found, but more importantly, each of these predictors significantly interacted with Group 0 mean score reduction, such that higher relative risks (ß = -1.22), lower reliability (ß = .36), and higher variance (ß = -.15) in Group 0 compared to Group 1 each reduced the association between Group 0 mean score reduction and improvement in AUC.

Demographic adjustment only improves AUC when the mean reduction in scores due to measurement bias is sufficiently high while risk for impairment, test reliability and test score variances are sufficiently equivalent among the demographic groups. When this is not the case, demographic adjustment can be counter-productive, reducing the AUC of the test. We conclude by proposing a novel method for adjusting test scores.

The National Institutes of Health-Toolbox cognition battery (NIH-TCB) is widely used in cognitive aging studies and includes measures in cognitive domains evaluated for dimensional structure and psychometric properties in prior research. The present study addresses a current literature gap by demonstrating how NIH-TCB integrates into a battery of traditional clinical neuropsychological measures. The dimensional structure of NIH-TCB measures along with conventional neuropsychological tests is assessed in healthy older adults.

Baseline cognitive data were obtained from 327 older adults. The following measures were collected: NIH-Toolbox cognitive battery, Controlled Oral Word Association (COWA) letter and animals tests, Wechsler Test of Adult Reading (WTAR), Stroop Color-Word Interference Test, Paced Auditory Serial Addition Test (PASAT), Brief Visuospatial Memory Test (BVMT), Letter-Number Sequencing (LNS), Hopkins Verbal Learning Test (HVLT), Trail Making Test A&B, Digit Span. Hmisc, psych, and GPARotation packages for R were used to conduct exploratory factor analyses (EFA). A 5-factor solution was conducted followed by a 6-factor solution. Promax rotation was used for both EFA models.

The 6-factor EFA solution is reported here. Results indicated the following 6 factors: working memory (Digit Span forward, backward, and sequencing, PASAT trials 1 and 2, NIH-Toolbox List Sorting, LNS), speed/executive function (Stroop color naming, word reading, and color-word interference, NIH-Toolbox Flanker, Dimensional Change, and Pattern Comparison, Trail Making Test A&B), verbal fluency (COWA letters F-A-S), crystallized intelligence (WTAR, NIH-Toolbox Oral Recognition and Picture Vocabulary), visual memory (BVMT immediate and delayed), and verbal memory (HVLT immediate and delayed. COWA animals and NIH-Toolbox Picture Sequencing did not adequately load onto any EFA factor and were excluded from the subsequent CFA.

Findings indicate that in a sample of healthy older adults, these collected measures and those obtained through the NIH-Toolbox battery represent 6 domains of cognitive function. Results suggest that in this sample, picture sequencing and COWA animals did not load adequately onto the factors created from the rest of the measures collected. These findings should assist in interpreting future research using combined NIH-TCB and neuropsychological batteries to assess cognition in healthy older adults.