Introduction
Streptococcus pneumoniae remains the most commonly identified bacterial cause of community-acquired pneumonia (CAP), the leading infectious cause of hospitalization and death in both children and adults. Reference Jain, Self and Wunderink1 The pneumococcal urinary antigen test (pUAT), which detects the C-polysaccharide antigen common to all serotypes of S. pneumoniae, has emerged as an important diagnostic method for pneumococcal pneumonia. Reference Viasus, Calatayud, McBrown, Ardanuy and Carratala2 Compared to traditional microbiological methods like blood or sputum cultures, which are affected by inadequate sample collection, delayed processing, and prior antimicrobial therapy, pUATs have increased the rate of etiologic diagnosis for CAP by 11%–23%. Reference Jain, Self and Wunderink1–Reference Song, Eun and Nahm4 Identification of S. pneumoniae as the causative pathogen is important, as doing so enables early targeted treatment and improved antimicrobial stewardship. Reference Kim, Deshpande and Rothberg5
It remains unclear, however, which patients should undergo pUAT. Pneumococcal urinary antigen test utilization varies widely among hospitals across the U.S. One large retrospective cohort study of adults admitted with CAP to 170 U.S. hospitals showed that hospital rates of UAT utilization ranged from 0% to 69%, underscoring the lack of consensus on when this test should be ordered. Reference Schimmel, Haessler and Imrey6 The 2019 American Thoracic Society/Infectious Diseases Society of America (ATS/IDSA) guidelines argue against routine pUAT because small randomized trials have failed to demonstrate improved outcomes following pUAT and targeted therapy. Reference Metlay, Waterer, Long and Anzueto7,Reference Falguera, Ruiz-Gonzalez and Schoenenberger8 However, because other large observational studies Reference Costantini, Allara, Patrucco, Faggiano, Hamid and Balbo9,Reference Uematsu, Hashimoto, Iwamoto, Horiguchi and Yasunaga10 have reported reduced mortality for patients receiving pUAT, especially among severe and very severe CAP cases, the 2019 ATS/IDSA guidelines make a weak recommendation to perform pUAT only in patients with severe CAP. A recent prospective study of 1941 patients, however, showed that only 4% of patients who met ATS/IDSA indications for pUAT had a positive result. Reference Bellew, Grijalva and Williams11 This suggests that current guidelines for pUAT utilization identify a population with low diagnostic yield, resulting in unnecessary tests and wasted resources.
The optimal patient population for pUAT thus remains unknown. Stronger predictors of pUAT positivity must be identified to improve test efficiency and identify opportunities for improved antimicrobial stewardship. Previous attempts to develop prediction models for pUAT positivity have been limited by small sample sizes, absent multivariable analyses, and the inclusion of variables that are not easily extracted from the electronic medical record (EMR). Reference Bellew, Grijalva and Williams11–Reference West, McCauley, Sorensen, Jephson and Dean16 Importantly, to date, no prediction model has been validated. The objective of our study was to derive and externally validate a risk model to predict pUAT positivity in adults hospitalized with CAP and compare the efficiency of our model to that of guideline-recommended testing.
Methods
This study was approved by Cleveland Clinic Institutional Review Board (18-1237, 15-1254, 16-1035). All analyses were conducted in R version 4.1.0. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines were followed. Reference Debray, Collins and Riley17
Study population
We conducted a retrospective cohort study of adults (aged ≥18 years) admitted with pneumonia to one of 177 U.S. hospitals in the Premier Healthcare Database from January 2010 to December 2015, or to a non-Premier hospital in the Cleveland Clinic Health System (CCHS) from January 2010 to December 2019. The Premier Healthcare Database is a large U.S. hospital discharge database that has been widely used for research and includes hospitals located in all regions of the U.S. Reference Rothberg, Pekow, Lahti, Brody, Skiest and Lindenauer18 It contains data elements including sociodemographic information, discharge International Classification of Disease, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes, hospital and physician information, and a date-stamped record of all items billed during hospitalization.
We included patients with a primary diagnosis of pneumonia (ICD-9-CM codes 480–488, 507.0) and a present-on-admission flag, or a principal diagnosis of respiratory failure (ICD-9-CM 518.81, 518.82, 518.84, 779.1) or sepsis (ICD-9-CM 785.52, 790.7, 995.91, 995.92, 038.x), paired with a secondary diagnosis of pneumonia present-on-admission. To increase our certainty of capturing patients with suspected CAP at the time of admission, we restricted our study population to patients who had chest imaging by hospital day 1 and at least three consecutive days of antimicrobial treatment. Patients were excluded if they had a diagnosis of cystic fibrosis or tuberculosis, or if they were hospitalized for less than 24 hours. If patients were hospitalized more than once, the most recent hospitalization was selected for inclusion.
Data extraction
For each hospitalization, the following data were extracted: patient age, sex, race, principal and secondary diagnoses, sociodemographic information, medical comorbidities, antibiotics prior to admission, treatments during admission, and microbiological data. Variables for patients in the Premier database were identified from hospital discharge records and physician claims using ICD-9-CM and daily charge codes, while those for patients in the CCHS database also included ICD-10-CM codes and clinical and laboratory data. A list of ICD-9-CM and charge codes that were used during data extraction is included in the Supplementary Information.
Derivation and validation cohorts
To derive a prediction model for pUAT positivity, we first identified patients with CAP in the Premier Healthcare Database who underwent pUAT. We a priori decided to split the Premier sample by hospital, rather than at the patient level. We randomly selected 80% of the Premier hospitals to serve as the derivation cohort and used the remaining 20% as the internal validation cohort. A prediction model was then developed using the derivation cohort dataset (Figure 1). We applied the same inclusion and exclusion criteria to create an external validation cohort of patients admitted to one of 12 hospitals in the Cleveland Clinic Health System and who underwent pUAT (Figure 1). Only a small fraction (<1%) of patients were excluded from analyses for missing candidate predictors.
Figure 1. Schematic representation of internal and external validation employed.
Candidate predictor variables
Potential predictors were identified by reviewing the literature for risk factors for pneumococcal pneumonia that would be available at the time of admission. Candidate predictors included patient demographics, admission source, smoking status, clinical findings on admission (leukocytosis, hyponatremia, thrombocytopenia, confusion, and fever), medical comorbidities (e.g., chronic lung disease, congestive heart failure, chronic kidney disease, chronic liver disease, hypertension, diabetes, obesity, substance abuse, alcohol abuse, dementia, stroke history, and seizure/epilepsy), risk factors for multi-drug resistant organisms (residence in a skilled nursing facility (SNF), immunosuppression, dialysis, admission in the previous 6 months), hospital characteristics (location and teaching status), history of pneumococcal infection in the previous year, history of IV antibiotic use in the previous year, and pneumonia severity on admission (severe vs. non-severe). Severe CAP was defined as the need for vasopressors, invasive mechanical ventilation, or intensive care unit (ICU) admission in the first 48 h of hospitalization; this definition utilized the major criteria for severe CAP listed in the 2007 ATS/IDSA guidelines. Reference Mandell, Wunderink and Anzueto19 History of pneumococcal infection in the previous year was defined as any positive test for S. pneumoniae (UAT, blood culture, sputum culture, or pleural fluid culture) in the previous 1 year prior to the admission date. Additional details on variable definitions used in the CCHS dataset are included in the Supplementary Information.
Statistical methods
In the derivation cohort, all candidate predictor variables were included as potential covariates in a full multivariable logistic regression model. The primary outcome was pUAT result. Several variable selection methods were explored to obtain a parsimonious model: a) stepwise backward elimination with a significance level of 0.005 for variable retention, b) backward elimination by Akaike information criterion (AIC), and c) random forest. The model that represented the best trade-off between number of variables and predictive power was then selected as the final model and applied to both the internal and external validation cohorts. Model discrimination was compared based on the C-statistic. Model calibration was evaluated graphically by grouped bar charts of observed vs. predicted proportions of positive pUATs within increasing deciles of predicted probability. In the external validation cohort, a multiplicative scalar that minimized the total error between observed and predicted probabilities was determined and applied (Supplementary Information).
To evaluate the clinical benefit of our prediction model, we compared the efficiency of our model to that of the 2019 ATS/IDSA guidelines, which recommend pUAT only in patients with severe CAP. To do this, we used the pUAT positivity rate for severe CAP patients in each sample as the predicted probability threshold. Patients with a predicted probability greater than this cut-off value would have been tested using our model. We then compared the number of tests that would have been performed and the number of cases that would have been detected using our model, as compared to testing only patients with severe CAP.
Results
The Premier dataset consisted of 198,130 patients admitted with CAP, while the CCHS dataset consisted of 115,404 patients admitted with CAP. Of 198,130 Premier patients admitted with CAP, 27,970 (14.1%) underwent pUAT, of whom 1962 (7.0%) tested positive. Characteristics of Premier patients with and without pUAT are shown in Supplemental Table 1. Overall, there were very small differences in patient-level characteristics between those who did and did not undergo pUAT. Patients with pUAT were not more likely to have severe CAP on admission, compared to those who did not undergo pUAT (22% vs. 23%).
Characteristics of the cohorts
Characteristics of patients in the derivation, internal validation, and external validation cohorts are shown in Table 1. The median age of patients ranged from 69 to 71 years. All three cohorts had similar proportions of patients with hypertension (65%–71%) and type 2 diabetes (32%–35%). Compared with Premier patients, CCHS patients were more likely to have multiple comorbidities and severe CAP on admission (40% vs. ∼23%). The rate of pUAT positivity was lowest in the CCHS external validation cohort (2.0%), compared to the derivation (7.4%) and internal validation (6.2%) cohorts.
Table 1. Baseline characteristics of patients in the derivation, internal validation, and external validation cohorts
Abbreviations: SNF, skilled nursing facility; SP, Streptococcus pneumoniae.
1Median (IQR) for continuous; n (%) for categorical.
Characteristics of patients with positive vs. negative pUAT in the derivation cohort
Several variables were significantly associated with a positive pUAT in univariate analyses. Table 2 shows characteristics of patients with positive and negative pUATs in the derivation cohort. Compared to patients with negative pUATs, those with positive tests were younger (median age 65 vs. 69 years), were current smokers (31% vs. 23%), had substance abuse (8.6% vs. 4.4%), had severe CAP at the time of admission (31% vs. 21%), had hyponatremia at the time of admission (20% vs. 15%), and had a history of pneumococcal infection in the previous 1 year (1.8% vs. 0.4%). They were less likely to have obesity (12% vs. 16%) and congestive heart failure (20% vs. 28%).
Table 2. Characteristics of Premier hospital patients with positive vs. negative pneumococcal urinary antigen tests within the derivation cohort
Abbreviations: SNF, skilled nursing facility; SP, Streptococcus pneumoniae.
1Median (IQR) for continuous; n (%) for categorical.
2Wilcoxon rank sum test; Pearson’s Chi-squared test; Fisher’s exact test.
Predictors of pneumococcal UAT positivity in the derivation cohort
Our final model was developed from stepwise backward elimination with a significance level of 0.005 for variable retention and included 14 variables: race, admission from an SNF, congestive heart failure, hypertension, obesity, substance abuse, diabetes, smoking, hyponatremia, teaching hospital, region, IV antibiotic use in the past year, severe CAP on admission, and history of pneumococcal infection in the past year. Odds ratios (ORs) and 95% confidence intervals (CIs) for predictors are shown in Table 3. History of pneumococcal infection in the past year (OR 6.99, 95% CI 4.27–11.46), severe CAP on admission (OR 1.76, 95% CI 1.56–1.98), substance abuse (OR 1.57, 95% CI 1.27–1.93), smoking (OR 1.23, 95% CI 1.09–1.39), and hyponatremia (OR 1.35, 95% CI 1.17–1.55) were most strongly associated with pUAT positivity. IV antibiotic use in the past year (OR 0.65, 95% CI 0.52–0.82), congestive heart failure (OR 0.72, 95% CI 0.63–0.83), obesity (OR 0.71, 95% CI 0.60–0.85), and admission from a skilled nursing facility (OR 0.60, 95% CI 0.45–0.78) were negative predictors of pUAT positivity. Results from alternative variable selection methods (backward elimination by AIC, random forest) are presented in Supplementary Table 2.
Table 3. Associations between patient/hospital characteristics and pUAT positivity using multivariable logistic regression in the derivation cohort
Abbreviations: SNF, skilled nursing facility; SP, Streptococcus pneumoniae.
Model intercept = −2.354.
Prediction model performance
Receiver operating characteristic curves for our model in the derivation, internal validation, and external validation cohorts are superimposed in Figure 2. Our model had a C-statistic of 0.63 in the derivation sample, 0.60 in the internal validation sample, and 0.67 in the external CCHS sample. Figure 3 shows the proportion of positive pUATs in the two validation samples by deciles of predicted risk. Our model demonstrated excellent calibration in the internal validation cohort, especially across the first nine deciles (Figure 3A). In the CCHS external validation sample, our model overestimated the risk of pUAT positivity (Supplemental Figure 1); a data-determined multiplicative scalar of 0.36 was applied to the original predicted probabilities to adjust the calibration (Figure 3B). Details of the re-scaling process are included in the Supplementary Information. In the Premier sample, of the 1962 patients who tested positive, 1114 (56.8%) were in the top four risk deciles. In the CCHS sample, of the 394 patients with positive pUATs, 242 (61.4%) were in the top four risk deciles.
Figure 2. Receiver operating characteristic (ROC) curves for derivation, internal validation, and Cleveland Clinic Health Systems (CCHS) external validation samples.
Figure 3. Observed vs. predicted pUAT positivity proportions in the Premier internal validation and CCHS external validation samples, based on a 14-variable prediction model derived from the Premier 80% derivation sample. (A) Premier 20% hold-out internal validation sample, (B) CCHS external validation sample. Rescaled calibration plot where a multiplicative scalar of 0.36 was applied to the original predicted probabilities.
Of the 27,970 patients in the Premier sample who underwent pUAT, 6265 (22.4%) were guideline-concordant and had severe CAP on admission, of whom 575 (9.2%) tested positive. At a threshold of predicted risk of 9.2%, 5971 patients would have been tested using our prediction model, of whom 707 (11.8%) would have tested positive. Compared to testing only patients with severe CAP, our model would have detected 23% more cases with 5% fewer tests in the Premier dataset. In the external validation cohort, our model would have detected a greater proportion of pneumococcal cases (3.8% vs. 2.6%, p < 0.001) with 37.6% fewer tests, when compared to testing only patients with severe CAP. Additional details are provided in the Supplemental Information.
Discussion
In this large retrospective cohort study, we derived and externally validated a prediction model for pUAT positivity in patients admitted with CAP. We derived a model that balanced clinical utility with predictive power and identified key risk factors for pUAT positivity, including history of pneumococcal infection in the past year, severe CAP on admission, substance abuse, smoking, and hyponatremia. Negative predictors included IV antibiotic use in the past year, congestive heart failure, and obesity. Importantly, because these factors can be easily extracted from the EMR, our model can be incorporated into automated clinical decision support. Compared to guideline-recommended testing, our model detected more cases of pneumococcal pneumonia with fewer tests.
Several studies have reported similar predictors of pUAT positivity in CAP patients. Reference Bellew, Grijalva and Williams11–Reference West, McCauley, Sorensen, Jephson and Dean16 One prospective study in Spain Reference Molinos, Zalacain and Menéndez12 found that Pneumonia Severity Index (PSI) risk class ≥IV was a significant predictor of pUAT positivity, while another in Utah Reference West, McCauley, Sorensen, Jephson and Dean16 reported ICU admission as a significant risk factor. Smoking, Reference Troy, Wong and Barnes15 hyponatremia, Reference Molinos, Zalacain and Menéndez12 and female sex Reference Molinos, Zalacain and Menéndez12,Reference Ito, Ishida, Tachibana, Nakanishi, Yamazaki and Washio14 have also been reported to be independent predictors, while prior antibiotic treatment Reference Molinos, Zalacain and Menéndez12 has been found to be a negative predictor.
To the best of our knowledge, only one other study has evaluated model performance. Reference Molinos, Zalacain and Menéndez12 Their model contained nine variables: female sex, heart rate ≥125 beats per minute, systolic blood pressure <90 mmHg, oxygen saturation <90%, no prior antibiotic treatment, pleuritic chest pain, chills, pleural effusion, and blood urea nitrogen (BUN) >30 mg/dL. With a C-statistic of 0.64 (95% CI, 0.58–0.70), their model’s discrimination was similar to ours. However, unlike our model, theirs required several variables that would not be easily extracted from the EMR, limiting the feasibility of their model to be incorporated into automated clinical decision support.
Our model is the first to be externally validated. In the external validation cohort, our model offered fair discrimination, but it overestimated the risk of pUAT positivity, likely due to the large differences in pUAT positivity rates between the CCHS (2.0%) and Premier derivation cohort (7.4%). These differences can likely be explained by significant geographic, hospital, and temporal dissimilarities between the two datasets. The observed proportion of positive pUATs, however, increased with every increase in decile of predicted probability, suggesting that our predictions could be multiplied by a scalar to adequately adjust the calibration. After applying a data-determined multiplicative scalar that minimized the total error between observed and predicted probabilities, our model’s calibration was significantly improved.
This study has several limitations. First, because the Premier database is primarily an administrative dataset based on discharge ICD-9 codes, it is possible that we may have underestimated the presence of risk factors, and the generalizability of our model could be limited. Additionally, some risk factors (e.g., substance use disorder) are very broad terms without granularity. However, the Premier database includes detailed date-stamped billing records during hospitalization, increasing our confidence in the specificity of the diagnosis and procedure codes. Second, the Premier dataset lacks many clinical variables that could not be included in our model. However, we validated our model on a clinically rich dataset of CCHS patients. Third, due to limitations of the Premier database, we used surrogate markers of severity to define severe CAP, rather than utilizing both the major and minor criteria of the 2007 ATS/IDSA guidelines. Next, information on patients’ pneumococcal vaccination status was not available in our datasets. Finally, our model was derived using data only from patients who underwent pUAT, which is not routinely performed in all CAP patients and thus may limit the generalizability of our model.
Our study has important implications. Current ATS/IDSA guidelines recommend pUAT only in patients with severe CAP, Reference Metlay, Waterer, Long and Anzueto7 but it appears that clinicians are not targeting this population when ordering pUATs. Among Premier patients, we found that those who underwent pUAT were not more likely to have severe CAP on admission, and there were very small differences in patient-level characteristics between those who were and were not tested. Even when guideline recommendations are followed, the pUAT costs approximately $425 per positive test result due to the low positive test prevalence. Reference Bellew, Grijalva and Williams11,Reference Dinh, Duran and Davido20 By detecting more cases of pneumococcal pneumonia with fewer tests, our model could significantly improve efficiency of testing, making the pUAT more cost-effective. More importantly, however, early identification of S. pneumoniae as the causative pathogen increases opportunities for antimicrobial stewardship by enabling early targeted treatment and antibiotic de-escalation. The next logical step would be to incorporate our prediction model into the EMR to provide point-of-care clinical decision support to providers. A well-designed randomized controlled trial could then determine whether use of our model can promote antibiotic de-escalation and improve outcomes.
In summary, we derived and externally validated a prediction model for pUAT positivity. Because our model contains variables that can be easily extracted from the EMR, it has the potential to be incorporated into the EMR to improve efficiency of testing, facilitate early diagnosis of pneumococcal pneumonia, and enable antibiotic de-escalation.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ash.2023.421
Acknowledgments
This work was supported by the Agency for Healthcare Research and Quality [R01HS024277] to Dr. Michael B. Rothberg and the IDSA Foundation (2022 Grants for Emerging Researchers/Clinicians Mentorship award) to Priscilla Kim.
Disclosures
Abhishek Deshpande is a consultant for Merck and has received research support from Seres Therapeutics unrelated to the current study. All other authors report no potential conflicts of interest.
Open access
