Introduction
Following the publication of the Institute of Medicine’s landmark paper, To Err is Human, 1 US children’s hospitals formed collaboratives to measure and improve outcomes for healthcare-acquired conditions (HACs). Reference Lyren, Brilli, Bird, Lashutka and Muething2–4 These collaboratives have dramatically reduced rates of harm, Reference Bucholz, Toomey and Schuster5 but their measurement systems, as well as those used for public reporting such as by Departments of Health, have been limited to reporting crude harm rates. Children’s hospitals have seen a gradual increase in the acuity and complexity of their patients. Reference Hall, Berry, Hall and Goodwin6,7 The lack of risk adjustment impedes internal benchmarking, potentially obscuring relative improvements in performance over time, and may limit the validity of comparative analyses or external benchmarking. Reference Goudie, Dynan, Brady and Rettiganti8,9 Central line-associated bloodstream infection (CLABSI) is one of the most prevalent pediatric healthcare-associated infections (HAI) and is associated with morbidity and mortality. 10,Reference Hord, Lawlor and Werner11 The existing approach to risk adjustment for CLABSI is the National Healthcare Safety Network (NHSN) Standardized Infection Ratio (SIR) which adjusts for hospital characteristics. Reference Adler, Yaniv and Steinberg12 Apart from birthweight in infants hospitalized in neonatal intensive care units (NICUs), it does not control for individual patient characteristics, which are strong predictors of CLABSI risk. Reference Adler, Yaniv and Steinberg12,Reference Cui13 Our primary hypothesis is that failure to account for acuity and complexity of hospitalized patients could bias comparative analyses of CLABSI rates within pediatric hospitals over time.
To support valid comparisons of children’s hospital performance in reducing CLABSI rates over time, we used electronic health record (EHR) data from a large quaternary children’s hospital to develop a risk adjustment model for CLABSI in hospitalized children and compared observed to expected rates to determine the impact of adjusting for patient factors.
Methods
Study design, setting, and participants
We employed a prospective cohort design using Epic EHR data stored in the Children’s Hospital of Philadelphia (CHOP) data warehouse. CHOP is a quaternary free standing children’s hospital with 601 beds including 100 NICU beds, 74 pediatric ICU (PICU) beds, 32 cardiac ICU (CICU) beds, and 49 beds for children hospitalized with cancer. This study was granted a waiver of HIPAA authorization by the Institutional review board at CHOP.
We included hospitalized patients aged 0–21 years with documented central catheters in place (ie, “catheter days”) between July 1, 2013 and December 31, 2022. Inclusion criteria for catheter days were aligned with NHSN definitions. Reference Cui13 Central catheters included peripherally inserted central catheters (PICCs), central venous catheters (CVCs), and port-a-caths (‘ports’).
CLABSI events were prospectively identified and verified to match NHSN definitions by certified infection preventionists as part of the hospital-wide HAI surveillance program. In alignment with the NHSN CLABSI reporting approach, we excluded mucosal barrier injury lab-confirmed bloodstream infections (MBI-LCBI) as there is insufficient evidence that standard CLABSI prevention practices are effective against this category of infections. Reference Srinivasan and Padula14,Reference Coffey, Marino and Lyren15 To account for changes in the NHSN definition of CLABSI during the study period, we retrospectively re-adjudicated and excluded 18 CLABSIs that did not meet the NHSN January 2022 CLABSI definition. Reference Srinivasan and Padula14
Predictors of CLABSI
In developing our risk adjustment model we considered factors shown in prior studies Reference Armbrister, Finke, Long, Korvink and Gunn16–Reference Gaur, Bundy and Gao29 to be associated with CLABSI, including (1) catheter-related factors such as cumulative inpatient catheter days (total number of inpatient days a patient had any central catheter, reflecting inpatient days at risk of CLABSI), dwell time of current central catheter (days from insertion, including outpatient and inpatient days, for the oldest indwelling catheter), catheter type (port vs other), and total parenteral nutrition (TPN) use; (2) patient demographics including age, sex, race/ethnicity, and primary language; (3) patient diagnoses and comorbidities such as presence of ostomy, neutropenia, lymphatic conditions, and behavioral health conditions; and (4) treatment related factors such as hospital length of stay and admission to the oncology service (Supplemental Table 1).
Because our hypothesis was that failure to account for acuity and complexity of hospitalized patients might bias internal and external comparisons, we adapted a previously published framework Reference Spaulding, Watson and Dreyfus30 that used information on clinical care to identify children receiving ICU level of care, even if they were not hospitalized in an intensive care unit, which is a frequent occurrence in large children’s hospitals. The framework designates ICU level of care by organ system: cardiovascular, respiratory, renal, hematologic, and neurologic (Table 1). In our modeling, we considered ICU level of care as a dichotomous variable (presence of any organ system receiving ICU level of care vs none) to represent the acuity on that catheter day. We created a count variable (number of organ systems receiving ICU level of care) to represent complexity of care.
Table 1. Bivariable and multivariable analyses of CLABSI
a OR = odds ratio, CI = confidence interval, Ref = reference group, NA = not applicable
b Not included in multivariable model due to strong correlation with number of organs requiring ICU level of care
c Expressed as per 100 days
d Not included in multivariable model due to strong correlation with dwell time
Most variables in this analysis were previously validated as part of data analytic and quality improvement efforts. Reference Hsu, Mathew and Wang20 We validated the ICU level of care variables extracted from the EHR by performing manual reviews of 60 charts selected through structured random sampling: 1500 data points representing 25 severity factors used to define the ICU level of care variables were reviewed by clinical experts. Of 16 data points flagged as incorrect upon review, 14 were resolved with better localization of data in the EHR and code in the entire data set was updated. Two were determined to be potential errors in the medical record, representing an error rate of 0.13% in the overall data set.
Analysis
Analyses were conducted in SAS (version 9.4) and R (version 4.1.0). Approximately 1.5% of catheter days had missing values for dwell time and were excluded.
We used generalized estimating equations (GEE) with independent correlation structure to account for repeated observations (catheter days) within patients. Risk factors were considered for inclusion in multivariable modeling based on the following considerations: (1) association with CLABSI risk in univariate modeling (p value < 0.10); (2) collinearity review of risk factors; and 3) clinical expert input. We included an interaction term between dwell time and port catheter type because ports may modify the relationship between dwell time and CLABSI. Typically, long dwell times are associated with increased CLABSI risk, however, ports are associated with reduced CLABSI risk because they remain encased under the skin and are accessed less frequently. Reference Harris, Sbarra and Leekha31,Reference Martinez, Baugnon and Vergnaud32
We considered several multivariable models to assess the relative importance of significant but collinear variables. Models were assessed for fit based on quasi-likelihood under the independence model criterion (QICu). Reference Rabelo, de Alvarenga and Miranda33 The final risk adjustment model generated a predicted probability of CLABSI for each catheter day, based on patient characteristics. We summed probabilities for every catheter day in each year to calculate the expected number of CLABSIs based on the model. Reference Umscheid, Mitchell, Doshi, Agarwal, Williams and Brennan17 We calculated expected and observed CLABSI rates per 1000 catheter days for each year. We conducted bootstrap resampling with replacement at the patient level 500 times within each year to calculate the 95% confidence interval for each year’s expected CLABSI rate.
Results
Between July 1 2013 and December 31 2022, 16,411 inpatients had a total of 520,209 central catheter days. Among them, 633 patients experienced 796 CLABSIs. Among patients, children younger than 1 year of age made up the largest proportion (46%), 55% were male, 47% non-Hispanic white, 19% non-Hispanic black, 14% Hispanic, and 10% with primary language other than English. In Supplemental Table 2, we compare characteristics in CLABSI-associated catheter days and CLABSI-free catheter days. During all catheter days, the respiratory (39%) and cardiovascular (21%) organ systems most commonly required ICU level of care. Compared with non-CLABSI catheter days, CLABSI catheter days were more frequently associated with any organ system (57% vs 45%), 1 organ system (34% vs 28%) and 2 organ systems requiring ICU level of care (23 vs 16%). Figure 1 shows selected risk factors that increased during the study period, demonstrating an increase in the acuity and complexity of patients, need for prolonged central venous access, and behavioral health conditions.
Figure 1. CLABSI risk factors with significant increases* over time in the overall cohort. (a) Changes over time in catheter days in patients with behavioral health conditions or lymphatic conditions. (b) Changes over time in the number of organs requiring ICU level of care. (1) Changes over time in catheter dwell time and cumulative inpatient catheter days.
*Significant increase at P < 0.10.
Table 1 lists factors that were associated with CLABSI in bivariable and multivariable analyses. In our final multivariable risk adjustment model, infants, dwell time of central catheter, TPN use, behavioral health condition, non-English speaking, oncology service, non-port catheter type, lymphatic condition, and number of organs requiring ICU level of care were significantly associated with CLABSI. An interaction between dwell time and port catheter type suggests the protective or neutral effect of longer dwell time on CLABSI risk with a port, compared with increased CLABSI risk associated with longer dwell times with other catheter types (OR 0.89; 95 CI 0.83–0.96). Although oncology service displayed decreased risk in bivariable analysis, this was likely mediated by the protective effect of ports (used frequently in Oncology patients); in the final risk-adjusted model, oncology service was associated with increased risk of infection. Presence of any organ system requiring ICU level of care was associated with a 63% increased odds of CLABSI in bivariable analysis. For every additional organ system receiving ICU level of care, there was a 24% increased odds of CLABSI (OR 1.24; 95% CI 1.12–1.37).
Using the multivariable model, we calculated risk-adjusted (‘expected’) CLABSI rates and compared yearly expected to observed rates (Figure 2). In the earliest years of the study (2014–2015), when a smaller proportion of catheter days involved ICU level of care and dwell times were shorter, expected CLABSI rates were lower than those observed (range of difference 0.013–0.144 per 1000 catheter days lower). However, in the most recent years (2021–2022), when a higher proportion of catheter days involved ICU level of care and dwell times were longer, observed rates were lower than expected (range of difference 0.132–0.234 per 1000 catheter days lower) (Figure 2).
Figure 2. Observed CLABSI rate compared to the expected rate.
The expected CLABSI rate for each year was computed by calculating the predicted probability of CLABSI for each catheter day, and summing those probabilities to calculate the “expected” number of CLABSIs each year. The error bars show the 95% confidence intervals for the expected CLABSI rate, which were computed via bootstrap resampling with replacement at the patient level within each year 500 times to calculate the 95% confidence interval for each year’s expected CLABSI rate. All catheter days were included for each patient selected in the resampling. Expected CLABSI rates were re-calculated for each resample using model probabilities as described, and 2.5th and 97.5th percentiles of CLABSI rates were used as the confidence intervals for each year.
Discussion
In this prospective cohort study, we developed a risk adjustment model for CLABSI using data from a 9.5-year period. We saw an increase in the prevalence of multiple risk factors associated with CLABSI, notably, the proportion of children receiving ICU level of care and the duration of need for central venous access. On adjusting for these risk factors, observed CLABSI rates were lower than expected rates in later study years when prevalence of CLABSI risk factors was higher. Our crude CLABSI rate showed no statistically significant decrease over the last 5 years, despite quality improvement interventions and increased resource deployment toward CLABSI prevention, leading to an erosion of confidence in whether ongoing bundles of prevention practices could achieve further improvement. Truly beneficial interventions could risk being discarded due to lack of perceived effectiveness based on assessment of crude CLABSI rates. Our model suggests that failure to risk-adjust could obscure improvements in performance, while longitudinal analysis of CLABSI rates adjusted for key patient characteristics may unmask actual improvement.
On a larger scale, the Children’s Hospitals’ Solutions for Patient Safety (SPS) collaborative of 145 pediatric hospitals noted a recent upward centerline shift in CLABSI rates, following earlier improvements in rates. Reference Maude, Fitzgerald and Fisher34 Without controlling for changes in patient-level risk factors, it is impossible to know whether this shift in CLABSI rate represents a true change in quality of patient care, or is a function of increasing acuity and complexity of hospitalized children. Reference Hall, Berry, Hall and Goodwin6,7 The same measurement problem may exist when comparing CLABSI prevention performance across institutions when patient characteristics differ substantially; most pronouncedly in quaternary pediatric hospitals, where many patients are transferred or admitted for complex chronic conditions, or require ICU level of care.
Risk adjustment methods to date have not adequately addressed adjustment for patient-level factors. Reference Goudie, Dynan, Brady and Rettiganti8,Reference Cui13,Reference Lyren35 The CDC SIR model adjusts for medical school affiliation, facility type, facility size, and PICU size. Reference Adler, Yaniv and Steinberg12 A multi-center risk adjustment study of CLABSI in adults demonstrated that neither medical school affiliation nor facility size were associated with CLABSI and suggested that adjustment for patient-level risk factors provided better discrimination. Reference Umscheid, Mitchell, Doshi, Agarwal, Williams and Brennan17 The only patient risk factor employed in pediatric SIR is birthweight category as a proxy for degree of prematurity (a known risk factor for CLABSI) in measurement of NICU CLABSI rates. But most children are not hospitalized in NICUs, and many quaternary children’s hospital NICUs have a high proportion of full-term infants admitted with severe and complex chronic conditions, whose comorbidities and need for central venous access are associated with a CLABSI risk rivaling that of premature infants. Reference Delgado-Rodriguez and Llorca36
To our knowledge, our study provides the first risk adjustment model for pediatric CLABSI utilizing patient-level risk factors. Data were abstracted from the EHR in an automated process, making it potentially feasible for health systems to adopt this methodology, as analytics capabilities continue to advance. We have validated an EHR derived physiologic definition of a new variable—ICU level of care—as a measure of acuity and complexity of illness irrespective of patient location. Criteria for ICU admission are influenced by hospital guidelines, census, and competing demands for ICU beds, as exemplified during the COVID-19 pandemic. Reference Weiner-Lastinger, Dudeck and Allen-Bridson37 A definition that objectively measures the utilization of resources that qualify as ICU level of care is likely to be a more accurate measure of patient acuity and complexity. Another factor, catheter dwell time, serves as a marker of chronic medical need, and contributes additive risk for CLABSI. These findings highlight that patient complexity, acuity, and chronic medical need taken together, may be useful for risk adjustment efforts.
Some risk factors in our model are potentially modifiable. Our analysis unmasks disparate risk in non-English speaking patients and those with behavioral health conditions. We propose that such modeling could drive customized preventive strategies in high-risk subgroups of patients. Knowledge of changes in patient risk factors can help improvement teams better prepare for heightened CLABSI risk and interpret trends in CLABSI rate. Quality improvement interventions in recent years at our institution have expanded ‘beyond the bundle’: to pro-active identification of high-risk patients (behavioral health, lymphatic conditions, non-English speaking), with targeted interventions for these patients in addition to standard CLABSI prevention practices. These data-driven targeted strategies may partly explain the greater difference between observed and expected CLABSI rates in recent years.
Our study has some limitations. The model was derived from a single quaternary Children’s Hospital and needs validation. It is possible that some risk factors identified (eg, behavioral health conditions, lymphatic disorders) may not be generalizable to other pediatric hospitals. There may be unmeasured risk factors, such as antibiotic exposure. We could not define the relationship between staffing ratios, acuity and CLABSI rates. We used a hospital-wide CLABSI rate as the outcome for this study to demonstrate a feasible model for risk adjustment that could be used for trending within and across hospital systems and in collaboratives. We intend to pursue subgroup analyses by units in future studies, as certain patient risk factors may be more influential within subgroups.
Our findings have several policy implications. First, hospitals that track CLABSI rates and have active CLABSI prevention efforts should consider measuring annual risk-adjusted results in addition to their control charts to discern changes in CLABSI rates and effectiveness of interventions over time. 38 Patient safety collaboratives and reporting agencies that benchmark children’s hospitals based on outcomes, should partner in efforts such as creation of multi-institutional data sets to develop and validate risk adjustment models.
In conclusion, failure to adjust for patient factors, particularly chronic illness, and acuity and complexity of disease, may miss clinically significant differences in CLABSI rates within and across hospitals. With growing EHR analytic capabilities available across health systems, a patient-level risk adjustment approach to pediatric metrics such as CLABSI is increasing in feasibility. Risk adjustment allows organizations to measure the impact of quality improvement efforts more accurately in the context of changing patient characteristics and could foster novel improvement approaches, with targeted interventions designed for high-risk patient populations.
Supplementary material
For supplementary material accompanying this paper visit https://doi.org/10.1017/ice.2024.111
Acknowledgments
We thank several clinicians at Children’s Hospital of Philadelphia who aided us in identifying and defining variables in the EHR for the analysis—Andi Fu, MD; Marissa Brunetti, MD; Mark Weber, MSN BSN CRNP; Kathleen Gibbs MD; David Munson MD; Aaron Dewitt MD; and Richard Aplenc, MD PhD MSCE. We would like to thank Daniel Hyman, MD MMM; and Anne Lyren, MD MSc (clinical Director of Children’s Hospitals’ Solutions for Patient Safety National Network) for providing us with contextual information about CLABSI rate trends from the Solutions for Patient Safety collaborative. We also thank Jennifer Faerber, PhD (Children’s Hospital of Philadelphia) for her input regarding the data analysis. We thank Susan Coffin MD MPH, Jeffrey Gerber MD PhD, and Julia Sammons MD MSCE (Children’s Hospital of Philadelphia) for reviewing our manuscript prior to submission and providing their expert input. The individuals listed received no compensation.
Contributors’ statement
Dr Srinivasan and Dr Keren conceptualized and designed the study and were involved in acquisition, analysis and interpretation of data. They drafted the initial manuscript and critically reviewed and revised the manuscript.
Ms. Oliver, Ms. Huang, Dr Roberts, and Dr Shu also participated in the design of the study and were involved in acquisition of the data and statistical analysis. They aided in interpretation of the data and critically reviewed and revised the manuscript.
Ms. Donnelly and Ms. Harrison were involved in acquisition of data and interpretation of findings. They critically reviewed and revised the manuscript.
All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
Financial support
The study was funded through Dr Keren’s Gerald D. Quill Distinguished Chair in the Department of Pediatrics at Children’s Hospital of Philadelphia.
Role of the Funder/Sponsor
The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Competing interests
The authors have no financial disclosures or conflicts of interest related to this work.
Abbreviations
CHOP: Children’s Hospital of Philadelphia
CICU: Cardiac intensive care unit
CLABSI: Central line-associated bloodstream infection
CVC: Central venous catheter
GEE: Generalized estimating equations
HAC: Healthcare-acquired condition
HAI: Healthcare-associated infection
EHR: Electronic health record
ICU: Intensive care unit
MBI-LCBI: Mucosal barrier injury lab-confirmed bloodstream infections
NHSN: National Healthcare Safety Network
NICU: Neonatal intensive care unit
PICC: Peripherally inserted central catheter
PICU: Pediatric intensive care unit
QICU: Quasi-likelihood under the independence model criterion
SIR: Standardized Infection Ratio
TPN: Total parenteral nutrition
