Introduction
The World Health Organization (WHO) classified antimicrobial resistance (AMR) as one of the ten main global health issues to be tracked in 2021.1 High use of antibiotics, use of multiple broad-spectrum antibiotics, and lack of proper antimicrobial stewardship are considered some of the foremost factors responsible for the increase in AMR.Reference Vivas, Barbosa, Dolabela and Jain2 Hospitals are an example of a setting where antibiotic usage is showing evident increase, and the recent years have shown a significant increase in the use of broad-spectrum antibiotics such as vancomycin, carbapenems, and fourth-generation cephalosporins.Reference Davey, Brown and Charani3,Reference Baggs, Fridkin, Pollack, Srinivasan and Jernigan4 The proper antimicrobial stewardship of broad-spectrum antibiotics has shown to reduce their unnecessary overuse.Reference Cheng, To and Li5
Pneumonia is a major infectious disease with higher incidence, mortality, and rates of antimicrobial treatment in older adults.Reference Miyashita and Yamauchi6,Reference Henig and Kaye7 It’s often treated with broad-spectrum antibiotics including anti-MRSA (Methicillin-resistant Staphylococcus aureus) and antipseudomonal agents.Reference Klompas8 However, research findings and guidelines emphasize the significance of appropriate antimicrobial therapies, warning that the liberal use of antimicrobial combinations will eventually generate untreatable resistant infections.Reference Martin-Loeches, Rodriguez and Torres9 In Japan, pneumonia is growing in significance due to the ageing population.Reference Miyashita and Yamauchi6 As it’s primarily affecting those aged over 64, pneumonia is requiring higher rates of medical treatment and ranking as the fifth leading cause of death, making its management increasingly important.Reference Miyashita and Yamauchi6,Reference Morimoto, Suzuki and Ishifuji10
Quantifying antibiotic use is crucial for antimicrobial stewardship programs. Although one definitive metric or method has yet to be determined, quantitative evaluation can aid the development of stewardship programs for monitoring and promoting their responsible use in hospitals.Reference Kallen, Natsch and Opmeer11–Reference Bitterman, Hussein, Leibovici, Carmeli and Paul13 Since comparisons across hospitals using unadjusted metrics are discouraged, various risk adjustment approaches for antimicrobial use have been developed, incorporating patient or hospital-related predictors in regression modeling.Reference Ibrahim and Polk14,Reference Wattier, Thurm, Parker, Banerjee and Hersh15 An example of that is the observed-to-expected (O/E) ratio, which has served as a quality indicator in the past for various measures related to inpatient care, including antimicrobial use.Reference Polk, Hohmann, Medvedev and Ibrahim12,Reference Wattier, Thurm, Parker, Banerjee and Hersh15–Reference Tan, Vermeulen, Wang, Zvonar, Garber and Daneman17 While the basic O/E ratio is useful, it is still vulnerable to sample variation. Thus, smoothing techniques, such as those applied by the Agency for Healthcare Research and Quality, can help stabilize estimates.18 Therefore, this study aimed to develop an improved method for benchmarking broad-spectrum antibiotic use in the empiric treatment of pneumonia in older adult inpatients by applying the smoothed O/E ratio and adjusting for varying patient risk profiles across hospitals.
Methods
Data source
Patient and hospital data were obtained from the Diagnosis Procedure Combination (DPC) research group’s database, funded by the Ministry of Health, Labour and Welfare (MHLW). DPC system is a measuring tool intended to standardize Japanese medical care by making acute patient care transparent.Reference Hayashida, Murakami, Matsuda and Fushimi19 The database includes survey data defined by MHLW, containing information from acute care hospitals, such as patient clinical information and disease classifications based on the 10th Revision of the International Classification of Diseases (ICD-10), as well as claims data.Reference Hayashida, Murakami, Matsuda and Fushimi19,Reference Uematsu, Yamashita, Kunisawa, Fushimi and Imanaka20 Acute care hospitals periodically and officially submit DPC data to MHLW, and copies of this data were provided to the DPC research group.
Study population
We selected patients hospitalized and discharged between April 1st, 2018, and March 31st, 2020. Patients were included in the study if they were over 64 years old, admitted for pneumonia, and received antibiotic treatment within the first 48 hours of admission. Pneumonia diagnosis, recorded as both the admission-precipitating diagnosis and the most resource-consuming diagnosis, was based on the ICD-10 codes J13, J14, J15.x, J16.x, J17.x, and J18.x. However, patients who had undergone surgery prior to treatment were excluded, as this is uncommon in pneumonia admissions, and they were considered as perioperative prophylaxis cases. Patients were also excluded if they had missing data related to the variables needed in risk adjustment. In addition, hospitals with less than a total of 100 patient cases were excluded.
Outcome
The primary outcome in this study was the smoothed O/E ratio of the use of broad-spectrum antibiotics across the hospitals. Broad-spectrum antibiotic use at the patient level was quantified as the exposure to these antibiotics during empiric treatment (ie, whether or not the patient received them during these 48 hours).Reference Stanic Benic, Milanic and Monnier21 In this study, broad-spectrum antibiotics were defined as antibiotics reserved for patients with risk of MDR pathogens, according to the Japanese Association for Infectious Diseases/Japanese Society of Chemotherapy guideline for pneumonia treatment.Reference Mikasa, Aoki and Aoki22 Additionally, some agents not in the guideline were considered as antibiotics reserved for last-line treatment, following the WHO’s “AWaRe” classification.23 Consequently, the broad-spectrum antibiotic agents in our study were: Aztreonam, Biapenem, Cefepime, Cefozopran, Ceftolozane-Tazobactam, Ciprofloxacin, Colistin, Daptomycin, Doripenem, Faropenem, Fosfomycin, Imipenem, Lascufloxacin, Linezolid, Meropenem, Minocycline, Panipenem, Pazufloxacin, Piperacillin-Tazobactam, Polymyxin B, Prulifloxacin, Tebipenem, Tedizolid, Teicoplanin, Tigecycline, Vancomycin. It is worth mentioning that the definition of broad-spectrum antibiotics could differ for some agents depending on the resources, so our definition applies to this study.
O/E ratio and smoothed O/E ratio
The O/E ratio, which follows the concept of the standardized mortality ratio, is usually calculated by dividing the observed value, as the numerator, by the expected value, which is obtained through a prediction model, as the denominator.Reference Mohammed, Manktelow and Hofer24
$$Observed - to - Expected\;\left( {O/E} \right)\;Ratio = {{Observed\;Value\;}\over{Expected\;Value}}$$
As a simplified general concept, a hospital’s O/E value below 1 will indicate that the broad-spectrum antibiotic use is lower than expected, while a value above 1 is higher.
Mohammed et al. differentiated between several methods of deriving the standardized mortality O/E ratio. Including the standard one above, he introduced another which can be referred to as the “smoothed” O/E ratio. It differs than the standard O/E ratio by utilizing the smoothed observed values as the numerator instead of the crude (raw) observed values.Reference Mohammed, Manktelow and Hofer24
$$Smoothed\;O/E\;Ratio = {{Smoothed\;Observed\;Value\;}\over{Expected\;Value}}$$
The smoothed observed value, which is estimated using a multilevel model with a random intercept, shrinks sample variation in the crude observed values.
Baseline variables and model development
A multilevel prediction model is needed to calculate the smoothed O/E ratio of broad-spectrum antibiotic use among hospitals. The model can be represented as:
$$\log {{{p_i}\;}\over{1 - {p_{ij}}}} = \alpha + \beta {x_{ij}} + {\gamma _j}$$
p ij is the probability of broad-spectrum antibiotic use for a certain patient (i) treated at a certain hospital (j), x ij is the vector of risk adjustment variables, α represents the model intercept, β is the coefficient for x ij, and γj is the random intercept.Reference Mohammed, Manktelow and Hofer24
Predictors for our model included patient-related variables reported in previous literature as risk factors for resistance in pneumonia, as well as disease severity criteria.Reference Mikasa, Aoki and Aoki22,Reference Arancibia and Ruiz25–Reference Imamura, Miyazaki and Watanabe29 The disease severity variables were based on the A-DROP (Age, Dehydration, Respiratory failure, Orientation disturbance, Blood pressure) score, a modified version of the CURB-65 (Confusion, Uremia, Respiratory rate, Blood pressure, Age≥65) score for pneumonia severity, proposed by the Japanese Respiratory Society, with predictive power similar to that of the CURB-65.Reference Shindo, Sato and Maruyama28 Afterwards, the following variables were selected as predictors: age, sex, ICD-10 codes, variables composing A-DROP score (blood urea nitrogen (BUN), oxygen saturation (SpO2), orientation disturbance, systolic blood pressure), immunodeficiency, previous hospitalization within the last 90 days, admission from home, admission from nursing home, intensive care unit (ICU) admission, being on mechanical ventilator, tube feeding, and various comorbidities on admission (congestive heart failure, hypertension, coronary obstructive pulmonary disease, diabetes, dementia, cerebrovascular disease, renal disease, rheumatic disease, cancer, pulmonary circulation disorder).
A multilevel logistic prediction model, with the hospital identifiers as the random intercept, was developed based on the previously mentioned predictors. The random intercept allows the calculation of the smoothed O/E ratio and accommodates the clustering effect of the hospitals in our dataset.Reference Debray, Collins and Riley30 Since we included multiple predictors in our model, we recognized the potential for multicollinearity. By checking the variance inflation factor, we found that most predictors displayed low levels, with only a couple showing slightly moderate levels, confirming that multicollinearity wasn’t a significant concern. Furthermore, following the recommendations of the “Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis” (TRIPOD) statement, we performed validation for our prediction model. The data was split non-randomly by fiscal years (FY) into a training dataset (FY2018: from April 1st, 2018 till March 31st, 2019) and a validation dataset (FY2019: from April 1st, 2019 till March 31st, 2020).Reference Moons, Altman and Reitsma31 Afterwards, model validation was performed by assessing and comparing performances in the training, validation, and overall datasets. The parameters for the performance assessment included: concordance statistic (C-statistic), the means of the observed use and the predicted probabilities, the calibration slope, and the accuracy.Reference Steyerberg32
Statistical analysis
We used our validated multilevel prediction model to obtain the risk-adjusted broad-spectrum antibiotic use and calculate the smoothed O/E ratios for each hospital as follows:
$$Smoothed\;O/E\;Ratio\; = {{Smoothed\;Observed\;Use\;}\over{Expected\;Use}} = {{\mathop \sum \nolimits_{i=1}^{{n_j}} {{\left( {1 + \;{e^{\left[ { - \alpha - \beta {x_{ij}} - {\gamma _j}} \right]}}} \right)}^{ - 1}}}\over{\mathop \sum \nolimits_{i=1}^{{n_j}} {{\left( {1 + \;{e^{\left[ { - \alpha - \beta {x_{ij}}} \right]}}} \right)}^{ - 1}}}}$$
The numerator is the sum of the probabilities of broad-spectrum antibiotic use from the multilevel model with the random intercept, while the denominator reflects the sum of the predictions from the same model without the random intercept. The confidence intervals (CIs) for the obtained Smoothed O/E ratios were generated through bootstrapping with 500 iterations, providing reliable estimates of variability.
All the statistical analyses were performed using R statistical software version 4.1.1 (R Foundation for Statistical Computing, Vienna, Austria), and the “lme4” package (ver 1.1.31) was used for fitting multilevel models. A P-value < .05 was considered statistically significant.
Ethics approval
The study was conducted in accordance with the “Ethical Guidelines for Medical and Health Research Involving Human Subjects” issued by the Ministry of Education, Culture, Sports, Science and Technology, the Ministry of Health, Labour, and Welfare, and the Ministry of Economy, Trade, and Industry. Moreover, it utilized anonymous data, and the need for informed consent from the study participants was waived by disclosing research information to the public. It was also approved by the ethics committee of the Graduate School of Medicine, Kyoto University (R0135).
Results
A total of 263,271 patients from 1,264 hospitals who met the inclusion criteria were extracted from the database. Figure 1 shows the selection process. Hospitals with fewer than 100 total patient cases were excluded, as some hospitals had very low numbers of total cases, while our model included 24 predictors. There was no notable difference in the data characteristics before and after excluding these cases (Supplementary Table). Eventually, 244,747 patients from 958 hospitals were analyzed. Furthermore, the training and validation datasets resulting from splitting the data consisted of 129,584 and 115,163 patients, respectively.
Figure 1. Flow diagram showing the selected dataset and the model validation subsets. Abbreviations: ICU, intensive care unit; FY, fiscal year.
Table 1 summarizes the baseline characteristics of the study population. The mean age was 81 (± 8.30) years, and 41.7% of the patients were female. Moreover, among all patients, 35.3% (n = 86,316) showed use of broad-spectrum antibiotics. In addition, the proportion of use of broad-spectrum antibiotics was higher among variables such as previous hospitalization in the last 90 days (44.4%), being on mechanical ventilator (62.5%), and being in the ICU (51.1%).
Table 1. Baseline patient characteristics in the dataset
Abbreviations: BUN, blood urea nitrogen; SpO2, saturation of peripheral oxygen; ICD, international classification of diseases; ICU, intensive care unit.
Table 2 sums up the model’s performance for the training set, validation set, and the combined full dataset through the previously mentioned performance parameters. There was minimal change in the performance parameters across the three sets.
Table 2. Model validation and performance
The C-statistics of our multilevel model were 0.722 and 0.632 with and without the random intercept, respectively. Figure 2 shows the smoothed O/E ratios of the use of broad-spectrum antibiotics as empiric therapy and their confidence intervals (CIs) across hospitals (sorted in ascending order). The graph demonstrates variability among the hospitals, with the lowest and highest ratio values being 0.13 (95% CI: 0.09–0.20) and 2.81 (95% CI: 2.64–2.97), respectively. Among the 958 hospitals, 312 had an upper 95% CI of the O/E ratio less than 1, indicating that the use of broad-spectrum antibiotics was less than the risk-adjusted average, while 331 hospitals showed a lower 95% CI greater than 1, indicating that the use was higher. Consequently, 315 hospitals had CIs crossing 1. Overall, there was notable variation in the smoothed O/E ratios among the hospitals.
Figure 2. Smoothed observed-to-expected (O/E) ratios of the use of broad-spectrum antibiotics among the hospitals. The dots and bars indicate the Smoothed O/E ratios and the 95% confidence intervals (CI), respectively. The left part represents hospitals with an upper 95% CI of the O/E ratio that is less than 1 (n = 312), while the right part represents hospitals showing a lower 95% CI of the O/E ratio greater than 1 (n = 331). Consequently, the middle black part represents the hospitals of which 95% CI includes 1 (n = 315).
Discussion
Using a nationwide inpatient database, we quantitatively benchmarked the use of broad-spectrum antibiotics in empiric therapy for older adult pneumonia inpatients through a risk-adjusted smoothed O/E ratio. Results showed wide variations among hospitals, and our method enabled the assessment of how much hospitals are using these antibiotics according to the risk-adjusted average. Subsequently, we were able to indicate which hospitals show a high ratio of broad-spectrum antibiotic use without being affected by patient characteristics and hospital differences.
Our study showed considerable variation among hospitals in the use of broad-spectrum antibiotics after performing risk adjustment. These findings align with previous studies that have also used a diagnosis-related system to establish risk-adjusted O/E ratios for benchmarking the use of various antibiotics.Reference Polk, Hohmann, Medvedev and Ibrahim12,Reference Yu, Moisan and Tartof16,Reference Tan, Vermeulen, Wang, Zvonar, Garber and Daneman17 In contrast to Polk et al., who used indirect standardization, and Tan et al., who utilized facility-level data and single-level regression, our study employed multilevel regression analysis with patient-level data and a random effect for facilities. This approach provides more detailed information for risk adjustment and appropriately accounts for the clustered nature of the patient data within hospitals, enhancing the accuracy of our estimates and improving comparisons among facilities.
Although Yu et al. also employed a multilevel model in one of their approaches, a key distinction in our study is the application of the smoothed O/E ratio rather than the standard O/E ratio. The smoothed O/E ratio has not been previously applied for benchmarking antimicrobial use, making its inclusion in this study a novel approach. The numerator of the smoothed O/E ratio (ie, Smoothed Observed Use) is a refined model-based estimate that minimizes the influence of extreme values and sample variation in the crude observed data. This approach enhances estimation and assessment in our study by providing a more reliable measurement. Moreover, our graph visualization of the ratio and CIs for each hospital offers a clearer indication of variability and statistical uncertainty compared to other studies, thereby enhancing the transparency of our benchmarking process.
While the previously mentioned studies used various quantity metrics for antibiotic use, such as days of therapy and defined daily dose (DDD), we used patient exposure as our metric.Reference Polk, Hohmann, Medvedev and Ibrahim12,Reference Yu, Moisan and Tartof16,Reference Tan, Vermeulen, Wang, Zvonar, Garber and Daneman17 Although the WHO recommends the defined daily dose (DDD) for measuring antibiotic consumption, there’s no gold standard for the magnitude of antimicrobial use, and DDD alone doesn’t always show a realistic picture in practice, as different metrics reflect different attributes of use.Reference Polk, Hohmann, Medvedev and Ibrahim12,33,Reference Berrington34 Since we are benchmarking the use of broad-spectrum antibiotics in empiric therapy, ie the first 48 hours of admission, the metric of exposure was suitable for our scope, and it was selected from a set of evidence-based and validated quantity metrics for antibiotic use.Reference Stanic Benic, Milanic and Monnier21 Nonetheless, we acknowledge that the duration of therapy is another important factor that should be considered or can be reflected using other metrics.
The inter-hospital variation in our study could be attributed to other factors such as resistance rates. However, since hospitals across Japan don’t drastically differ in rates of resistant pneumonia pathogens, the variation could be due to other modifiable factors.35 Thus, our method can aid in reducing the use regardless. Moreover, a previous study, aimed to identify patient and hospital factors that may contribute to differences in antibiotic use, has pointed out that university-affiliated hospitals showed a lower level of antibiotic unitization than other institutions.Reference Haug, Berild, Walberg and Reikvam36 Similarly, in our study, when we calculated the smoothed O/E ratio among hospitals grouped by teaching status, the group of non-teaching hospitals showed a slightly high ratio (1.22) compared to the teaching hospitals (0.98). It is worth noting that this is merely an observation for now, but further investigation could lead to more reliable inferences.
Our findings identified hospitals with high O/E ratios of broad-spectrum antibiotic use that may require further revision. By providing findings as feedback, hospitals can utilize such data as a resource for identifying potential needs for improvement and aligning with antimicrobial stewardship objectives. By providing more reliable and stable estimates than the standard O/E ratio, the smoothed O/E ratio enhances the accuracy of hospital comparisons, supporting better-informed decisions. Additionally, these findings can also assist policymakers and healthcare providers in understanding the situation regarding the use of these antibiotics for older adult pneumonia patients, guiding efforts to decrease their use in hospitals. Furthermore, this study may serve as a benchmark for future investigations of antibiotic use in hospitals, allowing subsequent research to explore areas of improvement in identified hospitals, to improve antimicrobial stewardship. It can also serve as a guide for future studies focusing on other types of infections.
This study was strengthened by the large sample size across multiple hospitals. Additionally, considering the hospital random intercept and smoothed O/E ratio provided an advantage for our approach and assessment over the standard O/E ratio. Moreover, presenting the 95% confidence intervals derived from bootstrapping demonstrated the uncertainty in our estimations.
Our study also had some limitations. First, the clinical appropriateness of the use of broad-spectrum antibiotics for patients individually was not examined. While that is important in clinical practice, our analysis focused on risk-adjusted benchmarking rather than clinical appropriateness. In addition, data that would allow for this direct assessment, such as microbiology results, was not available in DPC. Second, this study only included older adults treated for pneumonia from Japanese acute care hospitals. Although this indicates that the findings cannot be generalized to full populations or other infections, they do, however, emphasize a significant aspect of an important infectious disease primarily affecting that age group. The study’s purpose was not to benchmark for a full population, but rather to focus on pneumonia as a main infection with a high volume of antibiotic use.
In conclusion, we were able to demonstrate variability in broad-spectrum antibiotic use in the empiric treatment of pneumonia in older adults among hospitals through a risk-adjusted smoothed O/E ratio that considers patient-level risk factors and random variations. Additionally, our method identified the hospitals with high O/E ratios, where reflecting on practices may lead to beneficial changes.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ice.2025.5
Financial support
Y.I. received the JSPS KAKENHI Grant [JP23H00448] from the Japan Society for the Promotion of Science and the Health and Labour Sciences Research Grants [JPMH23HA2002 and JPMH24AA2006] from MHLW. The funders played no role in the study design, data collection and analysis, publishing decision, or manuscript preparation.
Competing interests
All authors declare no conflicting or competing interests.
Manuscript preparation
We thank Dr. Jung-ho Shin for his valuable support during the study.
Open access
