Ambulatory settings account for ∼60% of antibiotic expenditures, Reference Suda, Hicks, Roberts, Hunkler and Danziger1 and it has been estimated that ∼30% of antibiotic use in these settings is unnecessary. Reference Fleming-Dutra, Hersh and Shapiro2 In ambulatory settings, audit and feedback has been shown to decrease inappropriate antibiotic prescribing and was recently recommended among the core elements of ambulatory stewardship. Reference Gerber, Prasad and Fiks3–Reference Sanchez, Fleming-Dutra, Roberts and Hicks12 Behavioral science research shows individual data provision for quality improvement may be more effective than group data. Reference Brehaut, Colquhoun and Eva13 A recent before-and-after, quasi-experimental quality improvement initiative in 22 pediatric clinics in our healthcare system showed marked improvement in guideline-concordant care for 3 acute respiratory tract infections (ARTIs) when individual peer comparison data were provided monthly over time. Reference Clegg, Bean, Ezzo, Hoth, Sheedy and Anderson7 However, provision of these individual feedback reports is time- and resource-consuming and may not provide improvement over group feedback. In fact, we found no trials of ambulatory antimicrobial prescribing feedback comparing individual clinician performance with feedback on group performance.
In this context, we designed a cluster randomized trial to directly compare the effect of individual peer comparison feedback to group feedback on guideline-concordant care for 3 pediatric ARTIs among family medicine clinicians. Family medicine clinics were the focus for 2 reasons. First, previous studies show significant variability in guideline-concordant care for pediatric ARTIs between pediatricians and non-pediatricians. Reference Nyquist, Gonzales, Steiner and Sande14–Reference Frost, McLean and Chow16 Secondly, there are limited antimicrobial stewardship data for pediatric patients cared for by family medicine clinicians. We hypothesized that individual feedback would be superior to group feedback in improving appropriate antibiotic prescribing.
Methods
Trial design
In August 2015, retrospective review of data from January 1, 2014, to June 30, 2015, was conducted by the team. We then set eligibility criteria for a parallel group, cluster-randomized, controlled trial as follows: (1) family medicine practices within the healthcare system (n = 85); (2) upper respiratory tract infection or common cold measure (URI) of <83% (mean score in 2013 Healthcare Effectiveness Data Information Set in patients aged 3 months to 18 years, 17 n = 34); (3) ≥20 pediatric illness encounters with a diagnosis of URI in the previous 6 months (n = 28); and (4) consent by the lead clinician by e-mail for practice participation (n = 26). A practice consisted of at least 1 cluster or clinic site, and several practices in the system had multiple sites.
Participants
The 85 family medicine practices (some of which had multiple sites or clinics) in North Carolina, South Carolina, and Virginia, affiliated with 13 hospitals in the Novant Health system, included 341 physicians and 143 advanced practice providers. Novant Health uses Epic software (EpicCare,Verona, WI) for their electronic health records. The analytics and informatics group develops data sets from which quality and safety measurement data are abstracted. Our ambulatory antimicrobial stewardship team, developed in 2013, consisted of 2 quality specialists, a pharmacist, and 4 physicians (a pediatrician and 3 infectious disease specialists).
Interventions
All participating practices received the following:
-
1. A 1-hour, in-person, educational session, in September–October 2015, with the lead clinician (and occasionally other clinicians at the clinic), clinic administrator, and stewardship team physicians describing the project and clinical guidelines for URI, acute bacterial sinusitis (ABS), and acute otitis media (AOM). Reference Hersh, Jackson and Hicks18–Reference Chow, Benninger and Brook21
-
2. A tip sheet detailing how to improve scores (including appropriate codes and documentation strategies) (Supplementary Digital Content 1 online).
-
3. An after-visit summary for clinicians to give to patients and parents discussing antibiotic use and side effects (Supplementary Digital Content 2 online).
-
4. Clinic-level baseline performance data for all pediatric and family medicine clinics for the 3 ARTIs. Following the educational session, baseline and then monthly data were provided by e-mail to the lead clinician and clinic administrator. They were asked to forward via e-mail or to hand deliver the data to clinicians at the site and to share the data at monthly clinic meetings. Compliance with information flow to individual clinicians was measured only informally through multiple follow-up discussions in-person or via email with each clinic. The lead clinicians in the control group received clinic-level performance data only (Supplementary Digital Content 3 online), and lead clinicians in the intervention group additionally received clinician-specific data with individual comparisons limited to the specific site (Supplementary Digital Content 3 and 4 online). Thus, all practices could view performance data for all pediatric and family medicine practices by practice name at the clinic level. Intervention practices could also view individual scores by clinician name for their clinic sites but not others. For the very few clinicians who practiced in >1 site, performance data were provided for each site. Clinical decision support was not used.
-
5. Discussion about a new clinical pathway for acute bacterial sinusitis with a request for adoption and implementation.
Interventions were applied and measured at the cluster (ie, clinic) level and not at the patient level.
Data collection
Baseline performance data were collected retrospectively from January 1, 2014, through October 31, 2015. The intervention period was November 1, 2015, through December 31, 2017. The postintervention period was January 1 through December 31, 2018, during which time only clinic-level data were provided monthly to all 39 clinics. Visit-level data included International Classification of Disease, Ninth Revision (ICD-9) codes and, starting in October 2015, ICD-10 codes (see Supplementary Digital Content 5 online) associated with a patient’s illness encounter and listed as a visit diagnosis. Illness encounters were defined as evaluation and management visits for new patients (with codes 99201–99205) and for established patients (with codes 99212–99215).
Outcomes
Using guidelines from the American Academy of Pediatrics Reference Hersh, Jackson and Hicks18–Reference Lieberthal, Carroll and Chonmaitree20 and the Infectious Diseases Society of America, Reference Chow, Benninger and Brook21 customized clinical quality measures for the 3 ARTIs were developed and validated by selective, manual electronic chart review of the electronic record.
In brief, ARTIs were defined as follows. URI was defined as the percentage of children aged 3 months to 18 years diagnosed with URI who were not dispensed an antibiotic prescription on or within 3 days after the illness encounter. ABS was defined as the percentage of children aged 1–18 years diagnosed with ABS who were dispensed a first-line antibiotic (ie, amoxicillin or amoxicillin-clavulanate). AOM was defined as the percentage of children aged 6 months to 12 years diagnosed with AOM who were dispensed the first-line antibiotic (amoxicillin). Illness encounters were excluded if an antibiotic was prescribed within 30 days prior (if URI) or within 60 days prior (if ABS or AOM) to the illness encounter (for complete measure definitions, see Supplementary Digital Content 5 online).
The proportions of illness encounters receiving appropriate care for each ARTI over the baseline, intervention, and postintervention periods, respectively, were measured for each site as the number of illness encounters with appropriate care divided by the total number of illness encounters involving the ARTI. The primary outcomes were the relative changes in the proportion of illness encounters receiving appropriate care between the baseline and intervention periods for the 3 ARTIs. The same assessments between the intervention and postintervention periods were secondary outcomes. A target of ≥90% for URI was set using the HEDIS 2013 90th percentile 17 and, at 80% for ABS and AOM, consistent with targets suggested by an outpatient antibiotic use target-setting workgroup. 22
Other secondary outcomes were the baseline-to-intervention period change and intervention-to-postintervention period change in broad-spectrum antibiotic prescribing percentage (BSAP%), determined monthly by enumerating all antibiotics given for pediatric illness encounters, stratifying by narrow and broad spectrum (see Table 3 for definitions of each) and dividing by the total number of antibiotics prescribed for any condition, not limited to the 3 ARTIs. For this calculation, patients were excluded if their record showed an allergy to an antibiotic listed as narrow or broad spectrum and/or one of the listed antibiotics had been given in the prior 60 days.
To determine whether code shifting occurred or total antibiotic utilization changed after the intervention began, we recorded the mean number of encounters per clinic for the 3 ARTIs and the total of all pediatric illness encounters and antibiotics prescribed for all pediatric patients seen for illness in the baseline and intervention periods in both groups.
Randomization
Although outcomes were assessed at the clinic level, we randomized at the practice level to avoid intrapractice contamination of the intervention. The 26 participating practices were stratified by size: very large (2 practices, each with 5–10 clinic sites), large (4 practices with ≥9 clinicians), small (10 practices with 6–8 clinicians), and very small (10 practices with <6 clinicians). All but the very large practices were at a single site.
Using PROC SURVEYSELECT in SAS software, 23 half of the practices in each stratum were randomly selected to be in the intervention group, and the remaining practices were assigned to the control group (see Fig. 1).
Fig. 1. Flow diagram for clinic selection process.
Practices were unaware of randomization allocation until educational visits occurred. Practices could not be masked after allocation, and all continuously participated for the duration of the trial. Evaluators were unmasked after randomization.
Statistical methods
Power was calculated at the cluster (ie, clinic) level and was based on the primary outcomes, the change in appropriate care between the baseline and intervention periods for the 3 ARTIs. Following randomization of the available practices, the numbers of clusters were determined to be 22 (22 clinics in 13 practices) for the intervention group and 17 (17 clinics in 13 practices) for the control group. Based on retrospective data from the baseline period, the average cluster sizes (ie, the average numbers of encounters per clinic over 24 months or the average length of the baseline and intervention periods) for URI, ABS, and AOM, respectively, were estimated to be 229.1, 79.2, and 140.9 for the intervention group and 230.2, 71.5, and 145.7 for the control group. These data provided >85% power to detect a study group difference in mean baseline-to-intervention period change of 0.4 standard deviations, corresponding to a medium effect size Reference Cohen24 with 2-sided α = 0.05 for intracluster correlations ranging between 0.01 and 0.15. The power analysis was performed using PASS 15 software. 25
The clinic was the unit of analysis for describing changes in clinical decision making, whereas the illness encounter was the unit of observation. Clinic-level baseline variables are summarized for the intervention and control groups in Table 1.
Table 1. Baseline Characteristics of Practice Sites Allocated to Intervention and Control Groups
Note. ARTI, acute respiratory tract infection; URI, upper respiratory infection; ABS, acute bacterial sinusitis; AOM, acute otitis media.
All other analyses were performed using SAS software, 23 and graphics were created using R software. 26 To assess the influence of the intervention on clinical decision making, a generalized linear mixed model (GLMM) was analyzed for each outcome using PROC GLIMMIX in SAS software. The response variable in the models was of the form y/n, where y is the number of appropriate treatment occurrences and n is the total number of relevant encounters. The response distribution was specified as binomial with a logit link function. The independent variables were the study group (intervention and control), measurement period (baseline, intervention, and after intervention), and a study-group-by-period interaction term. The models included random-intercept terms at both the clinic and practice levels to account for relatedness of clinical decisions made in the same clinic or practice. The group-by-period interaction term was the estimate of interest used to compare the baseline period-to-intervention period (or intervention-to-postintervention period) relative change in appropriate prescribing between the intervention and control groups. Adjusted odds ratios and corresponding 95% confidence intervals were used to quantify the group difference in prescribing change. Analysis of the secondary outcome, BSAP%, was conducted using the same approach.
To account for multiple testing, the Holm step-down procedure Reference Holm27 was used to adjust P values, where the overall α level was prespecified at 0.05 and the family of inferences included the tests of group-by-period interaction for all the aforementioned models.
For each outcome variable, the intra-cluster correlation (ICC) was calculated using the level-2 (ie, random intercept for clinic) variance from the GLMM and a level-1 variance component assumed to be π Reference Fleming-Dutra, Hersh and Shapiro2 /3 = 3.29 for a logistic random intercept model. The ICC is a quantitative measure of within-cluster correlation, defined as the proportion of the total variance in the outcome attributed to the variance between clusters. Reference Wu, Crespi and Wong28
Ethical considerations
The protocol was reviewed and approved by the Institutional Review Board of Novant Health Presbyterian Medical Center, Charlotte, North Carolina.
Results
Figure 1 details the flow of participants. The intervention group (13 practices and 22 clinics) had 124 clinicians during the baseline period and 134 clinicians during the intervention period. For the control group (13 practices and 17 clinics), there were 101 clinicians during the baseline period and 103 for the intervention period. During the 4-year period, there were 173,131 pediatric illness encounters in the intervention group and 126,125 pediatric illness encounters in the control group. All clinics participated throughout all 3 periods studied.
Both groups were comparable at baseline with respect to all variables examined (Table 1). Both groups demonstrated improvement during the intervention period compared to baseline for each ARTI as shown graphically (Fig. 2). The intervention group improved performance for URI by 23.2% (absolute percentage change), ABS by 24.9%, and AOM by 26.2% (mean improvement for all 3 of 24.8%). The control group improved for URI by 17.0%, ABS improved by16.5%, and AOM improved by 18.5%, with mean improvement overall of 17.3% (Table 2).
Fig. 2. Proportion of illness encounters with appropriate prescribing for the intervention and control groups during the 3 periods studied.
Table 2. Comparison of the Percentage of Patients With Appropriate Treatment for ARTI in Intervention and Control Groups for 3 Study Periods
Note. ARTI, acute respiratory tract infection; URI, upper respiratory infection; ABS, acute bacterial sinusitis; AOM, acute otitis media.
a Proportions are totals for patients with appropriate treatment for the specific ARTI/specific pediatric illness encounters.
b Adjusted odds ratios and P values are generated from the generalized linear mixed model (GLMM) and compare the intervention and control groups on the change from baseline to intervention period, and on the change from the intervention period to the postintervention period. P values are adjusted for multiple testing.
Table 3. Comparison of Broad-Spectrum Antibiotic a Prescribing Percentage in Intervention and Control Groups for 3 Study Periods
a Broad-spectrum antibiotics include amoxicillin-clavulanate, azithromycin, cefaclor, cefdinir, cefixime, cefpodoxime, ceprozil, ceftriaxone, cefuroxime, ciprofloxacin, clarithromycin, clindamycin, erythromycin, levofloxacin, linezolid, loracarbef, and moxifloxacin.
b Adjusted odds ratios and P values are generated from the generalized linear mixed model (GLMM) and compare the intervention and control groups on the change from baseline to intervention period, and on the change from the intervention period to the postintervention period. P values are adjusted for multiple testing.
However, the intervention group had significantly greater improvement compared with the control group. For URI, the odds ratio (OR) was 1.62 (95% confidence interval [CI], 1.37–1.92; P < .01). For ABS, the OR was 1.45 (95% CI, 1.11–1.88; P < .01), and for AOM, the OR was 1.59 (95% CI, 1.24–2.03; P < .01) (Table 2).
When individual peer data were withdrawn for the intervention group for January–December 2018, gains in performance were sustained and scores continued to increase for all 3 measures comparing the intervention period and the postintervention period: URI increased from 87.1% to 91.7%; ABS increased from 77.0% to 78.5%; and AOM increased from 82.4% to 82.6%. For the control group, performance improved for URI from 77.1% to 83.1% and for AOM performance improved from 76.8% to 84.9% (Table 2). Neither group met the goal of ≥80% for ABS. The control group’s performance declined in the postintervention period for ABS from 73.8% to 70.0% (Table 2).
For the secondary outcome, reduction in BSAP%, the baseline mean for the intervention group was 50.5% and for the control group it was 44.8%. The 2 groups reached nearly identical means for the intervention period (40.3% for the intervention group and 39.5% for the control group) and continued to decrease in the postintervention period. However, reduction in BSAP%, comparing baseline-to-intervention periods, was significantly greater for the intervention group (OR, 0.80; 95% CI, 0.74–0.87; P< .01). Further declines were noted in the postintervention period to 34.4% and 35.9%, respectively, for the intervention and control groups (Table 3).
For both groups, compared to baseline, the mean number of illness encounters per clinic for URI and ABS increased for the intervention periods from 210 to 330 and from 73 to 107 for the intervention group. For the control group the mean number of illness encounters per clinic for URI and ABS increased from 211 to 265 and from 65 to 113, respectively. The mean number of illness encounters per clinic for AOM declined in both groups from 129 to 94 and from 134 to 97, respectively.
Total antibiotic utilization was 25% for each group at baseline and, in the intervention and postintervention periods, 25% and 24%, respectively, for the intervention group, and 27% and 27%, respectively, for the control group. Intracluster correlations, estimated from the GLMMs, for URI, ABS, AOM, and BSAP% were 0.172, 0.117, 0.138, and 0.063, respectively, indicating that 17%, 12%, 14%, and 6%, respectively, of the variability in the outcomes can be explained by between-clinic differences.
Discussion
In this cluster randomized trial among family medicine clinics, individual clinician feedback resulted in superior performance compared to clinic-level feedback for pediatric ARTIs. Specifically, individual peer comparison data yielded greater performance for URI, AOM, and ABS and a significantly greater reduction in BSAP%. Although a number of studies have demonstrated the value of individual clinician feedback, this trial was unique in directly comparing individual clinician feedback with clinic feedback.
Similar to our findings, in several pediatric patient studies where individual peer feedback was provided for ARTIs (but not compared to clinic feedback), prior studies have reported that individual clinician feedback is very effective for improving appropriate antibiotic prescribing. Gerber et al Reference Gerber, Prasad and Fiks3 evaluated antibiotic prescribing appropriateness for ABS, streptococcal pharyngitis, and pneumonia in a hospital-affiliated network of 18 pediatric practices (162 clinicians) that received personalized, private feedback and found that BSAP% was reduced in recipients compared with controls who received no feedback. Reference Gerber, Prasad and Fiks3 Kronman et al Reference Kronman, Gerber and Grundmeier4 also showed improvement in antibiotic prescribing with a distance learning program including individual clinician feedback. In our prior work, 10 pediatric practices that received individual feedback viewable only by other clinicians in their practice significantly improved performance on URI, ABS, and AOM, reaching ≥ 90% performance for each and significant reduction in BSAP% within 18 months. Reference Clegg, Bean, Ezzo, Hoth, Sheedy and Anderson7
As anticipated, improvements were maintained for the postintervention year because data provision continued for both groups at the clinic level, and we had shown significant improvement with this intervention alone for the control group.
Diagnostic shifting did occur with increases for both URI and ABS and declines in AOM, each occurring in both groups. Similar changes have been seen with pediatric clinics and individual peer comparison. Reference Clegg, Bean, Ezzo, Hoth, Sheedy and Anderson7 We suspect that these changes reflect more appropriate diagnostic coding resulting from education, as suggested by others. Reference Sharp, Hu and Shen29,Reference Madaras-Kelly, Hostler and Townsend30
Total antibiotic prescribing changed little after the intervention, as expected, since we did not focus on prescribing reduction but rather on guideline-concordant care. The reduction in use with improved URI performance likely represented a small proportion of total antibiotic use. In addition, whereas reduction in BSAP% was greater in the intervention group, it may have been even lower for both groups had we focused only on pediatric ARTIs and not included all pediatric illness encounters.
Our study had several strengths: stratified randomization of practices, continued participation and engagement of all clinics in each group during the course of the trial, and the relatively large number of geographically dispersed clinics in each trial group. Our study also had several limitations. It was conducted within a single healthcare system with a common electronic medical record and full support of an analytics group. Thus, these findings may not be broadly generalizable. Second, there was risk of contamination between groups because family medicine clinics regularly meet regionally and otherwise communicate with each other. Third, we did not estimate the extra time and resources for the intervention group, and although we received very few requests for patient data or discussion of the data provided, individual reports did not require additional time to compile each month. Fourth, we did not assess whether therapy was warranted for the diagnoses of ABS and AOM, but rather whether the chosen antibiotic was guideline-concordant. The sinusitis metric may have included patients with viral sinusitis who did not receive an antibiotic and may have penalized clinicians if no antibiotic was prescribed. We expected both groups to be affected equally, however. Finally, while the groups appeared similar at the clinic level, we did not analyze patient-level variables. We believe, however, that patient characteristics for illness encounters had little bearing on outcomes because the ARTIs studied are common in pediatric subpopulations, and our randomization approach likely provided sufficient balance in these variables between study groups.
Based on our cluster randomized trial, we conclude that individual peer comparison feedback achieves significantly greater performance improvement for 3 pediatric ARTIs and BSAP% compared with group (clinic) performance feedback. These findings add to the developing information about optimal approaches to improving appropriate antibiotic usage in the ambulatory setting. Further research is needed to understand how best to leverage peer influence to improve antibiotic prescribing, including evaluating automated clinician dashboard presentation of performance compared with direct email, and determining optimal feedback frequency.
Acknowledgments
We acknowledge the assistance of other members of the Novant Health Ambulatory Antimicrobial stewardship team including Rebecca A. Bean, BS, PharmD; Jacinta E. Chin, BA, PharmD, BCPS; Caitlin E. Falcone, BS; Robin C. Hahn, MPH; Matthew G. Hestwood, MPH; Alicia N. Hoth, BA, RN, BSN; Lisa M. Matthis, RN, MHA; Anaite Moore, BS; David H. Priest, MD, MPH; Crystal J. Redfern, MS, MBA; David J. Sheedy, MPH; Aaron M. West, BS, CPPS; and James B. Willis, MHA, CPHQ. We thank Rhett L. Brown, MD, FAAFP; Janice E. Huff, MD, FAAFP; and Mark L. Higdon, DO, FAAFP for their critical reviews of the manuscript.
Financial support
No financial support was provided relevant to this article.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ash.2021.212
Open access
