Introduction
Inappropriate antibiotic prescribing often results from diagnostic uncertainty or error. This gap in care quality has been identified as a primary, modifiable contributor to the global increase in antibiotic-resistant bacterial infections. Reference Ventola1 In the U.S. alone, 2.8 million antibiotic-resistant infections occur each year with 35,000 associated deaths. 2 Thus, there have been multiple “calls to action” related to antibiotic stewardship, including those highlighting the emergency department (ED). Reference May, Cosgrove and L’Archeveque3–6
Skin and soft tissue infections (SSTIs) account for approximately two percent of all ED encounters (~3 million annual visits) and available reports indicate frequent suboptimal antibiotic prescribing in this setting. Reference Kamath, Sudhakar, Gardner, Hemmige, Safar and Musher7–Reference Cairns and Kang10 Specifically, cellulitis is overdiagnosed in 30% of cases in which patients with noninfectious mimics, termed pseudocellulitis, are prescribed antibiotics. Reference Weng, Raff and Cohen8,Reference Arakaki, Strazzula, Woo and Kroshinsky11,Reference Pulia, Schwei and Alexandridis12 Among cases of accurately diagnosed cellulitis, providers often use two antibiotics despite randomized controlled trials demonstrating this does not improve clinical outcomes. Reference Pallin, Binder and Allen13–Reference Leman15 Additionally, uncomplicated abscesses are often treated with one or more antibiotics despite high cure rates being observed with incision and drainage alone across two large RCTs (number needed to treat (NNT) for antibiotics to prevent treatment failure = 14–26). Reference Pulia and Fox16–Reference Talan, Mower and Krishnadasan19
Our previous qualitative work characterized drivers of antibiotic decision-making for SSTIs among emergency medicine (EM) physicians which were then mapped to potential interventions. Reference Pulia, Schwei, Hesse and Werner20 Intervention mapping is a systematic framework used for the development, implementation, and evaluation of clinical interventions through incorporation of evidence-based practices and the social and cognitive determinants of clinical decision-making. Reference Bartholomew Eldredge21,Reference Fernandez, Ruiter, Markham and Kok22 As there are limited resources to implement stewardship-focused interventions in the ED, we sought to conduct an experiment to quantitatively assess the potential impact of the developed interventions. In order to capture changes in antibiotic prescribing decisions for SSTIs based on these interventions, we utilized case vignettes that presented brief clinical scenarios with various combinations of the proposed interventions and asked EM physicians if they would treat with antibiotics. Case vignettes have historically been used to study clinical decision-making due to their flexibility in manipulating multiple clinical factors while being cost-effective and overcoming ethical limitations of experimental research with real patients. Reference Evans, Roberts and Keeley23
There is a clear need to evaluate and prioritize interventions that can optimize antibiotic use in the management of skin and soft tissue conditions evaluated in the ED. Thus, the objective of this study was to evaluate the effectiveness of a set of interventions in minimizing inappropriate prescription of antibiotics for presumed SSTIs in the ED using a case vignette survey design to simulate the decision-making process.
Methods
We conducted a case vignette study via an electronic survey with a national sample of EM physicians. The case vignettes followed a factorial design where the inclusion of each intervention factor was systematically varied, enabling assessment of the relative influence of each intervention on the provider’s decision to prescribe an antibiotic for presumed SSTI.
Participants
Our target population was practicing EM physicians, which we operationalized as post-residency physicians who self-reported working at least three shifts a month in an ED and who identified EM as their primary specialty. The University of Wisconsin Survey Center facilitated the recruitment of EM physicians for our electronic survey. We purchased a random list of 2000 EM physicians licensed in the U.S. through the American Medical Association from IQVIA. Three percent of surveys were distributed to EM physicians ages 71–80, 13% to EM physicians ages 61–70, 20% to EM physicians ages 51–60, 33% to EM physicians ages 41–50, and 31% to EM physicians ages 31–40. We selected this distribution based on the age distribution of the entire IQVIA dataset and excluding EM physicians > 80 and < 31 to reduce the number of retired or training EM physicians approached. We conducted three rounds of survey outreach from January 2020 to June 2020 and approached a total of 1000 EM physicians with a pre-notification letter including a cash pre-incentive, invitation email, and follow-up emails. The University of Wisconsin Institutional Review Board deemed this study to be exempt.
Case vignettes
The case vignettes consisted of a base clinical scenario for cellulitis (Figure 1) or abscess (Figure 2) which were modifiable by inclusion of the intervention factors. We developed the base clinical scenarios to portray an ambiguous dermatologic complaint, for which the differential would include SSTI, to better capture differences in antibiotic prescribing. We completed pilot testing and cognitive interviewing with EM physicians to get feedback on the content and wording of the case vignettes to ensure they evoked clinical uncertainty.
Figure 1. Cellulitis vignette base clinical scenario and mapped interventions.
Figure 2. Abscess vignette base clinical scenario and mapped interventions.
The case vignettes contained five binary factors that were either present or absent in unique combinations for each vignette. Using a systems-engineering informed qualitative approach, we previously identified barriers to appropriate antibiotic prescribing and mapped interventions to address these barriers. Reference Pulia, Schwei, Hesse and Werner20 The five factors included 1) lack of access to care, 2) patient expectations, 3) diagnostic uncertainty, 4) fear of adverse outcomes, and 5) provider knowledge gaps. The phrases in bold and underlined in Figure 1 and Figure 2 indicate where each of the mapped interventions would be inserted into the vignette. Table 1 lists each mapped intervention and its associated factor and clinical scenario by condition.
Table 1. Mapped interventions for appropriate antibiotic prescribing for cellulitis and abscess
All combinations of the five interventions being present/absent produced 32 case vignettes per condition, with 64 case vignettes in total. Which factors are included in each numbered vignette is described in a design matrix (Supplement Table 1). It was unrealistic to ask participants to read 64 vignettes, so we utilized a balanced incomplete block design where participants responded to 8 vignettes, 4 for cellulitis and 4 for abscess. Reference Box, Hunter and Hunter24 We chose 4 vignettes per condition to balance participant burden (response rate) and factor coverage. The respondents for each factor block (e.g. 10001 factor block: first and fifth factors are present) were shown both cellulitis and abscess vignettes for that assigned block.
We randomized which four factor blocks were presented to participants and whether a cellulitis or abscess vignette was presented first. The survey then alternated between presenting a cellulitis or abscess vignette. We randomized the order within the lineup of cellulitis and abscess vignettes such that the cellulitis and abscess vignettes of the same factor block may or may not have ended up next to each other in the survey (Supplement Figure 1).
Following each vignette, we asked participants to indicate how likely they were to prescribe antibiotics for that case (not very likely, slightly likely, somewhat likely, very likely, extremely likely) and if they would prescribe an antibiotic in their everyday practice (yes/no). For cellulitis cases only, we also asked participants how likely it was that the patient had cellulitis (not very likely, slightly likely, somewhat likely, very likely, extremely likely).
Statistical analysis
We described the cohort of survey respondents according to their basic demographics, including gender (female, male, not reported), ethnicity (Hispanic/Latino, not Hispanic/Latino, not reported), race (American Indian or Alaskan Native, Asian or Hmong, Black or African American, Native Hawaiian or Other Pacific Islander, White, other), training (American Board of Emergency Medicine Certified (yes/no), years of post-residency clinical experience (<5, 5 to 9, 10 to 14, 15 to 19, 20 to 24, 25+), and characteristics of their primary practice site (ED setting (urban, suburban, rural), type of ED (community, academic, veterans affairs, critical access, other), U.S. census region (Midwest, Northeast, South, West).
Utilizing judgment analysis methods, we built multilevel logistic regression models for each participant to infer how they weighed the intervention factors in their decision whether to prescribe an antibiotic. The analysis consisted of a hierarchical logistic model possessing two regression equations, one modeling the vignette effects within the participants, and the other modeling participant effects between participants (see Supplement Material 1 for model specification). Reference Brown, Brown, Saunders, Castelaz and Papasouliotis25–Reference Hox, Kreft and Hermkens27
We conducted two analyses according to these methods in order to estimate antibiotic prescribing probabilities, one to estimate probabilities for each factor and one to estimate probabilities for each vignette. The factor analysis incorporated indicator variables for each of the five factors into the model so that individual vignettes could be represented by the presence/absence of each factor. Alternatively, the vignette analysis included a variable for the vignette number (1–32). Both analyses included a cellulitis/abscess indicator and interaction terms between the cellulitis/abscess indicator and factor indicators or vignette number so prescribing probabilities for each condition could be distinguished. The factor model aggregated the prescribing probabilities across vignettes with each factor to estimate the overall prescribing probability per factor. The vignette model simply produced individual prescribing probabilities for each of the 64 case vignettes (32 per infection type). We adjusted for the following covariates in both models: years of experience, confidence in diagnosis, attitude about inappropriate antibiotic prescribing, self-reported inappropriate antibiotic prescribing behavior, and familiarity with the Infectious Diseases Society of America antibiotic prescribing guidelines.
In a final analysis of the cumulative effect of including multiple mapped interventions, irrespective of the specific interventions, we categorized the vignettes as having zero, one, two, three, four, or five intervention factors present. For instance, the cellulitis vignette with the thermal imaging camera intervention and clinical decision support intervention would be considered a vignette with two intervention factors. We then calculated the proportion of participants that would prescribe an antibiotic for vignettes grouped by number of factors and condition and assessed differences in prescribing via a Chi-square test. All analyses were conducted in Stata 17.
Results
Of the 1000 survey invitations sent out, 139 surveys were started (13.9% response rate) and 113 surveys were completed (13 respondents were ineligible, 13 surveys were incomplete). Participants were primarily male (71.6%), not Hispanic/Latino (90.3%), White (84.1%), and American Board of Emergency Medicine certified (85.8%) with 20 or more years post-residency clinical experience (51.4%) (Table 2). Participants most commonly worked at suburban (40.7%) and community EDs (69.0%) in the Midwest (38.1%).
Table 2. Physician and practice setting characteristics (n = 113)
Each of the 32 vignettes had a median of 14 (Min: 11, Max: 16) responses (Supplement Table 1) with each intervention appearing 224 to 228 times per condition. Across all vignettes, participants would have been prescribed antibiotics in 39.2% of cellulitis cases and 59.3% of abscess cases.
A response of “A great deal” to “How much of a problem is inappropriate antibiotics in the emergency department?” was associated with significantly lower antibiotic prescribing than participants responding “A little” with an odds ratio (OR) of 0.127 (95% CI: 0.028 – 0.588, p = 0.01) in the factor model with similar results in the vignette model. More years of clinical experience was also associated with lower antibiotic prescribing in the factor model (and similarly in the vignette model) with 20 to 24 years (OR 0.134 95% CI: 0.023 – 0.797, p = 0.03) and 25 or more years (OR 0.137 95% CI: 0.024 – 0.782, p = 0.03) of clinical experience having significantly lower prescribing than participants with less than five years of experience (Supplement Table 2 and 3).
Figure 3a compares the adjusted antibiotic prescribing probabilities for cellulitis when the five mapped interventions were present versus absent in the case vignettes. The thermal imaging camera intervention showed the largest decrease in antibiotic prescribing probability of 17.2% (95% CI: 16.7% – 17.7%). The patients raising concern over potential antibiotic side effects/adverse reactions (representing education materials or a shared decision-making intervention) had a similarly large decrease in antibiotic prescribing probability of 15.2% (95% CI: 14.8% – 15.6%). The remaining interventions of paramedicine or telehealth follow-up, inappropriate antibiotic prescribing flag, and clinical decision support had smaller decreases in antibiotic prescribing probability of 6.1% (95% CI: 5.9% – 6.3%), 1.3% (95% CI: 1.2% – 1.4%), and 6.8% (95%CI: 6.6% – 7.1%) respectively.
Figure 3. Adjusted prescribing probabilities by mapped interventions for cellulitis (Figure 3a) and abscess (Figure 3b).
Figure 3b compares the adjusted antibiotic prescribing probabilities for abscess when the five mapped interventions were present versus absent in the case vignettes. The largest decrease in antibiotic prescribing probability for abscess was 20.9% (95% CI: 20.2% – 21.5%) for the patient education materials or shared decision-making intervention followed by 16.9% (95% CI: 16.3% – 17.5%) for the rapid MRSA PCR intervention. As with cellulitis, the interventions of paramedicine or telehealth follow-up, inappropriate antibiotic prescribing flag, and clinical decision support had smaller decreases in antibiotic prescribing probability for abscess of 4.2% (95% CI: 4.0% – 4.4%), 1.3% (95% CI: 1.2% – 1.4%), and 2.7% (95%CI: 2.7% – 2.8%) respectively.
Figure 4 displays the adjusted antibiotic prescribing probabilities for all 64 case vignettes in a radar plot. The numbers 1 to 32 circling the plot indicate the vignette number (see associated interventions for each vignette number in Supplement Table 1) with the adjusted antibiotic prescribing probabilities plotted radially with 0% probability at the center and extending to 100% probability at the edge. This shows that in the majority of cases, participants had a higher probability of prescribing antibiotics for abscesses compared to cellulitis. The figure also demonstrates that the modification of the case vignettes through the inclusion of the mapped interventions resulted in considerable differences in antibiotic prescribing behavior.
Figure 4. Radar plot of adjusted prescribing probabilities by vignette for cellulitis and abscess.
The abscess case vignettes with the four lowest adjusted antibiotic prescribing probabilities (23, 7, 15, 24) all included both the patient education materials or shared decision-making intervention and the rapid MRSA PCR intervention. Similarly, the cellulitis case vignettes with the three lowest adjusted antibiotic prescribing probabilities (15, 23, 16) all included both the patient education materials or shared decision-making intervention and the thermal imaging camera intervention.
The proportion of participants that would prescribe antibiotics for vignettes with zero, one, two, three, four, or five intervention factors present was significantly different for both cellulitis (χ2 = 20.8, P = 0.001) and abscess (χ2 = 25.2, P < 0.001) (Figure 5). Note that the number of responses for each category of zero to five intervention factors present was quite different due to the number of ways vignettes could be constructed (e.g. one vignette with zero intervention factors vs. ten vignettes with three intervention factors). The lowest proportion of antibiotics were prescribed for vignettes with four intervention factors for cellulitis (26.1%) and for vignettes with three intervention factors for abscess (47.6%).
Figure 5. Proportion prescribed antibiotics for cellulitis and abscess by number of intervention factors.
Discussion
To our knowledge, this study represents the first judgment analysis applied to hierarchical prioritization of antibiotic stewardship interventions. Our vignette design effectively facilitated judgment analysis as demonstrated by the variety of prescribing responses elicited by the different vignettes. The significant difference in proportion prescribed by number of factors also shows that manipulation of the factors produced different prescribing decisions. Our overall findings indicate that the application of diagnostic tools for skin and soft tissue infections has the highest potential for reducing inappropriate antibiotic prescribing. This finding reinforces the documented difficulty in distinguishing cellulitis from pseudocellulitis and the need for novel diagnostic methods to assist in differential diagnosis. Reference Pulia, Schwei and Alexandridis12,Reference Ko, Raff and Garza-Mayers28
Skin surface thermal imaging allows for potential real-time testing that can decrease misdiagnosis and prevent unnecessary antibiotic prescribing. Recently, our team published a large diagnostic validation study which found that differences in detection of skin surface temperature through thermal imaging, with or without the additional predictive tool ALT-70 (asymmetry, leukocytosis, tachycardia, and age ≥70 years) would result in improved diagnosis of cellulitis and could serve as a valuable tool to decrease overdiagnosis in the ED. Reference Pulia, Schwei and Alexandridis12 Also in agreement with our findings, prior research has found rapid MRSA PCR testing following incision and drainage of an abscess improves appropriate usage and de-escalation of anti-MRSA antibiotic agents. Reference May, Rothman and Miller29,Reference Bridwell, Bajaj, Gress, Hambuchen and Clay30
The other highly effective intervention in reducing antibiotic prescribing for cellulitis and abscess was patients raising concerns for potential side effects/adverse reactions (e.g. Clostridiodes difficile). This intervention was designed to represent the provision of educational materials and/or shared decision-making tools that encourage patients to directly ask the provider about the potential risks and benefits of antibiotics. Increasingly the importance of patient engagement in the diagnostic process has been recognized as essential to improving patient care. Reference McDonald, Bryce and Graber31–Reference Schoenfeld, Kanzaria and Quigley34 Qualitative data shows that patients want to be involved in the decisions surrounding their care in the ED, but often do not feel empowered to do so. Reference Schoenfeld, Goff, Downs, Wenger, Lindenauer and Mazor33 Additionally, there are EM physician-identified barriers that make it challenging to consistently engage in shared decision-making with patients. Reference Schoenfeld, Goff and Elia35 There is a paucity of literature examining how to pragmatically implement patient engagement strategies like shared decision-making in the ED for infectious conditions and their downstream impact on diagnostic and antimicrobial stewardship metrics.
The community paramedicine program or telehealth follow-up, provider flag for inappropriate antibiotic prescription requests, and clinical decision support interventions had a smaller impact on EM physician antibiotic prescribing decisions than diagnostic tools or patient education. We propose that these interventions have potential and should be considered as part of intervention bundles but would be lower on the prioritization list as individual interventions.
It is important to highlight the limitations of our study. First, physicians made decisions based on case vignettes that do not contain all clinically relevant factors that would be present in a real case. Additionally, the influence of interacting with a patient face-to-face could not be replicated in the vignettes. Another limitation was that the second and third rounds of survey distribution occurred during the first four months of the COVID-19 pandemic in the U.S. Response rates may have been lower due to the burden placed on EM physicians by the pandemic. The observed low participation from providers working in critical access settings is likely due to our inclusion criteria requiring primary specialization in emergency medicine. It is important to note that these findings may not be generalizable to settings with limited resources or primarily staffed by providers without specialization in emergency medicine. Finally, our sample size was too small to incorporate the interactions between factors in our models to directly assess the impact of combinations of interventions. However, we found the case vignettes with the lowest adjusted prescribing probabilities for both abscess and cellulitis included the patient education materials or shared decision-making intervention and diagnostic tool (rapid MRSA PCR or thermal imaging camera) intervention, indicating the combination of these two interventions may be the most effective in reducing inappropriate antibiotic prescribing.
Our study successfully identified several rigorously designed ED antibiotic stewardship interventions for SSTIs that had a significant impact on prescribing behavior in simulated case vignettes. Prioritization of stewardship interventions in experimental contexts prior to implementation is critical to preserve resources and optimize the likelihood of real-world effectiveness. Based on our results, we recommend future research focused on the development and integration of novel diagnostic tools, such as thermal imaging, and interventions focused on patient engagement and shared decision-making to improve diagnosis and management of SSTIs in the ED.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ice.2024.211
Financial support
This work was supported by the Central Society for Clinical & Translational Research Early Career Development Award, by the Agency for Healthcare Research and Quality (award no. K08HS024342), and by a Clinical and Translational Science Award program, through the National Institutes for Health National Center for Advancing Translational Sciences (NCATS grant no. UL1TR002373).
Competing interests
All authors report no conflicts of interest relevant to this article.
Open access
