Hostname: page-component-586b7cd67f-t8hqh Total loading time: 0 Render date: 2024-11-23T00:49:21.608Z Has data issue: false hasContentIssue false

Regional needs assessment for emergency physician audit and feedback

Published online by Cambridge University Press:  13 May 2020

Rakesh Gupta
Affiliation:
Division of Emergency Medicine, McMaster University, Hamilton, ON
Mathew Mercuri
Affiliation:
Division of Emergency Medicine, McMaster University, Hamilton, ON
Amy McCulloch
Affiliation:
McMaster University, Hamilton, ON
Sandra Monteiro
Affiliation:
Department of Health Methods, Evidence, Research and Impact, Center for Simulation Based Learning (CSBL), McMaster Program Education, Research, Innovation, and Theory (MERIT), McMaster University, Hamilton, ON
George Mastoras
Affiliation:
Department of Emergency Medicine, University of Ottawa, Ottawa, ON
Ashley Krywenky
Affiliation:
University of Ottawa, University of Ottawa Skills and Simulation Center, Ottawa, ON
Teresa Chan
Affiliation:
Division of Emergency Medicine, Department of Medicine at McMaster University, Program of Faculty Development in Faculty of Health Sciences at McMaster University, Hamilton, ON
Shawn Mondoux*
Affiliation:
Division of Emergency Medicine, Department of Medicine at McMaster University, Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON
*
Correspondence to: Dr. Shawn Mondoux, 237 Barton Street E., Hamilton, ONL8L2X2; Email: [email protected]

Abstract

Objectives

Audit and feedback is widely used to improve physician performance. Many data metrics are being provided to physicians, yet most of these are driven by the regulatory environment. We sought to conduct a needs assessment of audit and feedback metrics that were most useful to clinicians within our health care region.

Methods

We conducted a Web-based survey of five clinical practice sites in our region and asked that physicians rank 49 clinical practice metrics. In addition, we assessed their readiness for audit and feedback and their preferences for data confidentiality. We collected data on duration of training, gender, and site of practice (academic v. community) allowing for comparison between groups.

Results

A total of 104 emergency medicine physicians participated in the survey (52.3% response rate). There was a significant readiness for participation in audit and feedback activities. Top ranked metrics were emergency department return rates and colleague's assessment of collegiality and quality of care, which were common across all sites. Small yet significant differences were noted between genders and academic v. community practitioners.

Conclusion

This study represents the first regional analysis of physician preferences for audit and feedback activities and implementation. It demonstrates that physicians are interested in audit and feedback activities and provides a roadmap for the development of a regional audit and feedback structure. It will also be used as a guiding document for regional change management.

Type
Original Research
Copyright
Copyright © Canadian Association of Emergency Physicians 2020

CLINICIAN'S CAPSULE

What is known about this topic?

Physicians are increasingly being provided with feedback on their own clinical practice metrics, called audit and feedback.

What did this study ask?

We assessed whether physicians wanted to participate in audit and feedback activities, asked which data were most important, and then sought differences among subgroups.

What did this study find?

We found physicians wanted audit and feedback, had clear and common preferences for the data being reported with some notable subgroup differences.

Why does this study matter to clinicians?

Physicians want audit and feedback and have clear data preferences that should be used to drive implementation success and improvement.

INTRODUCTION

Audit and feedback is a widely used strategy for assessing and improving physician performance.Reference Ivers, Jamtvedt and Flottorp1Reference Cooke, Duncan, Rivera, Dowling, Symonds and Armson3 It involves measurement of performance on several metrics and feedback of those results to physicians for comparison against peers or other benchmarks. There is evidence that audit and feedback is both demanded by physiciansReference Lavoie, Plint, Clifford and Gaboury4 and leads to modest improvements in professional practice.Reference Ivers, Jamtvedt and Flottorp1

With the growth and increasing availability of robust electronic health records systems, there is now a greater ease in measuring and reporting physician clinical performance measurement and reporting. Much of these activities have been driven by hospital administrators, government agencies, and quality assessment organizations rather than physicians themselves.Reference MacLean, Kerr and Qaseem5 Limitations in the available data, in addition to insufficient engagement with clinicians, has led to the use of metrics poorly suited for the goal of improving care, including measures of practices unsupported by evidence.Reference MacLean, Kerr and Qaseem5

Meanwhile, when initiating new educational initiatives, it is common to engage in some level of needs assessment to establish the stated and hidden needs of one's intended audience.Reference Stefan, Hall, Sherbino and Chan6,Reference Chan, Jo and Shih7 Needs assessments may be a powerful tool to understanding how best to engage in audit and feedback implementation but are lacking in the current literature.

Recognizing the local peculiarities of all audit and feedback activities and their implementation, we conducted a regional needs assessment of emergency medicine (EM) physicians in the Southern Ontario. A needs assessment process such as this has not previously been completed for EM physicians in Canada. We aimed to assess EM physician's impressions of audit and feedback, including their perceptions of specific metrics from diverse practice categories and their attitudes toward who should have access to practice data.

METHODS

We surveyed EM physicians in our region to assess their audit and feedback needs (online Supplemental Appendix A). The data metrics used in the survey were based on a framework currently used in the emergency department (ED) at the University of Calgary.

Survey development

The survey included participant demographics, as well as questions regarding attitudes toward audit and feedback. The survey covered the following broad feedback categories included in the University of Calgary framework: Operational, Flow, Balancing, Resource Utilization, Patient Satisfaction, and Disease-Specific Measures. We added Collaboration, Education, and Patient-Reported Outcome Measures categories to the survey to unify current feedback activities with newer metrics and to include novel ideas that are presented in the literature. Further questions to assess respondents’ preferences for audit and feedback frequency and data anonymity were also included. All questions were evaluated on a 10-point Likert scale.

Two experienced researchers (M.M. and K.W.) assessed the first draft for face and content validity. The revised survey was then uploaded to SurveyMonkey®, which includes a proprietary survey review tool. A local qualitative researcher then reviewed the survey for content, methods, and language (T.C.). Our chief resident with methodological expertise in Quality Improvement then reviewed the survey for readability, consistency, and comprehension (R.G.). At this stage, the survey was reviewed by a survey expert and qualitative researcher at McMaster Education Research, Innovation, and Theory (MERIT) (Sa.M.). A local physician colleague then performed a cognitive review of the survey for the research team and provided feedback on modifications, improvements, and clarity (A.M.). Finally, the survey was completed and reviewed by two external clinicians from the University of Ottawa Department of Emergency Medicine to again assess the face and content validity, as well as readability (G.M. and A.K.).

Participants

Respondents were recruited through seven local health systems. Emails requesting participation were sent to each clinical chief for dissemination. Respondents were entered into a draw for a $150 gift card. The recruitment email was sent at the end of January 2019. Reminder emails were sent in mid-February and early March 2019. The final email soliciting participation included a “leaderboard” of participating sites in an effort to gamify survey responses. The target for total survey responses was 100 respondents. The project was granted Quality Improvement (QI) exemption from Hamilton Integrated Research Ethics Board (HiREB).

Analysis

Means and standard deviations were calculated for each question. Overall rankings of metrics were conducted based on averages and standard deviations. In instances where averages were the same, those metrics with smaller standard deviations were ranked more highly. Unpaired t-tests were used for gender-group and practice-type comparisons. The statistical testing was only applied to questions in which there was a mean difference of greater than 0.5 on the 10-point Likert scale, as the research team believed that to be the lower limit value for significance in real practice. Data regarding duration of practice were stratified into three duration-based classifications for each question. Analysis of variance was not performed due to the significant number of study questions (49) and the likelihood of positive finding due to chance alone. Our study purpose was primarily descriptive, and any statistical comparisons were meant for hypothesis generation only.

RESULTS

Reporting guidelines as described by Kelley et al.Reference Kelley, Clark, Brown and Sitzia8 were followed for this publication. A total of 104 EM physicians responded to the survey from a total of six hospitals and one regional health system (Niagara Health - containing five clinical ED sites). Among these institutions, one hospital treated only a pediatric population (McMaster Children's). Two regional community hospitals had only one respondent to the survey, both of whom were included in summary data but eliminated from site-specific descriptions. After excluding the sites with single respondents, there were a total of 195 EM physicians, yielding an overall response rate of 52.3%. The percentage of respondents from each clinical site ranged from 27.8% to 81.0%.

Five respondents identified both as academic and community practitioners and their data were included in both the academic and community cohorts for analysis as it was believed that this was a better representation of their actual practice and was likely to diminish statistically significant differences rather than accentuate them. A total of 69 (66.3%) practitioners identified as having an academic practice and 40 (38.5%) as having a community practice. Thirty-eight (36.5%) respondents identified as female and 65 (62.5%) as male. The total percentage of males employed at each site ranges from 57.1% to 83.3%. Respondents were for the most part within the first 10 years of practice (42 at <5 years, and 65 at <10 years). Only 17 respondents had more than 20 years of practice.

Attitudes on audit and feedback

Respondents indicated the desire to engage in more audit and feedback regarding their clinical practice and believed audit and feedback could improve their clinical practice. The distribution of responses to questions regarding attitudes on audit and feedback are presented in Table 1.

Table 1. Responses for attitudes towards audit and feedback

Overall metric rankings

The entire family of proposed metrics was sorted by medians, then by the lower quartile followed by the upper quartile value. The 10 metrics that were rated highest by respondents are displayed in Table 2, and the bottom 10 rated metrics are displayed in Table 3. A full ranking on individual metrics can be found in the online Supplemental Appendix B.

Table 2. Top 10 metrics as evaluated by respondents

Table 3. Bottom 10 metrics as evaluated by respondents

Site-specific comparisons

All five clinical sites that were surveyed had the following four data elements in common within their top 10 lists: 72 hour return and admit rate, 72 hour return admitted to the operating room/intensive care unit (OR/ICU), EM physician evaluation of my collegiality, and EM physicians evaluation of the quality of care that I provide. The remaining elements in the top 10 at each site are found in Table 4 in rank order.

Table 4. Other top 10 elements for each specific site included in the survey

Data privacy and dissemination

Questions around data anonymity had only 90 respondents. Of these, 82 (91.1%) preferred seeing the distribution of data within their department but only being able to identify their own practice data. Two respondents hoped to all un-anonymized data, and six respondents believed they should have access to only their data without comparators. Table 5 details whom respondents believe should have access to their own practice data.

Table 5. Privacy and dissemination preferences of respondents

Notable differences between academic and community practitioners

Overall, few statistically significant differences were observed between academic and community practitioners. Comparisons between both groups identified that community physicians were more interested in having their transfer rates reported to them (5.71 v. 4.52; p = 0.03), were more interested in registration to physician assessment time (5.17 v. 4.19; p = 0.01), were less interested in 72 hour ED return rate with ICU/OR admission (8.35 v. 9.05; p = 0.04), and were significantly more interested in the management of stroke (7.33 v. 5.69; p = 0.004).

Notable differences between genders

There were statistically significant gender differences in some metrics. Female respondents were less likely to believe that receiving more practice data would help them improve (7.30 v. 8.30; p = 0.02), were less likely to want more data about their clinical practice (7.22 v. 8.11; p = 0.02), and were less likely to think that providing individuals with practice data was important (7.16 v. 8.36; p = 0.01). The only other statistically significant data point was that female respondents rated the metric of “Number of patients seen per shift” lower than their male counterparts (5,82 v. 6.87; p < 0.01)

Notable trends between years in practice

Those individuals with less than 5 years of practice ranked almost all metrics higher than their peers with the exception of the disease-specific metrics and patient outcomes categories. Those practitioners with between 16 and 20 years of practice placed a higher value on disease-specific and patient outcome metrics. Those practitioners between 5 and 10 years or those with greater than 20 years of experience generally rated the value of audit and feedback metrics the lowest through the majority of categories.

DISCUSSION

The results of this survey show a strong desire by EM physicians to engage in local audit and feedback. This is in keeping with previous findings described in the literature.Reference Lavoie, Plint, Clifford and Gaboury4 Across all sites, 72-hour return metrics figured prominently and has been described in the literature.Reference Calder, Pozgay and Riff9 Surprisingly, peer ER physician assessments of collegiality and competence were highly sought-after metrics, both of which are also described.Reference Norcini10Reference Ramsey, Wenrich, Carline, Inui, Larson and LoGerfo13 In addition, this study shows that patient satisfaction metrics and patient-reported outcome metrics were important for physician audit and feedback in many centers. Finally, metrics focused on adherence to clinical practice guidelines (CPGs) were meagerly represented within the top metrics rated by clinicians, with a focus only on adherence to sepsis among the top 10. In contrast, our regional pediatric center rated five metrics related to adherence to CPGs among their top 10.

Of interest, many of the lowest-ranked metrics were focused on ED resource use. Other literature has documented that, when surveyed, EM physicians believe that there is an overuse of medically unnecessary advanced diagnostic imaging.Reference Kanzaria, Hoffman, Probst, Caloyeras, Berry and Brook14 Our study allowed respondents to rate the importance of resource use among other practice metrics. It may be that participating physicians place less value on resource awareness in the ED when compared with other practice metrics, supporting the need for initiatives such as Choosing Wisely Canada.15 An alternative explanation may be that the physicians believe there is little room for improvement in these areas, which could be the case in their local environments. For some of these metrics, the notion that there is little room for improvement would not be supported by the literature (de Wit, unpublished findings: Qualitative assessment of physician practices of DVT and PE diagnostic ordering practices, October 2019). Combined with the presence of the “consultation rate” among the lowest-ranked metrics, it is also possible that the picture may represent the physician's desire to preserve independent decision-making and practice.

We noticed that female respondents rate audit and feedback activities in general and volume-based metrics of lower value. One hypothesis to explain this is that female physicians’ actual practice reflects a more patient-centered approach with more time spent at the bedside. This would lead to less emphasis on metrics outside of that interaction, particularly volume-centered metrics. Previous large studies have demonstrated more patient-centered practices in womenReference Bertakis, Helms, Callahan, Azari and Robbins16Reference Roter and Hall19 and lower mortality and 30-day readmission.Reference Tsugawa, Jena, Figueroa, Orav, Blumenthal and Jha20 This was demonstrated in an internal medicine population and has not been explored in an ED setting. The signal for gender preference in our survey should be considered hypothesis generating rather than generalizable across populations, and we are currently pursuing a qualitative study to further explore these findings.

Responses from practitioners with less than 5 years of experience were almost universally positive regarding audit and feedback activities. This may indicate a shift toward the need for objective measurements of practice, or a reflection of a persistent desire for feedback, which physicians in training are increasingly accustomed to receiving. This said, there is no literature to support these hypotheses.

Physicians had very specific preferences for which individuals and entities should have access to their practice data. Among all presented options in Table 5, only four entities retained the approval of more than half of the respondents (answered “yes” or “yes to a subset”): The ED chief, the ED quality of care committee, program directors, and the process improvement physician in the group. It seems that physicians are quite invested when they know that practice data will be seen by individuals who understand the clinical context in which they practice. It could be hypothesized that physicians are hopeful that these data may be used internally to drive positive change but should not be used as external pressure for change.

Finally, it should be noted that physician preference should not make up the entirety of how we choose appropriate audit and feedback metrics. By choosing areas outside of these preferences, physicians and departments may have identified unperceived learning needs and practice gaps that were not appreciated by physicians alone. These areas represent important targets for practice improvement.

LIMITATIONS

As in all surveys, it is possible that, despite its response rate, the survey attracted largely those individuals who have an interest in having practice data. Those who have not participated may have more negative impressions of audit and feedback for physicians and may have ranked different elements as more desirable. The sequelae of this study would inform a starting point for audit and feedback in other centers, leading groups who are developing this functionality to include “late-adopters” early in the project development and derivation phase.

Due to the significant number of metrics (49) that were ranked by respondents, conducting analyses of significance on all variables would yield a high probability of spurious findings. As such, analyses were limited to differences between results that were externally valid (0.5). It is possible that there were some significant results that did not meet this threshold and were not analyzed.

Within the group of respondents, there were some sites that have begun activities of audit and feedback and others that have not. Those that are already conducting this practice may have a more pragmatic and experienced view of what data are truly useful to their practice. Those who have not yet participated in this activity may have ranked elements that will later prove to be less useful. This is less likely due to the homogeneity of the data between clinical sites, yet remains possible.

CONCLUSION

This is the first needs assessment described in the literature that examines EM physicians’ readiness and desired metrics with audit and feedback within an entire health care region. The results of this study should help inform the development and implementation of physician-centered audit and feedback activities. Understanding what information physicians value with respect to their practice is an important aspect of the change management cycle during the implementation of such activities.

Supplemental material

The supplemental material for this article can be found at https://doi.org/10.1017/cem.2020.348.

Clinical trials registry

Not applicable.

Ethics

This project received QI exemption from the Hamilton Integrated Research Ethics Board (HiREB).

Financial support

Dr. Mondoux holds an early career research award through the Department of Medicine at McMaster University and receives financial support for his role as the Quality and Safety Lead at St. Joseph's Hospital Hamilton.

Competing interests

None declared.

References

REFERENCES

Ivers, N, Jamtvedt, G, Flottorp, S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane database Syst Rev 2012;6(6):CD000259. doi:10.1002/14651858.CD000259.pub3.Google Scholar
Brehaut, JC, Colquhoun, HL, Eva, KW, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med 2016;164(6):435441. doi:10.7326/M15-2248.CrossRefGoogle ScholarPubMed
Cooke, LJ, Duncan, D, Rivera, L, Dowling, SK, Symonds, C, Armson, H. The Calgary Audit and Feedback Framework: a practical, evidence-informed approach for the design and implementation of socially constructed learning interventions using audit and group feedback. Implement Sci 2018;13(1):136. doi:10.1186/s13012-018-0829-3.CrossRefGoogle ScholarPubMed
Lavoie, CF, Plint, AC, Clifford, TJ, Gaboury, I. “I never hear what happens, even if they die”: a survey of emergency physicians about outcome feedback. CJEM 2009;11(6):523528. doi:10.1017/s1481803500011787.CrossRefGoogle ScholarPubMed
MacLean, CH, Kerr, EA, Qaseem, A. Time out - charting a path for improving performance measurement. N Engl J Med 2018;378:17571761. doi:10.1056/NEJMp1002530.CrossRefGoogle ScholarPubMed
Stefan, A, Hall, JN, Sherbino, J, Chan, TM. Faculty development in the age of competency-based medical education: a needs assessment of Canadian emergency medicine faculty and senior trainees. CJEM 2019;21(4):527534. doi:10.1017/cem.2019.343.CrossRefGoogle ScholarPubMed
Chan, TM, Jo, D, Shih, AW, et al. The Massive Online Needs Assessment (MONA) to inform the development of an emergency haematology educational blog series. Perspect Med Educ 2018;7(3):219223. doi:10.1007/s40037-018-0406-0.CrossRefGoogle ScholarPubMed
Kelley, K, Clark, B, Brown, V, Sitzia, J. Good practice in the conduct and reporting of survey research. Int J Qual Heal Care 2003;15(3):261266. doi:10.1093/intqhc/mzg031.CrossRefGoogle ScholarPubMed
Calder, L, Pozgay, A, Riff, S, et al. Adverse events in patients with return emergency department visits. BMJ Qual Saf 2015;24(2):142148. doi:10.1136/bmjqs-2014-003194.CrossRefGoogle ScholarPubMed
Norcini, JJ. Peer assessment of competence. Med Educ 2003;37(6):539543. doi:10.1046/j.1365-2923.2003.01536.x.CrossRefGoogle ScholarPubMed
Rodgers, KG, Manifold, C. 360-Degree feedback: possibilities for assessment of the ACGME core competencies for emergency medicine residents. Acad Emerg Med 2002;9(11):13001304. doi:10.1197/aemj.9.11.1300.CrossRefGoogle ScholarPubMed
Sherbino, J, Bandiera, G, Frank, JR. Assessing competence in emergency medicine trainees: an overview of effective methodologies. CJEM 2008;10(4):365371. doi:10.1017/S1481803500010381.CrossRefGoogle Scholar
Ramsey, PG, Wenrich, MD, Carline, JD, Inui, TS, Larson, EB, LoGerfo, JP. Use of peer ratings to evaluate physician performance. JAMA 1993;269(13):16551660. doi:10.1001/jama.1993.03500130069034.CrossRefGoogle ScholarPubMed
Kanzaria, HK, Hoffman, JR, Probst, MA, Caloyeras, JP, Berry, SH, Brook, RH. Emergency physician perceptions of medically unnecessary advanced diagnostic imaging. Acad Emerg Med 2015;22(4):390398. doi:10.1111/acem.12625.CrossRefGoogle ScholarPubMed
Choosing Wisely Canada - Emergency Medicine. Available at: https://choosingwiselycanada.org/emergency-medicine/ (accessed October 29, 2019).Google Scholar
Bertakis, KD, Helms, LJ, Callahan, EJ, Azari, R, Robbins, JA. The influence of gender on physician practice style. Med Care 1995;33(4):407416. doi:10.1097/00005650-199504000-00007.CrossRefGoogle ScholarPubMed
Krupat, E, Rosenkranz, SL, Yeager, CM, Barnard, K, Putnam, SM, Inui, TS. The practice orientations of physicians and patients: the effect of doctor-patient congruence on satisfaction. Patient Educ Couns 2000;39(1):4959. doi:10.1016/S0738-3991(99)00090-7.CrossRefGoogle ScholarPubMed
Roter, DL, Hall, JA, Aoki, Y. Physician gender effects in medical communication: a meta-analytic review. JAMA 2002;288(6):756764. doi:10.1001/jama.288.6.756.CrossRefGoogle ScholarPubMed
Roter, DL, Hall, JA. Physician gender and patient-centered communication: a critical review of empirical research. Annu Rev Public Health 2004;25(1):497519. doi:10.1146/annurev.publhealth.25.101802.123134.CrossRefGoogle ScholarPubMed
Tsugawa, Y, Jena, AB, Figueroa, JF, Orav, EJ, Blumenthal, DM, Jha, AK. Comparison of hospital mortality and readmission rates for medicare patients treated by male vs female physicians. JAMA Intern Med 2017;177(2):206213. doi:10.1001/jamainternmed.2016.7875.CrossRefGoogle ScholarPubMed
Figure 0

Table 1. Responses for attitudes towards audit and feedback

Figure 1

Table 2. Top 10 metrics as evaluated by respondents

Figure 2

Table 3. Bottom 10 metrics as evaluated by respondents

Figure 3

Table 4. Other top 10 elements for each specific site included in the survey

Figure 4

Table 5. Privacy and dissemination preferences of respondents

Supplementary material: File

Gupta et al. supplementary material

Gupta et al. supplementary material 1

Download Gupta et al. supplementary material(File)
File 17.2 KB
Supplementary material: PDF

Gupta et al. supplementary material

Gupta et al. supplementary material 2

Download Gupta et al. supplementary material(PDF)
PDF 1 MB