Background
Point-of-care ultrasound (PoCUS) and the use of simulation in training have become an integral part of emergency medicine practice. PoCUS is a core competency in emergency medicine and other clinical specialties, and gives physicians the ability to rapidly and noninvasively assess patient physiology, anatomy and pathology, thus improving diagnosis and guiding resuscitation.Reference Frederiksen, Juhl-Olsen and Sloth 1 , Reference Moore and Copel 2 Emergency PoCUS has been shown to help differentiate causes of pleuritic chest pain in patients with a normal chest x-ray,Reference Volpicelli, Cardinale and Berchialla 3 to improve diagnostic accuracy and narrow the differential diagnosis in patients with undifferentiated hypotension,Reference Jones, Tayal and Sullivan 4 to improve time-to-diagnosis and mortality of cardiac trauma patients,Reference Plummer, Brunette and Asinger 5 and to offer benefit in the accurate and timely diagnosis of acute cardiorespiratory emergencies,Reference Mandavia, Hoffner, Mahaney and Henderson 6 , Reference Volpicelli, Elbarbary and Blaivas 7 in addition to other documented benefits.Reference Hwang, Kimberly and Liteplo 8
Numerous studies have shown that competence in such PoCUS protocols, including the Focused Assessment with Sonography for TraumaReference Gogalniceanu, Sheena and Kashef 9 and Prehospital Assessment with Ultrasound for EmergenciesReference Chin, Chan and Mortazavi 10 can be achieved after short intensive training. The Abdominal and Cardiac Evaluation with Sonography in Shock (ACES) protocol, previously described by Atkinson et al. in 2009,Reference Atkinson, McAuley and Kendall 11 is a structured six-view PoCUS approach for patients in shock (Figure 1). It was recently found that medical learners demonstrated competency in simulated PoCUS for the ACES protocol after a focused training program involving specified reading, videos, and an interactive workshop. Competency was measured by image acquisition and interpretation, with learners successfully generating 99.7% of the ultrasound windows and correctly interpreting 93% of the scans.Reference Parks, Atkinson and Verheul 12
Figure 1 The Abdominal and Cardiothoracic Evaluation with Sonography in Shock (ACES) protocol: ultrasound windows consist of 1) one or more cardiac views, 2) an inferior vena cava view, 3) a screen of the abdominal aorta, 4) right and 5) left flank views for pleural and peritoneal fluid, and 6) a pelvic view for bladder size and free fluid.
High-fidelity simulation offers many benefits in medical education, including decreased reliance on patient or volunteer availability, and the ability to customize and repeat virtually any clinical scenario. Unfortunately, the fidelity, or realism, of the simulated human body is restricted by the available technologies, which limits the ability to realistically approximate actual ultrasound images.Reference Blum, Rieger and Navab 13
The purpose of this study was to investigate the impact of integrating simulated PoCUS into standard human patient simulation by combining high-fidelity clinical and PoCUS simulators. The primary objective was to determine whether medical learners who have completed training in simulated PoCUS demonstrate an improvement in their diagnostic accuracy, confidence, and precision during simulated cardiorespiratory emergency scenarios with the addition of ultrasound into simulation (USIM).
Methods
Study design
This was a prospective observational study involving medical school clerks (third- and fourth-year medical students) and resident physicians (PGY1–3) completing rotations in an urban regional hospital. The study was approved by the Dalhousie University Research Ethics Board.
Study setting and population
The study was conducted in the simulation centre located in a multidisciplinary teaching facility at the New Brunswick Community College from March to September 2012. All participants enrolled voluntarily, after recruitment using local communication channels and signed consent and confidentiality forms. Inclusion criteria were 1) voluntary participation, 2) being available and willing to attend the training and testing session, and 3) being either a third- or fourth-year medical student clerk or a physician currently in residency training. Participants had no prior ultrasound training. Because this was a pilot study, no power calculation was undertaken.
Study protocol
All participants underwent a standardized training protocol in simulated PoCUS, involving pre-reading, didactic teaching, and supervised hands-on experience with a high-fidelity CAE Vimedix ultrasound simulator (Figure 2). The training protocol we utilized has been described previously by our group.Reference Parks, Atkinson and Verheul 12
Figure 2 Picture of the CAE Vimedix high-fidelity ultrasound simulator used in the study.
Following PoCUS training, participants were introduced to the patient simulation suite and the high-fidelity clinical simulator, the Laerdal SimMan 3G. During this time, participants were oriented to the simulator and interviewed the simulated patient through the built-in microphone, listened to heart and lung sounds, familiarized themselves with the monitor, and identified decreased pulses and diaphoresis. Each of the participants then participated in a full orientation scenario to solidify their confidence with the simulators. Training continued until the participants felt comfortable with the simulation setting and agreed to proceed to the testing scenarios.
To determine whether the addition of PoCUS to standardized simulated scenarios improves diagnosis, each participant completed a set of six cardiorespiratory medical scenarios. Each of these scenarios included pathology that was potentially detectable by PoCUS, including aortic stenosis, cardiac tamponade, dilated cardiomyopathy, myocardial infarction, pleural effusion, and pulmonary embolism (features of the simulators for each scenario are included in Appendix 1). Each of the 12 participants completed all six scenarios for a total of 72 datasets. None of the participants had advance knowledge of the diagnosis in the case that they were managing. The sequence of scenario delivery was generated using a Latin square design to ensure a counterbalanced approach. This arrangement was intended to minimize carryover effects due to practice and learning. Participants were instructed to consider only the diagnosis and were not expected to provide any interventions or treatment options for the simulated patient.
For each scenario, the simulation suite was equipped with both simulators (programmed for the given scenario), a confederate role-playing nurse who presented the case to the participant, and a confederate in an adjacent room playing the role of the patient via a built-in microphone in the high-fidelity clinical simulator. Scenarios began with the participant being provided with a clinical summary by the nurse. This summary included the patient’s presenting complaint, vitals, and an electrocardiogram. The participant would then approach the high-fidelity clinical simulator, perform a focused history and physical exam, and then formulate a differential diagnosis. Participants were allowed a maximum of five possible diagnoses, ranked in order of decreasing confidence of likelihood and given a numeric percentage between 1% and 100%.
Following clinical examination, the participant would proceed to the PoCUS simulator and perform an ACES scan to assess for any pathology. Using the information gathered with the PoCUS simulator, the participant would then formulate a revised differential, again with a maximum of five diagnoses and a corresponding percentage confidence for each.
This method of testing was repeated for each participant until all six scenarios had been completed. Following the final scenario, participants reviewed each case in detail with the instructor. Appendix 2 outlines details of scenario development and technical specifications.
Data collection and measurement
Each participant was observed by one of the study investigators and was assessed for completeness of the clinical examination and image acquisition during PoCUS scans. Scans were defined as complete or incomplete, with specific omissions being recorded.
Written differential diagnoses were collected from participants following each scenario. Each of these sheets contained the participant’s ranked differential diagnosis and percentage confidence before and after PoCUS. These differential diagnoses were analysed to assess the change in diagnostic accuracy, confidence, and precision.
Diagnostic accuracy was determined by scoring the ranking of the correct diagnosis on the participant’s differential. For example, if the participant’s first-ranked diagnosis was the correct one, he or she received five points; if it was second-ranked, the participant received four points, and so on. Diagnostic confidence refers to the change in percentage confidence that the participant assigned to the correct diagnosis before and after PoCUS. Diagnostic precision assessed the difference between the pre- and post-PoCUS differentials, and whether PoCUS helped narrow the participant’s differential.
Data analysis
All data analyses was completed using R software, version 2.15 (R Foundation for Statistical Computing, Vienna, Austria). For each participant, an improvement in diagnostic accuracy was defined as either a higher score on his or her diagnostic accuracy after simulator usage or a perfect score on both the before and after diagnostic accuracy with an increase in diagnostic confidence. For each participant, an average of the diagnostic scores for the six scenarios before and after PoCUS was taken. A one-sided Wilcoxon signed-rank test was used to examine any change in diagnostic scores following PoCUS.
Similarly, an average of the confidence in diagnoses measures and the number of viable diagnoses were taken before and after PoCUS was determined. The difference in confidence level was tested with a one-sided paired sample t-test. Diagnostic precision was assessed by the difference in the number of viable diagnoses and tested with a one-sided Wilcoxon signed-rank test.
Results
A total of 12 medical learners were recruited to participate in the study (six third- and fourth-year Dalhousie University medical student clerks, and six PGY1–3 emergency medicine and non-emergency medicine resident physicians. Each of the 12 learners completed the necessary preparation before attending the session, completed training, and completed six testing scenarios during the day of the session. There were no indeterminate results, missing responses, or adverse events during the study.
One participant, during one scenario, failed to complete a scan of the lung bases and thus did not complete a full ACES scan. In addition, one participant during one scenario did not complete a satisfactory respiratory examination.
Differential diagnostic ranking
The addition of PoCUS to the standard simulated assessments improved the differential diagnostic ranking during simulated cardiorespiratory cases. The correct diagnosis was placed higher in the differential after PoCUS in a median of four (interquartile range [IQR]=3, 4.5) out of the six scenarios for each participant. Participants were more likely to have a correct primary diagnosis at the top of their differential after PoCUS simulator usage. Prior to use of PoCUS, 45 correct primary diagnoses were made during the simulated scenarios (62.5%, n=72). Following simulated PoCUS, 64 (88.9%) correct primary diagnoses were made. This increase in a correct primary diagnosis was statistically significant (χ2=14, 1df, p=0.0002). Of the 27 incorrect diagnoses before PoCUS, a total of 20 (74.1%) were improved after PoCUS. There were only three occasions where an incorrect diagnosis superseded a correct diagnosis after the addition of PoCUS, and it was on the same scenario for three different participants (i.e., dilated cardiomyopathy).
Diagnostic accuracy
The mean diagnostic accuracy score before PoCUS was 3.8 (median=4, IQR=3,5, 4.1), whereas the mean score after PoCUS was 4.5 (median=4.6, IQR=4.1, 5.0). The median of the post-PoCUS scores was significantly greater than that of the pre-PoCUS scores (W=66, p=0.004) (Figure 3).
Figure 3 Box and whisker plot of participants’ median of diagnostic accuracy before and after PoCUS.
Diagnostic confidence
Similar to diagnostic accuracy, the confidence increased for all participants in at least two of the scenarios, with a median of four scenarios with confidence level increases (IQR=3, 4.5; Figure 4). All participants who had an improvement in their diagnostic accuracy score also had an improvement in their confidence. There were 44 tests where the participant accurately identified the diagnosis both before and after use of PoCUS. Of these, 29 (65.6%) had increased confidence in the diagnosis after PoCUS.
Figure 4 Box and whisker plot of participants’ median diagnostic confidence before and after PoCUS.
The mean confidence in the diagnosis score pre-PoCUS was 52.2 (SD=14.7), whereas the mean post-PoCUS score was 81.7 (SD=9.5), and the difference in these means was statistically significant (t=−7.71, p<0.0001).
Diagnostic precision
Participants were further able to reduce the number of diagnoses in their differential. Participants narrowed their diagnoses to a median of 4.5 times out of 6 scenarios (IQR=3, 5). Two participants narrowed their number of diagnoses in all six scenarios. The median number of diagnoses for each patient pre-PoCUS was 3.5 (IQR=3.0, 3.8) with a median of 2.3 (IQR=1.5, 2.9) diagnoses post-PoCUS. The difference was statistically significant (W=0, p<0.001) (Figure 5).
Figure 5 Box and whisker plot of participants’ median number of viable diagnoses before and after PoCUS.
Discussion
The results in this study demonstrate that the addition of an ACES simulated PoCUS protocol to standardized simulated cardiorespiratory scenarios leads to improved diagnostic accuracy, confidence, and a narrowed differential for medical learners following a brief and focused standardized training protocol. These findings show that PoCUS protocols can provide medical learners with useful diagnostic information in simulated settings and facilitate both rule in and rule out determinations for a number of conditions.
The improvement in diagnostic characteristics with PoCUS during simulated cardiorespiratory scenarios supports the utility and practicality of teaching focused PoCUS protocols with high-fidelity simulators, particularly in medical emergency scenarios. One of the scenarios, a patient with dilated cardiomyopathy, proved difficult for participants. All three occasions where a worse diagnosis was given post-PoCUS usage involved this specific scenario. It is believed that the counterbalancing or the sequencing approach removed participant fatigue as a cause for this. Possible explanations include 1) the CAE Vimedix ultrasound simulator does not present participants with a valid depiction of dilated cardiomyopathy; 2) the training protocol lacked sufficient emphasis on identifying this pathology; or 3) learners were unfamiliar with this pathology and require more experience to familiarize themselves with the typical clinical and PoCUS characteristics of dilated cardiomyopathy.
Only one participant during one scenario performed an inadequate ACES scan by failing to scan the lung bases during the pleural effusion scenario. Because this respiratory pathology has ultrasound findings near the lung bases, the participant was unable to accurately identify and interpret the effusion. Interestingly, both the pre- and post-PoCUS differentials did not include pleural effusion. This example illustrates the importance of completing a thorough PoCUS scan.
The results in this study are consistent with the previously reported benefits of PoCUS protocols in the clinical setting. Jones et al. reported the diagnostic benefit of PoCUS in patients presenting to the emergency department with undifferentiated hypotensionReference Jones, Tayal and Sullivan 4 and found that a PoCUS protocol decreased the median number of viable diagnoses (from four to three median diagnoses per differential) and increased the proportion of differentials with the correct diagnosis ranked first (from 50% to 80%). Similarly, it was found that a simulated PoCUS protocol decreased the median number of viable diagnoses by 1.2 (from 3.5 to 2.3) and increased the proportion of differentials with a correct primary diagnosis by 26.4% (from 62.5% to 88.9%).
Limitations
Because USIM was limited entirely to simulation, no clinical conclusions can be drawn from these findings. In terms of the realism of the simulated scenarios, the protocol did not include treatment options. The high-fidelity mannequin simulator is capable of responding to treatment, and it could have been made more realistic if participants were given the option of treating the simulated patient. Appropriate responses and clinical improvement or deterioration may provide participants with an increased feeling that a scenario is realistic and an increased opportunity to hone treatment skills.
Study participants were medical student clerks and residents with varying degrees of experience and knowledge. Given the small number of participants, their heterogeneity, and the fact that the study was conducted at a single centre, the external validity of the results may be limited.
While unlikely to fully explain the findings, it is possible that the improved diagnostic performance could be due to participants having more time to reconsider the case, as they performed the ultrasound after their initial assessment. The diagnostic accuracy score has not been validated elsewhere.
As previously discussed, there are limitations to the effectiveness of high-fidelity simulation training, primarily due to shortcomings of simulation technologies. Although the high-fidelity simulators employed in this study were state of the art, it is unlikely that these simulated scenarios exactly mirrored real-life events and suspended participant disbelief.
In addition, the current study did not focus on the translation of diagnostic benefits into a clinical setting or the long-term retention of PoCUS competency.
Future Research and Follow-Up
The participants will be followed to determine the extent of their long-term retention of this training and knowledge, particularly in terms of their ongoing training and clinical implementation of PoCUS. Future research is required to determine whether the effects found will translate from the simulation setting into the real-life clinical setting.
Conclusions
This pilot study suggests that for medical learners newly competent in PoCUS, the addition of an ACES PoCUS protocol to standard clinical assessment improves diagnostic accuracy, confidence, and precision during simulated cardiorespiratory scenarios. This is consistent with clinical studies and supports the use of ultrasound during medical simulation.
Supplementary material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/cem.2014.41.
