CHD accounts for the most common birth defect and affects nearly 1% (about 40,000) of live births annually in the United States of America. Reference Gilboa, Devine and Kucik1 Furthermore, with surgical and medical advancements, an increasing number of patients survive into adolescence and adulthood with CHDs. These factors require a comprehensive understanding of the diagnosis, treatment, and management of CHD, beginning with a broad range of medical trainees.
Developments in technology-enhanced learning have led to meaningful improvements in understanding and practices in medical education in a variety of clinical scenarios. Reference Cook, Hatala and Brydges2,Reference Real, DeBlasio and Beck3 Specifically, virtual reality is emerging as a unique learning modality to better demonstrate the anatomy, physiology, and surgical considerations of various congenital heart lesions as well as normal cardiovascular relationships. Reference Falah, Charissis and Khan4 Virtual reality involves the use of headsets containing dual display screens to mimic binocular vision. An immersive three-dimensional (3D) environment is then rendered from a host computer, allowing the user to explore and interact with the 3D world. This learning experience has the potential to more effectively teach learners about clinically important nuances of cardiovascular physiology in both the normal and abnormal state.
While virtual reality has been introduced as an educational tool, its use has not been extensively explored or refined in CHD training and education. Virtual reality may better teach the complexities associated with the anatomy and physiology of CHDs compared to traditional lectures, diagrams, and even online visual representations. Our goal in this study was to integrate a prototype virtual reality congenital heart experience into the general paediatric residency training curriculum. In this multi-centre, prospective, interventional case–control study, we assessed the efficacy, efficiency, and user experience of this novel teaching modality, with the hypothesis that this programme would efficiently improve learners’ knowledge and understanding of CHDs in a unique and enjoyable learning experience.
Methods
We performed this prospective case–control study of paediatric residents and 3rd/4th year medical students at three tertiary academic medical centres from July 2018 to June 2020. After obtaining institutional review board approval, all learners in a 1–4 week outpatient cardiology elective were included in our study, regardless of previous completion of a cardiology rotation. Learners were assigned a “Learner Identification” number for anonymous learner tracking at the beginning of their rotation. We used The Stanford Virtual Heart (SVH; Lighthaus Inc., Long Beach, CA), a CHD virtual reality experience developed at Stanford University and specifically upgraded for this study. Learners can interact with a virtual 3D model of the heart, including immersing themselves “inside the heart,” following blood flow within the chambers and vessels, and exploring non-traditional views of the heart, such as the posterior aspect of the heart. The software includes integrated audio recordings to guide learners step-by-step through learning modules for the anatomy and physiology of a normal heart, as well as six of the most common congenital cardiac defects (atrial septal defect, ventricular septal defect, aortic stenosis, pulmonic stenosis, coarctation of the aorta, and patent ductus arteriosus). From July 2018 to June 2019, learners were categorised as the control group and were not exposed to the virtual reality educational intervention. The intervention group was intended to include learners exposed to the hour-long virtual reality intervention from July 2019 to June 2020, though due to COVID-19 university closures and limitations, the intervention was ended in February 2020. Sample text of the audio script and a sample audio recording is included in the Supplemental Materials.
To evaluate the efficacy of the virtual reality curriculum, we created and validated a formal assessment tool, described in detail by Wilson et al Reference Wilson, Lim and Axelrod5 This same assessment was administered at the completion of all learners’ rotations, and scores between the control and intervention groups were compared. The assessment tool was developed by seven paediatric cardiology fellows and faculty and was reviewed and vetted by education and assessment experts from other specialties. The final assessment included twenty-seven items testing anatomic and physiologic concepts of six common CHDs including atrial septal defect, ventricular septal defect, coarctation of the aorta, patent ductus arteriosus, pulmonary stenosis, and aortic stenosis. The assessment tool was externally validated by six experienced paediatric cardiology faculty member “content experts (CE)” (at the Associate Professor or Professor levels) from Stanford University, University of Michigan and University of California San Diego in accordance with methods described by Polit et al. Reference Polit and Beck6 CEs were not involved in creation of the assessment tool nor the remainder of this research protocol. There were four items referencing echocardiogram images, one matching item, one free-text item, and twenty-five multiple choice items.
From July 2018 to June 2019, learners in the control group did not interact with the virtual reality program and completed the assessment at the completion of their paediatric cardiology rotation. From July 2019 to February 2020, learners in the intervention group completed the hour-long virtual reality experience and then completed the same assessment at the end of their paediatric cardiology rotation. Learners who used the virtual reality software completed an additional qualitative survey to assess the virtual reality technology and experience. Prior to this study, none of the learners had previously used the SVH nor seen the assessment tool. Individual item responses for the assessment tool were stored via a REDCap database using unique and deidentified learner IDs, as assigned by the centre. The total assessment score was derived by recording the number of items that learners answered correctly, ranging from 0 to 27.
Data are presented as frequency (percentage) for categorical variables and mean ± standard deviation for continuous variables. Group comparisons were made in learners’ information, total assessment scores, and individual items using Chi-square test or Fisher’s exact test for categorical variables and two-sample t-test for continuous variables. A p-value < 0.05 was considered statistically significant. All analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC, USA).
Results
There were 80 learners in the control group and 62 in the intervention group for a total of 142 study patients. At Centre 1, all eligible learners completed the assessment in the control and intervention groups, 35 and 25 learners, respectively (100%). At Centre 2, 15 of 22 (68%) eligible learners completed the assessment in the control group, while 17 of 22 (77%) eligible learners completed the assessment in the intervention group. At Centre 3, 30 of 34 (88%) eligible learners completed the assessment in the control group with four learners switching off of the cardiology elective prior to completion of the assessment. At Centre 3, 20 of 21 (95%) eligible learners completed the assessment in the intervention group, as one learner completed the virtual reality experience but did not complete the assessment. Learners in both groups were similar with regard to medical centre location, duration of cardiology rotation, year in training, and previous rotations in Paediatric Cardiology (Table 1). The largest percentage of learners were PGY1 trainees (45.8%). Learners most frequently completed a four-week long Paediatric Cardiology rotation (36.6%).
Table 1. Trainee information (N = 142)
Data are presented as N (%).
MS, Medical Student; PGY, Post-Graduate Year.
P-value from Chi-square test.
Table 2. Total assessment scores by centre
Data are presented as mean ± standard deviation.
P-value from two-sample t-test.
Learners in the intervention group performed better on the assessment overall (20.4 ± 2.9 versus 18.8 ± 3.8 correct answers, p = 0.01 ; Table 2). Learners in both groups scored lowest on items addressing the clinical management or long-term complications of CHDs (Supplemental Table). Individual item analysis showed significantly improved scores in the intervention group for items testing visuospatial concepts (Supplemental Table). There was a significantly higher performance on three out of five items that tested the concept of chamber dilation with various intracardiac shunts in the intervention group (Supplemental Table, all p < 0.05). One of the two items that tested identification of chamber dilation based on echocardiogram images demonstrated a significant difference between groups (Q18 in Supplemental Table, p = 0.0002). There was no significant difference among the two groups in items involving diagnosis of a congenital heart lesion based on echocardiographic images (Supplemental Table).
Of the 62 learners in the intervention group, 45 (72.6%) responded to the qualitative survey regarding the virtual reality experience (Table 3). All survey respondents indicated that the educational programme either significantly improved or improved their understanding of cardiovascular physiology and anatomy. All survey respondents also indicated that the virtual reality experience was either very enjoyable or enjoyable. About 75.6% of respondents highly recommended and 24.4% of respondents recommended integration of the virtual reality program into the residency curriculum and specifically into the cardiology outpatient rotation at their respective institutions.
Table 3. VR experience Feedback (N = 45)
* Data are presented as N (%).
Discussion
The anatomy and physiology of CHDs are difficult concepts that require a fundamental knowledge-base and, just as importantly, a visuospatial understanding of the normal and abnormal heart. Various technological advancements including the development of 3D printing, simulations, and virtual reality programs offer new educational strategies for medical trainees to augment traditional lecture and case-based teaching. Previous studies have highlighted the utility of 3D-printed heart models in teaching CHDs, though no robust large-scale studies have been performed. Reference Illmann, Ghadiry-Tavi and Hosking7,Reference White, Sedler and Jones8 White et al specifically used 3D-printed heart models of ventricular septal defect and Tetralogy of Fallot and compared test scores of paediatric and emergency medicine residents who used the 3D-printed models versus those who did not. While there was no significant difference in test scores between the two groups, trainees subjectively reported improvement in understanding of these CHDs. Reference White, Sedler and Jones8 The use of patient simulations for paediatric residents in the diagnosis and management of various cardiac diseases (including left to right shunts, ductal-dependent lesions, and cardiogenic shock) has been evaluated on a small scale with varied improvement in objective assessments of knowledge retention. Reference Mohan, Follansbee and Nwankwo9,Reference Harris, Adler and Unti10 Subjectively, learners reported improvement in understanding and comfort with managing paediatric cardiology patients.
Virtual reality has emerged as an innovative tool specifically used to better teach anatomy to medical trainees. Reference Izard, Juanes Méndez and Palomera11,Reference Moro, Stromberga and Raikos12 Based on user feedback, virtual reality increased user confidence and perceived knowledge and understanding of anatomical concepts. However, Stepan et al did not show significant differences in pre- or post-intervention, nor in knowledge retention assessments in learners who participated in a virtual reality experience focused on neuroanatomy. Reference Weeks and Friedman13 This lack of significant improvement may have been due to the brief nature (10 min) of the virtual reality experience.
Upon review of the current literature, there have been no large-scale, formally validated studies that evaluate the efficacy of virtual reality as an educational tool using a validated assessment tool. Our study highlights the multi-centre use of a novel, guided virtual reality program to augment medical trainee education in paediatric cardiology through a unique hour-long interactive audio/visual experience. As opposed to learner self-evaluation, we also utilised a rigorously validated assessment tool, described by Wilson et al, to more accurately measure the objective impact of the intervention. Reference Polit and Beck6
The results from our study demonstrated that learners who were exposed to the interactive virtual reality program performed better on the validated assessment tool. Visuospatial understanding, specifically, resultant chamber dilation from various intracardiac shunts, was particularly improved in learners who used the virtual reality program, highlighting the ability of virtual reality to better convey these visuospatial and physiologic concepts than traditional teaching methods. Scores on the assessment tool were higher in the intervention groups at all three centres, suggesting that these results are reproducible at other such large academic training institutions. Interestingly, learners in the intervention group had significantly higher assessment scores than those in the control group at Centre 2, which may indicate that the virtual reality experience particularly addressed concepts or scenarios not widely encountered at Centre 2. This may also translate to other centres if implemented on a large scale. The overwhelmingly positive response from learners about the virtual reality experience at all three centres and support for its integration into the residency curriculum suggests a potential gap in the current traditional educational tools made available to learners.
Given the complexity and subtleties in the diagnosis and management of patients with CHDs, and the relatively common prevalence of CHDs in the paediatric population, it is imperative that trainees develop a solid understanding and comfort with managing these patients. Weeks et al suggested that the education of paediatric residents in the evaluation of CHD should include not only evidence-based medicine, lectures, but also interactive and hands-on learning and must be dynamic with the ever-changing landscape of technological advancements.Reference Weeks and Friedman 13
Limitations to our study include the inability to completely match learners by their level of training and previous exposure to cardiology. Though learners were similar amongst intervention and control groups in terms of proportion of learners from each site, duration of cardiology outpatient rotation, and year of training, there are inevitable differences in rotation experiences and supplemental education at the various sites that may have affected scores. Also, given that the control group and intervention group completed the assessment one year apart, there may have been small curriculum changes that could have served as confounders, though none of the participating sites reported any formal curricular changes during the study period. There may have also been differences in learners’ interest and general knowledge of paediatric cardiology. This may be most evident with the large difference in scores on the assessment tool from the Centre 2 (16.7 ± 4.8 in the control group versus 20.4 ± 3.3 in the intervention group, p = 0.02). As this centre had the smallest number of learners of all three centres (15 learners in the control group versus 17 learners in the intervention group), higher scores in a proportionally smaller group of learners in the intervention group may have accounted for this large difference. Furthermore, academic closures related to the COVID-19 pandemic may have affected learners’ experiences during cardiology rotations as well as ended the intervention period early (February 2020 instead of June 2020 as originally planned). We have no evidence that there was discussion about the assessment between residents amongst intervention and control groups, which could have inflated scores in the intervention group. Also, the higher scores in the intervention groups may actually reflect the integrated audio script of the virtual reality program itself rather than the virtual reality interface. Long-term understanding and knowledge retention were not assessed, though this may be an area for further exploration.
In this study, learners who completed the virtual reality experience overall performed better than those who did not, based on comparison of scores from our validated assessment tool, particularly on items testing visuospatial concepts. Based on learner feedback from our study as well as several others in the current literature, trainees clearly support the integration of new teaching modalities into the traditional paediatric residency curriculum. Our study demonstrates that a virtual reality program such as the Stanford Virtual Heart is an effective teaching modality to enhance learner engagement and teaching of visuospatial concepts of CHDs. Further efforts investigating long-term knowledge retention and expansion to include more complex CHDs or different subspecialties are warranted. Our positive results support continued virtual reality development and expansion to other clinical scenarios and amongst other medical trainees.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1047951122000890
Acknowledgements
The authors would like to acknowledge Lighthaus, Inc. (www.lighthaus.us), the VR education company that developed The Stanford Virtual Heart.
Financial support
This study was supported, in part, by a grant from the Gilbert Whitaker Fund for the Improvement of Teaching, awarded through the University of Michigan. The Stanford Virtual Heart was developed by Lighthaus Inc. and funded by The Betty Irene Moore Children’s Heart Center at Stanford and OculusVR.
Conflicts of interest
David M. Axelrod is the lead medical advisor and a shareholder at Lighthaus Inc., the company that produces the Stanford Virtual Heart. The remaining authors disclose no conflicts of interest.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national guidelines on human experimentation (respect for potential and enrolled subjects, favorable risk-benefit ratio and fair subject selection) and with the Helsinki Declaration of 1975, as revised in 2008. The University of Michigan IRB reviewed the study design of this publication and deemed this study to be IRB exempt on May 11, 2018.
