Manuscript

Canadian neurology residency programs transitioned to Competency-Based Medical Education (CBME) in 2020. CBME shifts the focus of residency training to intended outcomes, ultimately aiming to produce competent neurologists and improve patient outcomes. Under this system, residents must demonstrate competence in Entrustable Professional Activities (EPAs) of each graduated stage of training and are assessed on an entrustment scale. Training programs must also have a robust assessment system to foster learner development and evaluate residents’ achievement of intended outcomes. While outcomes-focused education makes sense in theory, its implementation may be challenging, as it requires significant shifts in culture. The success of CBME as a transformative innovation ^{Reference Van Melle, Frank, Holmboe, Dagnone, Stockley and Sherbino1} therefore hinges on its pairing with iterative evaluation. ^{Reference Hamza, Ross and Oandasan2,Reference Oandasan, Martin, McGuire and Zorzi3} As with other educational innovations, adapting CBME to local contexts is inevitable and necessary for maximum benefit to be realized. ^{Reference Varpio, Bell and Hollingworth4} This study therefore aimed to (1) examine the variability in CBME implementation across neurology programs, (2) determine the barriers toward uptake of CBME, and (3) identify the benefits and pitfalls of CBME to neurology residency training. This interim assessment of CBME implementation across neurology residency programs will help inform the iterative process.

Residents in the CBME curriculum for at least 6 months and staff neurologists involved in clinical teaching in Canadian adult or pediatric neurology programs were invited to anonymously complete a 20-minute resident- or faculty-oriented survey, as appropriate. Participants were recruited by email invitations distributed by each institution’s neurology program director or administrator and through the CNSF. Surveys were administered via the Qualtrics online platform, and a letter of information was embedded in the survey. Each survey included multiple-choice questions and free-form responses. A $5 gift card was provided to resident participants. The project was approved by Western University’s Research Ethics Board.

Survey questions were agreed upon by the investigators and developed with attention to established core components ^{Reference Van Melle, Frank, Holmboe, Dagnone, Stockley and Sherbino1} of CBME curricula, including tailored learning experiences, competency-focused instruction, and programmatic assessment. Questions were designed to identify potential areas of program improvement. ^{Reference Oandasan, Martin, McGuire and Zorzi3} Multiple-choice questions yielding nominal data were analyzed by frequency of each response option. Free-form responses were individually reviewed by the investigators, and common themes and sentiments were identified.

Responses were received from 38 residents (30 adult neurology and 8 pediatric neurology, out of approximately 120 adult neurology and 20 pediatric neurology residents who were eligible) and 48 faculty (40 adult and 8 pediatric). Some survey questions were not answered by some respondents. Responses were received from residents and faculty from 11 of 16 Canadian neurology training programs. The largest number of resident responses were from McGill and Toronto, with seven and six respondents, respectively. The largest number of faculty responses were from Western and Calgary, with 12 and 8 respondents, respectively.

Most residents felt that CBME was clearly explained to them. They felt that senior residents had a good understanding of CBME and EPAs, but that faculty did not. Senior residents were perceived to offer to complete EPAs more often than faculty. Residents also reported that senior residents were more reliable than faculty at completing EPAs without needing reminders (Fig. 1a). There was variability in the ease of obtaining EPAs during different clinical settings. The perceived best settings to obtain EPAs were after reviewing a consult in person and at the end of a consultant’s time on service, while the most difficult setting was after reviewing a consult by phone (Fig. 1b). A large majority of residents felt that “not wanting to bother the staff” was a barrier to requesting EPAs, and many residents doubted that staff would complete EPAs even if asked. By contrast, a potential poor performance review was not felt to be a barrier to obtaining EPAs. Residents were divided on whether feedback from EPAs was useful (Fig. 1c). Resident opinions also varied on whether CBME has led to timely or frequent feedback. Most residents did not feel that CBME has been helpful with prompting reflection on their progression or with producing high-quality feedback. There was a strong consensus among residents that assessors do not have a common understanding of the entrustment scale (Fig. 1d).

Figure 1: Selected nominal responses from resident survey. a . Resident perception of program buy-in to CBME. b . Resident perception of best clinical settings to obtain EPA. c . Resident perception of barriers to obtaining EPAs. d . Resident perception of the impact of CBME on training experience.

Free-form responses revealed recurrent themes of EPA-based feedback not being useful or specific enough to guide improvement. Residents were concerned about bearing the burden of initiating EPAs, resulting in increased stress and burnout. The main challenges of completing EPAs were with service being too busy, high frequency of assessments, and additional documentation burden. Residents felt the CBME experience could be improved by better standardizing faculty understanding of the entrustment scale and by implementing reminders to staff to complete EPAs.

Faculty members felt that CBME was clearly explained and well understood, and that adequate training was provided. However, most were unsure if a colleague would provide a similar assessment given the same clinical scenario (Fig. 2a). Most faculty were split between feeling neutral or positive about the impact of CBME on the resident learning experience, though the vast majority agreed that a positive impact on patient care was not evident. Most faculty felt that CBME significantly increased the amount of paperwork (Fig. 2b).

Figure 2: Selected nominal responses from faculty survey. a . Faculty perception of program buy-in to CBME. b . Faculty perception of the impacts of CBME on residents, patients, and paperwork.

In free-form responses, many faculty members commented that it was too early to determine the impact of CBME on resident learning, but felt that the system should, at least in theory, yield more frequent and timely feedback. Some worried that CBME required too much work to implement. Common barriers to faculty completing EPAs included being too busy, residents not asking or requiring encouragement to ask, and difficulty recalling the details of the clinical encounter when trying to complete the EPA afterward. Some faculty felt that the electronic platforms used to track EPAs were slow or unintuitive. Moving forward, some suggested adding discipline-specific faculty development around EPAs, coaching, and providing feedback to trainees.

The theory of programmatic assessment ^{Reference Torre, Rice and Ryan5,Reference Torre, Schuwirth and Van der Vleuten6} suggests that many low-stakes data (EPAs, in the case of CBME) should be obtained and integrated to inform higher-stakes decisions, such as advancement to the next stage of training. It is felt that the sum of many assessments, which may individually be imperfect, can provide a richer and more credible representation of a trainee’s progress. ^{Reference Torre, Rice and Ryan5,Reference Torre, Schuwirth and Van der Vleuten6} However, both faculty and residents perceived a lack of standardization with use of the entrustment scale and expressed negative feelings toward the heterogeneity in the use and understanding of the entrustment scale. This shared discomfort may indicate a fundamental uneasiness with subjectivity in medical education, which may not be resolvable solely by updating assessment tools and training frameworks.

Although neurology residency programs have a relatively small number of EPA types and required encounters compared to other Royal College-accredited programs such as internal medicine, ^7–9 neurology residents still feel it is stressful to navigate the CBME system. It is unclear if there is any relationship between the number of EPAs required and the perceived burden or quality of training under the CBME system. The balance between additional paperwork and stress, and the desired enhancement of feedback regularity and specificity, needs optimization. Residents agreed that certain clinical contexts were more conducive to requesting EPAs than others, such as in-person interactions being preferable to phone calls. Delay of EPA completion impairs the reliability and accuracy of feedback. One strategy may be to leverage the convenience of certain clinical contexts such as reviewing a consult in person and integrate EPA completion into the institutional culture of these scenarios.

Our study echoes some sentiments reported in a study of residents and faculty primarily based at Queen’s University, which found that faculty were concerned about residents’ reluctance to participate in CBME, and that residents were hesitant to engage because of a lack of perceived benefit of this system. ^{Reference Crawford, Cofie, McEwen, Dagnone and Taylor10} Similar to our data, a 2021 survey of CBME residents training in Quebec highlighted increased administrative burden and burnout without clear pedagogical benefit. ¹¹ Some of the challenges we identify, including delayed EPA completion and inconsistent application of the entrustment scale, recapitulate findings from a 2020 survey of Canadian program directors and CBME program leads. ¹² More time and experience are needed to delineate the benefits of CBME and facilitate its integration and commitment from both faculty and residents. One of the limitations of our study is the incomplete response rate from eligible trainees and faculty. The data were insufficient to draw conclusions about the variability in CBME implementation across different neurology residency programs.

Residents and faculty are still struggling with the transition to CBME. Orientation to CBME was considered adequate, though there is still room to improve the quality and reliability of feedback and coaching provided to residents. There was near-unanimous sentiment that CBME has increased workload for both faculty and residents, contributing to stress and burnout. Several common barriers to EPA completion were identified, which may be high-yield targets to optimize CBME implementation. A key challenge is the lack of standardization between assessors, which could be targeted by carefully planned faculty development. It is too early to appreciate any benefit that CBME may have on resident training or patient care. This question should be revisited in the coming years once the first cohort of residents undergoing CBME have completed their training.