1. Outcomes at the micro- & meso-level

There are several short-term outcomes that could serve as easy, measurable outcomes to demonstrate the value of CBME. For instance, many educators endorsed measuring the number of entrustable professional activities (EPAs) observations and determining the quality of feedback. Previous studies on workplace-based assessment systems, such as In-Training Evaluation Reports (ITERs) have shown that examining the artifacts generated in a systematic way can improve the overall quality of assessment data.^{Reference Chan and Sherbino10}

Going forward, it may be prudent for educators to form chains of evidence linking these immediately measurable outcomes (e.g., EPAs acquired during the first year) and later outcomes (e.g., likelihood of remediation required,^{Reference Ross, Binczyk and Hamza11} likelihood of completing areas of specific enhanced competency).^{Reference Binczyk, Babenko, Schipper and Ross12} Program leaders might harness the power of ongoing program evaluation or quality improvement, and that under our national research ethics policy (Tri-council Policy Statement 2),¹³ these efforts are usually deemed exempt from research ethics processes. As such, measuring the more accessible and top-rated short-term outcomes is both necessary and pragmatic for robust program improvement.

2. Wellness and trainee efficacy

Trainee wellness and resident self-efficacy were ranked in the top five educational outcomes to be measured in the short term, which aligns well with the Institute for Healthcare Improvement's Quadruple Aim. We might aim to determine if we are training of physicians committed to continuing professional development, and who are receptive to feedback and adept in providing it. Poor faculty feedback to trainees has been cited as a contributing factor to depression in trainees,^{Reference Pereira-Lima, Gupta, Guille and Sen14} whereas improved feedback has the potential to foster a sense of competence, autonomy, and relatedness.^{Reference Sargeant15} Because CBME advocates for an increase in the direct observation and provision of objective, high-quality feedback, program evaluators should attempt to study their effects on learner well-being.

3. Macro-level outcomes

Patient safety and quality of care metrics (e.g., evidence-based practice, efficiency and effects on hospital flow) for CBME trainees was one of the clinical outcomes that was ranked as most important to follow both in the short-term and the long-term. From a practical perspective, this appears daunting both in terms of data gathering and in establishing causal links between specific clinical outcomes and care provided. Collaboration between educators and quality improvement researchers will be essential to examine these complex systems. Other clinical outcomes, such as multi-source assessments for CBME residents in the clinical settings are more easily trackable and can provide useful information especially when linked to the educational outcomes of individual EPA metrics.

1. Health systems impact

Postgraduate training programs should focus on the endorsed long-term educational outcomes, as they sit in the broader health care system where budgets and human health resource management are important. Specifically, one of the most endorsed long-term educational impacts might be a change in the number of trainees pursuing sub-specialty training after their qualifying exams. Traditionally, the RCPSC training programs in EM have had an embedded area of focused learning that was part of the Postgraduate year 4 of most programs.^{Reference Thoma, Mohindra and Woods16} With regards to costs, the expense of time-variable training within CBME was likely believed to be important for our nationally funded system. Time-variability may be ideal for the trainee^{Reference Nousiainen, Mironova and Hynes17} but can add cost and uncertainty to the health care system. Such financial outcomes are more system-focused and easier to measure than outcomes that attempt to quantify trainee quality.

2. Trainee focused impact

Trainee-focused clinical outcomes may serve as proxies for better physicians and are typically cited by education scholars as potential benefits of CBME^{Reference Frank, Snell and Ten18}; however, defining and measuring these outcomes will be quite difficult. How can we measure physician quality in EM?^{Reference Pines, Alfaraj and Batra19} Especially in environments like EM where trainees can so readily access their supervising physicians, how can we tease apart their individual performance?^{Reference Schumacher, Holmboe, Van Der Vleuten, Busari and Carraccio7,Reference Sebok-Syer, Chahine, Watling, Goldszmidt, Cristancho and Lingard9} Or should we be aiming to find resident-sensitive markers of performance?^{Reference Schumacher, Holmboe, Van Der Vleuten, Busari and Carraccio7,Reference Sebok-Syer, Chahine, Watling, Goldszmidt, Cristancho and Lingard9,Reference Sebok-Syer, Goldszmidt, Watling, Chahine, Venance and Lingard20} These considerations are important but are beyond the scope of this present project. The quality of physician care is undoubtedly a complex construct that involves numerous competing factors.

3. New methods for evaluation of outcomes

While outcome measurement is important, Gruppen and colleagues warn us not to focus too much on demonstrating that graduates of CBME programs are “better.”^{Reference Gruppen, Frank and Lockyer2} Instead, we must match the inherent complexity of outcomes with sophisticated study designs, such as collaborative and multicenter studies, focusing on individuals, teams, institutions, and settings. We should also aim to include novel models for program evaluation such as rapid-cycle evaluation^{Reference Hall, Rich and Dagnone21} and realist evaluation (see Appendix 2). Recent evaluation work in EM evaluating early implementation of CBME using rapid evaluation has yielded important lessons about fidelity of implementation, and perhaps even short-term outcomes such as behavior change among stakeholders.^{Reference Hall, Rich and Dagnone21} Moving forward, these strategies and others will help us tackle the difficult task of high-quality outcome-based evaluation.