Introduction: Much of the literature investigating the challenges associated with completing high quality work-based assessments (WBAs) have raised specific concerns over the appropriate documentation of assessments of underperforming trainees or trainees in difficulty. The purpose of this study was to examine the relationship between resident performance and the quality of assessments documented by supervisors on Daily Encounter Cards (DECs). The effect of trainee proximity (i.e. on-service versus off-service status) on this relationship was also examined. Methods: A series of DECs from the Department of Emergency Medicine at the University of Ottawa was scored by two raters using the Completed Clinical Evaluation Report Rating (CCERR). The CCERR is a 9-item instrument that has previously demonstrated reliable ratings and the ability to discriminate the quality of completed DECs. A proxy measure of resident performance was calculated by averaging the scores across performance items on the DEC to produce a “mean DEC rating”. Linear regression analysis was conducted with “mean DEC rating” as the independent measure and CCERR score as the dependent measure. Separate linear regression analyses were repeated for DECs completed for on-service versus off-service residents. Results: Linear regression analysis demonstrated a small but significant inverse relationship between mean DEC rating and CCERR score (p<0.001, r=-0.184), suggesting that when residents performed poorly, their supervisors tended to document higher quality assessments, and conversely, when residents performed well, their supervisors provided lower quality assessments. Further analysis demonstrated that this relationship was present for the on-service group (p<0.001, r=-0.24). However, no relationship was observed in the off-service group (p=0.62, r=-0.05). Conclusion: Resident performance and trainee proximity are important factors impacting the quality of documented clinical performance assessments. Greater attention needs to be given to determining ways of improving the quality of assessments reported for residents who are appropriately progressing in their clinical competence as well as for off-service trainees.

Introduction: Barriers to completing high quality work-based assessments (WBAs) include relational factors such as the episodic and fragmented interaction that often exists between clinical supervisors and trainees. In an effort to increase supervisor-trainee continuity, the Department of Emergency Medicine at the University of Ottawa created Clinical Teaching Teams (CTT) in which a resident and clinical supervisor work matched shifts together throughout the year. The aim of this study was to determine the impact of supervisor-trainee continuity on the quality of assessments documented on Daily Encounter Cards (DECs). Methods: DECs completed by 20 clinical supervisors were collected and sorted into three groups representing differing degrees of supervisor-trainee continuity (Group 1: CTT emergency resident; Group 2: non-CTT emergency resident; Group 3: non-CTT off-service resident). DECs were scored using the Completed Clinical Evaluation Report Rating (CCERR), a 9-item instrument that has been shown to have reliable ratings and the ability to discriminate the quality of completed DECs. Scores were analyzed using a univariate ANOVA with “mean CCERR score” as the dependent variable and “continuity group” and “supervisor” as between-subject variables. The relationship between CCERR scores and number of CTT encounters over time was examined using a repeated measures ANOVA with “encounter number” as the within-subject factor. Results: Mean CCERR scores for the CTT (21.0, SD=5.8), non-CTT (21.9, SD=4.2), and off-service (20.7, SD=4.0) groups differed (p=0.019). A subsequent pairwise comparison demonstrated a statistically significant difference in means between the non-CTT and off-service groups (p=0.04); however, this 1.2 difference on the 45-point CCERR scale is unlikely to be of any educational significance. The number of repeated encounters did not have a statistically significant effect on CCERR scores (p=0.43) indicating that DEC quality did not improve with greater supervisor-trainee interaction. Conclusion: DEC quality as scored by the CCERR was low for all three groups. Increasing supervisor continuity alone did not result in higher quality assessments of clinical performance. Additional research focusing on the educational alliance that develops between supervisor and trainee may hold greater promise.

Introduction: In response to concerns in the literature over the quality of completed work-based assessments (WBAs), faculty development and rater training initiatives have been developed. The Completed Clinical Evaluation Report Rating (CCERR) was designed to evaluate these interventions by providing a measure of the quality of documented assessments on In-Training Evaluation Reports (ITERs). Daily Encounter Cards (DECs) are a common form of WBA used in the Emergency Department setting. A tool to evaluate initiatives aimed at improving the quality of completion of this widely used WBA is also needed. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Methods: This study was conducted in the Department of Emergency Medicine at the University of Ottawa. Six experts in resident assessment grouped 60 DECs into three quality categories (high, average, poor) based on their perception of how informative each DEC was for reporting judgments of the resident’s performance. Eight clinical supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared using a univariate ANOVA to determine if the CCERR was able to discriminate DEC quality. Reliability for the CCERR scores was determined using a generalizability analysis. Results: Mean CCERR scores for the high (37.3, SD=1.2), average (24.2, SD=3.3), and poor (14.4, SD=1.4) quality groups differed (p<0.001). A pairwise comparison demonstrated that differences between all three quality groups were statistically significant (p<0.001), indicating that the CCERR was able to discriminate DEC quality as judged by experts. A generalizability study demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. Conclusion: There is strong validity evidence to support the use of the CCERR to evaluate DEC quality. It can be used to provide feedback to supervisors for improving assessment reporting, and offers a quantitative measure of change in assessor behavior when utilized as a program evaluation instrument for determining the quality of completed DECs.