Introduction: For many years, Emergency Medicine (EM) educators have used narrative comments to assess their learners on each shift, either in isolation or combined with some type of Likert scale ranking. Competency based medical education (CBME), soon to be fully implemented throughout Canadian EM educational programs, encourages this type of frequent low-stakes narrative assessment. It is important to understand what information is currently garnered from existing narrative assessments in order to successfully and smoothly transition to the CBME system. The purpose of this study was to explore how one Canadian undergraduate EM program's narrative assessment comments mapped to two competency frameworks: one traditional CanMEDS-based and one competency-based, built on entrustable professional activities (EPAs). Methods: A qualitative and quantitative content analysis of 1925 retrospective, narrative assessments was conducted for the 2015/2016 and 2016/2017 academic years. The unprompted comments were mapped to the Royal College CanMEDS framework and the Association of Faculties of Medicine of Canada EPA Framework. Using an iterative coding process as per accepted qualitative methodologies, additional codes were generated to classify comments and identify themes that were not captured by either framework. Results: 93% and 85% of the unprompted narrative assessments contained comments that mapped to at least one CanMEDS role or EPA competency, respectively. The most common CanMEDS role commented upon was Medical Expert (86%), followed by Communicator, Collaborator and Scholar (all at 23%). The most common EPA competency mentioned related to history and physical findings (62%) followed by management plan (33%), and differential diagnosis (33%). However, 75% of narrative comments contained within the assessments, included ideas that did not fall into either framework but were repeated with frequency to suggest importance. The experiential characteristics of working with a learner were commented upon by 22% of preceptors. Other unmapped themes included contextual information, generalities and platitudes, and directed feedback for next steps to improve. Conclusion: While much of the currently captured data can be mapped to established frameworks, important information for both learner and assessor may be lost by limiting comments to the competencies described within a particular framework, suggesting caution when transitioning to a CBME assessment program.