Introduction: Competence committees (CCs) struggle with incorporating professionalism issues into resident progression decisions. This study examined how professionalism concerns influence individual faculty decisions about resident progression using simulated CC reviews. Methods: In 2017, the investigators conducted a survey of 25 program directors of Royal College emergency medicine residency training programs in Canada and those faculty members who are members of the CCs (or equivalent) at their home institution. The survey contained twelve resident portfolios, each containing formative and summative information available to a CC for making progression decisions. Six portfolios outlined residents progressing as expected and six were not progressing as expected. Further, a professionalism variable (PV) was added to six portfolios, evenly split between those residents progressing as expected and not. Participants were asked to make progression decisions based on each portfolio. Results: Raters were able to consistently identify a resident needing an educational intervention versus those who did not. When a PV was added, the consistency among raters decreased by 34.2% in those residents progressing as expected, versus increasing by 3.8% in those not progressing as expected (p = 0.01). Conclusion: When using an unstructured review of a simulated resident portfolio, individual reviewers can better discriminate between trainees progressing as expected when professionalism concerns are added. Considering this, educators using a competence committee in a CBME program must have a system to acquire and document professionalism issues to make appropriate progress decisions.