Introduction: Short-term gains in knowledge and skills of critical emergency procedures are demonstrated after simulation, but there is uncertainty regarding long term retention. Our objective was to determine whether simulation of critical emergency procedures promotes long term retention of procedural skills in non-surgical physicians likely to perform them. Methods: MEDLINE and Embase (from start of database to June 2018) and the CENTRAL Trials Registry of the Cochrane Collaboration (May 2018 Issue) were searched using a peer-reviewed strategy. Studies were eligible if they (1) were observational cohorts, quasi-experimental or randomized controlled trials, (2) assessed intubation, cricothyrotomy, periocardiocentesis, tube thoracostomy or central line placement performance by non-surgical physicians, (4) utilized any form of simulation (all levels of realism and technology), and (4) assessed skill performance immediately after and at ≥3 months post-simulation. There was no language restriction. Two reviewers independently assessed article eligibility. One reviewer extracted data and assessed study quality. Primary outcome was skill performance 3 months post-simulation. Secondary outcomes included skill performance at 6 and ≥12 months post-simulation, and skill competency at 3 months post-simulation. Results: 1370 citations were identified. 12 studies were eligible. Methodological quality was uniformly poor with high risk of bias, lack of defined primary outcomes, inadequate sample sizes, and non-standardized, unvalidated tools of unclear clinical significance. Given significant heterogeneity in design, populations, procedures, and outcome timing, a narrative synthesis of results was undertaken. In 10 studies participants’ performance at 3, 6 and 12 months retention testing remained above baseline assessment. However, 3 studies showed a significant decrease in performance at 3 months post-simulation compared to immediately post-simulation. Performance was also lower in 2 studies at 6 months post-simulation, and 2 studies at ≥12 months post-simulation. Four studies assessed competency and 3 demonstrated maintenance of competency. Conclusion: There was significant heterogeneity and poor methodological quality among the eligible studies. Results were conflicting for retention of procedural skills and competency. Future directions should include development of robust assessment tools, and improved research methodology of simulation education targeted at critical procedural skills.