Over the last twenty years, NSF and the engineering community have called for systemic changes in engineering education, including an emphasis on contextual understanding; increased teaming skills, including collaborative, active learning; and an improved capacity for life-long, self-directed learning. In addition, ABET has called for engineering graduates that demonstrate an ability to apply science and engineering, and ABET requires assessment processes designed to measure student achievement of learning outcomes. Olin College has responded to these calls for change by embracing new learning approaches and assessment techniques, and by developing project-based courses that encourage experiential understanding of content and aid the development of life-long learning skills. To address the assessment needs of new pedagogical approaches, Olin recently instituted a competency based assessment system to accompany the traditional course grading system already in place. The thread of competency assessments provides grading coherency for both faculty and students, and it provides students with valuable information concerning their development of nontraditional skills that they could use to identify shortcomings and further their learning. In this paper, we describe the new pedagogical approaches in Olin's introductory materials science course, and we explain our implementation of the competency assessment system to measure student attainment of both materials science knowledge and broader skills such as teaming, communication, and experimental inquiry.