Editors' Commentary

In this chapter John Holcomb raises difficult questions about the ethics of experimental design when investigating questions about student learning, which led him to forego the traditional approach of using a control group. This decision resulted in difficulties with getting his work published in statistics education journals, despite the work having received extremely positive feedback at conferences. He also discusses the meaning of reliability and validity in measurement and provides an example of a simple scheme for analyzing open-ended responses to surveys.

Introduction

My SoTL study involved developing data analysis group projects for an introductory statistics course that required students to write technical reports. To determine if students were learning the desired skills with these projects, I designed take-home mid-term and final examinations where students were given individualized data sets and investigative questions. In contemplating how to present evidence of effectiveness, I thought a great deal about the gold standard of evidence in statistics, namely that to conclude that a treatment causes an effect, subjects have to be randomly assigned to a treatment or a control group. In education, this is not always possible. Even when the same instructor teaches both the treatment and control, he can bias the results, consciously or unconsciously. My anxieties about my study lacking a control group were well founded because I experienced difficulty in publishing my results in a statistics education journal. I believe my paper was held to a different standard from that for other articles. This chapter discusses the decisions I made concerning what evidence to collect, the reviewing process, and the contradiction that my work was well received at regional, national, and international conferences, but never was published in a peer-reviewed journal.

Beginning Ideas

In the late 1990s, when I submitted my application to the Carnegie Academy for the Scholarship of Teaching and Learning (CASTL) program, mathematics departments were developing courses to improve the quantitative literacy of their undergraduate students.