If CALT is different from other forms of testing, should computer-based testing be evaluated against a different set of standards from that used to evaluate other tests? Chapter 3 focused specifically on the potential threats that may limit the validity of inferences and uses of CALT. Such threats have occupied the public's attention, but this chapter takes up the more technical issue of how CALT should be evaluated overall. Surely, an evaluation of the quality of a test should not be centered entirely on the aspects of the test that catch the attention of the informed layperson. This chapter addresses CALT evaluation in view of specific suggestions that have been made by CALT developers. These suggestions in particular and CALT evaluation more generally are discussed from the perspective of the profession's views of how all language tests should be evaluated. It begins with a look at guidelines and advice suggested to promote quality during CALT development and evaluation, and then summarizes research focused on specific aspects of CALT quality. A subsequent section demonstrates the types of findings obtained through the use of Bachman and Palmer's (1996) test usefulness framework. The final part discusses CALT evaluation within more recent perspectives in educational measurement (Kane, 2001) about developing a validity argument through the definition of inferences intended to underlie score interpretation and use. We argue that specific technology-related issues should be placed within a broader framework of test evaluation.

Guidelines and advice

From the time that computer-assisted testing began to be established in the 1970s, the unique set of issues posed by the medium was recognized by educational measurement researchers.