Objective: To identify and compare clinical practice guideline appraisal instruments.

Methods: Appraisal instruments, defined as instruments intended to be used for guideline evaluation, were identified by searching MEDLINE (1966–99) using the Medical Subject Heading (MeSH) practice guidelines, reviewing bibliographies of the retrieved articles, and contacting authors of guideline appraisal instruments. Two reviewers independently examined the questions/statements from all the instruments and thematically grouped them. The 44 groupings were collapsed into 10 guideline attributes. Using the items, two reviewers independently undertook a content analysis of the instruments.

Results: Fifteen instruments were identified, and two were excluded because they were not focused on evaluation. All instruments were developed after 1992 and contained 8 to 142 questions/statements. Of the 44 items used for the content analysis, the number of items covered by each instrument ranged from 6 to 34. Only the instrument by Cluzeau and colleagues included at least one item for each of the 10 attributes, and it addressed 28 of the 44 items. This instrument and that of Shaneyfelt et al. are the only instruments that have so far been validated.

Conclusions: A comprehensive, concise, and valid instrument could help users systematically judge the quality and utility of clinical practice guidelines. The current instruments vary widely in length and comprehensiveness. There is insufficient evidence to support the exclusive use of any one instrument, although the Cluzeau instrument has received the greatest evaluation. More research is required on the reliability and validity of existing guideline appraisal instruments before any one instrument can become widely adopted.