1. The survey was conducted by Jun Yin, who also analyzed the data. Technical questions about the administration and results of the surveys should be addressed to Dr. Yin ([email protected]).

The survey of full professors sampled white males at a ratio of one-third and all others at a ratio of one. The survey of departments contacted chairs of all Ph.D.-granting departments, half of the M.A.-granting departments, and one-third of the B.A. political science and B.A. combined/joint programs. Responses for B.A. political science and B.A. combined/joint programs have been combined in the tables.

In all cases, the data have been appropriately weighted and can be treated as random samples.

2. The surveys, which were originally scheduled for November, were eventually circulated in May, a time when end-of-the-academic-year chores competed for respondents' attention.

The response rate was lower than might have been expected or desired. Moreover, the response rate was uneven with, one suspects, those who have had experience with the solicitation and writing of outside letters—that is, those with something to say on the subject—selectively more likely to respond. Among full professors, those in Ph.D.-granting departments and those with professional visibility were more likely to return the questionnaires, which would have the effect of overstating the average burden imposed by the solicitation of outside letters. Among department chairs, those in B.A.-granting departments had markedly lower rates of return. Thus, the already low figure for the proportion of B.A. departments that solicit letters is probably inflated. Response rates were as follows:

3. The comments appended to the questionnaires bear slight elaboration. Although the survey addressed full professors in their capacity as writers of, and department chairs in their capacity as solicitors of, external reviews, respondents made their comments from the perspectives of many roles. Thus, on the basis of their experiences as senior members of departments, full professors raised issues about the use of outside letters in making decisions. Similarly, department chairs mentioned requests to serve as external referees. Respondents to both surveys referred to experiences as deans, as members of university promotions committees, and the like.

4. The stakes are especially high for all concerned when the decision entails the awarding of tenure. Hence, reviewers may take special pains to dispense their responsibilities conscientiously. It is also possible that tenure reviews are more time-consuming because the external reviewer would be less likely to be familiar with the work of an emerging scholar than of a more established member of the profession.

5. Although we cannot measure the extent to which members of Ph.D.-granting departments or research or professionally active faculty are overrepresented among respondents, it is clear that the respondents do not constitute a representative sample of full professors, or even of full professors who are members of the Association. Presumably, one source of selection bias would be that those who are asked to write outside letters would be more likely to want to sound off about the process. Moreover, it is also possible that those who act as good citizens—by writing outside letters and by taking on other professional responsibilities—are also more likely to return questionnaires.

6. Of course, department chairs may have little autonomy in deciding which personnel decisions require external documentation.

7. The disparity in the recollections of the two groups of respondents might reflect a tendency for department chairs to begin the mental accounting at the time of the initial inquiry and the full professors at the time of the receipt of written materials—or a natural tendency of each group to give a positive account of their behavior.

8. Although the bias in the direction of the positive was a consistent theme in the comments, several respondents raised the issue of the consternation caused by the occasional letter that is not simply negative, but relentlessly, perhaps even gratuitously, negative.

9. It should be noted that departments differ in the extent to which they are in a position to promise confidentiality to external referees. Departments in public institutions seem to be less likely to be in a position to make promises of confidentiality. In the comments accompanying the questionnaires, one lone respondent argued that candidates for promotion should be able to see a “sanitized” version of the outside letters.

10. These are, of course, pseudonyms. However, the criss-crossing of letters to faculty with similar first names and utterly dissimilar surnames did take place.