There is a considerable body of literature suggesting that standardized rater training is a critical element to achieving inter-rater reliability. With the advance of clinical and research trials to worldwide locations there have been challenges in tailoring instruments not only in terms of language but also culture. In this study we compare different cohorts of raters receiving training from seven different countries. We asked if there was any consistent by-country impact on certain items, factors, or subscales on the instrument.

The data set comes from rater training events held in Russia, India, Singapore, Austria and the United States including participants from seven different countries. Raters (n=221) scored videotaped interviews of schizophrenic patients using the Positive and Negative Syndrome Scale (PANSS). Results were compared across countries and intraclass correlation coefficients (ICC) compared by total score, subscale score and by item.

There were no significant differences between rater cohorts in terms of PANSS total score. Differences emerged in terms of the assessment of negative symptoms in general and the assessment of unusual thought content in particular.

The lack of differences in rater performance across countries indicates that standardized rater training is broadly effective in cohorts selected to participate in clinical trials. By item analysis indicated that there are some important differences in the way that different groups conceptualize items. This suggests a need to tailor training to those items that are less reliable in certain groups to ensure reliability and validity in the use of this instrument.