This paper summarises the methods used in a five-nation European study to establish the reliability of the European Version of the Camberwell Assessment of Need (CAN-EU), presents the results, and discusses the implications of these findings. The paper should be red in close conjunction with other related papers in this series, which give more detailed accounts of key related aspects of the study (Becker et al, Reference Becker, Knapp and Knudsen1999, Reference Becker, Knapp and Knudsen2000, this supplement; Reference Knudsen, Vázquez-Barquero and WelcherKnudsen et al, 2000, this supplement; Reference van Wijngaarden, Schene and Koetervan Wijngaarden et al, 2000, this supplement).

Within the domain of the assessment of individual patient outcomes, increasing importance has recently been attached to the needs of those who suffer from mental illness. This emphasises the active role of the users of mental health services, and also raises a series of important questions: How can needs be defined, and by whom ? How can they be measured and compared ? What importance should be accorded to both met and unmet needs in the assessment of individual patients, and in the planning and evaluation of mental health services as a whole ? How should the needs of those suffering from schizophrenia be prioritised in relation to the needs of other diagnostic groups ?

CAMBERWELL ASSESSMENT OF NEED - EUROPEAN VERSION (CAN-EU)

The assessments of need in the European Psychiatric Services: Inputs Linked to Outcome Domains and Needs (EPSILON) Study were made using the Camberwell Assessment of Need - European Version (CAN-EU), based on the CAN Research Version 3.0 (Reference Phelan, Slade and ThornicroftPhelan et al, 1995). This is an interviewer-administered instrument comprising 22 individual domains of need: accommodation, food, looking after the home, self-care, physical health, psychological distress, psychotic symptoms, information about condition and treatment, daytime activities, company, safety to self, safety to others, alcohol, drugs, intimate relationships, sexual expression, basic education, child care, transport, using a telephone, money and welfare benefits. There is good agreement between staff and user ratings of the overall numbers of needs, although there may be substantial differences for individual items. It is important to note that in the present study the service user (patient) ratings are used.

Each item of the CAN-EU contains the same question structure. The first question asks whether a need exists, and, if it does, whether it is a met or an unmet need. If there is no need in any particular area, then the interviewer proceeds straight to the next item. If a met or unmet need does exist, then further questions relating to service receipt for that item are asked. The first of these finds out how much care is received from friends or relatives (0=no help, 1=low help, 2=moderate help, 3=high help). The same question is asked about care received from formal services and also how much care is required from formal services. Finally, the person being interviewed is asked whether overall they receive the right sort of help, and whether they receive the right amount of help. Both of these are rated as zero (no) or one (yes).

Summary scores of the total number of needs (the number of 1s or 2s), the met needs (the number of 1s) and the unmet needs (the number of 2s) are computed. If the number of valid items (i.e. excluding missing values) needs is 18 or more, a prorated total is computed from the valid items, otherwise the summary score is regarded as missing.

AIMS OF THE EPSILON STUDY

The aims of the EPSILON Study are :

1. (a) To produce standardised versions of five instruments in key areas of mental health service research in five European languages (Danish, Dutch, English, Italian and Spanish) by a rigorous conversion process from the original version into the other four languages. This involves (i) accurate and independent translation and back-translation from the original into the other four languages, (ii) checks of cross-cultural applicability using focus groups, and (iii) assessment of instrument reliability. Full details of these procedures are given elsewhere (Becker et al, Reference Becker, Knapp and Knudsen1999, Reference Becker, Knapp and Knudsen2000, this supplement; Reference Knudsen, Vázquez-Barquero and WelcherKnudsen et al, 2000, this supplements; Reference van Wijngaarden, Schene and Koetervan Wijngaarden et al, 2000, this supplement). This paper reports the reliability study on the CAN-EU.

2. (b) To obtain and compare data from five regions in different European countries, each with its particular system of health care, about social and clinical variables, characteristics of mental health care and its costs. The results of this study component are being prepared for publication.

3. (c) To test both instrument-specific and cross-instrument hypotheses. Full details of this stage of the study will be published in due course.

Although it is now relatively common for the authors of outcome scales to publish details of scale reliability in the original language, it is rare for the authors of translated versions to repeat the reliability exercise in the new languages, or indeed to do more than undertake a literal translation. This study therefore aims to undertake the conversion and cultural adaptation of each of the five main study scales into all the study languages in a comprehensive and scientifically rigorous manner.

METHOD

RESULTS

Table 1 shows summary statistics for the primary rater at time 1, including the results of a homogeneity of variance test. Mean total needs and unmet needs differed significantly between sites, with Amsterdam and London tending to show higher values than the other three sites on both measures. This will be discussed further in a paper on the substantive results of the study. Of more relevance to the reliability estimation is the lack of homogeneity in variance for met and total needs, as shown by the Levene test in Table 1.

The α coefficients, which reflect correlations between individual CAN items, were moderate to low, as shown in Table 2. For total needs, the pooled α was 0.64 (95% CI 0.58-0.70). Only for met needs (pooled mean 0.48, 95% CI 0.40-0.56) was there strong evidence for differences between sites, with Santander having the lowest value at 0.16. For unmet needs (pooled mean 0.58, 95% CI 0.51-0.64) the differences were less marked, but Copenhagen showed a somewhat higher value than the other sites, at 0.70.

The ICCs between the two time points are given in Table 3, which shows that test-retest reliability is at an acceptable level. There were no significant differences between sites, except for unmet needs. Pooled values were 0.85 (95% CI 0.82-0.88) for total needs, 0.69 (95% CI 0.63-0.74) for met needs and 0.78 (95% CI 0.74-0.82) for unmet needs.

For estimating κ coefficients for the individual items, unmet, met and total needs were each expressed as binary variables in turn. Table 4 shows that κ coefficients for test-retest reliability were high for total needs (0.55-0.84), and moderately high for met needs (0.40-0.76, excluding κ for met needs for drugs, which was zero) and unmet needs (0.34-0.85). Standard errors for these κ estimates were typically 0.06, 0.08 and 0.09 respectively. Only one item had a κ coefficient below 0.4 (unmet needs for physical health).

There was evidence of site differences in interrater reliability (Table 5) for total, met and unmet needs. However, all the sites had coefficients for total and met needs above 0.8. In the case of unmet needs, coefficients for Amsterdam, Santander and Verona were under 0.8, although they were all still over 0.65. The pooled estimates were 0.93 (95% CI 0.92-0.95) for total needs, 0.85 (95% CI 0.81-0.87) for met needs and 0.79 (95% CI 0.75-0.83) for unmet needs.

Interrater reliability for individual items, pooled over sites (Table 6), was very good for total needs (0.75-0.99) and met needs (0.57-0.95), and moderately good for unmet needs (0.41-0.83). Standard errors were typically 0.04, 0.07 and 0.10 respectively. All items had κ coefficients over 0.4.

Paired sample t-tests revealed a tendency for a decrease in the rating of total needs over time, pooled across sites, but this was significant only at a borderline level (P=0.053). At individual sites, there were no significant differences between mean scores at test and retest, with the exception of total needs in Verona, where the time 2 total values were rated lower: 4.39 at time 2 compared with 5.11 at time 1, difference 0.72 (95% CI 0.52-0.92), P=0.001. This is most likely to be a chance finding, given the large number of tests employed.

DISCUSSION

This paper has described how the reliability of the CAN-EU was tested in five different centres. Face validity was not specifically tested. However, focus groups were used in the translation process and these indicated that the CAN-EU was largely acceptable in its format and content. This confirmed the attainment of face validity for the original English version (Reference Phelan, Slade and ThornicroftPhelan et al, 1995). Very high internal consistency between the items for the CAN-EU is not expected or even necessarily desirable, and the moderate levels of α are quite acceptable in this context. Indeed, they are not surprising, given the diverse range of needs assessed with the instrument, which were deliberately selected to cover the entire range of difficulties commonly encountered by people suffering from severe mental illnesses. In this context the α coefficients are not that informative, but have been reported so as to maintain consistency with the other papers in this supplement.

The very low value for α for met needs for Santander is interesting, and may be connected with the lower level and smaller degree of variation in met needs at that particular site, as shown in Table 1. Alternatively it may be connected with one particular item, “help with psychotic symptoms”. When this item is removed, the α is doubled to 0.32, more in keeping with its value at other sites.

Overall, the test-retest reliability is at least moderately good, although usually lower than interrater reliability. Lack of reliability may, in some cases, have been due to changes in patient status that occurred between the two time points, in addition to lack of consistency in a patient's responses from one time point to the next. However, interviews were generally made within intervals of 1-2 weeks, so real changes in status were unlikely.

Interrater reliability is excellent, with only a slight fall-off for unmet needs. Although there were significant differences between sites, all values of interrater reliability coefficients were over 0.65. The two slightly lower reliabilities for unmet needs (Amsterdam and Verona) are due to higher standard errors of measurement rather than the differences in variances between the samples shown in Table 1. It should be noted that Verona had a larger pool of primary raters and, in this respect, the data from that site may more realistically reflect the range of raters who might use the instrument in practice. The very small standard error of measurement in London might reflect a longer history of CAN training and use.

For individual items, both for testretest and interrater, the items with the lowest k values tend to be those where there are low base rates for the need: for example, drugs. Two items showed both low k and low percentage agreement in the test-retest comparison: ‘psychological distress’ (item 9) and ‘company’ (item 14). These two items are not of this character, and there seems to be no obvious pattern in the inconsistent responses over time. It may be that these two items are hard to rate because they are very much related to mood and reflect relatively transient situations.

A point which applies generally, both over time and also between raters, is that there are greater levels of agreement for total needs than for the component items. However, the very skewed nature of the data relating to individual items (i.e. the low base-rates in many cases) makes reliability tests problematic. Indeed it reduces the feasibility of analysing these variables individually, except in very large samples.

Mean scores did not differ significantly between test and retest, except for one score in one site. The pooled ratings for total needs did decrease slightly over time (at a borderline level of significance) but in general there is little evidence for substantial increase or decrease over time, a problem which might occur if patients tended to reflect on and modify their ideas following an interview. In these respects the CAN can be seen to be stable over time.

This analysis has concentrated on the three total needs scores, rather than individual items. This is because the 22 CAN items, while clearly important in considering the needs of individual patients, are of limited use for analytical purposes when treated in isolation, since most of them are encountered infrequently in individual cases. Similarly, the sections of the CAN relating to levels of formal and informal care received, and formal care required, are most relevant for clinical rather than research purposes. With large samples, the data on particular needs and on care required or received could be analysed, but such samples have hitherto been scarce.

Bearing in mind these caveats, we suggest that the summary scores for the CAN-EU (total, met and unmet needs) are generally reliable over time and between raters. Despite some evidence for differences in levels of reliability between sites for unmet needs at test-retest, and between raters for all three total scores, the results are good at each site, and encouraging for the use of this instrument in its five translations.