Governmental policy statements on mental health practice over the past decade have emphasised the importance of routinely measuring individual patient outcomes (Department of Health, 1991, 1998; Secretary of State for Health, 1999). Despite the availability of various standardised tools with which to measure the symptom severity of common psychiatric disorders, patient need and wider quality of life and health status, little is known about the actual use of standardised outcomes measures by clinicians (Reference Slade, Thornicroft and GloverSlade et al, 1999). One previous survey of 73 consultant psychiatrists from 1989 established which of a pre-specified range of symptom-based clinical measures were in use at that time (Reference Rice and DonnellyRice & Donnelly, 1992). In view of the central role given to outcomes measurement in the recent National Service Framework for Mental Health (Secretary of State for Health, 1999) we decided to undertake a survey of the current use of outcomes measures in psychiatric practice in the UK.
NATIONAL SURVEY
Questionnaire design
We constructed and piloted a self-completion/self-report questionnaire that sought to identify the routine standardised outcomes measures used by adult psychiatrists for the purpose of:
-
(a) identifying and assessing the severity of clinical disorders;
-
(b) identifying patients' needs and deficits in social functioning, and quality of life;
-
(c) monitoring patient progress;
-
(d) clinical audit.
Respondents were asked about the use of outcomes measures for the following problems: depression, anxiety and related disorders; schizophrenia and other psychoses; cognitive impairment; drugs and alcohol. Respondents also were asked what outcomes measures their trusts routinely collected or asked them to collect.
Survey method
We conducted a postal questionnaire survey between July and September 2000 of 500 consultant psychiatrists practising in the UK National Health Service (NHS) chosen randomly from UK NHS clinicians responsible for the psychiatric care of working-age adults listed in the Medical Directory.
Questionnaire responses
In total, 369 (74%) of the 500 questionaires were returned, of which 29 were incomplete or ineligible (raw eligible response rate 340/500, 68%; modified response rate 340/471, 72%). Most were general psychiatrists (82%) and worked in non-teaching hospital settings (66%). Details of their responses are given in Table 1.
Table 1 The reported use of standardised measures by clinicians (the percentage values are given with 95% confidence intervals)
| Use by clinicians of standardised measures, by problem category | Case identification and severity measurement | Assessing social functioning, quality of life and needs | Measuring clinical change over time | Clinical audit |
|---|---|---|---|---|
| Depression/anxiety | ||||
| Never | 55.3% (49.8-60.1) | 80.6% (75.9-84.7) | 58.2% (52.8-63.5) | 76.5% (71.6-80.9) |
| Occasionally | 34.1% (29.0-39.4) | 12.9% (9.6-17.0) | 30.5% (25.7-35.8) | 15.3% (11.6-19.6) |
| Routinely | 10.5% (7.5-14.4) | 6.5% (4.1-9.6) | 11.2% (8.0-15.0) | 4.1% (2.2-6.8) |
| Measures used | ||||
| BDI, 61/340 | HoNOS, 20/340 | BDI, 49/340 | BDI, 18/340 | |
| HAD, 53/340 | SAS, 9/340 | HAD, 41/340 | HoNOS, 18/340 | |
| HRSD, 46/340 | SFQ, 5/340 | HRSD, 23/340 | HRSD, 13/340 | |
| HoNOS, 11/340 | GAF, 4/340 | HoNOS, 18/340 | HAD, 12/340 | |
| MADRS, 10/340 | CAN, 3/340 | MADRS, 10/340 | ||
| GAF/CGI, 9/340 | ||||
| Schizophrenia/other psychoses | ||||
| Never | 72.9% (67.9-77.6) | 75.6% (70.4-79.8) | 73.5% (68.5-78.1) | 78.8% (74.1-83.0) |
| Occasionally | 20.6% (16.4-25.3) | 13.5% (10.1-17.6) | 20.0% (15.9-24.7) | 11.8% (8.5-15.7) |
| Routinely | 6.5% (4.1-9.6) | 11.2% (8.0-15.0) | 6.5% (4.1-9.6) | 9.4% (6.5-1.3) |
| Measures used | ||||
| PANSS, 25/340 | PANSS, 20/340 | HoNOS, 33/340 | HoNOS, 24/340 | |
| HoNOS, 20/340 | BPRS, 13/340 | BPRS, 13/340 | PANSS, 6/340 | |
| BPRS, 17/340 | HoNOS, 16/340 | PANSS, 12/340 | BPRS, 8/340 | |
| Manchester Scale, 9/340 | GAF/CGI, 9/340 | |||
| PSE/SCAN, 6/340 | ||||
| GAF, 5/340 | ||||
| Cognitive impairment | ||||
| Never | 44.7% (39.3-50.2) | 83.5% (79.2-87.3) | 66.5% (61.2-71.5) | 86.5% (82.4-89.9) |
| Occasionally | 40.6% (35.3-46.0) | 10.6% (7.5-14.4) | 24.7% (20.2-29.6) | 10.6% (7.5-14.4) |
| Routinely | 14.7% (11.1-18.9) | 5.9% (3.6-8.9) | 8.8% (6.0-12.4) | 2.9% (1.4-5.3) |
| Measures used | ||||
| MMSE, 134/340 | HoNOS, 13/340 | MMSE, 60/340 | MMSE, 13/340 | |
| WAIS, 9/340 | QL Checklist, 3/340 | HoNOS, 13/340 | HoNOS, 9/340 | |
| WAIS, 6/340 | ||||
| Drugs/alcohol | ||||
| Never | 83.3% (79.1-87.3) | 88.8% (84.9-91.9) | 91.2% (87.6-94.0) | 91.2% (87.6-94.0) |
| Occasionally | 10.6% (7.5-14.3) | 5.9% (3.6-8.9) | 4.1% (2.3-6.8) | 3.5% (1.8-6.1) |
| Routinely | 5.9% (3.6-8.9) | 5.3% (3.2-8.2) | 4.7% (2.7-7.5) | 5.3% (3.2-8.3) |
| Measures used | ||||
| CAGE, 10/340 | HoNOS, 12/340 | HoNOS, 10/340 | HoNOS, 8/340 | |
| SADQ, 3/340 |
Case identification and severity measurement
Depression/anxiety and cognitive impairment were the disorders where outcomes measures were used most commonly for identifying and assessing the severity of psychiatric disorders, with around half of clinicians using these measures either routinely or occasionally. The most commonly used measures were the Beck Depression Inventory (Reference Beck and WardBeck & Ward, 1961) (61/340), the Hospital Anxiety and Depression Scale (Reference Zigmond and SnaithZigmond & Snaith, 1983) (53/340) and the Hamilton Rating Scale for Depression (Reference HamiltonHamilton, 1967) (46/340).
The most commonly used measure in detecting cognitive impairment was the Mini-Mental State Examination (Reference Folstein, Folstein and McHughFolsteinet al, 1975). For disorders such as schizophrenia, the majority of consultants (72.9%; 95% CI 67.9-77.6) never used standardised measures. For drug and alcohol problems, the most commonly reported measure was the CAGE questionnaire (Reference Mayfield, Millard and HallMayfieldet al, 1974) (10/340).
Assessing social functioning, quality of life and needs
Only a small minority used a standardised questionnaire for detecting patients' needs and deficits in social functioning and quality of life. The most commonly used was the Health of the Nation Outcome Scale (HoNOS; Reference Curtis, Beevor and WingCurtis & Beevor, 1995). Specific ‘needs assessment’ tools, such as the Medical Research Council (MRC) Needs for Care and the Camberwell Assessment of Needs, rarely were reported as being used in routine practice.
Measuring clinical change over time
Standardised measures most commonly were used in order to measure change over time for depression and anxiety problems, with 19.4% (95% CI 15.3-24.0) of consultants using them either routinely or occasionally. The most commonly reported measures were the Beck Depression Inventory (49/340); the Hospital Anxiety and Depression Scale (41/340) and the Hamilton Rating Scale for Depression (23/340). For those with dementia, the most commonly used questionnaire was the Mini-Mental State Examination (60/340). For other conditions, clinicians rarely used standardised measures.
Clinical audit
Overall, standardised questionnaires were used even less for clinical audit than for the other purposes outlined above. The most commonly reported condition for which they were used was depression/anxiety, where the Beck Depression Inventory was the most common instrument; for other conditions (schizophrenia; drugs and alcohol; and dementia) the HoNOS was used.
Outcomes measures routinely collected by hospitals/trusts
Very few clinicians (46/340, 13.5%; 95% CI 10.0-17.6) reported being required routinely to collect standardised outcomes measures for all patients by their trust. When asked specifically about the HoNOS, 88/340 (26%; 95% CI 21.3-30.1) reported being asked to collect these data for certain patients, and only 28/340 (8.2%; 95% CI 5.5-11.7) reported using specific needs assessment tools (such as the Camberwell Assessment of Need and the MRC Needs for Care).
In contrast to standardised questionnaires, such as the HoNOS, trusts commonly collected routine administrative data, such as length of stay (86.2%; 95% CI 82.7-89.9) and readmission (70.6%; 95% CI 65.4-75.4).
General comments relating to the use and experience of outcomes measurement
Approximately one-third (120/340) used the questionnaire to give comments. Forty respondents expressed a negative view of standardised outcomes measures, questioning their ability to capture the subtlety of multi-faceted outcome and to describe the individual patient. Terms such as “simplistic” and “ pseudo-scientific gloss” were used. One respondent expressed reservation about any scale that “divides continuous fluctuating process into arbitrary categories”. Twenty-eight respondents explicitly questioned the basic psychometric properties of validity, reliability and sensitivity to change for available measures.
Respondents stated that the valid and reproducible use of outcome measures requires a robust infrastructure, particularly in terms of administration and information technology resources, to support the process (n=20). Respondents generally felt that these features were lacking and that this represented a barrier to their use. Additionally, 29 respondents felt that more time and resources would be needed if outcome measurement were to be carried out and used routinely. Twenty-two respondents stated that they did not find the results of standardised outcomes measures particularly useful in clinical practice. One respondent stated that they were more “research tools” rather than instruments that are useful in clinical practice. Another stated that the “use of scales detracts from the therapeutic relationship”.
Comments were largely critical of the HoNOS (n=21) and related to: time to complete (n=16); inadequate psychometric properties (n=8); the lack of value added to routine clinical assessment (n=5); and the lack of enthusiasm among staff (n=7). Positive comments (n=7) included the fact that it could be completed by non-clinicians (n=4) and that it acted as a useful aide mémoire in clinical decision-making (n=3). One person stated that “the HoNOS, although scientifically flawed, is useful for bringing together all members of the multi-disciplinary team”.
IMPLICATIONS FOR MENTAL HEALTH PRACTICE, RESEARCH AND POLICY
Outcomes measurement forms a central component of the National Service Framework, with, for example, the stipulation of a minimum data set (Reference Glover, Knight and MelzerGlover et al, 1997), which includes the HoNOS, to be collected alongside care planning for all of those with severe mental illness and a number of outcomes indicators to be implemented on a routine basis (Secretary of State for Health, 1999). The results of this survey, in particular the barriers identified to outcomes measurement, will be of particular interest to those charged with implementing the National Service Framework for Mental Health. Several areas are worthy of further discussion.
Our main finding is that the majority of psychiatrists do not use outcomes measures in their day-to-day practice. Patient needs and psychosocial problems are measured infrequently in any standardised or consistent way, despite explicit Government policy (Reference Glover, Knight and MelzerGlover et al, 1997; Secretary of State for Health, 1999) to adopt measures such as the HoNOS and needs assessment tools. This may reflect a wider indifference towards and failure to address psychosocial outcomes and needs, or simply a failure to use standardised measures to assess them.
The HoNOS does seem to have found a place in measuring outcome in UK mental health services, albeit a small one. Initial hopes that it would be used on a servicewide basis in order to measure outcome, define population needs and to judge the performance of individual trusts and teams (Reference Wing and DelamontheWing, 1994) have, however, clearly not been realised. Specific concerns expressed by clinicians relating to the time taken to complete the instrument and the poor psychometric properties will need to be addressed if its use is going to be increased. On the other hand, positive aspects of outcomes measures were identified that may be useful in deciding their place in mental health services. In particular, the comments that non-medical personnel can complete instruments such as the HoNOS and that it encouraged discussion within the wider clinical team are worthy of note.
Administrative measures collected by trust — such as length of stay and readmission rates — although easier to collect, may bear little relation to the psychosocial outcome of the individual patient or clinical population. It is administrative outcomes that will form the basis of the performance elements of the NHS (Secretary of State for Health, 1999). Of particular concern is that these figures are the easiest to manipulate or ‘improve’, without conferring any overall health gain on the population or service under consideration (Reference Nutley and SmithNutley & Smith, 1998).
Teams need to be resourced adequately to collect outcomes data, and the instruments that are used must be reasonably concise and easily administered (Reference Slade, Thornicroft and GloverSlade et al, 1999). Additionally, the reluctance among clinicians to collect data on a routine basis may reflect the fact that they see little benefit to themselves or their patients' care. Outcomes measurement is a ‘technology’ that has opportunity costs and therefore must be shown to be beneficial in improving the quality of care as measured by actual outcomes of patients or communities. No direct evidence exists that there is a benefit in this respect for those working and being cared for in either primary care or specialist psychiatric services (Gilbody et al, Reference Gilbody, House and Sheldon2001a ,Reference Gilbody, House and Sheldon b ). Successful implementation of outcomes management will need to overcome the barriers that we identified in this survey of current UK psychiatric practice.
Acknowledgements
S. M. G. was supported by the UK Medical Research Council Health Services Research Training Fellowship Programme. The NHS Centre for Reviews and Dissemination and the Department of Health Studies are part of the Medical Research Council Health Services Research Collaboration (MRC HSRC).
eLetters
No eLetters have been published for this article.