Introduction
In some instances, acquired brain injury (ABI) can result in extended periods of reduced consciousness where individuals may be described as ‘slow-to-recover’. According to the Royal College of Physicians (2003) guidelines, between 10 and 20% of people who experience moderate to severe brain injury may remain in either a vegetative state (VS) or minimally conscious state (MCS). Impaired awareness and responsiveness are characteristic of disordered consciousness and the stages of recovery of consciousness have been defined by consensus-based diagnostic criteria which differentiate coma, VS and MCS (Giacino et al., Reference Giacino, Zasler, Katz, Kelly, Rosenberg and Filley1997, Reference Gill-Thwaites and Munday2002).
Coma requires absence of eye opening, verbalisation or mouthing of words, response to commands and intentional movement. Emergence from coma to VS requires periods of eye opening and return of autonomic functions, but there is no evidence of the ability to follow commands, make intelligible verbal responses, make verbal or gestural attempts to communicate or exhibit localising or automatic motor responses (Giacino & Kalmar, Reference Giacino and Kalmar1997). Furthermore, VS has been defined as ‘no evidence of sustained, reproducible, purposeful or voluntary behavioural responses to visual, auditory, tactile or noxious stimuli’ (The Multi-Society Task Force on Persistent Vegetative State, 1994, p. 1499). Minimally conscious state, as defined by the Aspen Workgroup on the Vegetative and MCS, is ‘a condition of severely altered consciousness in which the person demonstrates minimal but definite behavioural evidence of self or environmental awareness’ (Giacino et al., Reference Giacino, Zasler, Katz, Kelly, Rosenberg and Filley1997, p. 84). One or more of the following behaviours needs to be observed and reproduced (at least inconsistently) to determine emergence from VS to MCS: comprehension of simple commands, manipulation of objects, gestural or verbal yes/no responses, intelligible verbalisations or stereotypical (not reflexive) movements (Giacino et al., Reference Giacino, Zasler, Katz, Kelly, Rosenberg and Filley1997).
There are a number of behavioural assessments which are used by clinicians to assess consciousness. Behavioural assessments of consciousness involve observation of a range of behaviours associated with emergence of consciousness, such as arousal, responsiveness to structured and naturally occurring stimuli, communication and purposeful movement. Clinicians use such assessments to monitor changes in consciousness over time. Having a reliable and valid assessment measure is a necessity in order to ensure results are accurate. An additional principal element required to maximise utility is responsiveness. Responsiveness is the sensitivity of a measure to detect true or reliable changes occurring in the individual over time, above and beyond random fluctuations or error inherent in the test score. Responsiveness is essential for assessing the effectiveness of treatment and longitudinal change over time (Guyatt, Deyo, Charlson, Levine, & Mitchell, Reference Guyatt, Deyo, Charlson, Levine and Mitchell1989). If measures are sensitive and able to reliably detect change over time, they can help clinicians to determine readiness for rehabilitation in a timely manner, plan and implement appropriate treatments, reliably detect changes in response to treatment and justify treatment efforts and funding. Multiple methods have been suggested to determine responsiveness including paired t-test, receiver operating characteristic curve and effect size statistics (Kazis, Anderson, & Meenan, Reference Kazis, Anderson and Meenan1989; Portney & Watkins, Reference Portney and Watkins2000); however, there is no consensus on the most ideal method (Husted, Cook, Farewell, & Gladman, Reference Husted, Cook, Farewell and Gladman2000; Wright & Young, Reference Wright and Young1997).
A recent systematic review by Seel and colleagues (Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010) of behavioural assessment scales of disorders of consciousness resulted in evidence-based recommendations about their clinical use. The review examined the available evidence for content validity, reliability, ability to predict functional outcomes and diagnostic validity of assessments of disorders of consciousness, and was conducted by an expert panel of 14 members (Seel et al., Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010). The systematic review included 37 articles which represented 13 assessment scales for disorders of consciousness (Seel et al., Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010). The evidence was rated using several evidence-rating schemes, including: the Aspen Workgroup criteria for transitioning from VS to MCS and emerging from MCS (used to assess content validity), the American Academy of Neurology (AAN) four-tier level of evidence scheme (used to classify studies addressing diagnostic and prognostic validity) and the task force-developed three-tier evidence scheme (used to classify studies addressing reliability) (Seel et al., Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010). Clinical practice recommendations were based on the evidence and used a modified AAN strength of recommendation scheme to classify assessments as recommended for use with minor, moderate or major reservations, not recommended at this time, or not recommended (Seel et al., Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010). Ultimately, they recommended that the Coma Recovery Scale – Revised (CRS-R) may be used to assess disorders of consciousness with minor reservations and the Disorders Of Consciousness Scale (DOCS), Sensory Modality Assessment and Rehabilitation Technique (SMART), Sensory Stimulation Assessment Measure (SSAM), Wessex Head Injury Matrix (WHIM), and the Western Neuro Sensory Stimulation Profile (WNSSP) may be used with moderate reservations. The remaining seven assessment scales included were either recommended for use with major reservations, or were not recommended for use at this time or not recommended. Measures that were recommended for clinical use with minor or moderate reservations in the review by Seel and colleagues (Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010) were included in the current review.
Summary of Recommended Measures
Coma Recovery Scale – Revised
The JFK Coma Recovery Scale – Revised (Giacino, Kalmar & Whyte, Reference Giacino, Kalmar and Whyte2004) is a clinician-administered scale that assesses patients with disorders of consciousness, assists with differential diagnosis between coma, VS and MCS and can be used for treatment planning and monitoring. This revised scale was developed in line with the diagnostic recommendations of the Aspen Workgroup on MCS and was designed for patients with ABI with Ranchos Los Amigos Levels of Cognitive Functioning (RLAS) from Level I through to IV. The 23 items are divided into six subscales, each hierarchically organised from reflexive to cognitively mediated behaviours, including auditory, visual, motor, oromotor, communication and arousal processes. Scoring is based on the presence of a behavioural response to stimuli. Psychometric properties are sound; inter-rater reliability is high (r = 0.84), as is test–retest reliability (r = 0.94) (Giacino et al., Reference Giacino, Kalmar and Whyte2004). It has demonstrated concurrent validity with the Disability Rating Scale (DRS) (Rappaport, Hall, Hopkins, Belleza, & Cope, Reference Rappaport, Hall, Hopkins, Belleza and Cope1982) and internal consistency is high (r = −0.90 and Cronbach's α = 0.83, respectively) (Giacino et al., Reference Giacino, Kalmar and Whyte2004).
Disorders of Consciousness Scale (DOCS)
The DOCS (Pape, Heinemann, Kelly, Hurder, & Lundgren, Reference Pape, Heinemann, Kelly, Hurder and Lundgren2005a) was designed to detect and distinguish changes in consciousness, providing a description of neuro-behavioural recovery. The DOCS includes 23 test stimuli organised into eight subscales: social knowledge, taste and swallowing, olfactory, proprioceptive and vestibular, auditory, visual, tactile and testing-readiness. A baseline observation is taken whereby a checklist is completed by the examiner after observing the person at rest. Testing stimuli are administered following baseline observation, where scores are allocated based on the best behavioural response to stimuli on a three-point scale (0 = no response, 1 = generalised response, and 2 = localised response). A higher score on the DOCS scale indicates a higher level of neuro-behavioural integrity. The DOCS demonstrates good internal consistency (Cronbach's α = 0.85) and construct validity was determined through Rasch modelling, which showed some evidence of unidimensional hierarchic interval characteristics (Pape et al., Reference Pape, Heinemann, Kelly, Hurder and Lundgren2005a).
Sensory Modality Assessment and Rehabilitation Technique (SMART)
The SMART was designed as an assessment and treatment tool to distinguish accurately patients in VS and MCS and then direct and evaluate rehabilitation efforts to facilitate recovery (Gill-Thwaites & Munday, Reference Gill-Thwaites and Munday2004). The SMART assesses eight modalities including five sensory modalities (visual, auditory, tactile, olfactory and gustatory) and three other modalities comprising motor function, functional communication and wakefulness/arousal. Each modality is rated on a five-level hierarchical scale (level 1 = no response; level 2 = reflexive; level 3 = withdrawal; level 4 = localization; and level 5 = discrimination). The five levels relate directly to RLAS Levels I–IV. If a patient is observed to perform at level 5 in any of the seven modalities, such behaviour is incompatible with a diagnosis of VS. The total score can range from 7 to 35; however, in a clinical context, SMART scores are not summed, rather levels of function in modalities are reported. The formal component of the SMART assessment is conducted by SMART-trained and accredited assessors and comprises a sensory assessment and behavioural observation assessment. The informal component of the SMART assessment consists of information gathered from family and carers regarding observed behaviours and previous interests and preferences. Psychometric properties are sound: inter-rater reliability is excellent (r = 0.96), as is test–retest reliability (ICC of 0.97), with moderate concurrent validity with the WNSSP (r = 0.70) (Gill-Thwaites & Munday, Reference Gill-Thwaites and Munday2004).
Sensory Stimulation Assessment Measure (SSAM)
The SSAM (Rader & Ellis, Reference Rader and Ellis1994) was developed to distinguish changes in consciousness, to provide an accurate, replicable assessment of responsiveness in people with ABI between RLAS Levels II through V, and to assist treatment planning. Stimuli are presented to five senses and the best response is measured using the sensory stimulation response scale (SSRS) within three categories: eye opening, motor and vocalisation. Each category has six response choices, ranging from ‘no response’ to ‘able to follow commands or communicate ideas’. Scores are summed and range from 3 to 36, giving a total response score. Test–retest reliability is excellent (r = 0.93), inter-rater reliability is high (r = 0.89), and the SSAM has demonstrated moderate concurrent validity with the GCS (r = 0.70) and the DRS (r = 0.61) (Rader, Alston, & Ellis, Reference Rader, Alston and Ellis1989; Rader & Ellis, Reference Rader and Ellis1994).
Wessex Head Injury Matrix (WHIM)
The WHIM was developed by a multidisciplinary team (Shiel, Wilson, McLellan, Horn, & Watson, Reference Shiel, Horn, Wilson, Watson, Campbell and McLellan2000a) following observations of 88 patients recovering from severe traumatic brain injury (TBI). It is designed to detect subtle changes in the early recovery of cognition, social behaviour, communication and attention, and to identify a sequence of recovery from coma to emergence from post-traumatic amnesia (PTA). The WHIM contains 62 items which are hierarchically organised, with items rank ordered in terms of their sequence of recovery. Spontaneous behaviours, responses to presentation of standardised stimuli and responses to naturally occurring stimuli are examined. The WHIM score is the rank number of the highest behaviour successfully passed in the sequence of 62 behaviours. Psychometric properties include good inter-rater reliability (k = 0.86) and test–retest reliability (k = 0.74) (Shiel et al., Reference Shiel, Horn, Wilson, Watson, Campbell and McLellan2000a). It has demonstrated strong concurrent validity with the Glasgow Coma Scale (GCS) (initial WHIM r = 0.83 and final WHIM r = 0.95) (Majerus & Van de Linden, Reference Majerus and Van de Linden2000).
Western Neuro Sensory Stimulation Profile (WNSSP)
The WNSSP was developed for use with people with TBI who are classified as RLAS Level II through V, which includes people in VS and MCS (Ansell & Keenan, Reference Ansell and Keenan1989). The tool was designed to evaluate cognitive status, monitor progress and predict improvements in patients who are in VS and MCS. The WNSSP consists of 33 items divided into six subscales (arousal/attention, auditory comprehension, visual comprehension, visual tracking, object manipulation and expressive communication). Total scores range from 0 to 113, with higher scores indicating greater abilities. Psychometric properties include concurrent validity with RLAS (r = 0.73) and excellent internal consistency (Cronbach's α = 0.95) (Ansell & Keenan, Reference Ansell and Keenan1989).
Objectives
The review by Seel and colleagues (Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010) provided a highly valuable examination of the available evidence for the reliability, diagnostic validity (including criterion validity) and prognostic validity of different assessment scales for disorders of consciousness. Their review did not examine responsiveness of the different scales. Thus, to the best of the authors’ knowledge there is no known examination of the evidence available about the ability of the scales to detect meaningful change. This knowledge has obvious clinical utility. It is essential that clinicians are able to determine whether their patients are getting better (or worse) through accurate, objective means. To address this gap in the literature, and complement the findings of Seel and colleagues (Reference Seel, Sherer, Whyte, Katz, Giacino, Rosenbaum and Zasler2010), the present review examines the literature to explore the state of the evidence in relation to the responsiveness of the CRS-R, DOCS, SMART, SSAM, WHIM, and WNSSP.
Methods
Search Strategy for Identifying Literature
Relevant studies were identified from the following five electronic bibliographic databases: Medline, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Cochrane Database of Systematic Reviews (DSR) and Cochrane Central Register of Controlled Trials (CCTR). A search was conducted on 16 August 2012. Search strategies used keywords of the six full assessment titles: Coma Recovery Scale Revised, Disorders of Consciousness Scale, Sensory Modality Assessment and Rehabilitation Technique, Sensory Stimulation Assessment Measure, Wessex Head Injury Matrix, and Western Neuro Sensory Stimulation Profile. Other resources searched were reference lists of included trials and the authors’ personal collections.
Inclusion Criteria
Criteria for including trials were as follows: (1) human adult participants; (2) reported in the English language; (3) reported on the use of at least one of the disorders of consciousness measures (CRS-R, DOCS, SMART, SSAM, WHIM, WNSSP); (4) took repeated measures of participants using the MCS tool on at least two occasions at different points in time; (5) reported total scores on at least two different time points and/or reported on change in scores between at least two different time points; (6) reported or described the participants as involved in a treatment programme during the assessment period, even if the stated purpose of the study was not to test the effectiveness of a treatment. For example, studies were included that did not specifically state that the aim was to test the effectiveness of a treatment but reported outcomes of participants over time where the participants were, for instance, inpatients in a brain injury rehabilitation setting. These studies were included despite the lack of description of specific details of treatment (i.e., nature, intensity, duration) because it was assumed that the patients in these studies were receiving some treatment or management of their condition during the assessment period, given their presence in those settings.
Procedure
Results of searches were tracked through EndNote, a reference-management system. Titles and abstracts were reviewed against inclusion criteria by E.D. If the abstracts and titles did not provide adequate information to evaluate criteria for inclusion, a full-text version of the article was reviewed by E.D. to confirm inclusion or exclusion. Templates were developed and used by both authors to extract relevant information from each included paper about responsiveness, including assessment time-points, statistical methods used to determine change and findings. Data about participants (injury severity and demographic information) as well as nature, intensity and length of treatment interventions were also extracted to enable the authors to consider potential reasons for any variations in findings between studies. Data about change as measured by any other assessments of consciousness that were administered at the same time-points as the six assessments that were the subject of this review were also extracted. These data were included to compare the responsiveness of the six measures relative to other measures used. Both authors reviewed and extracted data for all included papers.
Cohen's d measures the size of the treatment effect. It is important to determine effect size, in addition to statistical significance, because statistical significance does not necessarily imply that meaningful change has occurred. For example, a statistically significant change may be observed solely because of a large sample size, regardless of whether this change is clinically meaningful (Olejnik & Algina, Reference Olejnik and Algina2000). Thus, calculating the effect size is a way to demonstrate that meaningfully significant change has occurred. Cohen's d was calculated using the difference between two means divided by the pooled standard deviation weighted by sample size (Cohen, Reference Cohen1988; Zakzanis, Reference Zakzanis2001). According to Cohen, a d value of 0.8 or higher represents a large effect; 0.5, a moderate effect; and 0.2, a small effect. Meta-analysis was not conducted due to the heterogeneity of the studies identified.
Results
Results of all searches yielded 132 references. Of these, 40 trials met the inclusion criteria and full text articles were reviewed. Following review of the full text articles, 25 were included in the final review: these include articles related to the CRS-R (n = 11), DOCS (n = 2), SSAM (n = 1), WHIM (n = 5) and WNSSP (n = 6). Of the six relating to the WNSSP, one of these concurrently reported on the SMART. Data were extracted from included papers and are presented in Tables 1–4. Cohen's d is reported in the findings column of the summary tables, where it was possible to calculate it.
TABLE 1 Summary of studies that have measured change using CRS-R
Abbreviations for Tables 1–4: ABI, acquired brain injury; AVM, arterio-venous malformation; Ax, assessment; BIRU, Brain Injury Rehabilitation Unit; C/NCS, Coma Near Coma Scale; CRS, Coma Recovery Scale; CRS-R, Coma Recovery Scale-Revised; CVA, cerebro-vascular accident; DOCS, Disorders of Consciousness Scale; DRS, Disability Rating Scale; EEG, electroencephalogram; GCS, Glasgow Coma Scale; GOS, Glasgow Outcome Scale; ICH, intracranial haemorrhage; ICU, intensive care unit; LOS, length of stay; M, mean; MCS, minimally conscious state; NS, not significant; NTBI, non-traumatic brain injury; OBI, other brain injury; PALOCS, Post Acute Levels of Consciousness Scale; PTA, post traumatic amnesia; RLAS, Ranchos Los Amigos Cognitive Levels of Functioning Scale; SAH, subarachnoid haemorrhage; SD, standard deviation; SMART, Sensory Modality Assessment and Rehabilitation Technique; SSAM, Sensory Stimulation Assessment Measure; TBI, traumatic brain injury; TPI, time post-injury; UK, United Kingdom; USA, United States of America; VS, vegetative state; WHIM, Wessex Head Injury Matrix; WNSSP, Western Neuro Sensory Stimulation Profile; ♂, male; ♀, female.
In relation to the CRS-R (Table 1), only 3 of the 11 papers (27.3%) utilised statistical analysis methods to measure change in the CRS-R scores. Data were often presented solely via description or graphical representation. The vast majority (10 of the 11 papers, 91%) reported a change in CRS-R scores following treatment in at least some study participants. Of the three studies reporting results of statistical analysis, two of the three demonstrated a significant change in CRS-R score following treatment. It was possible to calculate the effect size of the change in scores in one paper (DeFina et al., Reference DeFina, Fellus, Thompson, Eller, Moser, Frisina and Prestigiacoma2010), where a large effect size (d = 2.01–5.11 for different groups) was observed. This provides evidence that the CRS-R is responsive to change. Additionally, some studies provided information about changes on the CRS-R simultaneously with changes on other outcome measures examining disorders of consciousness (i.e., concurrent validity). In these studies it was observed that the CRS-R and DRS provided similar patterns of responsiveness to change in participants. However, the level of agreement between the CRS-R and Coma/Near Coma Scale (C/NCS) (Rappaport, Dougherty, & Kelting, Reference Rappaport, Dougherty and Kelting1992) varied. Primarily, there was a consistent pattern of results for both the CRS-R and C/NCS when correlated with the level of drug concentration in two studies (McMahon, Vargus-Adams, Michaud, & Bean, Reference McMahon, Vargus-Adams, Michaud and Bean2009; Vargus-Adams, McMahon, Michaud, Bean, & Vinks, Reference Vargus-Adams, McMahon, Michaud, Bean and Vinks2010). This provides preliminary evidence for concurrent validity; however, future studies would benefit from measuring this directly by correlating the scores of the two measures.
The six papers measuring change using the WNSSP, one of which concurrently measured change using the SMART, are outlined in Table 2. Of these studies, 3/6 used statistical analysis to determine significance of change over time, one of which related to the SMART. It was possible to calculate effect size for four papers (Lammi, Smith, Tate, & Taylor, Reference Lammi, Smith, Tate and Taylor2005; Patrick et al., Reference Patrick, Wamstad, Mabry, Smith-Janik, Gurka, Buck and Blackman2009; Smith, Taylor, Lammi, & Tate, Reference Smith, Taylor, Lammi and Tate2001; Wilson & Gill-Thwaites, Reference Wilson and Gill-Thwaites2000), mostly indicating effect sizes over 0.8. These data support the responsiveness of the WNSSP (Lammi et al., Reference Lammi, Smith, Tate and Taylor2005; Patrick et al., Reference Patrick, Wamstad, Mabry, Smith-Janik, Gurka, Buck and Blackman2009) and SMART (Wilson & Gill-Thwaites, Reference Wilson and Gill-Thwaites2000). Two studies concurrently reported on change on other consciousness scales indicating agreement between the WNSSP and RLAS, Rader's scale and GCS (Hall, MacDonald, & Young, Reference Hall, MacDonald and Young1992) and between the WNSSP and the Post Acute Level of Consciousness Scale (PALOCS) (Eilander et al., Reference Eilander, van de Wiel, Wijers, van Heugten, Buljevac, Lavrijsen and Prevo2009). However, sample sizes were small (n = 6 and n = 3) and agreement was not determined using statistical analysis, rather descriptive analysis.
TABLE 2 Summary of studies that have measured change using WNSSP and/or SMART
See the footnote to Table 1 for abbreviations.
The findings of the five studies measuring change using the WHIM are detailed in Table 3. Three of the five papers used statistical analysis to determine the significance of change and Cohen's d was calculated for three papers (Wilson, Coleman, & Pickard, Reference Wilson, Coleman and Pickard2008; Wilson, Elder, McCrudden, & Caldwell, Reference Wilson, Elder, McCrudden and Caldwell2009; Wilson, Harpur, Watson, & Morrow, Reference Wilson, Harpur, Watson and Morrow2003). Effect sizes were in excess of 0.8 for two of the studies (Wilson et al., Reference Wilson, Coleman and Pickard2008, Reference Wilson, Elder, McCrudden and Caldwell2009) with moderate effect sizes for a smaller sample of seven participants (Wilson et al., Reference Wilson, Harpur, Watson and Morrow2003). This same study by Wilson and colleagues (2003) compared improvement according to the GCS and RLAS with improvement on the WHIM. The WHIM indicated improvements in all seven cases whereas the GCS and RLAS indicated concurrent improvements in only four of the seven cases (Wilson et al., Reference Wilson, Harpur, Watson and Morrow2003).
TABLE 3 Summary of studies that have measured change using the WHIM
See the footnote to Table 1 for abbreviations.
Very few articles measured change on the SSAM (n = 1) or DOCS (n = 2) (Table 4). The two papers relating to the DOCS descriptively reported change during inpatient brain injury rehabilitation of a large sample showing change over time (Pape et al., Reference Pape, Tang, Guernon, Lundgren, Blahnik, Yongliang and Soneru2009) and three detailed case studies showing change over time and responsiveness to medication changes influencing arousal (Pape, Senno, Guernon, & Kelly, Reference Pape, Senno, Guernon and Kelly2005b). The regular serial assessment over time using the DOCS in the three cases highlights the value of regular assessment for intervention planning to maximise consciousness (Pape et al., Reference Pape, Senno, Guernon and Kelly2005b). The study by Davis and Gimenez (Reference Davis and Gimenez2003) used statistical analysis to demonstrate greater change in an intervention group (N = 9) receiving auditory sensory stimulation compared to a small control group (N = 3), and showed greater sensitivity than the GCS and RLAS in demonstrating these changes.
TABLE 4 Summary of studies that have measured change using SSAM or DOCS
See the footnote to Table 1 for abbreviations.
Discussion
This review has summarised the 24 studies identified in our literature search which have used the CRS-R, DOCS, SMART, SSAM, WHIM and WNSSP to measure change over time in response to treatment. The identified studies mostly aimed to investigate the efficacy of treatment programmes over time, or describe/predict outcomes by conducting follow-up assessments at varying lengths of time after initial assessment and injury. Overall, no study that was identified aimed specifically to examine the responsiveness of the measures. The quality of the data on responsiveness is limited across all measures because the majority of studies reported primarily descriptive analysis and only 8 of the 25 studies reported statistical analysis of change. However, the identified studies did provide some evidence for the ability of these measures to detect change following treatment. The CRS-R, WNSSP and WHIM have established responsiveness to change, based on the findings of more than one study that have incorporated statistical analysis in order to demonstrate significant change. Additionally, all of these measures had at least one study where it was possible to calculate the effect size of the change using Cohen's d. Effect sizes for all measures were large. The SMART has demonstrated responsiveness to change based on a single study using statistical analysis to detect change and a large effect size over the course of the entire study. The SSAM has moderate levels of evidence for responsiveness from the findings of one study incorporating statistical analysis of change; however, it was not possible to calculate effect size. The DOCS has preliminary evidence for responsiveness in the form of a study that did not statistically analyse change in scores over measurement time-points but did, however, use descriptive analysis to demonstrate changes in data over time.
Despite the need to establish the responsiveness of available behavioural assessments of people with disordered consciousness, there are obvious limitations associated with relying on observational assessment of consciousness in isolation. For example, the constellation of impairments resulting from the related injury may mean that the person is unable to demonstrate their underlying ability or awareness on some assessment items. For example, physical limitations preventing movement or speech may prevent demonstration of a response. Particularly in these cases, as discussed in a review of the behavioural evaluation of consciousness by Majerus et al. (Reference Majerus, Gill-Thwaites, Andrews and Laureys2005), individualised assessment approaches such as those outlined by Whyte and colleagues (Reference Whyte and DiPasquale1995, Reference Whyte, DiPasquale and Vaccaro1999) are necessary to detect subtle but clinically significant responses. Responsiveness can be assessed individually if a voluntary response is present. This can be determined by comparing the frequency of the behaviour between baseline and other phases, where the person is required to produce the behaviour (i.e., in response to a command or condition) (Majerus, Gill-Thwaites, Andrews, & Laureys, Reference Majerus, Gill-Thwaites, Andrews and Laureys2005). An example of this is a series of studies by Lancioni and colleagues (Reference Lancioni, O'Reilly, Singh, Buonocunto, Sacco, Colonna and Bosco2009, Reference Lancioni, Singh, O'Reilly, Olivetti, De Tommaso, Navarro and Sacco2011), where single-case experimental designs are used to compare the frequency of individual responses (i.e., eye blink to activate an optic sensor micro-switch) between baseline and intervention phases. These studies, which used an individualised assessment approach, were able to provide evidence that, in these cases, there was a clinically meaningful change (i.e., the person was able to learn to use micro-switch technology to request assistance from a caregiver) (Lancioni et al., Reference Lancioni, O'Reilly, Singh, Buonocunto, Sacco, Colonna and Bosco2009, Reference Lancioni, Singh, O'Reilly, Olivetti, De Tommaso, Navarro and Sacco2011). These studies also highlight the value of technology in enabling access and overcoming limitations, which may prevent a person from interacting with their environment and being able to give a response (Lancioni et al., Reference Lancioni, O'Reilly, Singh, Buonocunto, Sacco, Colonna and Bosco2009, Reference Lammi, Smith, Tate and Taylor2011). Also, it is important to consider the use of other assessment techniques, such as functional magnetic resonance imaging (Rodriguez Moreno, Schiff, Giacino, Kalmar, & Hirsch, Reference Rodriguez Moreno, Schiff, Giacino, Kalmar and Hirsch2010) and functional electroencephalography (Cruse et al., Reference Cruse, Chennu, Chatelle, Fernandez-Espejo, Bekinschtein, Pickard and Owen2012), to ensure a comprehensive approach to assessing consciousness.
While the responsiveness of our measures to detect change in a statistical sense is important, this should always be assessed in conjunction with clinical judgement. Outcome measures of disorders of consciousness may not be representative of the amount of change which constitutes a clinically meaningful change for an individual. For example, a very slight improvement in one area on a consciousness scale may result in other changes that are meaningful to the person and their quality of life, but may not constitute statistically significant change overall. For example, an improvement in tactile responsiveness may enable increased interactions with family; improved object manipulation may enable some participation in tasks; or increased arousal and improved and consistent tracking may result in being able to make choices and communicate using a yes/no visual communication system. Determining clinically meaningful difference for consciousness measures poses great challenges because people in VS or MCS are unable to communicate what constitutes meaningful change, and meaningful change is different for each individual. Perhaps the answer to capturing rehabilitation outcomes with patients in MCS, where consciousness assessment is indicating slight but not statistically significant changes in response to treatment efforts, is to use individualised goal attainment to ensure that the targets of our treatment are captured. It is interesting to note that most of the studies reviewed used group statistical analysis of change. Furthermore, in the case of n-of-1 studies or studies with small samples included in our review, descriptive statistics were used to analyse change rather than analysis of clinically or statistically significant change (i.e., reliable change index; Perdices, Reference Perdices2005).
The main limitations of this review are that the methodological quality of the included studies was not evaluated, because, to the authors’ knowledge, there are no tools that specially evaluate the quality of studies designed to investigate sensitivity to change. However, included papers have been comprehensively summarised. Furthermore, this review was not a systematic review using rigorous methods characteristic of such a process, such as selection of included studies by two independent reviewers and strict inclusion and exclusion criteria tailored to a highly specific search question. Rather, a systematic search strategy was used to explore the available literature on responsiveness of the six measures of consciousness. The narrow search strategy utilised (i.e., a keyword search of databases using the full titles of the six measures) also limited the scope of the literature search, and further available studies may have been identified if the search strategy had been broader. However, to counteract this, the reference lists on included studies were checked to identify any previous related studies. There are also numerous other measures of consciousness available that were not the subject of this review and may be useful for monitoring change. Some of these measures include the C/NCS (Rappaport et al., Reference Rappaport, Dougherty and Kelting1992), the Comprehensive Levels of Consciousness Scale (Stanczak et al., Reference Stanczak, White, Gouview, Moehle, Daniel, Novack and Long1984), the Full Outline of Unresponsiveness Scale (Wijdicks, Bamlet, Maramattom, Manno, & McClelland, Reference Wijdicks, Bamlet, Maramattom, Manno and McClelland2005), the Innsbruck Coma Scale (Benzer et al., Reference Benzer, Mitterschiffthaler, Marosi, Luef, Pühringer, De La Renotiere and Schmutzhard1991), the Loewenstein Communication Scale (Borer-Alafi, Gil, Sazbon, & Korn, Reference Borer-Alafi, Gil, Sazbon and Korn2002), the Swedish Reaction Level Scale- 1985 (Stalhammar et al., Reference Stalhammar, Starmark, Holmgren, Eriksson, Nordstrom, Fedders and Rosander1988) and the Glasgow–Liege Coma Scale (Born, Albert, Hans, & Bonnal, Reference Born, Albert, Hans and Bonnal1985).
Due to the lack of consensus on the most ideal statistical method for determining responsiveness of a measure, future studies should perhaps seek to examine responsiveness using a range of available methods. Furthermore, a wider retrospective examination of responsiveness could have been possible; however, many studies did not report the required data (i.e., means, standard deviations and/or raw scores for case studies or small samples at different time points). Treatment studies in this area are faced with the challenge of the rare condition and subsequently, small sample sizes. However, if such data are reported, it would enable data to be pooled to conduct meta-analyses. To enable between-study comparisons and pooling of data, future studies would also need to include information about: measurement time-points; treatment content, intensity and duration; and thorough participant information, including injury severity and time post-injury.
Conclusions
The findings of this review indicate that there is some evidence for the ability of the six measures to detect change following treatment. The CRS-R, WNSSP and WHIM have established responsiveness to change, given that several studies have incorporated statistical analysis to demonstrate significant change and there was at least one study where Cohen's d was able to be determined, demonstrating large effect sizes for each of these measures. The SMART has some evidence of responsiveness, based on one study demonstrating statistically significant change with a large effect size, as does the SSAM, based on one study incorporating statistical analysis of change only. The DOCS has preliminary evidence for responsiveness based on one study using descriptive analysis of change data over time without statistical analysis of change. Future studies should seek to examine responsiveness to further establish the utility of available measures of consciousness to reliably measure change in response to treatment.
