A concern of increasing importance to public health practice is limited health literacy, the inability of an individual to fully access, understand, appraise and use information to make informed decisions about his/her health( Reference Nutbeam 1 , 2 ). The Expert Panel on Health Literacy, led by the Canadian Public Health Association, reported that an estimated 55 % of Canadians aged 16–65 years did not have the health literacy skills needed to cope with the demands encountered in their daily health-care tasks( Reference Rootman and Gordon-El-Bihbety 3 ). Moreover, if one includes individuals over 65 years of age, there are more Canadians with low health literacy (60 %) than there are with low general (reading and writing) literacy (48 %)( 4 ).
Health literacy is a system issue and decisions about how best to provide health-related information involves many stakeholders. The Expert Panel on Health Literacy has called for more research to support policy and practice development, underscoring the need for a better understanding of the role that health literacy plays in health disparities, and for action ‘to enhance the capacities of systems that provide health information and service [and] to do so effectively for people with all levels of literacy and health literacy’( Reference Rootman and Gordon-El-Bihbety 3 ). Essential to that research are tools that can be used to efficiently study and measure the health literacy skills of Canadians.
Researchers and practitioners in medical and public health settings have consistently highlighted the need to develop appropriate measures of health literacy to facilitate research, develop interventions to promote health equity, and determine the effects of those interventions on health outcomes for individuals and populations( Reference Weiss, Mays and Martz 5 – Reference Weiss 7 ). While there are many assessment tools available to assess health literacy, most measures focus only on the functional aspects of basic literacy and numeracy skills( Reference Haun, Valerio and McCormack 8 ). Noted disadvantages of common health literacy measures are the length of time required for administration of the tool (e.g. Test of Functional Health Literacy in Adults (TOFHLA))( Reference Parker, Baker and Williams 9 ) and an incomplete measurement of functional competencies (e.g. Rapid Estimate of Adult Literacy in Medicine (REALM))( Reference Davis, Long and Jackson 10 ).
To overcome these limitations, Weiss and colleagues( Reference Weiss, Mays and Martz 5 ) developed the Newest Vital Sign© (NVS), an easy-to-administer screening tool involving interpretation of a nutrition label. The NVS is now widely used in the USA and has subsequently been adapted and validated for use in the UK( Reference Rowlands, Khazaezadeh and Oteng-Ntim 11 ), the Netherlands( Reference Fransen, Leenaars and Rowlands 12 ), Japan( Reference Kogure, Sumitani and Suka 13 ), Brazil( Reference Rodrigues, de Andrade and González 14 ) and Italy( Reference Capecchi, Guazzini and Lorini 15 ), as well as for use in different population groups including older adults( Reference Patel, Joel and Rovena 16 ), youth( Reference Linnebur and Linnebur 17 ), parents and children( Reference Driessnack, Chung and Perkhounkova 18 ), and people with chronic diseases such as diabetes( Reference Bailey, Brega and Crutchfield 19 ).
Despite its worldwide usage, the NVS has not yet been adapted for use in Canada. In addition, there has been limited research on whether the NVS or other health literacy assessments can be administered by computer and still provide valid results. One study using the short version of the TOFHLA found that computerized administration provided similar results to paper-based administration, but not all participants were able to complete the computerized version in the allotted time( Reference Chesser, Keene Woods and Wipperman 20 ).
The NVS was developed for one-on-one in-person administration by an interviewer, intended to mimic ‘real-world’ provider–patient interactions. Since the NVS became widely used, several studies have evaluated whether it can be self-administered using paper and pencil, which requires subjects to be able to read and understand the assessment questions and complete an answer form. These studies have found that self-administration of the NVS can be effective in populations with good literacy skills (e.g. adolescents in school)( Reference Linnebur and Linnebur 17 ), but self-administration does not perform well in low-literacy populations, in which as many as 46 % of individuals fail to complete the self-administered NVS( Reference Warren-Findlow, Patel and Dulin 21 ). Indeed, a recent systematic review evaluated eighty-seven reports of self-administered health literacy assessments and identified twenty-one instruments potentially suitable for self-administration; the NVS was not included among those instruments( Reference O’Neill, Gonçalves and Ricci-Cabello 22 ).
Administration of the NVS by computer, however, offers the possibility of self-administration using audio to deliver instructions and questions – a format that does not require subjects to read the instructions or questions to understand them. Such a format would allow the NVS to be easily and quickly administered, potentially provide valid results, and the ‘privacy’ of computer-based administration could help to reduce the stigmatization that some respondents have expressed in previous studies to interviewer-administered health literacy assessments( Reference Rootman and Gordon-El-Bihbety 23 , Reference Wolf, Williams and Parker 24 ).
We are unaware of any research evaluating computerized administration of the NVS. The purpose of the present study was to develop and evaluate the reliability of the NVS as a self-administered computerized multiple-choice tool via an electronic platform in a Canadian context (in English and French languages using the Canadian Nutrition Facts table and list of ingredients on the food label). Our specific objective was to assess whether self-administered computerized administration v. in-person, one-on-one, interviewer-led administration of the Canadian NVS results in a similar assessment of an individual’s health literacy skills.
Methods
Study design
A randomized crossover design with a washout period was used to compare results from administration of the computer-based NVS (C-NVS) with those of the traditional interviewer-administered NVS (I-NVS). Each participant was assigned to complete first either the C-NVS or the I-NVS health literacy assessment. After completing this first assessment, a follow-up appointment was scheduled (in 3–4 weeks) to complete a second health literacy assessment using the alternative version of the NVS tool.
Instruments
The original NVS is a six-question tool used to assess an individual’s ability to find and understand both text and numerical information on the nutrition label of an American ice cream container. This measure assesses both literacy and numeracy skills required for completion of a common health-relevant task (i.e. making food choices to meet a dietary goal or need). It has been shown to take approximately 3 min to administer during a face-to-face interview in which the nutrition label is held in the individual’s hand while an interviewer reads aloud six questions about the nutrition label( Reference Weiss, Mays and Martz 5 , Reference Welch, VanGeest and Caskey 25 ). Based on the number of correct answers, the respondent is categorized as having a high likelihood of (i.e. ‘likely’) low health literacy (score 0–1), ‘possible’ low health literacy (score 2–3) or ‘adequate’ health literacy (score 4–6).
The NVS tool has been validated both in English and Spanish for use in the USA( Reference Weiss, Mays and Martz 5 ) and, as noted earlier, has been adapted and validated for use in several other languages and countries. It has been used in both clinical and research settings to assess the likelihood of low health literacy in an individual( Reference Rowlands, Khazaezadeh and Oteng-Ntim 11 , Reference Fransen, Leenaars and Rowlands 12 ) and among subgroups of the population of varying ages and ethnicities( Reference Protheroe, Whittle and Bartlam 26 ).
Health Canada’s nutrition labelling experts in the Nutrition Labelling and Claims section of the Nutrition Regulations and Standards Division of the Bureau of Nutritional Sciences face-validated the Canadian version of the NVS. They adapted the American nutrition facts table and list of ingredients on the ice cream label to comply with the nutrition labelling guidelines of Canada’s Food and Drug Regulations in effect at the time of the research, including all optional elements that could be included in the nutrition facts table (i.e. servings per container, metric values, per cent daily value footnote). This labelling format, in English (Fig. 1) and French versions, was chosen over the bilingual format as a bilingual label would have altered considerably the demand for literacy skills in comparison with the original NVS. Ice cream was considered a culturally appropriate food product given its accessibility in food marketplaces and familiarity to most people in Canada.
Fig. 1 (colour online) Newest Vital Sign© instrument comprising the nutrition facts table and list of ingredients on the back of a 500 ml container of ice cream: (a) the original American version; (b) the adapted Canadian version (in English language)
The standard approach to administering the NVS is a one-on-one interview in which the interviewer reads the questions to subjects and subjects provide their answers to the six questions in an open-ended format. To use an open-ended format on a computer, however, would require the computer to recognize all possible variations of correct answers, which could include hundreds of minor variations in wording or spelling, and the computer would need to determine if each of those possible answers is correct or incorrect. Setting up such a computer program would be a ‘Google-level’ task, impractical for health literacy research. Instead, to allow presentation of the NVS on a computer, we developed a multiple-choice form of the NVS.
To do this, we determined appropriate multiple-choice distractors for each of the six NVS questions by having project staff conduct standard, interviewer-led open-ended NVS assessments with seventy-seven persons of varying health literacy levels (50 % scoring in the ‘adequate’ health literacy category, 50 % scoring in the low health literacy ‘possible’ or ‘likely’ categories) in French (n 23) and English (n 54) in Ottawa, Ontario, and Antigonish, Nova Scotia. We kept track of the most common incorrect answers given by subjects and used those most common incorrect answers as the distractor responses in the multiple-choice format. While this approach does not precisely mimic open-ended administration, it does give subjects the choice of selecting the incorrect answers that other subjects most commonly give during in-person administration. Those incorrect responses are shown in the multiple-choice French and English versions of the Canadian NVS. The complete tool, including the multiple-choice version and instructions for administration and scoring, were then professionally translated into Canadian French and back-translated into Canadian English.
In addition, for the C-NVS, a Microsoft® PowerPoint© presentation incorporating visual basic programming was developed to enable the integration of pre-recorded narration and programmed selection buttons to capture each participant’s answers. As the text (question and multiple-choice answers) and visuals (ice cream nutrition label showing the nutrition facts table and ingredient list) appeared on the computer screen, the participant would also hear the text as a narrative using the same questions used on the original interviewer-administered NVS.
To minimize computer skills and complexity of interaction we made use of the simplest and most available computer-based technologies. All participants used a computer mouse to interact with the PowerPoint presentation viewed on the computer screen. Narration was heard through computer headphones. An introductory module consisting of three slides with a pre-recorded narration overviewing the use of the computer, keyboard and mouse was completed by each participant before beginning the C-NVS PowerPoint module. Several versions of the complete computer-based tool, including the introductory module, were reviewed, pilot-tested by study researchers with consumers and revised to ensure clarity of sound and image.
For the I-NVS, participants were handed a copy of the ice cream nutrition label showing the nutrition facts table and ingredient list. The six NVS questions were read aloud to each participant in a one-on-one session with an interviewer. While the C-NVS used multiple-choice answer options for the reasons noted earlier, answers for the I-NVS were in the open-ended format used on the original NVS.
Scoring and timing
For both the I-NVS and the C-NVS, participants received 1 point for each of the six questions answered correctly, and 0 points for each incorrect answer. For both versions, participants had the option of skipping questions if they did not know the answer or chose not to answer. Unanswered questions were assigned a score of 0. This approach to skipped or unanswered questions is the standard approach recommended for administration of the NVS. The time to administer the C-NVS and the I-NVS was digitally recorded in each interview session.
Participants and recruitment
English- and French-speaking adults aged 18 years or older were recruited from multicultural catchment areas that include families, seniors and students of varying socio-economic status levels in Ottawa, Ontario and Antigonish, Nova Scotia.
Recruitment was done as follows: two local site coordinators, one in Ottawa, and one in Antigonish, met with community group leaders and allied health and adult education/literacy professionals (nurses, dietitians, physical activity promoters and adult educators) to make them aware of the project. An information sheet about the study was made available to these professionals to review in-person with their clients and participants in their various multicultural community-based groups of French- and English-speaking seniors, students and parents. The local site coordinator then met with those clients and groups that expressed interest in the project and screened interested participants in-person with the recruitment screening tool. Participants had to be 18 years or older, speak English or French, and understand and sign the information and consent form that was read aloud to them. Medical and demographic data were not collected, with approval from our research ethics boards, to avoid having potential subjects decline to participate because of the demand for functional literacy skills needed to complete forms and the concern for respecting participant privacy. The priority was to engage participants across the spectrum of literacy and health literacy levels to test the assessment tool. Furthermore, detailed demographic data collection was not needed to determine if both versions could generate similar scores with people of varying health literacy levels.
Participants also had to agree to attend the two scheduled sessions (to undergo a NVS with the alternative mode of administration), although as part of the informed consent process they were told they could withdraw from the study at any time. Each participant was randomly assigned to complete either the I-NVS or the C-NVS session first. After completing this first assessment, a follow-up appointment to complete the second health literacy assessment using the alternative format of the NVS was scheduled 3–4 weeks later.
There was no incentive or monetary compensation for participating in the study, although each participant was reimbursed for transportation, parking and/or childcare costs up to a maximum of $CAN 10/person for each interview session they attended. These study methods were approved by the Research Ethics Boards of Health Canada, University of Toronto and St. Francis Xavier University.
Sample size
We conducted sample size calculations for dependent-samples, two-sided t test with an effect size of 0·35 and α=0·05, with power >0·90, and calculated that we would need at least ninety participants to complete both the computerized and in-person administration of the NVS. We also assumed that at least 80 % of participants would remain in the study to complete both versions. Based on these calculations and assumptions, we aimed to enrol 110 English-speaking and 110 French-speaking participants to assure that ninety English and ninety French participants completed both versions of the tool.
Data analysis
One-way ANOVA models were computed to test whether there were mean differences between health literacy scores for French and English participants based on the type and order of the NVS (C-NVS, I-NVS). McNemar’s tests were also performed to determine whether the health literacy status of each participant (‘adequate’ health literacy v. ‘likely’ or ‘possible’ low health literacy) assessed with each of the two versions of the NVS were equivalent and if there was any effect of the order of the testing (computer administration first v. interviewer administration first). The data were summarized with histograms of NVS scores by type of NVS tool and analysed using the statistical software package SAS version 9.3.
Results
Enrolment and study completion
In total, 222 participants (112 English speakers/110 French speakers) including adults, seniors and students of various socio-economic status levels were recruited, with a distribution approximating the demographics of the community groups from which they were recruited. A total of 180 (ninety English-speaking and ninety French-speaking) participants completed both the I-NVS and C-NVS. Based on their NVS scores, their health literacy levels were categorized as low health literacy ‘likely’ or ‘possible’ for at least 25 % of participants in each language group and as ‘adequate’ among 63 % of English and 48 % of French participants using either tool (Table 1).
Table 1 Scores on the interviewer-administered and computerized versions of the Newest Vital Sign© (NVS) among Canadian adults aged 18 years or over recruited from multicultural catchment areas that included families, seniors and students of varying socio-economic status levels in Ottawa, Ontario and Antigonish, Nova Scotia
I-NVS, interviewer-administered version of NVS; C-NVS, computerized version of NVS.
a,bMean values were significantly different from the rest at P<0·05.
* Number of correct answers.
† There was no significant effect of format (C-NVS, I-NVS) and order of testing on participants’ NVS numerical scores (McNemar’s test; P=0·8375).
‡Cumulative percentage.
There were twenty-two English-speaking and twenty French-speaking participants who completed the first but not the second NVS assessment. Of these, 61 % were seniors, 95 % were female, 73 % scored in the low health literacy ‘likely’ or ‘possible’ limited health literacy category, and 79 % had been assessed with the I-NVS.
Comparison of scores on C-NVS and I-NVS
Scores for participants completing both assessments ranged from 0 to 6, with a mean of 3·63 (sd 2·11) for the C-NVS and 3·41 (sd 2·21) for the I-NVS. Figure 2 shows the distribution of scores on both the C-NVS and I-NVS overall (Fig. 2(a)) and by language (Fig. 2(b)). There was no significant effect of type of format (C-NVS, I-NVS) or order of testing on participants’ NVS numerical scores (P=0·8375). Although large proportions of participants (52/90 English, 58/90 French) had slightly different numerical scores on the C-NVS and I-NVS tools with an absolute mean difference of 0·91 (se 0·07, sd 0·98), only a small percentage (7·7 % of those tested in English and 12·2 % of those tested in French) had scores that placed them in different health literacy categories when assessed with the C-NVS v. the I-NVS (Fig. 3). Importantly, the proportion of those identified as low health literacy status (low health literacy ‘likely’ or ‘possible’) was not affected by the order of testing with the two formats of NVS (P values for McNemar’s test: I-NVS <C-NVS, P=0·1227; C-NVS<I-NVS, P=0·7456).
Fig. 2 Distribution of Newest Vital Sign© (NVS) scores by (a) version (
, C-NVS;
, I-NVS) and (b) language (, English;
, French;
, overall) among 180 (ninety English-speaking and ninety French-speaking) Canadian adults aged 18 years or over recruited from multicultural catchment areas that included families, seniors and students of varying socio-economic status levels in Ottawa, Ontario and Antigonish, Nova Scotia. Each respondent completed both I-NVS and C-NVS (C-NVS, computerized version of NVS; I-NVS, interviewer-administered version of NVS)
Fig. 3 (colour online) Test–retest reliability of the interviewer-administered with the computerized version of the Newest Vital Sign© (NVS) among 180 (ninety English-speaking and ninety French-speaking) Canadian adults aged 18 years or over recruited from multicultural catchment areas that included families, seniors and students of varying socio-economic status levels in Ottawa, Ontario and Antigonish, Nova Scotia. Each respondent completed both I-NVS and C-NVS. *Size of the bubble is proportional to the number of observations at that point (smallest bubble size represents one observation; largest bubble size=twenty-four observations). †Scores in the box labelled ‘Marginal health literacy’ are the scores of individuals who scored in the categories of low literacy possible or likely (i.e. score <4) on both the computerized and interviewer-administered versions of the NVS. ‡Scores in the box labelled ‘Adequate health literacy’ are those of individuals who scored in the category of adequate health literacy (score ≥4) on both the computerized and interviewer-administered versions (C-NVS, computerized version of NVS; I-NVS, interviewer-administered version of NVS)
Administration time
The average time required to administer and complete the C-NVS (4 min 22 s) was slightly longer than for the I-NVS (3 min 30 s), a difference of 52 s (P < 0·0001). The English versions of both NVS formats required 30 s less time to complete than the French versions (P=0·0018).
Discussion
To our knowledge, this is the first time the NVS health literacy assessment tool has been adapted for use in Canada, on paper or electronically. Furthermore, and perhaps most importantly, this is the first time the NVS has been adapted and administered in electronic form using a multiple-choice format with an integrated voice-over component in place of the traditional interviewer-based administration in any country. Indeed, a recent review of health literacy screening tools for eHealth applications( Reference Collins, Currie and Bakken 27 ) did not identify any validated computer-based health literacy screening instruments.
Our study showed that computer administration and interviewer administration yielded similar results when French speakers were tested with both French versions, and when English speakers were tested with both English versions. The ability to administer the NVS via computer with results similar to those obtained with in-person administration makes the tool an easily accessible and widely available approach to assessing the likelihood of limited health literacy when evaluating individuals in research, clinical and health promotion settings. Although the time to administer the C-NVS was slightly longer (by an average of 52 s) than the time to administer the I-NVS, the fact that an in-person interviewer does not need to be present during administration of the C-NVS makes the C-NVS more time-efficient and cost-effective from the point of view of staff time and personnel costs. Overall, therefore, our successful experience with this approach raises the possibility of using the instrument for large-scale, or even national-scale, health literacy assessments without the need for the in-person presence of research staff.
From an ethical perspective, the electronic interface minimizes the acknowledged discomfort or stigmatization that individuals sometimes report with interviewer-based tools, especially among those at highest risk of limited health literacy( Reference Parikh, Parker and Nurss 23 , Reference Wolf, Williams and Parker 24 , Reference Parker, Davis and Williams 28 ). The C-NVS will also be a useful addition to Internet-based questionnaires and survey tools to enable a better understanding of how health literacy may affect people’s ability to access, understand and use health information to make decisions to meet their health needs.
Limitations
There are limitations of our study that should be considered when interpreting the results. Although our participation rate was good, with 81 % of participants completing both the I-NVS and C-NVS, a higher number of individuals who scored in the limited health literacy categories (n 31) did not finish the study and complete both versions of the NVS compared with those who scored in the category of adequate health literacy (n 11). Furthermore, more of these individuals completed the I-NVS first, as opposed to the C-NVS, which may be a consequence of the discomfort or stigmatization with interviewer-based tools as discussed above, although limited ‘digital literacy’ may have caused some individuals not to complete the C-NVS. Whether and to what degree this might have affected our results could not be determined.
Implications for future practice and research
Availability of a computerized version of the NVS has important implications for health literacy research and programme planning, by allowing health literacy assessments to be performed without in-person interviews and potentially on a large-scale population basis. It also has the potential of allowing assessments to be performed with a lower risk of stigmatizing individuals who might be uncomfortable with in-person assessments. Furthermore, with the increasing importance of nutrition in the prevention and management of chronic disease, the Canadian multiple-choice adaptation of the NVS has particular value for assessing the influence of health literacy on food-based decision making( Reference Pleasant, McKinney and Rikard 29 ).
Conclusions
Based on the results of our study we conclude that both the English and French versions of a Canadian multiple-choice adaptation of the NVS can be successfully administered via computer, with results similar to those obtained with the standard question-and-answer, interviewer-led administration. Further validation studies with larger sample sizes and involving specific subgroups (e.g. ethnicity, age, gender) in a larger, more representative sample of Canadians and using additional modes of administration (e.g. mobile applications) would confirm these findings( Reference Haun, Valerio and McCormack 8 ).
Acknowledgements
Acknowledgements: The Newest Vital Sign© was reproduced with Pfizer’s permission to Health Canada, Copyright © Pfizer Inc. All rights reserved. Elaine de Grandpre, Manager of Knowledge Transfer and Education, Bureau of Nutritional Sciences, Health Canada, provided advice in the development of the research project and editorial support for the publication of the research report. Ashley Brooker, Dietetic Intern, St. Francis Xavier University, assisted with the collection of data. Disclaimer: The views expressed in the submitted article are the authors’ own and not an official position of the institutions or funders. Financial support: This work was supported by the Canadian Institutes of Health Research (CIHR) (grant number 201403MOP 137037). The CIHR had no role in the design, analysis or writing of this article. Conflict of interests: No author has any known conflict of interest. Authorship: E.D.M., D.E.G., B.D.W. and M.L. formulated the research question and designed the study; E.D.M., D.E.G. and R.W. carried out the study; E.D.M. analysed the data and wrote the article; R.W., D.E.G., B.D.W. and M.L. edited the article. Ethics of human subject participation: This study was conducted according to the guidelines laid down in the Declaration of Helsinki and all procedures involving the participants were approved by Health Canada’s Research Ethics Board. Written informed consent was obtained from all participants.
