Attempts to reduce health disparities in diet-related chronic diseases and their risk factors require valid and reliable diet assessment tools that can be used in diverse segments of the population. The most commonly used measures in epidemiological studies are food-frequency questionnaires (FFQs), as they can reflect long-term intake, have relatively low respondent burden, and can be either self- or interviewer-administered.
The FFQ technique poses a mental challenge for some respondents, asking them to compute averages of what may be seasonally or otherwise varied intakeReference Dwyer and Coleman1. This is likely to be more problematic for some groups than others. Validation studies of the most commonly used FFQs have generally been conducted with well-educated groups of white women and menReference Willett2–Reference Block, Coyle, Hartman and Scoppa4. Correlation coefficients with estimates of nutrient intake collected from multiple days of diet recalls or records typically range from 0.40 to 0.70. These values have been accepted as indicative of the ability of the FFQ to rank individuals correctly according to categories of nutrient intakeReference Zulkifli and Yu5. However, validation data from other populations (e.g. minority, low education) are scarce. Population subgroups with low socio-economic status (SES) have a higher degree of underreporting of energy intake than high-SES subgroupsReference Stallone, Brunner, Bingham and Marmot6. Two studies found that individuals with less than high school education perform poorly on FFQs, although there have been no studies to examine why this is the caseReference Kristal, Feng, Coates, Overman and George7, Reference Mayer-Davis, Vitolins, Carmichael, Hemphill, Tsaroucha and Rushing8. It is possible that the mental compilation and estimation required is simply too unfamiliar a task for these respondents to formulate answers to the questions and accurately report their intake. It is also possible that the actual food consumption of such persons differs from that of a higher-SES population, being more constrained by economic cycles (e.g. pay periods) than the seasonality typically built into FFQsReference Quandt, Arcury, McDonald, Bell and Vitolins9–Reference Wolfe, Frongillo and Valois11.
There have been several attempts to develop instruments that may be better suited for respondents who are elderly or have low educational attainment. These include a picture-sort food-frequency questionnaire developed and validated in the Cardiovascular Health Study, and another developed and validated in an elderly Utah populationReference Kumanyika, Tell, Fried, Martel and Chinchilli12–Reference Wengreen, Munger, Wong, West and Cutler14. In both cases the respondents were elderly, but were predominantly white and had high educational attainment. In the Cardiovascular Health Study, scores of 24 or higher on the Mini-Mental State Examination (MMSE) were required for inclusion regardless of education. Another instrument, the Diet History Questionnaire, was developed with modifications of the National Cancer Institute–Block Health Habits and History Questionnaire FFQ based on extensive cognitive interviewing with a wide range of participants, but its validation was conducted with a highly educated group of largely white and non-elderly personsReference Block, Hartman, Dresser, Carroll, Gannon and Gardner3, Reference Subar, Thompson, Kipnis, Midthune, Hurwitz and McNutt15.
Thus, while the need for alternative methods of collecting dietary data is acknowledged, they have not been tested in those populations for whom effective dietary assessment instruments are lacking. The present study was based on ethnographic and survey research begun in 1996 in a low-income, largely minority rural elderly population. That research included both qualitative exploration of food and dietary practices and quantitative documentation of dietary intake and other health behavioursReference Quandt, Arcury, McDonald, Bell and Vitolins9Reference Quandt, Vitolins, DeWalt and Roos16–Reference Bell, Quandt, Arcury, McDonald and Vitolins20. In the course of that research a number of issues were identified that were related to difficulties in administering FFQs. The combination of vision problems and low literacy made the use of self-administered instruments difficult. These older adults also found responding to fixed response questionnaires and abstract categories (e.g. Likert-type scales, ordered frequency categories) confusing. These tasks were unfamiliar; they seemed to be particularly challenging for those with little experience in formal education or occupations that required such reasoning. Some had difficulty in describing foods independent of meals. Many had difficulty maintaining attention through a repetitive food-frequency interview. These findings were used to produce two new dietary data collection instruments based on the food frequency approach.
The goal of the present study was to compare the performance of these two new dietary assessment tools and a modified National Cancer Institute–Block FFQ for assessing usual intakes of energy and other nutrients with repeat 24-hour recalls (24HR). The overall objective of the study was to identify the instrument best able to estimate dietary intake in an elderly, low-SES rural population.
Methods
Study population
Study participants were recruited in two rural North Carolina counties. Recruitment strategies included seeking referrals from senior and low-income housing personnel, senior centre and meal site staff, county Social Service agencies, and interviewers who were native to the research counties. Candidates were screened over the telephone, when possible, in their homes or at a local senior health fair. Local housing personnel allowed research staff to conduct screenings in several low-income apartment complexes. The study was described, brochures provided and screening completed. All participants signed informed consent upon enrolment. The study was approved by the Wake Forest University School of Medicine Institutional Review Board.
Participants were selected to equally represent men and women in three ethnic groups: white, African American and Native American. Eligibility criteria were: (1) residence in a study county; (2) age ≥ 65 years; (3) ≤ 12 years of formal education; (4) low income (defined as ≤ 150% of the national poverty level or Medicaid recipient); (5) community-dwelling; and (6) mentally able to answer interview questions without assistanceReference Dwyer and Coleman1Reference Block, Coyle, Hartman and Scoppa4–Reference Stallone, Brunner, Bingham and Marmot6. Additional exclusion criteria included (1) blindness or (2) being on kidney dialysis. Potential participants suspected of poor cognitive function were administered the MMSEReference Cockrell and Folstein21. Persons with scores of 23 or lower (for a ninth grade or higher education) or 17 or lower (for an eighth grade or lower education) were excluded based on recommendations from previous researchReference Crum, Anthony, Bassett and Folstein22.
A total of 137 elders enrolled in the study. Of these, 122 completed at least five 24HR and at least two diet assessment questionnaires and are included in these analyses. Of the 15 who did not complete the study, three died during the study period, five withdrew, one was admitted to a nursing home, one suffered significant cognitive decline and scored below the MMSE cut-off score during the course of the study, one moved out of the study counties, and four became too ill to keep interview appointments or were unable to answer questions without assistance. The 15 who did not complete the study were not significantly different from the 122 who did complete the study in age, education, income, receipt of Medicaid, gender or ethnicity (data not shown). Among the 122 persons included, 117 completed all six 24HR and three diet assessment questionnaires, two completed five 24HR and three questionnaires, and three completed six 24HR and two questionnaires.
Diet assessment
Data collection was completed in three phases (approximately 40 participants per phase) during a 16-month period from mid-February 2003 to early May 2004. Respondents were interviewed nine times in their homes over a 7-month period. The first six interviews were 24HR and were conducted at intervals of approximately 1 month. Each interview had a 7-day window for completion. At the final three interviews, the FFQ, a picture-sort administration of the FFQ (PSFFQ) and a meal pattern questionnaire (MPQ) (described below) were administered in a predetermined random order and recorded on paper forms. Each of these questionnaires was administered at least one day after the final 24HR and at least five days after the previous questionnaire. All three questionnaires were to be completed in a 3-week window after the final 24HR concluded. The study team allowed participants a slightly larger window if scheduling difficulties such as illness or holidays arose.
The 24HR data were entered by interviewers into a laptop computer on site, into the Nutrition Data System for Research (NDS-R) software, version 4.05_33 (Nutrition Coordinating Center, University of Minnesota). Interviewers used the ‘multiple-pass’ method to obtain maximum detailReference Johnson23. Of the six diet recalls, at least one and no more than two were collected for weekend days.
The FFQ used in this study is a validated, semi-quantitative, modified version of the National Cancer Institute Health Habits and History Questionnaire (HHHQ)Reference Mayer-Davis, Vitolins, Carmichael, Hemphill, Tsaroucha and Rushing8. This modified version includes a greater number of ethnic foods than the HHHQ. The FFQ required participants to report typical foods and beverages they had consumed over the previous six months. It consisted of 94 food and beverage line items. For foods, nine fixed response categories were provided and ranged from ‘never or less than once per month’ to ‘two or more times per day’. Beverage consumption ranged from ‘never or less than once per month’ to ‘six or more times per day’. Participants were asked to assign a typical portion size (small, medium or large) to each item with reference to other men/women their age.
The PSFFQ contained the same food and beverage line items as the FFQ. However, a colour photograph of each food or beverage category was presented to participants on a 4 in × 6 in card. Beneath each photo, a large print label described the food/beverage category. Participants selected and put aside those eaten ‘never or less than once per month’. The remaining cards were then sorted onto a 15 in × 36 in cardboard tray displaying each of the eight remaining frequency response categories. Interviewers assisted by reading the card labels or response categories to participants with low literacy skills and visual impairments. Interviewers recorded data on paper forms. Participants assigned a portion size (small, medium, large) to food categories (e.g. fruits, vegetables, beverages as used in the HHHQ). After assigning a general portion size, participants were queried on whether there were any portion size differences within a group, and those differences were recorded on the paper form by the interviewer.
The MPQ format was based on the diet history conceptReference Burke24. It consisted of participants reflecting over the past six months and recalling foods in a meal or snack pattern over the course of a typical week. The form consisted of six separate meal categories: breakfast, morning snack, lunch or midday meal, afternoon snack, dinner or evening meal, and evening snack. Within each meal or snack component, participants were asked to recall their usual intake of foods, guided by a series of prompts. The prompts for foods in each component were determined by the line items of the FFQ and from prior knowledge of eating patterns in the study countiesReference Bell, Quandt, Vitolins and Arcury19. The three snack sections were identical to each other; lunch and dinner consisted of identical food item lists. Open-ended questions were used to address types of sandwiches and mixed dishes consumed. Open-ended questions required interviewers to probe for ingredients and preparation methods for the foods recalled. Participants were probed about condiments and seasonings for items on the MPQ.
Quality control
Before the modified food-frequency instruments (PSFFQ, MPQ) were implemented in the field, they were pre-tested. The FFQ was not pre-tested, because it was used by the research team in previous studies and has been shown to be reliable elsewhereReference Mayer-Davis, Vitolins, Carmichael, Hemphill, Tsaroucha and Rushing8, Reference Vitolins, Quandt, Case, Bell, Arcury and McDonald18, Reference Bell, Quandt, Vitolins and Arcury19. The MPQ instrument, PSFFQ photographs and administration process were each pre-tested with four older adults in a county that was not included in the study sample. The PSFFQ pictures were professional photographs selected from stock photography or created for the research team to accurately represent all food categories and beverages queried. Participants were asked for comments, and interviewers noted any problems with the materials and procedures. For the PSFFQ, photograph content, colour and layout were adjusted as needed based on the comments of the pre-test participants.
Interviewers attended a two-day training session conducted by the investigators. Each was required to audiotape and submit multiple practice diet interviews to receive certification to collect data. Audiotapes were reviewed and compared with data collection forms to evaluate accuracy, as well as to assess interviewer style and skill. In order to prevent bias, one interviewer completed all 24HR interviews with a participant; a different interviewer blinded to the 24HR results conducted the final three interviews.
During the course of the study, 5% of diet recalls were tape-recorded and reviewed to ensure that drift did not occur in interviewer technique and accuracy of data entry. Sixty-four per cent of all questionnaire interviews were recorded. A random selection was reviewed, as well as questionnaires where ambiguous answers were recorded. To verify the interview process and assess participant experience with the interviewer, 19% of participants were contacted by phone regarding their diet recalls and 14% of the participants regarding their FFQs.
Additional measures
Demographic data were collected at baseline including self-reported ethnicity, education, date of birth, length of residence in the study counties, household size and income variables. Vitamin and supplement usage and overall health assessment data were also collected. Vitamin and supplement use data are not included in the present analyses.
Data management and statistical analysis
The 1998 Block Dietary Data Systems software was used for data entry of the FFQ, PSFFQ and the MPQ. Data screens for the FFQ and PSFFQ were developed based on the order in which the food item was presented on the form. For the MPQ, data entry screens were developed for three meals (breakfast, lunch, dinner) and three snacks (morning, afternoon, evening) to enable nutrient analyses by meal and snack. In addition, the data were combined in a SAS dataset to generate total daily energy and nutrient intake resembling the FFQ and PSFFQ nutrient datasets.
Descriptive statistics were used to summarise the demographic and health characteristics of participants and median nutrient intake (excluding supplements) for nutrients of interest. For macronutrients, percentage of energy was also included in this analysis. Agreement between the average of the 24HR and each of the dietary assessment instruments was assessed in two ways. First, Wilcoxon signed rank tests were used to determine if the median intake for the dietary assessment instrument (FFQ, MPQ, PSFFQ) differed from the average of the 24HR. Next, we considered the Pearson correlation between the dietary assessment instrument and the average of the 24HR. Due to the skewness of the nutrient data, the nutrient values were transformed to approximate normality using a Box–Cox transformation before the correlations were computedReference Box and Cox25. Both unadjusted correlations and correlations adjusted for energy intake using the residual method were computedReference Willett2. The same analytical procedure was used to perform pairwise correlations between the FFQ and either the MPQ or the PSFFQ. In order to determine whether correlation coefficients differed by gender, education, health status or interviewer assessment, we used Fisher's z-testReference Rosner26. All tests performed were two-sided with an α level of 0.05. SAS software version 8.2 (SAS Institute) was used for all statistical analyses.
Results
The characteristics of the study participants are shown in Table 1. The sample was 52.5% female and comprised African Americans (33.6%), Native Americans (36.1%) and whites (30.3%). Mean age was 75.2 ± 6.6 years. The level of formal education was low, with over half of each gender reporting an eighth grade education or less. Approximately 60% reported annual household income less than $US 10 000. Most had spent the majority of their lives in the study area. The most commonly reported occupations were machine operators, agricultural workers or service industry workers. Participants had high rates of chronic disease and risk factors. Over 75% reported being diagnosed with high blood pressure and over 50% with high cholesterol. For both males and females, the modal category of self-rated health was ‘fair’.
Data are presented as n (%), except for age (mean ± standard deviation).
† Percentages may not add to 100% due to ‘Don't know’ responses.
The median nutrient intakes estimated from the average of the 24HR and from the three summary instruments are shown in Table 2 by gender. Results for percentage of energy from alcohol are not presented because reports of alcohol consumption were extremely rare. For both males and females, the MPQ tended to overestimate intakes of most nutrients relative to the 24HR. For females, the FFQ and PSFFQ accurately estimated energy and intake of fats on average, but protein was significantly underestimated on the FFQ and carbohydrates were significantly underestimated on the PSFFQ. The PSFFQ accurately estimated all vitamins and minerals for females except vitamin C, potassium and sodium, while the FFQ underestimated most of them. For males, the FFQ tended to underestimate energy and nutrient intakes. The FFQ, and to a lesser extent the PSFFQ, underestimated most dietary vitamins and minerals for males. On average, the PSFFQ was the most accurate at estimating nutrient intake for both males and females.
%E – percentage of energy; RE – retinol equivalents; TE – tocopherol equivalents.
* Significantly different from 24-hour recalls, P < 0.05 (Wilcoxon signed rank test).
** Significantly different from 24-hour recalls, P < 0.01 (Wilcoxon signed rank test).
Table 3 shows the Pearson correlation coefficients for the associations of the average of the 24HR and each of the FFQ-type instruments. Most are statistically significant and positive; those adjusted for energy intake are lower than those unadjusted. In general, correlations are stronger and more often reach statistical significance for males than females. For females, the MPQ and PSFFQ are more highly correlated with 24HR than is the FFQ. For males, the FFQ and MPQ appear to have a stronger pattern of correlations with the 24HR than the PSFFQ. No comparisons of correlation coefficients for energy and nutrients between the average of the 24HR and each of the FFQ-type instruments by gender, education and health status were statistically significant (data not shown), except for one. The correlation coefficient for protein between the 24HR and PSFFQ was statistically significantly higher for men compared with women.
%E – percentage of energy; RE – retinol equivalents; TE – tocopherol equivalents.
Adjustment for energy intake done using the residual method (WillettReference Willett2).
* Correlation significant, P < 0.05.
Pearson correlation coefficients of the FFQ with the PSFFQ and MPQ are presented in Table 4. All of the correlation coefficients are statistically significant and positive. For both females and males, correlations for FFQ and PSFFQ are generally stronger than the corresponding correlations between FFQ and MPQ. Most of the correlation coefficients unadjusted for energy intake are stronger than adjusted, with exceptions for protein, vitamin C and fibre.
%E – percentage of energy; RE – retinol equivalents; TE – tocopherol equivalents.
Adjustment for energy intake done using the residual method (WillettReference Willett2).
All correlations significant at P < 0.05.
Discussion
The sample for this study was selected to represent an elderly low-SES group typical of that targeted for many community-based prevention and health promotion interventions. Despite their low educational attainment and occupational histories, the results of dietary assessment using standard data collection methods (repeat 24HR and FFQ) were similar to those reported elsewhereReference Willett2, Reference Subar, Thompson, Kipnis, Midthune, Hurwitz and McNutt15, Reference Wright, Wang, Kennedy-Stephenson and Ervin27, Reference Kumanyika, Mauger, Mitchell, Phillips, Smiciklas-Wright and Palmer28. The nutrient medians reported in the 24HR in this study are comparable to those obtained for persons 60 years and older in the 24HR portion of the National Health and Nutrition Examination Survey, 1999–2000Reference Wright, Wang, Kennedy-Stephenson and Ervin27. The deviation between the 24HR and FFQ seen here is assumed to be fairly typicalReference Willett2, Reference Subar, Thompson, Kipnis, Midthune, Hurwitz and McNutt15, although exceptions occurReference Kumanyika, Mauger, Mitchell, Phillips, Smiciklas-Wright and Palmer28.
In general, the correlation coefficients between 24HR and FFQ were slightly below the range observed in other studies of older or minority populationsReference Mayer-Davis, Vitolins, Carmichael, Hemphill, Tsaroucha and Rushing8, Reference Kumanyika, Mauger, Mitchell, Phillips, Smiciklas-Wright and Palmer28, Reference Morris, Tangney, Bienias, Evans and Wilson29. The correlations in the present study were stronger than those in another study on participants with low education levelsReference Mayer-Davis, Vitolins, Carmichael, Hemphill, Tsaroucha and Rushing8. This might be due to the methods of administration of the FFQ, as that study was conducted by telephone rather than with face-to-face interviews. The poorer performance of women than men on the standard FFQ has not been reported in other studies; however, most other studies that included both genders are non-minority, younger or higher-SESReference Subar, Thompson, Kipnis, Midthune, Hurwitz and McNutt15. In particular, women's reports of energy on the FFQ were uncorrelated with the 24HR, as reflected in non-significant correlations for total fat, unsaturated fats and fat-soluble vitamins. This suggests that the FFQ may not have captured particular foods or food preparation techniques in the same way as the 24HR. It should not necessarily be interpreted that men are able to more accurately report their diet. It is possible that men made consistent errors in reporting on the 24HR and FFQ (e.g. being unaware of added fats used in cooking or not knowing food item varieties such as low-fat dairy products served in the home).
Because adjustment for energy intake generally improves correlation coefficientsReference Willett2, it is surprising that this was not observed in the current study. In general, correlation coefficients were weaker after energy adjustment and for density measures (percentage of energy from fat and carbohydrate). It appears that participants did not underreport all macronutrients to the same degree, but were differentially underreporting some macronutrients. For example, for women (Table 2) on the PSFFQ, the median intakes of protein and fat are similar to the median 24HR amounts, but the median report for carbohydrates is higher and represents the largest proportion of total energy. Once adjustment for energy is made, due to the higher carbohydrate intake, percentage of energy from protein and fat are both lower than the percentage of energy reported on the 24HR. A similar pattern appears to occur for the correlation coefficients.
Picture-sort techniques have been tested by two other groupsReference Kumanyika, Tell, Fried, Martel and Chinchilli12, Reference Wengreen, Munger, Wong, West and Cutler14. The correlations with 24HR were somewhat higher in both of those studies. Neither of those studies administered a standard FFQ such as the Block. Thus, our study allows comparison of the picture-sort technique with both the 24HR and FFQ. The picture-sort technique used here produced stronger and more consistent correlations with the 24HR for women than did the FFQ. This suggests that the picture format may help older women, in particular, to more accurately report their usual dietary intake. This format was designed to make the task less abstract, more interactive and to cue memory. These results indicate that women in this sample responded more than men to such cues.
The MPQ technique developed for use in this study is patterned after the diet history approachReference Burke24. The diet history has been used as a semi-quantitative technique for obtaining descriptions of usual diet in clinical settings and, less frequently, for researchReference Bingham30. Most attempts to validate diet histories against 24HR or food records are somewhat dated (e.g. Reference Black, Prentice, Goldberg, Jebb, Bingham and Livingstone31–Reference Jain, Howe, Johnson and Miller34; exception is 35), as more recent research has focused on the food-frequency approach. Our findings are consistent with previous studies that showed an overestimation of intakes with the diet history method relative to recalls or recordsReference Hankin, Wilkens, Kolonel and Yoshizawa32, Reference Jain, Howe, Johnson and Miller34, Reference Black, Welch and Bingham35. One study confirmed this result using urinary nitrogen excretion and doubly labelled waterReference Black, Welch and Bingham35. The degree of overestimation in the present study was higher than reported in previous studies. This may be due to the tendency of this elderly population to eat leftovers from the noon meal at the evening mealReference Quandt, Vitolins, DeWalt and Roos16. When responding to the MPQ, they may have reported the amount prepared rather than the amount consumed for each meal.
This study has several limitations. Despite its use as a reference in this study, the 24HR is known to have measurement error associated with itReference Tooze, Subar, Thompson, Troiano, Schatzkin and Kipnis36. Unfortunately, the nature of this error is not known in the present study, as no validation using doubly labelled water or other technique was used. Since all methods of data collection used are subject to errors in memory, the errors in estimates in both the 24HR and the other measures are likely to be correlated. The study focused on older adults from one particular region (the rural southern USA) that is characterised by health and social disparities. We would expect the performance on dietary assessment to be better if we had used a more advantaged population of older adults. Also, sample size precluded examining ethnic differences in validity of the food-frequency instruments or the interaction of gender and ethnic effects.
Despite these limitations, this study provides evidence that older, community-dwelling adults of low SES, many of whom had serious health conditions, can complete an intensive dietary validation study. The use of five or six 24HR within a six-month period is a strength of our study, as it minimises within-person variability in nutrient intake reporting. Although the participants in the present study performed more poorly than more advantaged participants in other studies, our findings indicate that any of the three instruments will provide results that should permit study participants to be ranked in terms of dietary intake. More research is needed to understand the gender differences in performance on the instruments. In the present study, despite the knowledge that error can be attributed to both the data collection instruments and to the respondents, it is impossible to distinguish one type of error from the other.
Most alternative dietary data collection instruments have been developed to accommodate problems posed by low literacy and other characteristics of low-SES and older participants. The present study shows that for different types of instruments administered by an interviewer, the differences among the scores obtained are small. However, participant burden is often a concern when dietary assessment is part of a large battery of data collection instruments. If alternative dietary data collection instruments (e.g. a picture sort) are more enjoyable for study participants, they may assist with participant retention and therefore have value beyond data quality. Such attributes of alternative data collection techniques deserve further study.
The choice of dietary data collection instruments should be driven by the focus of the research, as well as consideration of the validity of the methods. The instruments tested here either underestimated (FFQ, PSFFQ) or overestimated (MPQ) intakes, consistent with other existing diet history or FFQ studies. For measuring absolute intakes, 24HR appear to be necessary. However, for epidemiological studies of diet and disease, the correlation of the questionnaire and the 24HR, not the absolute amount measured, is often more important. For women in particular, the correlation coefficients between the MPQ and 24HR were comparable to and often better than those of the PSFFQ and FFQ with 24HR. For men, the correlations between MPQ and 24HR were weaker than those between FFQ and 24HR, but comparable to or better than those between the PSFFQ and 24HR. These findings suggest that man and women may respond differently to dietary assessment instruments in different formats. Further research to better understand the source of these response differences is necessary before adopting either of these new instruments.
Acknowledgements
Sources of funding: This work was supported by a grant from the National Institute on Aging (AG 13469).
Conflict of interest declaration: None of the authors had a personal or financial interest in any company or organisation that could benefit directly from this research.
Authorship responsibilities: S.A.Q., T.A.A., M.Z.V., R.A.B. and R.F.D. contributed to the design of the study and interpretation of data. S.A.Q., M.Z.V. and S.L.S. were responsible for the acquisition of data. J.A.T. and C.C.D. designed and conducted the statistical analyses. S.A.Q. drafted the manuscript. All authors were responsible for the critical revision of the manuscript and gave final approval for submission.