To be effective in addressing nutrition and health challenges, food and nutrition policies must be based on an empirical understanding of food consumption patterns: dietary adequacy, the nutritional value of foods, food safety and food sources. In low- and middle-income countries (LMIC), the framing of food security and nutrition problems increasingly encompasses not only undernutrition but also diet quality(Reference von Grebmer, Saltzman and Birol1) and overweight and obesity, with their attendant chronic diseases(2,3) . Issues of the safety of the food supply, including microbiological and mycotoxin contamination(Reference Andrews-Trevino, Webb and Shively4,Reference Andrews-Trevino, Webb and Baral5) and potentially risky additives(Reference Hoffmann, Moser and Saak6), are emerging concerns of food policy in LMIC. Detailed information on individual food consumption, collected in a survey that is appropriately representative of the relevant population, is critical for making informed policies and developing effective programmatic responses to assure nutritional health(Reference Coates, Colaiezzi and Bell7,8) ; lack of individual-level dietary data is a barrier to developing effective solutions(9).
A wide variety of qualitative and semi-quantitative methods are currently used to assess the adequacy of individual dietary intakes in LMIC including dietary diversity scores(Reference Kennedy, Ballard and Dop10–Reference Arimond, Wiesmann and Becquey15) and FFQ(Reference Ang, Yap and Najwa16,Reference Forster, Fallaize and Gallagher17) , but their use as a measure of diet quality has been questioned(Reference Verger, Dop and Martin-Prével18), and quantitative estimates yielded by these methods are imprecise(Reference Eng and Moy19–Reference Zang, Luo and Chang22). Household food consumption data derived from household consumption and expenditure surveys have been used to estimate individual consumption but do not account for the intrahousehold allocation of food(Reference Coates, Patenaude and Rogers23–Reference Harris-Fry, Paudel and Shrestha25) and may miss consumption of food away from home(Reference Mwangi, Fiedler and Sununtnasuk26). Given these limitations, the detailed information derived from individual quantitative dietary recalls is more appropriate for many types of analyses and policy decisions.
Food policies at the national level manifest their effects at the level of the individual; to make informed food policies, governments need quantitative individual data. Such data allow for measuring important policy-relevant variables, including how foods are prepared (reflecting the preservation of their nutritional quality) and their source (important for understanding potential vehicles for fortification or other interventions and potential contamination risk). In most LMIC, individual quantitative dietary data have been collected infrequently if at all, due to issues related to cost, time, and complexity(Reference Coates, Colaiezzi and Bell7,Reference Pisa, Landais and Margetts27) .
Quantitative methods for assessing diets include single or multiple 24-hour dietary recalls (24HR), food diaries and weighed food record (WFR)(Reference Thompson, Subar, Coulston, Boushey and Ferruzzi28,Reference Wrieden, Peace and Armstrong29) . The interviewer-administered quantitative 24HR is often viewed as most appropriate for use in dietary surveys, providing more detailed information than the qualitative methods described above; recalls of time periods beyond the previous 24 h are not commonly used as they are subject to serious recall error(Reference Gibson, Charrondiere and Bell30,Reference Gibson and Ferguson31) . Barriers to use of the 24HR include the time required for the interview, need for more preparation in advance of data collection, more complex data analysis, and the need for a comprehensive set of dietary reference data to convert consumption data into detailed nutrient information.
A 24HR may be administered as a paper-based questionnaire (pen-and-paper interview (PAPI)) or on a tablet or other device (computer-assisted personal interview (CAPI)). Paper-based questionnaires require manual data entry, an error-prone process entailing a long lag between data collection and the availability of results. In contrast, CAPI-based methods have the advantage of electronic data capture and a direct link to reference data such as food composition information, eliminating the need for manual data entry and greatly reducing the time for data processing and analysis.
The INDDEX Project created a mobile data collection application (INDDEX24) for administering a 24HR using the multiple-pass method(Reference Gibson, Charrondiere and Bell30–Reference Steinfeldt, Anand and Murayi33), linked to the Global Food Matters Database (FMDB) where the dietary reference data are stored. The foods entered into the recall were drawn from a food list developed in advance and coded so as to link directly to the dietary reference data for analysis; the mobile application provides a standardised electronic data collection platform, customisable to meet the specific needs of a given survey, an architecture that links the data directly to the dietary reference data, and an automated analysis tool that streamlines the production of key results.
The present study reports the results of a comparison of a 24HR administered using a paper-based questionnaire (PAPI) with the CAPI approach using INDDEX24 in terms of accuracy (using the WFR as the benchmark) and cost-effectiveness. Comparisons of time required from initial preparation through production of key results and of cost are reported elsewhere(Reference Adams, Bell and Somé34). Few studies have compared the 24HR CAPI v. PAPI approach in a LMIC context. Several studies have assessed relative cost of 24HR using CAPI v. PAPI and identified key cost contributors(Reference Harris-Fry, Beard and Harrisson32,Reference Dickinson, McCauley and Madaj35–Reference Mukasa, Mushi and Maire39) . None of these studies related cost to the accuracy of the data collected.
Subjects and methods
Study setting, design and participants
The study took place in 2019 in Oubritenga province in the Plateau Central region of Burkina Faso. The validation study component was conducted in Moackin village in the rural commune of Absouya, and the interview time assessment study component in Ziniare in the urban commune of Ziniare and in Goue in the rural commune of Loumbila. While the 24HR can be administered as part of a representative survey, the goal of the study was not to represent the population but rather to compare the two administration methods; thus, the study site was purposively selected to be roughly comparable in key socio-demographic characteristics (e.g. economic status and education level) with the average levels in Burkina Faso.
The validation study was a cross-sectional comparative study calibrating two dietary recall methods against a benchmark WFR. The study had two arms. For the first arm (WFR-PAPI), respondents participated in the WFR observation during the first day of data collection and were interviewed about the foods consumed on the observation day using the 24HR PAPI the second day. For the second arm (WFR-INDDEX24), the same procedures were carried out except that participants were interviewed using the INDDEX24 the second day of data collection. To implement the interview time assessment study, a similar design was used with two sub-arms (INDDEX24 and PAPI) but with no WFR for this component.
Participants were women aged 18–49 years living in the study areas. One female member within the eligible age range was enrolled from each household. Exclusion criteria for participation included any mental or physical health issues that could affect the participant’s ability to complete the 1-d observation and the interview. Participants needed to agree to participate in both phases of data collection (observation for WFR and 24HR interview).
Recruitment and allocation of study participants
A household listing was conducted in Moackin village to identify households with eligible participants and was used to build the sampling frame from which households were randomly selected for the validation study. For the time assessment study, thirty households were systematically selected in each of the two other sites (Ziniare and Goue). In each household, one female member within the eligible age range who was available and consented was enrolled.
For the validation study, participants were contacted by a ‘consenting agent’ who was assisted in locating the selected households by a local guide. The consenting agent first obtained verbal agreement from the household head and then the agent talked with one eligible, available and interested woman of the household to obtain a written consent. Study procedures were clearly explained during the consenting process. Participants were informed that they would be participating in 2 d of data collection, the first for the WFR and the second for the 24HR. After that, the agent scheduled the WFR visit for a day within the next 2 d and arranged the time for the interviewer to arrive before the respondent started preparing the first meal of the day. The consenting agent provided the participant with a bowl and plate, instructed her to use them for taking her meals and explained that she should not change her diet. Equal numbers of participants were recruited in each study arm.
Data collection
Data collection for WFR and 24HR (INDDEX24 and PAPI) took place in September–October 2019 and was carried out on all days of the week. A total of eight 24HR interviewers and sixteen WFR interviewers were trained for 14 and 10 d, respectively, to collect the data. 24HR interviewers were trained on both INDDEX24 and PAPI and were assigned to conduct one or the other modality alternating by day so that all interviewers conducted interviews using both modalities on all days of the week. Data collection was overseen by two supervisor for the 24HR and three for the WFR. Two individuals trained on 24HR but not retained were hired as consenting agents in charge of participants’ recruitment.
Weighed food record
Interviewers arrived at the participant’s house in time to observe the preparation and/or the consumption of the first meal of the day and left after consumption of the last main meal of the evening. Interviewers were trained to remain as discreet and unobtrusive as possible when in the participant’s house between meal preparation and/or consumption periods. They accompanied the participant if she left the house and were trained to record any food consumed away from home. Throughout the WFR observation day, interviewers recorded all details about foods and mixed dishes prepared in the home (quantity of each ingredient, cooking method, total quantity of dish prepared, weight of the pot/pan), quantity of the food consumed by weighing the food and beverages before and after the participant had consumed them using a digital scale (MyWeigh KD 7000 with 1-g accuracy and 7-kg capacity), and time and place of preparation and consumption for all food items. For purchased foods, foods prepared outside of the participant’s household or leftovers from the previous day, the interviewers were instructed to record as many details as possible regarding the food or mixed dish, including the main ingredients, cooking method, and any other distinguishable characteristics, along with quantities. At the end of the WFR observation day, the interviewer arranged with the respondent the time for the 24HR visit the next day and later provided this information to the 24HR team for planning the visit of the 24HR interviewers.
24-Hour dietary recall
For the 24HR, a different group of interviewers visited the participant in her house the day after the WFR observation and carried out the 24HR. Before administering the 24HR, interviewers completed the registration of the participant, which consisted of recording a same set of information as for the WFR (e.g. date, household ID, location, birthdate, pregnant/lactating) to link 24HR and WFR data.
For the INDDEX4 interview, interviewers recorded first the full list of all foods, beverages and mixed dishes recalled as consumed by the participant during the previous 24 h and then probed for more details using the food and recipe list developed by the study and integrated in the INDDEX24 app (online Supplemental Information); if the food or recipe was not in the list, the interviewer directly input the required details. Following selection of the food or recipe from the food and recipe list in INDDEX24 (or inputting of a missing food item or non-standard recipe (NSR) name), the amount (portion size) consumed by the participant was estimated using the portion size estimation methods (PSEM) developed by the study team and integrated in the INDDEX24 app (online Supplemental Information). PSEM options used for INDDEX24 included direct weight, life-sized photos, proxy weight using play dough, proxy weight using dried sorghum, proxy weight using water, and standard units. In the case of mixed dishes, interviewers were instructed to select the standard recipe that best fit the description of the mixed dish, based on the participant’s report of ingredients used, with flexibility to accept minor divergences from the standard recipe. The acceptable differences included the addition/subtraction of minor ingredients (e.g. herbs, seasoning and spices), the use of a different type of cooking oil, the omission/addition of oil, and the use of condiments such as tomato, onion, bell pepper generally used in small amounts in the study context. If no close match was found for a reported mixed dish, it was considered as a NSR, and its details (ingredient description and quantities, and total quantity of recipe prepared) were collected by the interviewers with the participants or the dish preparer when possible. If the NSR was like an existing standard recipe, the interviewer could copy some/all the ingredients and use the copied version as a base from which to build the NSR. For recipes purchased or prepared by another person not available to provide details and considered as non-standard, participants were asked to report as many details as possible (ingredients, cooking method, place of purchase/acquisition and whatever other information could be provided).
For 24HR PAPI interview, interviewers used a paper data collection form and printed food and recipe booklets that contained all the same items as the INDDEX24 food and recipe list. During the 24HR PAPI interview, interviewers recorded the detailed description of the food or recipe reported by the participant and identified the best match in the food or recipe books, and then recorded the relevant code on the paper form. If the interviewer could not find a food or recipe in the books, they would describe it in detail and code it as ‘9999’ for food and ‘99 999’ for recipe. The same approach was taken for the standard and NSR selection in INDDEX24, and the same rules were applied to the PAPI data collection. To confirm whether mixed dishes described by the participants were among those listed as ‘standard recipes’, interviewers located the closest recipe in their printed recipe booklet and asked the respondent to list all the ingredients used in preparation of her recipe. Interviewers were trained to consider the recipe as non-standard if there were significant differences between the lists of ingredients (as described earlier); in that case, the recipe details were recorded including the amount prepared, the quantities and description of ingredients used. To estimate amounts consumed of foods and mixed dishes reported, PAPI interviewers had the same list of PSEM options (online Supplemental Information) as for INDDEX24 and were trained to identify which would be the best PSEM option for each food. A list of standard units and their codes was developed for use during PAPI interviews. When a standard unit was selected as the PSEM option, the interviewer asked the respondent about the standard unit size/volume (e.g. small or large bottle of Coca-Cola, French baguette – with weight or volume conversions automatically applied) of the food and the portion of the standard unit they consumed (e.g. 1 unit, half a unit, etc.). Then, interviewers selected the corresponding standard unit code within the list and recorded the code and the portion consumed in the PAPI form.
After the 24HR PAPI and INDDEX24, the interviewers administered the socio-demographic module and concluded the visit. For each day of 24HR data collection, interviewers were assigned two to three respondents with whom to conduct interviews using one type of 24HR modality (either INDDEX24 or PAPI). On average, fourteen WFR observations were conducted per day along with six to eight INDDEX24 and six to eight PAPI interviews following the WFR the next day.
Outcomes
Food group and nutrient intakes were estimated based on foods, beverages, and mixed dishes consumed and recorded during the WFR and both 24HR modalities (INDDEX24 and PAPI) using nutrient composition from the food composition table compiled by the study team (online Supplemental Information). The FAO/WHO GIFT food groups classification was used to group the foods, beverages and mixed dishes recorded in order to estimate intake by food group. For the time assessment study, the mean 24HR interview time and time spent on individual passes were estimated for both INDDEX24 and PAPI.
Sample size
The sample size was estimated as follows. First, we assumed that any correlation lower than 0·6 between the WFR-INDDEX24 and WFR-PAPI would be considered unsatisfactory. Second, we considered an increase in correlation of 0·15 or greater (correlation 0·75 or greater) a practically important improvement. Lastly, we set the power at 80 % and α level at 5 %. Given these conditions, we estimated the sample size to be 104 for each arm. It was inflated to 117 to account for approximately 12 % non-response. The final number of respondents was 116 for INDDEX24 and 115 for PAPI (Fig. 1).
Fig. 1. Flow diagram for INDDEX24 validation and cost studies in Burkina Faso. PAPI, pen-and-paper interview; WFR, weighed food record.
Ethical approvals and consent to participate
The study protocol and instruments were approved by the Ethics Committee for Health Research of the Ministry of Health in Burkina Faso and the Institutional Review Board of Tufts University in USA. All participants provided a written informed consent before the enrolment in the study.
Data management, cleaning and processing
For the INDDEX24, supervisors did some spot checking on tablets immediately after the interview, but most of the review was done by the investigators at the end of daily data collection after exporting the data in CSV. format. INDDEX24 data were reviewed for any outliers in gram weights reported, errors in food categorisation/coding, and completeness of NSR. Supervisors discussed and resolved any issues by early the following day.
Supervisors checked data quality of paper forms for WFR and 24HR PAPI daily, and any issues identified were discussed with the interviewers and resolved. WFR and 24HR PAPI data were entered into a programme built on CSPro version 7.3 using double entry. Hard copies of WFR and PAPI forms were used by the data entry supervisor to resolve any discrepancies. Study coordinators carried out subsequent cleaning to ensure appropriate use of food, recipe and ingredient codes, PSEM codes (PAPI only), value plausibility for amounts/quantities and number of portions, and household ID.
Any new foods or NSR (including food away from home and food prepared on the previous day) or PSEM conversion factors that did not already exist in the INDDEX24 dietary reference data were added to ensure that they had a match with the WFR, PAPI and/or INDDEX24. For the food data processing and conversion into nutrients, the INDDEX24 Global FMDB of dietary reference data (portion conversion factors, standard recipe information and food composition data) was used for WFR and PAPI using SAS v9.4. INDDEX24 analytical reporting feature was used to match all foods and convert them into nutrients automatically. The nutrient intake data sets for WFR, INDDEX24 and PAPI were then merged in order to complete the study analysis.
Statistical analyses
We conducted a comparison of the socio-demographic characteristics between the two study arms (WFR-INDDEX24 v. WFR-PAPI) to check the success of randomisation. Nutrient intake data were checked for outliers by reviewing histograms and scatter plots by nutrient. By convention, energy intakes <2092 kJ or >20920 kJ (< 500 or > 5000 kcal)/d were flagged as outliers(Reference Bell, Coates and Rogers40–Reference Tucker, Bianchi and Maras42). Any INDDEX4 or PAPI that differed from the respective WFR by > 4184kJ (1000 kcal) in either direction was also flagged for review. Both types of outliers were investigated by looking at the associated WFR or PAPI forms and the INDDEX24 analytical report exports for INDDEX24. Outliers that were due to data entry errors were corrected after reference to the paper forms. Remaining outliers were examined but maintained in the data set. The energy intakes from WFR for three INDDEX24 and five PAPI respondents were flagged as implausible values, and these respondents were excluded from the validation comparison analysis. Shapiro–Wilk tests were used to test for normality, and all nutrients were transformed (log or cube root) to correct for skewness. Nutrients included in the analysis were energy (kJ [kcal]), fat (g), protein (g), carbohydrates (g), total fibre (g), vitamin A (mcg RAE), vitamin C (mg), Ca (mg), Fe (mg) and Zn (mg). Bland–Altman plots were generated to assess visually the agreement between the WFR and the 24HR method for grams consumed, energy, and all nutrients of interest(Reference Bland and Altman43,Reference Bland and Altman44) . Bland–Altman plots show individual differences between the two methods and were used to examine the mean bias, 95 % limits of agreement, and the distribution of bias.
To assess the relative accuracy of the two recall methods, we compared each method to the benchmark (WFR-INDDEX24 and WFR-PAPI) for each nutrient of interest and for the energy contribution from the different FAO/WHO/GIFT food groups. Following the method of Arsenault et al.(Reference Batterham, Van Loo and Charlton45), equivalence tests were done using the two one-sided paired t test between recall method and WFR at the group level. A 10 % bound was used to assess equivalence, as suggested by other researchers(Reference Batterham, Van Loo and Charlton45,Reference Tugault-Lafleur, Black and Barr46) . A significant P < 0·05 for the equivalence test indicates that the WFR is equivalent to the 24HR at 10 % bound; non-significance means the two are statistically significantly different. After equivalence testing, we compared the two recall methods to each other and assessed the relative magnitude of their differences from the WFR using a difference-in-differences approach; comparing the WFR-PAPI difference to the WFR-INDDEX24 difference using a random intercept mixed effects regression model to compare slopes between 24HR methods; this comparison was also done at the group level. In addition, the percentage of INDDEX24 and PAPI respondents with energy and nutrient intakes that fell within specific percentage error categories compared with the WFR was used to assess accuracy of both 24HR modalities at the individual level. This approach allows for assessing the extent to which each modality (INDDEX24 and PAPI) is accurate at the individual level relative to the WFR estimates. All analyses were performed using SAS 9.4 or Stata 15 SE.
Cost-effectiveness analyses
In order to estimate and compare the cost of using the CAPI modality via INDDEX24 and the PAPI modality to conduct a 24HR and produce a clean, analysable 24 h data set, we conducted a costing study alongside the validation study in Burkina Faso(Reference Adams, Bell and Somé34). We took an activity- and ingredients-based approach to collecting and processing the cost data for each modality, and costs were collected from a societal perspective, meaning that all costs, including the opportunity cost of respondents’ time, were accounted for(47).
Prior to data collection, we defined the set of activities required to conduct the 24HR and prepare an analysable data set, including the preparation of dietary reference data, survey preparation (including the procurement of supplies and equipment), training, survey execution, data entry, and data cleaning and processing. Then, during the validation study, we recorded the types and quantities of inputs, or ingredients (e.g. personnel, facilities, transportation, equipment and supplies), that were used to carry out each activity. Throughout the validation study, this modality-specific time use and expenditure information was recorded by study personnel using a set of data collection instruments (paper-based quick logs for use in the field, time use logs housed in Google Forms, and Excel-based time and expenditure reporting logs). Time use data for study personnel were valued using wages and salaries, while respondent time was valued at the minimum wage in Burkina Faso(48). All expenditures were converted from West African CFA francs to USA dollars where necessary, adjusted to 2019 USA dollars, and summed in order to estimate the total cost of conducting the 24HR using CAPI and using PAPI. In addition to calculating the total cost of conducting the 24HR using each modality, we also disaggregated costs based on total time (person days), total time costs (person days valued at personnel wage rates/salaries) and total non-time monetary costs.
In addition to the ‘base model’ in which the cost of conducting the 24HR reflected the conditions under which the validation study was conducted, we modelled several cost scenarios to reflect anticipated development of the FMDB. The FMDB, which houses dietary reference data for INDDEX24, is just beginning to be populated with data. However, as future INDDEX24 users share their dietary reference data (e.g. food composition tables, standard recipes, portion conversion factors), other users of the platform may be able to borrow some of these data directly from the FMDB, thereby reducing the costs associated with the preparation of these data for users of INDDEX24. We modelled three alternative scenarios in which we assumed that 25, 50 or 75 % of dietary reference data were borrowed directly from the FMDB for the INDDEX24 modality.
To estimate the cost-effectiveness of using the CAPI modality relative to PAPI, we set these estimates of cost alongside measures of the accuracy of each modality compared with the WFR benchmark. We based our primary measures of cost-effectiveness on three accuracy measures: (1) average percentage accuracy in estimating the number of food items consumed (item count), (2) average percentage accuracy in estimating total gram amount of food intake, and (3) a composite measure of the average percentage accuracy in estimating each of ten nutrients of interest (energy, fat, protein, carbohydrate, fibre, vitamin A, vitamin C, Ca, Fe, and Zn), in the spirit of mean nutrient adequacy described in Hatløy, Torheim, and Oshaug(Reference Hatløy, Torheim and Oshaug13) and applied for validation of dietary metrics(Reference Steyn, Nel and Labadarios49–Reference Mirmiran, Azadbakht and Esmaillzadeh51).
The average percentage accuracy in estimating the number of food items and gram amount was calculated based on the average percentage error relative to the WFR, that is, we calculated the average percentage error as the absolute value of one minus the ratio of the average estimate based on each 24HR modality to the average estimate based on the WFR, and then average percentage accuracy was calculated as 100 minus the average percentage error. For example, if the average gram amount among the study sample in the PAPI arm was 3500 based in the 24HR and 4000 based on the WFR, then the average PAPI percentage error was calculated as
$\left| {\left( {1 - {3500}\over{{4000}}} \right)\, \times \,100} \right| = 12\cdot5\%,$
and the average PAPI percentage accuracy, or effectiveness, was
$100 - 12\cdot5 = 87\cdot5\%$
.
For the composite measure of nutrient intake, average percentage accuracy was calculated as the overall average of the average percentage accuracy for each nutrient as described for item count and gram amount. In addition to these primary measures of cost-effectiveness, we used the same methodology to calculate nutrient-specific effectiveness for each of the ten nutrients of interest. The average percentage accuracy of each outcome based on the validation study was assumed to be the same for each of the alternative scenarios.
We calculated cost-effectiveness, or cost per average percentage points of accuracy, based on total cost and each measure of disaggregated cost (time measured in person-days, time cost and non-time (monetary) cost). For the scenarios in which dietary reference data were assumed to be borrowed from the FMDB, effectiveness was based on the average percentage accuracy of each outcome from the validation study.
Results
The randomisation was well balanced, with no statistically significant differences in participants’ socio-economic and demographic characteristics between the two arms of the validation study (Table 1). A total of 231 women (116 INDDEX24 and 115 PAPI) were enrolled in the validation study in Burkina Faso. Due to implausible values and a matching issue between WFR and INDDEX24 because of errors in respondent ID, five INDDEX24 and five PAPI respondents were excluded from the analysis (Fig. 1). The mean age of participants in the full sample was 29·7 (sd 8·5) years. Most participants (83·5 %) did not have any formal education, and the rest of them had primary or secondary level education. The average household size was 11·7 (sd 5·8). About one-third of participants (35·2 %) were breast-feeding, and 13·4 % were pregnant at the time of the study. Almost all the participants’ households (95 %) owned agricultural land. Borehole was the primary source of drinking water for all the participants. Slightly less than two-thirds of participants (64·7 %) reported preparing food for others all or most of the time. The main household cooking fuel was wood, which was reported by 96·8 % of participants. Participants reported using mainly three types of lighting sources: lamp with rechargeable battery (37·5 %), flashlight with non-rechargeable battery (35·6 %) and solar energy (16·8 %).
Table 1. Socioeconomic and demographic characteristics of participants in INDDEX24 validation study in Burkina Faso
* PAPI = Pen and Paper Interview
† P-value from t-test (binary variables) or chi-2 test (categorical variables) comparison between and PAPI arms
‡ Other lightening sources included generator, woodfire and flashlight from mobile phone
§ Other wall materials included hides/tarpaulin and mat/straw
** Other toilet facilities included ECOSAN latrine, manually poured flush toilet, and simple SanPlat latrine
Differences in estimates by INDDEX24 v. pen-and-paper interview 24-h dietary recall modalities compared with the weighed food record benchmark: nutrient intakes
In general, INDDEX24 and PAPI were equivalent to WFR based on two one-sided paired t equivalence test for the item count, gram amount intake and all macronutrient intakes except fat intake for PAPI, as indicated by the significant equivalence test P-value (Table 2). Regarding the micronutrient and fibre intakes, neither of the 24HR modalities was equivalent to WFR except for vitamin A and Zn intakes. INDDEX24 and PAPI were equivalent to WFR for vitamin A intake, while only INDDEX24 was equivalent for Zn intake. For the difference-in-differences comparisons, there were very few instances where INDDEX24 and PAPI differed significantly from one another in approximating the WFR estimate except for energy and carbohydrates. For energy intake, INDDEX24 underestimated by 157·8 kcal while PAPI overestimated by 137·8 kcal when compared with WFR. Both INDDEX24 and PAPI underestimated carbohydrate intake, but INDDEX24 underestimated by 73·9 g, while PAPI underestimated by 10 g (Table 2). Thus, for these nutrients (energy, carbohydrates), INDDEX24 and PAPI did not yield comparable results, though both were equivalent to the WFR at the 10 % bound level.
Table 2. Comparison of mean differences of INDDEX24 and PAPI to WFR in estimated nutrient consumption for the participants in INDDEX24 validation study in Burkina Faso
(Mean values and standard deviations) *
* Note the WFR is the minuend, and therefore a negative difference between WFR and 24HR modalities indicates an overestimate, and a positive number indicates an underestimate. Equivalence test P-value is from the paired two one-sided t test (TOST) reported using the natural log geometric mean and 10 % bound. A significant P < 0·05 for the equivalence test indicates that the WFR is equivalent to the 24HR at 10 % bound; non-significance means the two are statistically significantly different. The difference-in-difference P-value is from calculating the difference between each 24HR modality and the WFR and then calculating the overall difference of the differences. A random intercept mixed regression model was used for the difference-in-difference comparison. Statistical significance P < 0·05 indicates that the INDDEX24 and PAPI differed from each other with respect to the accuracy of their estimates. FCT completeness for each nutrient based on the foods reported in the WFR: energy (100 %), fat (100 %), protein (100 %), carbohydrates (97·5 %), fibre (92·1 %), vitamin A RAE (98·3 %), vitamin C (84·2 %), Ca (100 %), Fe (99·2 %) and Zn (99·6 %). The distribution of missing FCT values for each nutrient occurred equally across the INDDEX24 and PAPI modalities.
† WFR, weighed food record
For the individual-level comparison, the proportion of INDDEX24 and PAPI respondents with energy intake within 10 % of their WFR estimates was, respectively, 13 and 21 %; 29 and 43 %, respectively, had energy intakes within the 20 % error range (Table 3; see also Bland–Altman plots in online Supplementary Fig. S1–S4). The proportions of respondents with micronutrient intakes within 10 % of their WFR were 20 % or less for all nutrients, with INDDEX24 having slightly lower percentages within ±10 and ±20 percentage points. Both modalities showed a higher percentage of respondents within those error ranges for macronutrients (fat, protein, carbohydrate) than for micronutrients. For the micronutrient intakes, estimates falling within the 10 % error range were under 10 %, and those falling within the 20 % error range were under 20 %, with no consistent difference between modalities. Accuracy of the estimates was even lower for Ca, where less than 5 % of PAPI and INDDEX24 respondents had intakes within 10 % of their WFR estimates; none of the INDDEX24 respondents had vitamin C intake within 10 % of the WFR estimate. Both PAPI and INDDEX24 overestimated intakes far more frequently than they underestimated them at the individual level based on an error range of ± 50 %. This was the case for both macronutrients and micronutrients (online Supplementary Tables S3 and S4).
Table 3. Percentage of respondent falling within ranges of percentage error in estimating energy and nutrient intakes with INDDEX24 and PAPI 24HR modalities compared with WFR *
* FCT completeness for each nutrient based on the foods reported in the WFR: energy (100 %), fat (100 %), protein (100 %), carbohydrates (97·5 %), fibre (92·1 %), vitamin A RAE (98·3 %), vitamin C (84·2 %), Ca (100 %), Fe (99·2 %) and Zn (99·6 %). The distribution of missing FCT values for each nutrient occurred equally across the INDDEX24 and PAPI modalities.
Differences in estimates by INDDEX24 v. pen-and-paper interview 24-h dietary recall modalities compared with the weighed food record benchmark: food group intakes
Table 4 shows the results from the INDDEX24 and PAPI compared with the WFR for the contribution of each food group to total energetic consumption. The largest contribution of energy comes from cereals: about 77 % according to the WFR and 65–66 % for either 24HR modality. The 24HR underestimate was statistically significant for both modalities and comparable between them. The second largest contributor of energy, fats and oils, showed no significant difference between the WFR and either modality, with estimated contributions between 7·1 and 9·2 % of total energy content. The average differences in the contribution of energy from the rest of the food groups were around or less than 5 %. The differences between the WFR and either modality were statistically significant only for vegetables, beverages, spices and condiments, and fish and shellfish. The contribution of beverages and spices/condiments was negligible for WFR and both 24HR modalities; the contribution of vegetables and fish/shellfish was negligible for the WFR but meaningful (around 4 % for fish/shellfish and 6–8 % for vegetables) for 24HR modalities, not different by modality. Overestimates by both 24HR modalities were far more frequent than underestimates (except for cereals), and the two modalities were consistent with each other in overestimating v. underestimating the contributions except for the roots and tubers group, which was consumed by only around 5 % of respondents in either study arm. Because small numbers render significance tests unreliable, P-values for the differences between the WFR and each 24HR are reported only for those food groups consumed by at least 10 % of respondents.
Table 4. Median percentage of energy intake from major FAO/WHO GIFT food groups
(Median values and 25th, 75th percentiles)
* WRS = Wilcoxon rank sum test is a nonparametric test that compares two paired groups. Bolded WRS P-value indicates a statistically significant difference between the WFR and the 24HR at or below, P = 0·05. Note the WFR is the minuend and therefore a negative difference between WFR and 24HR modalities indicates an overestimate and a positive number indicates an underestimate. Significance of the difference between WFR and 24HR is not shown for food groups consumed by less than 10 % of respondents. These are roots and tubers (5·4 % INDDEX24 and 4·5 % PAPI), meat (6.3% INDDEX24 and 3.6% PAPI), milk and milk products (7·2 % INDDEX24 and 1 % PAPI) and food for particular nutritional use (0·9 % INDDEX24 and 0·9 % PAPI).
Differences in cost and cost-effectiveness estimates by INDDEX24 v. pen-and-paper interview 24-h dietary recall modalities compared with the weighed food record benchmark: food group intakes
The total economic costs of using the INDDEX24 and the PAPI modalities to conduct the 24HR are presented in Table 5. Reflected in the base model, the cost of using INDDEX24 was slightly lower (by $1360) than the cost of using PAPI when no dietary reference data were borrowed from the FMDB. In all four scenarios, while the cost of non-personnel inputs (e.g. supplies, equipment, transport, facilities) was higher for INDDEX24 than PAPI, conducting the 24HR using PAPI was more human-capital intensive, and the higher personnel costs associated with PAPI meant the total cost of INDDEX24 was lower (Table 5).
* 24HR, 24-hour dietary recall; FMDB, Food Matters Database.
† PAPI=Pen and Paper Interview
‡ The difference was calculated as INDDEX24-PAPI.
§ FMDB=Food Matters Data Base
** These scenarios assumed that, for the INDDEX24 modality, the specified percentage of dietary reference data (e.g., food composition data, standard recipes, conversion factors) were borrowed from the FMDB.
The average percentage accuracy and cost-effectiveness of each modality based on the three primary measures of effectiveness (item count, gram amount, and a composite measure of nutrient intake) are presented in Table 6. PAPI was slightly more accurate than INDDEX24 (∼1–2 percentage points) for gram amount and item count, while the average percentage accuracy of INDDEX24 based on the composite nutrient intake indicator was ∼9 percentage points more accurate than PAPI. Given the small differences in cost of the two modalities for the base model, PAPI was more cost-effective than INDDEX24 in terms of cost per unit of accuracy in estimating item count, while INDDEX24 was more cost-effective for gram amount and the composite indicator of nutrient intake.
Table 6. Cost-effectiveness of conducting a 24HR using INDDEX24 and PAPI in Burkina Faso
* The difference was calculated as INDDEX24 minus PAPI.
† Composite nutrient intake is calculated as the overall average of the average percentage accuracy of PAPI (or of INDDEX24) in estimating intake for energy, fat, protein, carbohydrate, fibre, vitamin A, vitamin C, Ca, Fe and Zn relative to intake based on the weighed food record.
‡ These scenarios assume the specified percentage of dietary data inputs (e.g. food composition data, standard recipes, conversion factors) can be borrowed from existing dietary reference data in the INDDEX24 database.
For the modelled scenarios in which 25–75 % of dietary reference data were assumed borrowed from the FMDB for the INDDEX24 modality, the cost of INDDEX24 further decreased relative to PAPI, based on the assumption that the dietary reference data are integrated into the INDDEX24 platform, while the preparation and use of dietary reference data for PAPI would still require implementers to seek out the information and incorporate it into their analysis. Thus, as the percentage of dietary reference data borrowed from the FMBD for INDDEX24 increases, so does the divergence between it and PAPI, since the cost of preparing dietary reference data for PAPI is unchanged. Therefore, INDDEX24 was incrementally more cost-effective than PAPI based on each of the three primary measures of accuracy. Each of these estimates of cost-effectiveness based on time (measured in person days), time cost and non-time monetary cost is available in Table 6.
Discussion
We hypothesised that the INDDEX24 and PAPI approaches would be comparable in accuracy, and that the time savings resulting from the INDDEX24 approach would mean that cost-effectiveness would favour the CAPI approach, and this hypothesis was largely supported the study.
Beyond the current study and a parallel study in Vietnam, we are unaware of studies that compared the accuracy of a CAPI and PAPI approach using a reference benchmark of accuracy such as the WFR. Some studies have assessed accuracy of one or the other modality using WFR or biomarkers(Reference Timon, van den Barg and Blain52,Reference Xue, Yang and Liu53) . Numerous validation studies have reported results similar to the present study – namely, that intakes of many nutrients measured by 24HR were not statistically significantly different from WFR, but that some nutrients did show differences and that some modifications may be needed to reduce error(Reference Arsenault, Moursi and Olney54–Reference Savard, Lemieux and Lafrenière57). But these did not compare one 24HR modality with another, and few were conducted in LMIC. Comparisons of CAPI with PAPI have been done to assess acceptability and feasibility of the methods(Reference Zang, Song and Wang58), and differences in measured outcomes(Reference Htet, Fahmida and Do59), but without a benchmark to compare accuracy of the two methods.
A possible limitation of the study is that the comparability of the two modalities may be due in part to the fact that all interviewers conducted both INDDEX24 and PAPI interviews; learning undoubtedly occurred that affected (most likely improved) the quality of the interviews in both study arms. For example, the INDDEX24 platform provided tags and probes as part of the app; interviewers may have learned to ask these questions in the PAPI even when they did not appear in the paper form. Developing the INDDEX24 platform required the systematic collection of dietary reference data prior to survey implementation. These data facilitated the development of an analysable database for both the INDDEX24 and PAPI approaches; such data might not be available in advance in the case of a PAPI as commonly implemented. In particular, the PAPI interviews made use of the repository of standard recipes and followed the same approach to recording NSR using the standard recipes as a basis when possible; this option is not typically available in PAPI dietary surveys.
Another possible limitation is that using the WFR as a benchmark meant that all respondents to the 24HR in the validation study had already been exposed to dietary measurement during the WFR the previous day. This exposure might have increased their awareness of their food consumption, affecting their recall the following day. Previous studies have found that the accuracy of 24HR may be improved by asking respondents to record or otherwise pay attention to their consumption in the days prior to an anticipated dietary survey(Reference Pries, Filteau and Ferguson60,Reference Rollo, Williams and Burrows61) , so this ‘priming’ effect may even be an advantage. In the present study, our purpose was not to obtain information about usual or representative diets of the population, but specifically compare the accuracy with which diets were recalled as captured by INDDEX24 and PAPI. While administration of the WFR might have improved the accuracy of the recall, the comparison would not have been possible without administering the WFR as a point of reference, since comparisons were made on detailed consumption information by nutrient and food group.
A final limitation is that the study was implemented in a single geographic area of Burkina Faso, and only to rural women 18–49 years old; the conclusions drawn from the comparison thus may not be applicable to different age or sex groups or to contexts with more varied and complex (e.g. urban) dietary sources. A similar INDDEX24 validation study was conducted in Vietnam, a different geographic context and a country with a more diverse diet, using the same procedures and protocols(Reference Bell, Coates and Bakun62). This second study reached similar conclusions in that both found that INDDEX24 and PAPI performed comparably to each other in terms of accuracy with respect to the WFR. In that study, though, 24HR modalities were equivalent to WFR for more micronutrients, and a larger proportion of respondents fell within ±20 % of the WFR intakes than in the present study.
In addition to accuracy and cost-effectiveness, there are other benefits to using the INDDEX24 platform. It is standardised, so that results from any study using this platform may be comparable with results from other countries or (in the future) other time periods. The INDDEX24 platform is designed to be linked to dietary reference data available in the FMDB. As INDDEX24 becomes more widely used, dietary reference data (e.g. food composition data, portion conversion, standard recipes) will increasingly populate the FMDB, further reducing the initial costs of preparing for a dietary survey and enabling the smooth linkage from survey responses to food composition and other data needed to process the survey results. The intention is to build up the contents of the FMDB and make it widely accessible, allowing faster preparation, survey implementation, analysis and production of usable information. As more dietary reference data can be borrowed from the FMDB, cost-effectiveness becomes even more favourable to INDDEX24. This is important not only to reduce the initial cost of preparation but also to reduce significantly the lag time between completing data collection and producing usable results while they are still relevant. By streamlining the process of conducting individual dietary intake surveys, the intention is that the use of the INDDEX24 platform will contribute to greater availability and use of detailed dietary data to inform effective food and nutrition policies.
Conclusion
The goal of the study was to compare the INDDEX24 platform, a CAPI approach to conducting individual dietary recall surveys, with the commonly implemented PAPI approach. Results support the conclusion that INDDEX24 is not less accurate than PAPI and that it has distinct advantages in terms of both cost and the timeliness with which survey results can be obtained once data collection is complete. In the future, as the FMDB is populated, the time and cost required for survey preparation using INDDEX24 may be further reduced. Inaccuracies in the estimation of micronutrient consumption and individual-level consumption were comparable between the two modalities; further exploration into ways of increasing the accuracy of recall methods is needed. While innovative technologies for measuring food consumption are under exploration, the 24HR is likely the most feasible method for collecting individual dietary information on populations in LMIC, and the INDDEX24 platform demonstrates advantages that could facilitate such collection and thus encourage the use of dietary surveys in the formation of programmes and policies related to food and nutrition.
Acknowledgements
We are very grateful to the INSD and IRSS staff and interviewers who worked tirelessly to support this study and to respondents who generously shared their time, household space and dietary information for full-day observations and recalls. We also appreciate and Julia Matteson’s work in the preparation of the manuscript.
The INDDEX Project is implemented by the Tufts University, Gerald J. and Dorothy R. Friedman School of Nutrition Science and Policy with funding from the Bill & Melinda Gates Foundation.
B. R., J. C., J. W. S. and W. B. designed the research; B. R., J. C., J. W. S., W. B., D. A. C., and S. W. provided essential materials (developed the INDDEX24 platform); J. W. S. and S. W. conducted the research; P. B., K. P. A. and J. W. S. analysed data; B. R., J. W. S. and K. A. wrote the paper; B. R. had primary responsibility for final content. All authors read and approved the final manuscript.
The authors declare no conflict of interest.
Supplementary material
For supplementary material referred to in this article, please visit https://doi.org/10.1017/S0007114521004700
Open access
