Many low- and middle-income countries (LMIC) face a double burden of malnutrition, where undernutrition coexists alongside overweight and obesity along with increasing rates of non-communicable disease stemming from dietary risk factors(Reference Popkin, Adair and Ng1,Reference Di Cesare2) . In order to address these challenges, countries must monitor how the food system interacts with dietary patterns and health outcomes and how these forces trend over time(Reference Micha, Mannar and Afshin3–5).
Despite the need for dietary data to inform a wide array of food policies and programmes, many LMIC collect nationally representative, quantitative dietary data only rarely, if at all. According to the FAO/WHO GIFT dietary survey inventory, there have been just nine national surveys in Africa since 1981 and just four since 2005(6). One major constraint to more frequent dietary assessment in LMIC is the lack of investment in public research infrastructure, that is, the databases, technological tools and standards needed to facilitate high-quality dietary research for societal benefit.
The USA, Europe and other high-income countries have developed 24-h dietary recall (24HR) software that can be administered via the web or respondents’ mobile phones and links directly to robust food composition databases(Reference Bradley, Simpson and Poliakov7–Reference Foster, Lee and Imamura13). Evidence from high-income countries has shown that digital survey technology can mitigate a number of problems related to accuracy, respondent and researcher burden, monetary and time costs of data collection, efficiency and ease of data coding and analysis, and respondent participation rates(Reference Amoutzopoulos, Steer and Roberts12,Reference Del Gobbo, Khatibzadeh and Imamura14) . Due largely to resource constraints for research infrastructure development, many LMIC continue to rely on a system of pen-and-paper interview (PAPI) questionnaires and manual data entry, rather than technology-enabled alternatives. Survey tools and dietary reference data are often assembled ‘from scratch’ for each episodic survey, requiring significant advance preparation before a survey as well as extreme post-survey lags in generating useable results(Reference Micha, Coates and Leclercq15). Technology-enabled infrastructure, including computer-assisted personal interview (CAPI) applications and linked electronic databases of dietary reference data, such as food composition and portion conversions, has the potential to improve accuracy and reduce the time and cost of dietary assessments in LMIC.
As one potential solution, the International Dietary Data Expansion (INDDEX) Project developed the technology-enabled INDDEX24 Dietary Assessment Platform, designed to streamline dietary assessment in LMIC(Reference Coates, Colaiezzi and Bell4). INDDEX24 is comprised of a (1) mobile CAPI application (INDDEX24 Mobile App) for collecting individual-level 24HR data on a smartphone or tablet via an interviewer-administered questionnaire, (2) linked web application, called the Global Food Matters Database (FMDB), where food composition, conversion factors, and other dietary reference data are stored, publicly shared, and integrated with the INDDEX24 Mobile App at multiple stages of the survey process, and (3) linked analytical reporting feature to facilitate data processing and analysis(Reference Coates, Colaiezzi and Bell4).
Though technological solutions offer great promise, rigorously designed validation studies are needed to determine whether a technology-enabled approach to 24HR surveys is at least as accurate and cost-effective as the PAPI ‘standard of practice’ in LMIC. A limited number of studies have examined the potential benefits of a CAPI approach to 24HR data collection in LMIC, without formal validation(Reference Caswell, Talegawkar and Dyer16,Reference Harris-Fry, Beard and Harrisson17) . We identified two studies from LMIC that compared a CAPI with a PAPI modality of 24HR administration(Reference Zang, Song and Wang18,Reference Htet, Fahmida and Do19) . To our knowledge, no study in a LMIC context has compared CAPI (or a more elaborate ‘technology-enabled’ platform) and PAPI modalities of administering a 24HR survey with a benchmark method such as a weighed food record (WFR). Likewise, no validation study in a LMIC setting has yet examined the relative cost and cost-effectiveness of different dietary recall modalities, including whether a comprehensive technology-enabled solution comprised of CAPI and a synchronised dietary reference database reduces the time and cost of data collection, processing and analysis.
This study, implemented by the Tufts University Friedman School and the National Institute of Nutrition (NIN) in Viet Nam, alongside a parallel effort by the INDDEX Project in Burkina Faso(Reference Rogers, Somé and Bakun20), aims to fill the following knowledge gaps in the context of Viet Nam: (1) What is the relative accuracy of INDDEX24 compared with a PAPI 24HR for measuring intake of specific food groups and nutrients of public health concern in LMIC? (‘accuracy component’) (2) What is the relative cost-effectiveness of the INDDEX24 v. PAPI modality (‘cost-effectivenes component’), defined as cost-per-average percentage point of accuracy? The results are intended to guide LMIC governments and researchers in choosing the dietary assessment approach that offers the greatest returns on accuracy and cost in resource-constrained contexts.
Subjects and Methods
Study design and sample selection
Participants in the accuracy study component (Fig. 1) were recruited to undergo a full-day observed WFR, followed by a randomly allocated INDDEX24 or PAPI 24HR the following day. A separate sample of respondents participated in a 24HR with either INDDEX24 or PAPI only (no WFR), in order to yield a ‘naïve’ estimate of time-per-interview to inform the cost and cost-effectiveness study component (Fig. 2).
Fig. 1. Study design for INDDEX24 validation study in Viet Nam: accuracy component.
PAPI, pen-and-paper interview; INDDEX, International Dietary Data Expansion; WFR, weighed food record
Fig. 2. Study design for INDDEX24 validation study in Viet Nam: time-per-interview measurement for cost-effectiveness component.
INDDEX, International Dietary Data Expansion; PAPI, pen-and-paper interview
The sample size for the accuracy component was calculated based on Pearson’s correlation tests for the strength of association between measurements of the test modalities (INDDEX24 and PAPI) and the reference method (WFR). A correlation below 0·6 between the WFR-INDDEX24 and WFR-PAPI was considered unsatisfactory, whereas an increase in correlation of 0·15 or greater (correlation 0·75 or greater) was considered a practically important improvement. α and ß were set at 5 % and 80 %, respectively. Given these parameters, the sample size was estimated at n 104 for each arm and increased to n 117 to account for an estimated 12 % non-response (n 234).
The study sites were situated in the Thanh Oai and Hai Ba Trung Districts of Hanoi Province in the Red River Delta region, located in northern Viet Nam. NIN selected the study sites purposively for variability in urbanicity, logistical convenience and socio-demographic characteristics typical of the range of income, livelihoods and housing found in northern Viet Nam. Hanoi Province has some of the lowest poverty in the country. About 33 % of the employed population works in agriculture, 23 % in non-farm self-employment and 44 % in wage work(21). Thanh Oai District (accuracy and time-per-interview components) is located about 25 km southwest of the capital city, Hanoi. Though administratively classified as ‘rural’, given its proximity to Hanoi and the extent of migration to and from the city for day labour, some areas have peri-urban characteristics. Hai Ba Trung District (time-per-interview component), in contrast, is in the bustling centre of Hanoi.
For the accuracy study component, 234 eligible women aged 18–49 years (maximum one per household) were randomly selected from a list-based sampling frame of all female residents of six randomly selected communes (i.e. subdistricts) in Thanh Oai District. The sampling frame was provided to NIN for this study by the rural Thanh Oai District community health services. The study coordinator selected respondents from the list using the Excel RAND function. Sampled women (n 234) were allocated to either (1) PAPI and WFR (n 117) or (2) INDDEX24 and WFR (n 117) through the random assignment, to respondents, of enumerators who alternated between conducting a 24HR with INDDEX24 or PAPI every other day.
To inform the cost-effectiveness study component, time-per-interview was assessed by administering the PAPI and INDDEX24 24HR to sixty women aged 18–49 years who were not exposed to WFR prior to completing a 24HR. The sample was stratified between one randomly selected subdistrict in Thanh Oai District (n 30) that was not part of the accuracy study and one randomly chosen subdistrict in Hai Ba Trung District (n 30). As with the accuracy component, the time-per-interview sample was drawn from a list-based sample frame maintained by district health representatives of these two districts. Enumerators were randomly assigned to each respondent and alternated every other day between 24HR modalities. Half of the thirty respondents received the PAPI and half received the INDDEX24 24HR in each district.
Community health workers and one staff member from NIN contacted sampled respondents in early August 2019, before data collection began, in order to explain the objectives of the study and obtain informed consent. The protocol stipulated that potential respondents be excluded based either on noticeably poor health that could interfere with their ability to be observed and interviewed (e.g. dementia and visible illness) or on an unwillingness to participate in both the WFR and 24HR components of the study. One respondent was replaced for not meeting the eligibility criteria.
Dietary assessment methods
Weighed food record
The observed WFR was selected as the benchmark method in this validation study as it is considered to provide an accurate estimate of individual consumption(Reference Gibson22,Reference Wrieden, Peace and Armstrong23) . This method typically requires an enumerator to be present in the household throughout the day for all meal preparation and consumption periods (and accompany the respondent if she leaves the house). Using this method, the enumerator records detailed, real-time information on all foods and beverages prepared and consumed, including recipes, ingredients (henceforth referred to as ‘foods’ or ‘food items’), and their weights. The WFR instrument was developed and pilot-tested before the start of data collection and consisted of three paper forms: a registration form, a recipe form to capture all household foods prepared and a consumption form to capture foods eaten.
INDDEX24 and pen-and-paper interview individual quantitative 24-h dietary recall dietary recall
The multiple-pass 24HR method was used for both test modalities (INDDEX24 and PAPI) based on the version developed and validated by Gibson and Ferguson for use in LMIC(Reference Gibson and Ferguson24,Reference Gibson, Charrondiere and Bell25) . Prior to the validation study, the INDDEX24 Dietary Assessment Platform was rigorously developed and tested over a period of 4 years, guided by the Tufts INDDEX team and a technical advisory committee of globally recognised dietary assessment experts, with significant input from partners in both Viet Nam and Burkina Faso. A feasibility study was carried out in both countries in 2018 in order to assess the usability and usefulness of the initial version of the INDDEX24 Dietary Assessment Platform. Respondents offered feedback through cognitive debriefing, and enumerators and technical experts (in both Viet Nam and Burkina Faso) were engaged through focus group discussions(Reference Wafa, Bell and Coates26). The findings were positive, and most respondents found INDDEX24 quicker, easier and more convenient than a PAPI 24HR(Reference Wafa, Bell and Coates26). Enumerators also offered detailed suggestions for improvements to the INDDEX24 Mobile App interface and structure that were incorporated into v2.1 of the INDDEX24 Dietary Assessment Platform that was used for the accuracy validation study.
The PAPI arm of the validation study was a 24HR paper data collection instrument that was adapted to be as similar to the INDDEX24 instrument as possible. Both INDDEX24 and PAPI followed the 24HR multiple-pass technique; however, there were some notable differences by modality. When conducting an INDDEX24 24HR, after entering foods, time and place of consumption in the First Pass (i.e. the ‘quick list’), enumerators would move to the Second Pass and select the food or standard recipe from the pre-defined dropdown list after probing for more details, per the instructions in the app. Any foods that were consumed more than once could be selected and copied from the respondent’s food list, rather than going back to the master list. Following the selection of the food or recipe from the list (or inputting a missing food item or non-standard recipe (NSR) name), the portion size was then estimated in the Third Pass. If a recipe did not exist in the INDDEX24 database, then the enumerator would record all the details as a NSR by either copying and modifying a standard recipe or by entering a new recipe including total cooked weight and the quantity of each ingredient used in the NSR Pass. The Fourth Pass was meant for reviewing a summary of the interview, in which a summary screen provided an overview of all foods, quantities and approximate energy consumed. As additional foods for the food list were identified during piloting, training and data collection, the updated master INDDEX24 food list and conversion factors in the FMDB were updated onto all enumerator tablets via wireless connection.
In contrast to the INDDEX24 24HR, during the PAPI 24HR, enumerators followed the same four passes but used a printed food list that contained all of the same items as those added to the INDDEX24 food list up to the point where training ended (i.e. no new items were added to the PAPI printed version after the start of data collection, reflecting a real-life constraint of working with a PAPI approach). The enumerators recorded details on each food and then searched the printed food list for the corresponding item and code. If the enumerator could not find a food or recipe, they would describe it in detail and code it as ‘99 999’. If a standard recipe did not exist that matched what was reported by the respondent, then the enumerator recorded it as a NSR and collected detailed information about the amount prepared and the quantities of ingredients used. Unlike with INDDEX24, which pre-assigned portion size estimation aids (PSEA) and probes to foods, when enumerators were administering the PAPI, they were expected to identify which would be the best PSEA option for each food and probe to find the most accurate match.
Preparation of dietary reference data
The dietary reference data for the study consisted of a list of over 3000 foods, standard recipes and ingredients (henceforth ‘food list’), as well as portion conversions, yield factors, retention factors and food-specific survey probes. The dietary reference data were developed for this study as well as for the 2019 Viet Nam National Dietary Survey(27). The food list was collated from a variety of data sources, including the published Viet Nam Food Composition Table(28), NIN Food Photo Atlas(29), NIN recipe book ‘Nutritive Values of 500 Common Dishes’(30) (based on household recipes reported in the 2010 General Nutrition Survey), the NIN yield factors book with simple cooked foods(31), the INDDEX Project Feasibility Study (n 60) conducted in 2018 and the Nutritive Value of Common Street Foods with street food recipes from southern Viet Nam(32). Standard recipes for mixed dishes were calculated based on the methods described by Gibson and Ferguson (2008)(Reference Gibson and Ferguson24).
The items in the food list were matched primarily with the Viet Nam NIN Food Composition Table(28), and other FCT were selected based on the availability of the data and the similarity of the food or country of origin as advised by FAO(33–40). The 2017 Vietnamese FCT contained only raw foods. For the purpose of the validation study, the FCT was expanded to include simple cooked foods (e.g. boiled, baked and steamed). For any non-raw foods, adjustments were made with the appropriate cooking yield and nutrient retention factors. All values were checked using FAO/INFOOD guidelines(41).
Portion size estimation methods used for the 24HR included direct weight, a photo atlas, proxy weight (with rice) and proxy weight (with water). In a few select cases where food was purchased outside the home, respondents estimated the price or weight of the food purchased. Portion conversion factors accounted for the density and edible portion, as appropriate. Details about each component of the dietary reference data are available in the Online Supporting Material, Supplement I.
Training and data collection procedures
A total of twenty-one WFR enumerators and four WFR supervisors were trained for 5 d. For the 24HR, twelve 24HR enumerators and three 24HR supervisors were trained for a total of 10·5 d on both the INDDEX24 and PAPI modalities. Training consisted of classroom learning and field-based practice.
Data collection occurred in August 2019 and was evenly distributed across all 7 d of the week. The WFR took place over the course of a full day, the day prior to the 24HR interview. Enumerators arrived at the respondent’s house in time to observe the preparation and consumption of the first food and drink of the morning and stayed through the last main meal of the evening. WFR enumerators recorded detailed information on food name, food code, quantity prepared, weights of pans and plates, quantity served, and amount leftover, as well as the time and place of preparation and consumption for each item. For meals where the foods were eaten communally, the WFR enumerators asked respondents to apportion the respondent’s meal into a second set of serving bowls that the respondent would eat. These serving bowls (e.g. rice bowl, dipping sauce, etc.) were weighed before and after consumption to determine the amount consumed by the respondent. If mixed dishes (i.e. those derived from a recipe) were purchased outside the home, prepared elsewhere (e.g. by a relative or neighbour), or leftover from a dish prepared the previous day, then enumerators recorded as many details as possible, and these dishes were later matched with standard recipes. WFR enumerators coded all items throughout the course of the day in between eating occasions, and codes were later checked by supervisors. Any discrepancies were discussed and addressed between the supervisor and the enumerator.
The team of 24HR enumerators administered the INDDEX24 or PAPI 24HR interview the day after the WFR at the respondent’s house at an appointed time (either in the morning or the afternoon). The 24HR enumerators were assigned to conduct either INDDEX24 or PAPI interviews for the day and alternated modality by day. For both INDDEX24 and PAPI, the enumerator first registered the respondent, then conducted the 24HR followed by the socio-demographic module and self-reported respondent weight.
All food weighing (WFR, INDDEX24 and PAPI) was performed with digital scales (My Weigh KD 7000 7kg, accurate within 1g, that were calibrated by the supervisors every day using a standard weight). In addition, each time the scale was moved in the course of the day, the enumerator checked the scale with a 100-g reference weight afterwards to ensure that it was still accurate. Data collection for the time-per-interview component also occurred in August 2019. The approach for the INDDEX24 and PAPI interviews followed the same structure as described above for the accuracy portion of the study, except that the WFR was not conducted in this component. With INDDEX24, the time-per-interview was recorded automatically by the mobile app, yielding the time per pass and for the total interview as part of the master data file. For the PAPI 24HR, time-per-pass and time-per-interview were collected manually by each enumerator using a phone stopwatch and recorded on a paper form designed for this purpose.
This study was conducted according to the guidelines laid down in the Declaration of Helsinki, and all procedures involving human subjects were approved by the institutional review boards at Tufts University (#1 904 024) and Viet Nam NIN (45/VDD-QLKH). Written informed consent was obtained from all subjects. Participants in the accuracy component of the study received approximately 25 USD paid in VND (17 USD for the WFR day and 8 USD for responding to the 24HR) to help offset the cost of their time. Participants in the time-per-interview component who responded to the 24HR received the equivalent of 8 USD.
Data entry, cleaning and processing: accuracy component
For INDDEX24, no additional data entry was required after enumerators recorded data in the mobile app at the interview site. Matches with appropriate conversion factors, food codes and nutrients were made automatically through the INDDEX24 analytical reporting function. Dietary reference data for foods, NSR (including food away from home and food prepared on the previous day) or PSEA conversion factors that were reported in the survey but did not already exist in the INDDEX24 FMDB were subsequently added to the FMDB before analysis.
All WFR and PAPI data were double-entered using CSPro7.3(42) with data entry forms developed for the purpose of this study. Two separate sets of data entry clerks were trained to enter the data. A data entry supervisor resolved discrepancies between the double-entered data by consulting the hard copies of the questionnaires. Subsequent cleaning was carried out to ensure the appropriate use of food, recipe, and ingredient codes, PSEA codes, and household ID.
For both the WFR and PAPI, all foods and recipes were converted to nutrient values by manually matching with INDDEX24 FMDB dietary reference data that were entered into the CSPro database developed to house the WFR and PAPI data. Any new foods, NSR (including food away from home and food prepared on the previous day) or PSEA conversion factors that did not already exist in the INDDEX24 dietary reference dataset were added to the CSPro database to ensure that they had a match with the WFR and/or PAPI.
Given that the WFR recorded data from the first meal to the last meal of the day but the 24HR asked respondents to recall all foods consumed in the previous 24 h, the 24HR analysis was bounded to match the time of the observed WFR. For the 24HR, the respondents were asked to report the approximate time of consumption but may not have reported the exact time, so a 45-min buffer was added to either side of the WFR to ensure inclusion of all 24HR start and end times. This resulted in the inclusion of more than 95 % of all food items across the 24HR (96·27 % for INDDEX24 and 95·70 % for PAPI). On average, the recall window was from 06.38 (35 m sd) to 19.03 (1 h 14 m sd) for INDDEX24 and 06.44 (33m sd) to 19.09 (1 h 51 m sd) for the PAPI.
The following nutrients of public health concern in many LMIC were selected for analysis: energy (kcals), fat (g), protein (g), carbohydrates (g), total fibre (g), vitamin A (mcg retinol activity equivalent), vitamin C (mg), Ca (mg), Fe (mg) and Zn (mg). To better approximate a normal distribution, nutrients were log-transformed. All benchmark (WFR) daily intakes for both groups fell within a standard cut-off of 500–5000 kcal and were kept for analysis regardless of their recall values(Reference Bell, Coates and Rogers43–Reference Tucker, Bianchi and Maras45).
Despite extensive efforts to update the FCT, it remained incomplete for certain nutrients. Of the foods reported in the validation study for our nutrients of interest, 97·0 % had energy values, 92·5 % had protein values, 82·9 % had fat values, 92·5 % had carbohydrate values, 82·1 % had fibre values, 90·0 % had vitamin A retinol activity equivalent values, 79·7 % had vitamin C values, 88·5 % had Ca values, 85·3 % had Fe values and 69·0 % had Zn values. The distribution of missing FCT values for each nutrient was equivalent across the INDDEX24 and PAPI modalities and thus does not affect the results of the study, as the research questions centre on relative validity and not absolute intake.
Data analysis
The primary objective of this study was to measure the differences between INDDEX24 and WFR, the PAPI 24HR and WFR, and the difference-in-differences between the two recall modalities. As a first step, Bland–Altman plots(Reference Bland and Altman46,Reference Bland and Altman47) were generated for grams consumed and for consumption of energy and all nutrients of interest in order to assess visually the level of agreement between each 24HR modality and the WFR. Next, following Arsenault et al. (Reference Arsenault, Moursi and Olney48), the two one-sided t test approach was used, pairing each individual’s recall and WFR on all nutrients of interest. The two one-sided t test is a paired sample comparison of means. Unlike difference testing, it evaluates whether the difference between the two collection methods is within a pre-specified range and aims to demonstrate equivalence by way of the lack of difference:
H0: (µ2–µ1) ≤ –ϵ or (µ2–µ1) ≥ ϵ;
H1: –ϵ < (µ2–µ1) < ϵ;
where (– ϵ, ϵ) is the pre-specified range of acceptable difference. One test of equivalence is performed on each bound and uses the larger P-value to compare α. Rejecting the null hypothesis of a sufficiently large difference, the alternative hypothesis of ‘nearly equivalent’ is accepted. Post hoc power computation for the two one-sided t test using the n 234 sample size remained over 80 % for most outcomes tested at a delta range of −0·5 to 0·5.
For log-transformed data such as these, the test uses the lognormal distribution, and therefore the ratio of the geometric mean differences was used as the centre. The standard 10 % bound was chosen for the equivalence buffer(Reference Arsenault, Moursi and Olney48–Reference Tugault-Lafleur, Black and Barr50). Once statistical equivalence was established between the WFR and 24HR pairs for both INDDEX24 and PAPI, the magnitude of the difference between the WFR-INDDEX24 and the WFR-PAPI was then assessed using a difference-in-differences approach. In this unadjusted random subjects model, nutrient outcomes were modelled on collection type (WFR and 24HR) and modality (PAPI and INDDEX24), estimating the difference of the WFR and 24HR for each modality and comparing these differences between modalities. SAS 9.4 Proc mixed was used.
To quantify the individual accuracy of the INDDEX24 and PAPI 24HR, the percentage of respondents using each 24HR modality that fell within specific percent error categories compared with the WFR for energy and each nutrient was calculated. This approach assessed the extent to which each modality (INDDEX24 and PAPI) was accurate at the individual level relative to the WFR estimates. All analyses used SAS 9.4 and Stata 15 SE.
Data analysis: cost-effectiveness component
The cost-effectiveness of each modality was calculated by combining cost data with INDDEX24 and PAPI indicators of food and nutrient intake accuracy, with ‘effectiveness’ defined in terms of percentage points of accuracy gained through each of the two modalities. Cost data were derived from a costing study, reported in detail elsewhere, that used an activity and ingredients-based method to estimate the total and relative costs of each activity needed to complete the 24HR survey and produce a clean, analysable 24HR dataset using the INDDEX24 v. PAPI modality(Reference Adams, Bell and Somé51). The cost study adopted a societal perspective such that all costs were accounted for (including respondent time) regardless of who incurred them(52). Costed activities included the preparation of dietary reference data, survey preparation, training, survey execution (including time-per-interview for both enumerators and respondents), data entry, and data cleaning and processing. The reported time and monetary costs associated with completing each activity were combined with information on staff wages and salaries, converted from Vietnamese Dong to US dollars where necessary, adjusted to 2019 US dollars, and summed in order to estimate the total cost of conducting the 24HR using INDDEX24 and the PAPI modalities.
Cost-effectiveness was assessed based on three primary outcomes: average percent accuracy in estimating the number of food items consumed (item count), average percent accuracy in estimating total gram amount of food intake and a composite measure of the average percent accuracy in estimating each of ten nutrients of interest (energy, fat, protein, carbohydrate, fibre, vitamin A, vitamin C, Ca, Fe and Zn), hereafter referred to as the ‘ten-nutrient composite measure of nutrient intake’. For the number of food items consumed and gram amount, the average percent error was calculated as one minus the ratio of the average estimate based on INDDEX24 (or based on PAPI) to the average estimate based on the WFR, and then percent accuracy was defined as 100 minus the absolute value of the average percent error. For gram amount, for example, the average percent accuracy of INDDEX24 and PAPI was calculated as:
$$100 - \left| {\left( {1 - {{\overline {{\rm{gram}}\;{\rm{amoun}}{{\rm{t}}^{{\rm{PAPI}}}}} } \over {\overline {{\rm{gram}}\;{\rm{amoun}}{{\rm{t}}^{{\rm{WFR}}}}} }}} \right){\kern 1pt} \times {\kern 1pt} 100} \right|$$
For the ten-nutrient composite measure of nutrient intake, a measure of overall accuracy in estimating nutrient intake, we followed Hatløy, Torheim and Oshaug(Reference Hatløy, Torheim and Oshaug53) whose methods have been applied for validation of dietary metrics(Reference Steyn, Nel and Labadarios54,Reference Mirmiran, Azadbakht and Esmaillzadeh55) . First, the average percent accuracy in estimating the intake of each nutrient was calculated, using the same method as described for item count and gram amount, and then the overall average was taken across the ten nutrients. For each measure of accuracy, absolute and relative (i.e. the difference between INDDEX24 and PAPI), cost-effectiveness was calculated as total cost per average percentage points of accuracy.
Results
Table 1 compares key socio-demographic characteristics among respondents who completed INDDEX24 and PAPI 24HR. The two groups of respondents were similar for most characteristics, though educational achievement was higher among INDDEX24 respondents, and slightly more PAPI respondents were pregnant and breast-feeding.
Table 1. Socio-demographic characteristics of PAPI and INDDEX24 respondents (n 234) in Thanh Oai, Viet Nam
PAPI, pen-and-paper interview; INDDEX, International Dietary Data Expansion.
Differences in estimates of nutrient intakes by pen-and-paper interview v. INDDEX24 24-h dietary recall modalities, compared with weighed food record
The average level of overestimation and underestimation when comparing the INDDEX24 and PAPI 24HR to the WFR was minor with an equivalence bound set at 10 % (Table 2). The two one-sided t test indicated statistical equivalence between both 24HR modalities and WFR for energy and all nutrients of interest with the exception of vitamin A, which showed an overestimate for PAPI and a sizable underestimate for INDDEX24.
Table 2. Difference-in-difference comparison of group means for energy and nutrients for PAPI and INDDEX24 to WFR in Thanh Oai, Viet Nam
(Mean values and standard deviations, n 234)
PAPI, pen-and-paper interview; INDDEX, International Dietary Data Expansion; WFR, weighed food record; RAE, retinol activity equivalent.
a Note the WFR is the minuend, and therefore a negative difference between WFR and 24HR modalities (PAPI and INDDEX24) indicates an overestimate, while a positive number indicates an underestimate.
b Equivalence test P-value is from the paired two one-sided t test (TOST) reported using the natural log geometric mean and 10% bound.
c The difference-in-difference P-value is calculated based on the first difference between each respective 24HR modality (PAPI and INDDEX24) and the WFR and the second difference based on those differences ((WFR-PAPI) – (WFR-INDDEX24)).
The difference-in-difference regression results, also presented in Table 2, showed that none of the relative differences between the two arms ((WFR-PAPI) – (WFR-INDDEX24)) was statistically significant, with the exception of protein and Fe. In the case of protein, the PAPI diverged less from the WFR than INDDEX24 (–3·7 v. 7·9 g), whereas for Fe INDDEX24 diverged significantly less than PAPI from the WFR (0·9 v. −1·3 mg). Overall, the differences between each 24HR modality (INDDEX24 and PAPI) compared with the WFR, and those differences compared with each other, were small.
The small differences are visible in a series of Bland–Altman plots (online Supplementary Fig. A–J) which show individual differences between the two methods and were used to examine the mean bias, limits of agreement and the distribution of bias. Side-by-side comparisons for both WFR-PAPI and WFR-INDDEX24 showed similar patterns. The plots for protein and fat indicated a greater bias with increased intake. This bias, although small, was inverted, with higher intakes for the WFR method on the PAPI side and higher intakes for the recall method on the INDDEX24 side. Carbohydrate, on the other hand, exhibited a small mean difference below zero for both methods, indicating a pattern of higher intakes for the WFR method. Consistent bias towards higher WFR values for protein, fat and carbohydrates are illustrated in the energy plot for the PAPI-WFR comparison. For micronutrients, 95 % of values remained within the limits of agreement and the mean difference in relation to zero (bias) remained small.
On an individual level, 26 % of PAPI respondents and 32 % of INDDEX24 respondents had energy intakes within 10 % of their WFR recalls, while a total of 53 % of PAPI and 59 % of INDDEX24 respondents reported energy intakes within 20 % of the benchmark (Table 3). For the nutrient estimates, the percentage of respondents within 10 % of their WFR varied by modality. For fat, fibre, vitamin A and Fe, the PAPI tended to have slightly more respondents within 10 % of their WFR, while for carbohydrates and Ca more respondents in the INDDEX24 arm were within 10 % of their WFR values. For the remaining nutrients (protein, vitamin C and Zn), item count and gram amount, the percentage of respondents within 10 % of their WFR was essentially the same for PAPI and INDDEX24. While overall, the group averages in Table 2 indicated slight overestimates for many nutrients across both PAPI and INDDEX24, the individual level error results in Table 3 – when disaggregated by overestimates v. underestimates – show that at an individual level there were non-trivial underestimates, particularly in the outer tails of the distribution (comparing < –50 % error to > +50 %). This highlights that as error estimates grow larger, both the PAPI and INDDEX24 24HR modalities were more likely to underestimate, rather than overestimate, compared with the WFR (online Supporting Material, Table S1).
Table 3. Percent of respondents falling within ranges of percent error in estimating energy and nutrient intakes with PAPI and INDDEX24 24HR modalities compared with WFR (n 234) in Thanh Oai, Viet Nam
PAPI, pen-and-paper interview; INDDEX, International Dietary Data Expansion; WFR, weighed food record; RAE, retinol activity equivalent.
Differences in estimates of food group intakes by pen-and-paper interview v. INDDEX24 24-h dietary recall modalities compared with weighed food record
The median percent of energy intake from the major FAO/WHO GIFT food groups provides further information about differences between the 24HR modalities (Table 4). These results again underscore the negligible absolute and relative average differences between the two 24HR modalities and the WFR: the absolute differences in energy intake between the PAPI-WFR and INDDEX24-WFR did not exceed +/–3 percentage points for any food group. Neither PAPI nor INDDEX24 24HR modality was statistically different from WFR for the majority of the food groups.
Table 4. Median percent of energy intake from major FAO/WHO GIFT food groups (n 234) in Thanh Oai, Viet Nam
(Median values and percentiles)
PAPI, pen-and-paper interview; INDDEX, International Dietary Data Expansion; WFR, weighed food record; WRS, Wilcoxon’s rank sum test.
a Results for three food groups are not shown here as they were consumed by less than 5 % of respondents in each arm: savoury snacks (1·7 % PAPI and 3·4 % INDDEX24), insects and grubs (0·9 % PAPI and 0·9 % INDDEX24), and food additives (0·9 % PAPI and 0·0 % INDDEX24).
b Median results are based on all respondents in that arm (consumers and non-consumers).
c Note the WFR is the minuend, and therefore a negative difference between WFR and 24HR modalities indicates an overestimate and a positive number indicates and underestimate.
d WRS = Wilcoxon Rank Sum Test is a nonparametric test that compares two paired groups.
Cost-effectiveness of pen-and-paper interview v. INDDEX24
As detailed in Adams et al., 2021(Reference Adams, Bell and Somé51), the total economic cost of using INDDEX24 to conduct the 24HR was $111 004 (2020 USD; n 147), compared with $120 483 (2020 USD; n 147) for PAPI. While the non-time costs (e.g. equipment) were higher for INDDEX24 than PAPI, INDDEX24 had lower personnel costs, leading to a $9479 cost savings to conduct the 24HR using INDDEX24 compared with PAPI. The average percent accuracy in estimating the gram amount and the ten-nutrient composite measure of nutrient intake was slightly higher using the INDDEX24 modality than the PAPI modality, while the average percent accuracy in estimating item count was slightly higher using PAPI (Table 5). However, given the cost savings associated with INDDEX24, INDDEX24 was more cost-effective than PAPI for all three outcomes.
Table 5. Cost-effectiveness of conducting a 24HR using PAPI and INDDEX24 in Viet Nam
24HR, 24-h dietary recalls; PAPI, pen-and-paper interview; INDDEX, International Dietary Data Expansion.
* See Adams et al. (Reference Adams, Bell and Somé51) for detailed cost information. Cost-effectiveness was calculated as cost per average percentage point of accuracy.
† The difference was calculated as INDDEX24 minus PAPI. All estimates are rounded to the nearest whole number.
‡ Ten-nutrient composite measure of nutrient intake is calculated as the overall average of the average percent accuracy of PAPI (or of INDDEX24) in estimating intake for energy, fat, protein, carbohydrate, fibre, vitamin A, vitamin C, Ca, Fe and Zn relative to intake based on the weighed food record.
On a nutrient-specific basis (online Supporting Material, Table S2), there was variability in the average percent accuracy achieved using INDDEX24 relative to PAPI (INDDEX24 had a higher average percent accuracy for energy, fibre, Ca, Fe and Zn, while PAPI had a higher average percent accuracy for fat, protein, carbohydrate, vitamin A and vitamin C). However, INDDEX24 was more cost-effective than PAPI for all nutrients except vitamin A.
Discussion
To our knowledge, this study was the first to assess whether using a technology-enabled approach to dietary assessment in an LMIC context can confer added benefits to a PAPI approach in accuracy and cost-effectiveness.
The results showed that the INDDEX24 technology-enabled approach was an accurate modality for assessing food and nutrient intake at a population level. While the accuracy of INDDEX24 and PAPI were similar, the INDDEX24 Dietary Assessment Platform cost less, which meant that INDDEX24 was the more cost-effective modality for collecting and processing accurate dietary recall data.
Though INDDEX24 performed better than the PAPI 24HR across a number of different measures of accuracy, the two modalities were not significantly different from one another on most measures. Our individual-level analyses showed non-systematic overestimation and underestimation for both INDDEX24 and PAPI for most foods and nutrients which, on average, resulted in estimates that did not deviate significantly from the benchmark results. The results of other 24HR validation studies in LMIC show similar variability in under- v. over-reporting as our study. Some have documented systematic under-reporting(Reference Alemayehu, Abebe and Gibson56–Reference Scagliusi, Ferriolli and Pfrimer62), while others found that the 24HR resulted in non-significant differences, on average, from a WFR benchmark, including some negligible over-reporting at a population level(Reference Dop, Milan and Milan63–Reference Thakwalakwa, Kuusipalo and Maleta68). An extreme value for vitamin A intake in INDDEX24 but not in PAPI contributed to the poorer performance of INDDEX24 for this nutrient. There are a number of plausible explanations for extreme values in one or the other modality, including the possibility of an omission or intrusion of an easily overlooked or rarely consumed food high in vitamin A by an INDDEX24 respondent but not a PAPI respondent. Furthermore, this study did not conduct multiple 24HR on the same individuals because the aim was not to calculate usual intake but rather gauge the relative accuracy of a single-day recall by modality relative to the WFR. Conducting multiple non-sequential days of recall to enable the estimation of usual intake typically helps smooth intra-individual variability and would be recommended for most dietary surveys that use either INDDEX24 or PAPI modalities.
There are several reasons why the PAPI and INDDEX24 might have performed similarly in terms of accuracy. First, both modalities were carefully designed to collect 24HR data using the same rigorous multiple-pass method. A more typical PAPI is of lower technical quality and may have diverged more dramatically from both the INDDEX24 results and the WFR. Second, in our study, the same enumerators received intensive, high-quality training on both modalities and alternately carried out both PAPI and INDDEX24 interviews. This cross-training likely benefited both modalities. The alternative, randomising interviewers to implement either INDDEX24 or PAPI (but not both), was rejected due to the concern for systematic bias stemming from enumerator skill levels and interview execution. Third, both modalities likely benefited from the systematic approach to developing dietary reference data that is required by the FMDB component of INDDEX24. Fourth, some of the convergence in results may be due to the use of standard recipes for both INDDEX24 and PAPI, despite the fact that standard recipes are not typical of most PAPI approaches.
While no other documented studies to date have compared a technology-enabled approach with a PAPI 24HR and against a benchmark method in an LMIC context, a few other studies have compared data collection modalities or a CAPI approach on its own to a benchmark method such as WFR or biomarkers(Reference Zang, Song and Wang18,Reference Kirkpatrick, Subar and Douglass69–Reference Timon, van den Barg and Blain71) . The vast majority of these studies were conducted in high-income countries, and none assessed multiple modalities with a benchmark in a single evaluative framework. Of the small number of studies comparing a CAPI and PAPI 24HR data collection modality in an LMIC without a benchmark, Zang et al. (2015)(Reference Zang, Song and Wang18) did not attempt to assess accuracy but found that the CAPI and PAPI produced similar results relating to acceptability and feasibility of use among adults in the Shanghai, China region. Htet et al. (2019)(Reference Htet, Fahmida and Do19) found that applying a CAPI multiple-pass 24HR method in Indonesia to assess children’s diets resulted in a higher percentage of children classified as acceptable energy reporters (within 95 % CI of modelled total energy expenditure) when using CAPI v. PAPI. As in our study, they found no significant differences between estimated mean energy and nutrient intakes for the technology-enabled CAPI v. PAPI.
There are some potential limitations to the study results, in addition to the potential influence of instruments and experience across enumerators implementing INDDEX24 and PAPI described above. The INDDEX24 sample was slightly more educated than the PAPI sample, albeit non-significantly so. This difference in education level might have affected the ability of INDDEX24 respondents to better recall foods and nutrients relative to PAPI respondents. At the respondent level, there was a risk of ‘priming bias’, where the WFR conducted the previous day may have prompted respondents to better remember what they ate in the recall. While this explanation seems intuitively plausible, a 1994 study by Ferguson in Malawi that tested for priming bias from an observer-weighed food record found no evidence that this was a concern(Reference Ferguson, Gibson and Opare-Obisaw72). Priming bias would have been faced similarly across both the INDDEX24 and PAPI arms and, therefore, is not a threat to the overall objective of the study, which was to assess relative accuracy of the two modalities. And, while ‘priming’ is a factor to be explored further in future studies that do seek to assess the accuracy of 24HR methods, there is evidence to suggest that intentional priming (e.g. asking respondents to photograph the foods eaten, maintain a detailed food diary (for literate populations) or place a ‘check’ beside illustrations of foods the 1–3 d prior to the 24HR) could be a useful means of improving the accuracy of respondents’ recall, a desirable outcome in itself(Reference Pries, Rehman and Filteau73,Reference Rollo, Williams and Burrows74) .
Similiarly, other potential limitations, including reactivity bias (where WFR respondents may have modified their consumption while under direct observation by an enumerator) and the single rather than repeat WFR-24HR visits required to construct indicators of ‘usual intake’, were not an issue for the primary objective of our study, which was to test the relative difference between INDDEX24 and PAPI modalities. These factors could affect the absolute results of the 24HR compared with WFR benchmark, in terms of the accuracy of average intake estimates overall. A stronger benchmark than the WFR, that avoids some types of WFR-specific error as well as shared error with the 24HR modalities, would have been a biomarker method such as doubly labelled water or urinary nitrogen. However, these biomarker methods were ruled out for the current study, primarily because they would not have allowed us to disentangle the sources of error in the test modalities, which the WFR was better suited to enable.
Viet Nam is a lower-middle-income country with a complex food system, an intricate diet and customary eating practices (e.g. eating food from a common bowl with chopsticks) that create numerous challenges for accurate dietary assessment, whether done through direct observation and weighing or by recall. That the INDDEX24 Dietary Assessment Platform performed well in this context, together with similar results from a parallel study conducted by many of the same researchers in Burkina Faso(Reference Rogers, Somé and Bakun20), lends confidence to its use in other LMIC with potentially less varied eating patterns and overall diets. Additional testing and validation of INDDEX24 is warranted, on surveys of different scales (e.g. nationally representative studies) and for other population groups (e.g. men and children). Replication of the validation protocol in a heavily urban population where packaged, processed and street foods form a large part of the diet will also offer useful insights to improve INDDEX24. It would be helpful to perform additional validation in contexts where respondents speak multiple languages within or across study sites. The use of a common language for interviewing in Viet Nam (Vietnamese) meant the relative effects of language switching in the INDDEX24 Mobile App on accuracy and cost were not tested. INDDEX24 offers enumerators the flexibility to toggle among up to four different languages within a single app, a feature that could confer added benefit to the process over a PAPI that would need multiple paper questionnaires in a range of languages. This study compared INDDEX24 with PAPI, since PAPI is still the most widely used method in LMIC. Further research could assess the relative accuracy of INDDEX24 compared with other technology-enabled approaches for dietary assessment including other ‘innovative’ technologies such as active or passive food imaging that do not rely on recall.
Our cost and cost-effectiveness analyses suggested that the use of INDDEX24 cost $9000 less per respondent than the PAPI when the dietary reference data were largely assembled from scratch. Our modelling projections suggest that the total cost for INDDEX24 would be approximately $40 000 less than the PAPI approach when 75 % of the needed dietary reference data were available in the FMDB as a starting point (see Adams et al., 2021 for details of this and other modelled cost scenarios). Populating the FMDB with country-specific data is a priority since, as dietary reference data are pre-populated in the FMDB, the absolute cost of using INDDEX24 will likely decline significantly, further improving the cost-effectiveness of INDDEX24 relative to PAPI. Furthermore, the detailed comparative cost models of INDDEX24 and PAPI constructed by Adams et al., 2021 suggest that economies of scale favour INDDEX24; in larger studies, the cost of implementing a 24HR declines at a greater rate for INDDEX24 than for PAPI, further increasing the overall cost-effectiveness of INDDEX24.
Conclusion
The INDDEX24 Dietary Assessment Platform was developed with the aim of streamlining and standardising the collection of 24HR data, to enable more LMIC researchers and governments to collect quantitative dietary data to inform a wide range of health, nutrition, agricultural, food safety and environmental policies. This validation study in Viet Nam has shown that the INDDEX24 Dietary Assessment Platform yields results that are as accurate as a standard PAPI approach while costing less. The evidence from this validation study offers positive reinforcement of the benefit of adopting INDDEX24 for implementing high-quality, quantitative dietary assessments.
Supplementary material
For supplementary material/s referred to in this article, please visit https://doi.org/10.1017/S0007114522001507
Acknowledgements
The authors gratefully acknowledge the NIN staff and enumerators who worked tirelessly to support this study and respondents who generously shared their time, household space and dietary information for full-day observations and recalls. The authors also appreciate David Alex Carroll’s very able research assistance in the preparation of the manuscript and Julia Matteson’s very helpful contribution to copy editing and formatting the original and revised manuscripts. Any mistakes are the sole responsibility of the authors.
This work was supported by the Bill & Melinda Gates Foundation (grant number INV-001600) through a grant to the Tufts University Gerald J. and Dorothy R. Friedman School of Nutrition Science and Policy’s INDDEX Project. Under the grant conditions of the Foundation, a Creative Commons Attribution 4·0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission.
J. C., B. L. R., W. B. and J. W. S. designed the research; J. C., B. L. R., W. B., J. W. S. and B. C. provided essential materials (developed the INDDEX24 platform); W. B. and H. D. conducted research; P. B., K. A. and W. B. analysed the data; J. C., W. B. and K. A. wrote the paper; J. C. had primary responsibility for final content. All authors read and approved the final manuscript.
There are no conflicts of interest.
Open access
