Complete case analysis

Complete case analysis is often used in epidemiological studies as this is the standard approach that statistical software packages adopt when dealing with missing data. In complete case analysis, only participants with complete data for all study variables to be considered in the analysis are included. This means that the dietary exposure (e.g. nutrient intake or diet quality score) would not be derived for a participant who had missing data on one of the FFQ items used to compute the exposure; instead, this participant’s data would be completely discarded from the analysis. This method can lead to biased findings if the missing data are not missing completely at random (MCAR). Data can be assumed to be MCAR when there are no systematic differences between the missing data and the observed data; that is, the probability of data being missing is unrelated to observed values of other variables or to the unobserved values themselves (e.g. FFQ data could be assumed to be MCAR if a participant accidentally skipped over a question). In addition to potential biases, complete case analyses can lead to a large reduction in sample size if missing data are common, resulting in reduced statistical precision and power. Furthermore, it is inefficient to exclude participants from the analysis who provided a substantial amount of complete data for many FFQ items but failed to respond to a small number of items.

Single imputation

Single imputation approaches are appealing as these are relatively simple to conduct. This type of approach replaces each missing value with only one value in the imputation process, producing a filled-in data set that allows standard analytical methods to be used. Single imputation approaches include zero imputation (where missing values are replaced with zero to represent no consumption of that food item) or mean, mode and median imputation, which replace missing values with the sample mean, mode or median, respectively. More complex single imputation approaches include k nearest neighbours imputation⁽ Reference Parr, Hjartaker and Scheel ^{9 )}. This approach identifies k participants with similar responses on other variables to that of the participant with the missing data on the FFQ item, all of whom have complete data for the FFQ item to be imputed. The average value for that particular FFQ item among those k participants is taken as the FFQ item value for the participant with the missing response.

Single imputation methods have various shortcomings, with disadvantages including the very strong assumptions made. For example, assuming that the missing data equate to no consumption when using zero imputation has the potential to result in misleading findings if it is participants who do not want to report consumption of particular items (e.g. energy-dense, nutrient-poor foods) who have missing data on these items. Furthermore, single imputation methods do not reflect the uncertainty created by the missing data as they assume that the imputed values are real observations. This leads to underestimation of variances and poor coverage of confidence intervals⁽ Reference Schafer and Graham ^{15 )}, which can result in incorrect inferences about associations between exposures and outcomes. Thus, these approaches for dealing with missing data are not recommended, with journals and medical research guidelines increasingly turning towards multiple imputation as a preferred method for handling missing data in epidemiological studies⁽ Reference Ware, Harrington and Hunter ^{16 ,} Reference Klebanoff and Cole ^{17 )}.

Multiple imputation

Multiple imputation, like single imputation, produces filled-in data sets that enable standard analytical methods to be used, allowing partially observed information from all participants to be included in analyses. Each missing value is replaced with multiple imputed values that are generally drawn from a statistical model (known as an ‘imputation model’). This produces multiple filled-in data sets, each of which is then analysed separately using standard methods and the multiple results are then combined to give an overall estimate⁽ Reference Rubin ^{18 )}. A benefit of multiple imputation over single imputation is that it appropriately accounts for the uncertainty in the missing values, ensuring that the variance is not underestimated. It can also correct for bias, for example, if predictors of non-response are included in the imputation process. However, although multiple imputation approaches have been highlighted as potentially useful approaches for dealing with item non-response in FFQs⁽ Reference Parr, Hjartaker and Scheel ^{9 )}, these have not often been used within nutritional epidemiology, although some nutritional cohorts, such as the Adventist Health Study 2 (AHS-2) cohort, have begun to routinely adopt this method⁽ Reference Orlich, Singh and Sabate ^{19 ,} Reference Rizzo, Sabate and Jaceldo-Siegl ^{20 )}. This is perhaps due to the complexity of these techniques and the lack of guidance about when these methods are suitable to use.

Multiple imputation involves the specification of an imputation model which should generally include all covariates to be considered in the underlying analytical model (of dietary exposure and health outcome), in addition to other covariates which may be associated with the variables that have missing data or that are predictive of non-response. These additional covariates, known as ‘auxiliary variables’, have been shown to increase efficiency and reduce bias⁽ Reference Collins, Schafer and Kam ^{21 )}. A particular challenge when using dietary exposures in epidemiological analyses is deciding which variables to include in the imputation model. Considering nutrient intake, it would seem more appropriate to impute individual food items as each individual food will potentially contribute to the score of multiple nutrients. Similarly, it would be ideal to retain all of the complete information available for individual FFQ items on which the diet quality score is based, rather than simply the complete score, in the imputation model. This is because using the complete score can result in inefficiencies as this will fail to consider complete FFQ items within the score, meaning that participants with missing data for only one FFQ item would have a missing score. However, multiple imputation can be computationally intensive, particularly when many categorical variables are included or when there are substantial missing data. Imputation models may fail to converge when they involve large numbers of FFQ items⁽ Reference White, Royston and Wood ^{22 )}. One solution, although less efficient, could be to impute sub-scale scores in order to retain as much information as possible within a more parsimonious model, as discussed elsewhere⁽ Reference Plumpton, Morris and Hughes ^{23 )}. Although options for conducting multiple imputation are now available in standard statistical software, such as Stata and SAS, appropriate ways in which multiple imputation can best be used to address item non-response in FFQ data, and particularly when deriving dietary exposures, have received little attention in the literature to date.

A further challenge with multiple imputation is that most software implementations of multiple imputation assume that the missing data are missing at random (MAR). This means that the probability that data are missing does not depend on unobserved data (i.e. the missing values themselves), but may depend on observed data (i.e. the non-missing data)⁽ Reference Sterne, White and Carlin ^{24 )}. An example of MAR in the context of diet assessment is that males may be more likely to have missing fruit and vegetable intake. If sex is observed, then within each sex category there will be no systematic differences between observed and missing data. However, if data are missing not at random (MNAR), meaning the probability that an FFQ response is missing depends on unobserved data (i.e. the missing values themselves)⁽ Reference Sterne, White and Carlin ^{24 )}, then multiple imputation can give misleading results (as will most complete case analyses and single imputation methods). Given known issues affecting assessment of dietary intake, such as social desirability bias⁽ Reference Hebert, Hurley and Peterson ^{25 ,} Reference Miller, Abdel-Maksoud and Crane ^{26 )}, it may be the case that FFQ data are MNAR if participants who have missing data on ‘unhealthy’ FFQ items (e.g. energy-dense, nutrient-poor foods) are those with high levels of consumption. Unfortunately, it is not possible to determine from the data if the missingness is MAR or MNAR. Therefore, as highlighted by Sterne et al.⁽ Reference Sterne, White and Carlin ^{24 )}, researchers must consider the possible reasons why the missing data could have occurred in order to decide if MNAR could be a problem in their study. Strategies have been proposed to recover missing data when the MAR assumption may not be reasonable, such as guided multiple imputation where a random sub-sample of participants with missing data is contacted to recover the missing data. This information is then used in the imputation model⁽ Reference Fraser and Ru ^{27 )}. However, it is not always possible to re-contact a sub-sample of study participants to obtain this information. Therefore, it is important to consider potential strategies for dealing with missing FFQ data in this context as well as to perform sensitivity analyses to examine the robustness of results to departures from the assumptions made about the missing data mechanism. Commonly researchers employ a combination of these methods. For example, principles of complete case analysis may be applied in the first instance, with participants missing a large number of items excluded from the analysis. This has often been recommended as a form of global assessment for ensuring questionnaire fidelity and to allow exclusion of poorly completed questionnaires that would impact on validity of the dietary assessment⁽ Reference Willett ^{1 )}. Subsequently, missing data may be subjected to imputation, with zero imputation a common approach⁽ Reference Parr, Hjartaker and Scheel ^{9 )}.

Prior investigations of methods for dealing with item non-response in FFQs

Very few studies have compared approaches for dealing with missing data in FFQs. Of those that have, these investigations have typically compared differences in nutrient intakes when analyses include only individuals with no missing FFQ responses with analyses that include individuals whose initially missing responses have been subsequently filled in through resurveying them up to 2 years later⁽ Reference Michels and Willett ^{28 )}. Findings from these studies tend to suggest that the common approach of assuming that non-response equates to not eating that particular food can introduce bias⁽ Reference Michels and Willett ^{28 –} Reference Fraser, Yan and Butler ^{30 )}. One study examined a sub-sample of participants followed up 3 months after the initial FFQ to recover missing data in order to determine the influence of assuming that non-response equated to no consumption. Nutrient intake was derived using an eighty-four-item FFQ and results showed that average energy intake, among other dietary measures, increased following resurvey⁽ Reference Ahn, Paik and Ahn ^{29 )}. A second study followed up a sub-sample of participants approximately 1 year after the initial FFQ and used a guided multiple imputation approach to derive complete data for those with missing observations for the eighty foods considered. This study showed that typically the resurveyed values of those who had missing data for commonly consumed foods tended not to be zero so the potential for bias may be greater if focusing on these foods⁽ Reference Fraser, Yan and Butler ^{30 )}. Both studies highlighted that their findings were influenced by the number of items that had missing responses, with one noting that the greater the number of missing food items, the greater the likelihood that they truly reflected no consumption of that item⁽ Reference Fraser, Yan and Butler ^{30 )}; while the other noted that the greater the number of items missing, the higher the increase in average energy intake⁽ Reference Ahn, Paik and Ahn ^{29 )}. A third study involving a sixty-item FFQ noted that 37 % of the items that were initially missing were subsequently reported to be consumed at least monthly and so assuming no consumption would not be accurate for these participants⁽ Reference Michels and Willett ^{28 )}. However, the authors concluded that the assumption of zero intake was reasonable if examining nutrient intake expressed as a percentage of energy. One drawback to these comparison studies is that they assume that resurveying participants at a later time point captures the true consumption at the earlier time point. This assumption may not be valid in practice given the well-known difficulties associated with accurately recalling dietary intake⁽ Reference Kipnis, Subar and Midthune ^{31 )}. It is therefore difficult to use these studies as a basis to infer the likely impact of assuming item non-response equates to zero consumption of the food item in question, although re-contacting participants may at least provide an idea of the direction of potential biases (i.e. whether participants with missing data were more likely to have consumed those foods).

To date, Parr et al.⁽ Reference Parr, Hjartaker and Scheel ^{9 )} have provided the most comprehensive comparison of methods for dealing with item non-response in FFQs. In addition to recovering missing data by resurveying participants with missing data 3 months after they were originally surveyed, they compared various single imputation approaches for dealing with missing FFQ data, including zero imputation, mode imputation, median imputation and k nearest neighbours imputation. The authors were particularly interested in considering the implications of zero imputation as their review of the literature suggested that this was the most common method used when dealing with item non-response. In the context of the Norwegian Women and Cancer study, Parr et al. concluded that assuming no consumption of a food item for which no response was provided may lead to underestimation and misclassification of the dietary intake (measured as the calculated intakes of food groups and a range of nutrient intakes, rather than dietary patterns or quality) of study participants as the resurvey values suggested that it was not accurate to assume zero consumption of missing FFQ items. However, a major limitation of the study, acknowledged by the authors, is that they were unable to conclude which imputation method provided the most accurate estimate of consumption since they did not know the true values for the missing data. Thus, that study is unable to provide guidance on the optimal approach to address item non-response. Although the authors noted that an evaluation of imputation methods required a simulation study with a complete data set as reference, to our knowledge, no study of this type has been conducted. Furthermore, the authors did not examine the influence of different missing data assumptions on exposure and outcome associations involving FFQ measures. Therefore, the effect of these assumptions in epidemiological studies in practice is unclear.