The health benefits of consuming dairy foods, a major source of Ca( 1 ), are well documented in the scientific literature( Reference Miller, DiRienzo and Reusser 2 , Reference Heaney 3 ). Adequate intake across the life cycle is beneficial for the control of blood pressure( Reference Miller, DiRienzo and Reusser 2 ), reduction in cardiovascular mortality( Reference van der Pols, Gunnell and Williams 4 ) and reduced risk of osteoporosis( Reference Moore, Bradlee and Gao 5 ). Despite the importance of an adequate Ca intake, evidence consistently demonstrates that many individuals, and particularly women, have difficulty achieving dietary dairy/Ca recommendations( Reference Bannerman, Magarey and Daniels 6 – Reference Briefel and Johnson 9 ). National survey data from Australia, the UK and the USA report mean daily intakes for adult women of 663, 682 and 756 mg, respectively, and for men of 827, 860 and 962 mg, respectively, compared with an estimated average requirement of 840 to 1100 mg/d depending on age( 10 ).
In order to identify those at risk of suboptimal Ca intake in Western populations it is necessary to accurately assess dairy/Ca intake. Traditional methods of dietary assessment (24 h recalls, food records) are burdensome and/or costly to administer as a screening tool for application at either the population or individual clinical level( Reference Willett 11 ). Thus an ideal method would be a short, easy-to-administer tool. A key criterion that supports the use of a tool in practice and research is relative validity, or its ability to accurately measure what it purports to measure, determined by how closely the results match those of a reference test( Reference Jones 12 , Reference Lang and Secic 13 ). Ideally, validity is tested using sensitivity (the ability of a test to correctly identify true positives) and specificity (the ability of a test to correctly identify true negatives); or when using continuous data a measure of agreement such as Bland–Altman analysis( Reference Lang and Secic 13 ). Tools should also have good reliability such that they produce consistent results when performed under similar circumstances, either over different time points or when conducted by different researchers( Reference Lang and Secic 13 ).
The present paper is the second of two reviews with the overall aims to: (i) identify published tools that estimate dairy and/or Ca intake and allow classification of individuals according to whether intake requirements are met or not; and (ii) assess the testing of tool properties in order to recommend a tool(s) for use. The first paper focusing on tools for children and adolescents is published( Reference Magarey, Yaxley and Markow 14 ). The current paper focuses on tools developed for use with adults.
Methods
A comprehensive search was completed to identify existing tools that measure dairy and/or Ca intake. The search was conducted using the databases MEDLINE, Scopus, Ovid, Informit and Web of Knowledge, with the keywords ‘calcium’, ‘dairy’, ‘milk’, ‘diet’, ‘nutrition’ and ‘food’, combined with ‘tool’, ‘questionnaire’, ‘FFQ’, ‘survey’, ‘measurement’, ‘assessment’, ‘evaluation’ and ‘analysis’. The search was not limited by dates, but databases were searched from their year of inception, the earliest being 1948 in the case of MEDLINE, to February 2013. The search was limited to English-language papers only. In addition to this search strategy, an identical search was conducted in Google Scholar to identify any relevant tools or papers in the grey literature. Additional articles were identified by searching the reference lists of the articles found in the database searches.
The database searches identified 1113 articles which reduced to 1022 when duplicates were removed. These were screened for relevance, resulting in the identification of 121 potentially relevant articles. This was followed by a second screening phase which identified forty-eight articles that met the following inclusion and exclusion criteria. Articles that discussed (i) developing or testing the reliability or validity of a previously unpublished tool to measure dairy and/or Ca intake, (ii) testing the reliability or validity of an existing tool to measure dairy and/or Ca intake and (iii) tools intended for use in Western populations, were included for review. Articles that (i) referred to tools that did not assess dairy or Ca intake, (ii) utilised existing tools but did not test these for reliability or validity in the study sample, (iii) measured dairy and/or Ca intake in non-Western countries (due to differences in the major food sources of Ca), (iv) utilised traditional whole of diet methods such as 24 h recalls, food records or diet histories to measure dietary intake, (v) were not in English or (vi) were published abstracts only, were excluded from review. Two authors (K.M., L.B.) sorted the articles independently for relevance and where disagreement arose a third author (M.M.) provided input. Where controversy remained, the relevance of the article was discussed and a final decision made regarding inclusion. A third and final screening phase identified the articles as referring to tools developed for or tested with adults (n 30) or children/adolescents (n 18).
Tools described in the articles were classified as (i) dairy assessment tools that assess the quantity or frequency of intake of dairy foods or (ii) Ca assessment tools that assess quantity or are able to classify respondents into specific categories of Ca intake. Some tools collected information on intake of dairy foods and other Ca-containing foods and were considered to be both Ca and dairy assessment tools.
When assessing reliability and validity of tools, a sample size of at least 100 subjects was considered acceptable( Reference Willett 11 ), tests of association (correlation coefficients) were considered weak statistical analysis, whereas tests that measured agreement (Bland–Altman or κ) and/or sensitivity and specificity were considered to provide strong systematic analysis( Reference Peat, Mellis and Williams 15 ). A mean difference between two administrations or between test and reference method of 100 mg (representing about 10 % of the recommended daily intake, or one-third of a serving of dairy products) was considered clinically significant. Further a κ value >0·5 was considered moderate agreement, a value >0·7 as good agreement and a value >0·8 as very good agreement( Reference Peat, Mellis and Williams 15 ).
Results
The thirty articles report on thirty-six tools that had been used in those aged 18 years or over. Four articles report on two tools( Reference Angbratt and Möller 16 – Reference Smith, Morgan and Vaughn 19 ), one article reports on three tools( Reference Sebring, Denkinger and Menzie 20 ), and another reports on what is assumed an online and paper version of the same tool but this is not clearly stated( Reference Hacker-Thompson, Robertson and Sellmeyer 21 ). Two tools( Reference Clover, Miller and Bannerman 17 , Reference Block, Woods and Potosky 22 ) are each reported in second articles( Reference Wirfält, Jeffery and Elmer 23 , Reference Miller, Yeo and Khor 24 ). Four tools assess both dairy and Ca intake( Reference Angbratt and Möller 16 , Reference Gans, Risica and Wylie-Rosett 25 – Reference Welten, Kemper and Post 27 ) and thirty-two assess Ca intake alone. Details of each of the tools are provided in Table 1. The tools were used in a range of population groups of differing age, gender, race, menopausal status, living situation, educational status and disease state.
Table 1 Summary and key features of studies describing dairy and/or calcium assessment tools utilised in adult populations
D, dairy; Q-FFQ, quantitative FFQ; QL-FFQ, qualitative FFQ; SQ-FFQ, semi-quantitative FFQ; vit/min, vitamin and mineral; NHANES II, National Health and Nutrition Examination Survey II; Vit D, vitamin D; Vit K, vitamin K; CSFII, Continuing Survey of Food Intake by Individuals; NR, not reported; est, estimated; Vit C, vitamin C.
Tool characteristics
All tools used an FFQ, with varying response options covering a variable period. Nineteen tools were quantitative( Reference Angbratt and Möller 16 – Reference Cummings, Block and McHenry 18 , Reference Sebring, Denkinger and Menzie 20 , Reference Block, Woods and Potosky 22 , Reference Wirfält, Jeffery and Elmer 23 , Reference Goldbohm, Chorus and Garre 26 – Reference Varenna, Binelli and Zucchi 35 ) allowing an estimate of milligrams of Ca, fifteen semi-quantitative( Reference Smith, Morgan and Vaughn 19 – Reference Hacker-Thompson, Robertson and Sellmeyer 21 , Reference Hertzler and Frary 36 – Reference Magkos, Manios and Babaroutsi 44 ) and two qualitative (i.e. frequency of intake of specified items)( Reference Gans, Risica and Wylie-Rosett 25 , Reference Beck and Oveson 45 ). Quantitative tools allowed varying serving sizes, the semi-quantitative tools provided a standard serving size and the qualitative tools included dairy products and other foods that make important contributions to Ca intake. In terms of food coverage, all tools included dairy products and nineteen included other foods that make an important contribution to Ca intake. One tool was designed to assess several nutrients and the foods included reflected this( Reference Beck and Oveson 45 ). The remaining tools were general FFQ that were tested for their ability to assess Ca intake.
Two tools were completed via computer( Reference Hacker-Thompson, Robertson and Sellmeyer 21 , Reference Matthys, Pynaert and Roe 30 ) and most could be self-administered (31/36). Visual aids were provided with five tools to assist respondents to identify portion size and quantify foods. Six tools were reported to take less than 15 min to complete, demonstrating an adequate user-friendliness and efficiency. Time to complete was not provided for most tools, but where sufficient information was obtainable an estimate was made based on the number of items by comparing with a comparable tool for which time to complete had been reported. Most tools required computer analysis or professional assistance to determine total daily Ca intake and/or adequacy; however, a few tools were able to provide an immediate indication of daily Ca intake.
Tool reliability
Test–retest reliability was reported for only six tools, two were dairy/Ca tools and four were Ca tools (Table 2). Inter-rater reliability was tested for one Ca tool( Reference Beck and Oveson 45 ). The statistical analyses varied, with correlation (Pearson, Spearman or intra-class) the most frequently used test. One study reported a κ value within a range for all nutrients tested( Reference Beck and Oveson 45 ) and another used cross-classification( Reference Welten, Kemper and Post 27 ). The tools were mostly tested in samples of less than 100 with only two tools tested in a sample of 100 or greater( Reference Miller, Yeo and Khor 24 , Reference Hertzler and Frary 36 ). The period between the two administrations of the tool varied from a minimum of 4 d( Reference Beck and Oveson 45 ) up to 1 year( Reference Welten, Kemper and Post 27 ), with most being 2–3 weeks.
Table 2 Details of reliability testing of tools that assess dairy and/or calcium intake in adults
D, dairy; n, sample size; P, Pearson’s correlation; NR, not reported; S, Spearman’s correlation; ICC, intra-class correlation; CC, cross-classification; κ, kappa coefficient.
* Inter-rater reliability.
Welten et al. provided the most comprehensive range of tests and these suggested moderate to good reliability (mean difference of 80 mg Ca, Pearson’s correlation of 0·78, exact agreement of 62·1 % and gross misclassification of 3·4 %)( Reference Welten, Kemper and Post 27 ). Miller et al. had comparable moderate intra-class correlation values across the two versions of their FFQ (thirty-five-item and fifteen-item; r=0·5 and r=0·6, respectively)( Reference Miller, Yeo and Khor 24 ) but did not report any findings from additional tests. The other three studies testing reliability reported only correlations and these were moderate to high. Inter-rater reliability tested by Beck et al. using the κ statistic showed a good level of agreement (κ=0·81 to 0·88)( Reference Beck and Oveson 45 ).
Tool validity
Twenty-six articles reported tests of relative validity, on four dairy/Ca tools( Reference Angbratt and Möller 16 , Reference Gans, Risica and Wylie-Rosett 25 – Reference Welten, Kemper and Post 27 ) and thirty-four Ca tools (Table 3), using an array of common reference methods. Sixteen studies used an estimated food record ranging in length from 3 to 14 d, three studies used multiple 24 h recalls, two used a general FFQ, two used diet histories and two used a single 24 h recall. The sample sizes of the studies varied greatly, ranging from fifteen subjects( Reference Pritchard, Seechurn and Atkinson 33 ) to 2414 subjects( Reference Severo, Lopes and Lucas 43 ) with 12/26 having a sample size less than 100. A range of statistical tests were performed, including correlation, comparison of mean values, Bland–Altman analysis, agreement using κ, cross-classification and assessment of sensitivity and specificity (Table 3). While correlation values may be moderate to high, this analysis tests only association. Ideally tool validity should be assessed by tests of agreement such as sensitivity and specificity, the κ statistic or Bland–Altman analysis( Reference Peat, Mellis and Williams 15 ).
Table 3 Details of validity testing of tools that assess dairy and/or calcium intake in adults
D, dairy; n, number; S, Spearman’s correlation; P, Pearson’s correlation; T, paired t test; CC, cross-classification; BA, Bland–Altman; LOA, limits of agreement; κ, kappa coefficient; SE, sensitivity; SP, specificity.
The four dairy/Ca tools identified in the present review were tested for relative validity. Two reported non-significant mean differences between the tool and reference method of less than 100 mg Ca( Reference Angbratt and Möller 16 , Reference Goldbohm, Chorus and Garre 26 ) but reported no other tests. Only Welten et al. conducted higher-level tests and reported moderate κ values for both dairy foods and Ca (0·60–0·76)( Reference Welten, Kemper and Post 27 ). However, results of the Bland–Altman analysis conducted in the same study indicated only 52 % exact agreement and very wide limits of agreement (±1000 mg) indicating that this tool would not perform well at the individual level( Reference Welten, Kemper and Post 27 ).
A majority of the Ca tools (22/26) were tested for relative validity. Virtually all studies reported correlation between the tool and reference method, with five studies reporting no additional tests( Reference Cummings, Block and McHenry 18 , Reference Smith, Morgan and Vaughn 19 , Reference Blalock, Currey and DeVellis 28 , Reference Musgrave, Giambalvo and Leclerc 32 , Reference Hertzler and Frary 36 ). Due to the limited value of correlation tests no further discussion of these results is provided although values are reported in Table 3.
Sensitivity and specificity were calculated by seven studies( Reference Clover, Miller and Bannerman 17 , Reference Montomoli, Gonnelli and Giacchi 31 , Reference Szymelfejnik, Wdoowska and Cichon 34 , Reference Hung, Hamidi and Riazantseva 37 , Reference Plawecki, Evans and Mojtahedi 41 , Reference Magkos, Manios and Babaroutsi 44 , Reference Beck and Oveson 45 ) (Table 3). While sensitivity values ranged from 71 %( Reference Hung, Hamidi and Riazantseva 37 ) to 95 %( Reference Beck and Oveson 45 ), specificity ranged from 46 %( Reference Clover, Miller and Bannerman 17 ) to 97 %( Reference Beck and Oveson 45 ). Ideally both the sensitivity and specificity for a screening tool would be high; however, only two studies reported both sensitivity and specificity to be greater than 80 %( Reference Montomoli, Gonnelli and Giacchi 31 , Reference Beck and Oveson 45 ).
Cross-classification statistics, which identified the percentage of subjects correctly classified by the tool into quartiles or tertiles of Ca intake, were reported by eight studies( Reference Wirfält, Jeffery and Elmer 23 , Reference Welten, Kemper and Post 27 , Reference Matthys, Pynaert and Roe 30 , Reference Pritchard, Seechurn and Atkinson 33 , Reference Johansson 38 , Reference Osowski, Beare and Specker 39 , Reference Pasco, Sanders and Henry 40 , Reference Severo, Lopes and Lucas 43 ) (Table 3). Osowski et al. reported the lowest correct classification of only 33%( Reference Osowski, Beare and Specker 39 ) while Severo et al. reported the highest agreement between methods of 89%( Reference Severo, Lopes and Lucas 43 ). Gross misclassification, defined as classification of Ca intake by the tool in the opposite quartile or tertile of intake, ranged from 0 % as reported by Pritchard et al.( Reference Pritchard, Seechurn and Atkinson 33 ) to 8 % reported by Matthys et al.( Reference Matthys, Pynaert and Roe 30 ).
Three studies calculated the κ statistic for the level of agreement between the two methods, Matthys et al. reported κ=0·20( Reference Matthys, Pynaert and Roe 30 ), Pasco et al. reported κ=0·40( Reference Pasco, Sanders and Henry 40 ) and Severo et al. reported κ=0·75( Reference Severo, Lopes and Lucas 43 ).
A greater number of studies used Bland–Altman plots to illustrate the level of agreement( Reference Clover, Miller and Bannerman 17 , Reference Sebring, Denkinger and Menzie 20 , Reference Hacker-Thompson, Robertson and Sellmeyer 21 , Reference Montomoli, Gonnelli and Giacchi 31 , Reference Pritchard, Seechurn and Atkinson 33 , Reference Hung, Hamidi and Riazantseva 37 , Reference Pasco, Sanders and Henry 40 , Reference Plawecki, Evans and Mojtahedi 41 , Reference Severo, Lopes and Lucas 43 , Reference Magkos, Manios and Babaroutsi 44 ). The mean bias between the tool and reference method ranged from +5 mg/d( Reference Clover, Miller and Bannerman 17 ) to +576 mg/d( Reference Pritchard, Seechurn and Atkinson 33 ). Limits of agreement varied widely between studies extending from ±233 mg( Reference Montomoli, Gonnelli and Giacchi 31 ) to ±1254 mg( Reference Pritchard, Seechurn and Atkinson 33 ).
Discussion
The present review identified thirty papers using thirty-six tools that met the criteria for inclusion; four tools that assessed both dairy and Ca intake and thirty-two that assessed Ca intake only. Based on the review of methods used and results of the reliability and validity testing, one tool for assessing dairy and Ca intake( Reference Welten, Kemper and Post 27 ) and five tools for assessing Ca intake are recommended( Reference Clover, Miller and Bannerman 17 , Reference Sebring, Denkinger and Menzie 20 , Reference Hacker-Thompson, Robertson and Sellmeyer 21 , Reference Montomoli, Gonnelli and Giacchi 31 , Reference Severo, Lopes and Lucas 43 ) (Table 4). While appropriate testing methods for relative validity and adequate levels of sensitivity, specificity and/or agreement were reported for these tools, only two( Reference Clover, Miller and Bannerman 17 , Reference Welten, Kemper and Post 27 ) were tested for test–retest reliability which was shown to be moderate( Reference Miller, Yeo and Khor 24 , Reference Welten, Kemper and Post 27 ).
Table 4 Final recommendations for dairy and/or calcium tools that are well validated for implementation in the practice and research setting in adults
* Also tested for reliability.
†Tested for reliability in the study by Miller et al. (2010)( Reference Miller, Yeo and Khor 24 ).
The common limitations of the testing of tool properties were the lack of testing for reliability, the high reliance on correlation which assesses association only, and the lack of tests that provide a measure of agreement. In addition when assessing validity it is important to determine a clinically meaningful level of significance as opposed to relying on statistical significance alone. None of the papers defined a level of clinical significance at which the results were meaningful in terms of dietary adequacy. This lack of recognition between statistically and clinically significant results limits conclusions relevant to clinical practice. In order to define clinically meaningful results we applied a 100 mg Ca cut-off for bias when assessing studies, or approximately the amount of Ca that might be delivered by one-third of a standard serving of dairy foods.
The lack of testing for reliability limits the ability to be confident in recommending a tool for use. In addition, relative validity results for those tools that assessed Ca vary such that some should be considered with caution while others appear to have acceptable levels of agreement and/or sensitivity and specificity. With respect to Ca tools, those that appear to be best in levels of relative validity are those developed by Clover et al.( Reference Clover, Miller and Bannerman 17 ), Montomoli et al.( Reference Montomoli, Gonnelli and Giacchi 31 ), Hacker-Thompson et al.( Reference Hacker-Thompson, Robertson and Sellmeyer 21 ), Sebring et al.( Reference Sebring, Denkinger and Menzie 20 ) and Severo et al.( Reference Severo, Lopes and Lucas 43 ). Each of these studies included a minimum of 100 participants, considered to be the smallest acceptable sample size for a validation study( Reference Willett 11 ), had a sensitivity and specificity of >80 %, or a Bland–Altman mean bias of <100 mg, or a κ statistic >0·80, or a correct classification of >80 %. One exception is that Clover et al. reported a specificity of <80 %, but importantly this was the only one of these five tools that was tested for reliability( Reference Miller, Yeo and Khor 24 ).
There are some additional limitations to the findings presented here, in particular the quality of the study design. The key study design criteria include level of evidence, potential sources of error and bias, and sample size. These have been discussed in the companion paper and the issues identified there equally apply to the adult studies in the current paper( Reference Magarey, Yaxley and Markow 14 ). In brief, all eligible papers were identified as having III-2 level of evidence, as defined by the National Health and Medical Research Council evidence hierarchy for diagnostic accuracy( 46 ), and there was potential for recall bias, positive respondent bias and recruitment bias. Many of the studies reported here targeted specific populations and thus when selecting a tool for use it is important to consider the population in which the tool properties were tested. Validity in one population does not guarantee validity in another population of different age, gender or physiological state.
Conclusion
In conclusion, based on the present review we recommend one tool for assessing dairy and Ca intake and five tools for assessing Ca intake. However, these should be considered cautiously as there are inherent limitations to all the reported studies suggesting they may not perform as well if tested using a study design of a higher level and this should be considered in future application of the tool. Further, when selecting a tool for use the relevance of the tool items to the food culture of the target population should be considered. The present literature review has identified gaps in the literature which may inform future research. Overall few tools were tested for reliability; therefore further research should be conducted to ensure that other Ca or dairy tools are adequately reliable for use.
Acknowledgements
Financial support: This review was supported by Dairy Australia Inc. The funder had no role in the analysis or writing of this article. Conflict of interest: None. Authorship: A.M., A.Y. and M.M. designed the study, oversaw the implementation, checked data extraction, contributed to the writing of the manuscript and commented on all drafts. K.M. and L.B. undertook the literature search, extracted all of the data, contributed to writing of the manuscript and commented on all drafts. Ethics of human subject participation: Ethical approval was not required.
