The prevalence of overweight and obesity in school-aged children is estimated to be 10 % worldwide and increasing(Reference Lobstein, Baur and Uauy1). Obesity is recognised as a rapidly growing threat to the health of populations in an increasing number of countries around the world(2), placing significant burden on health-care systems. Prevention is recognised as the most realistic and cost-effective strategy to deal with childhood obesity(Reference Lobstein, Baur and Uauy3).
While positive energy balance leads to the accumulation of excess weight, the aetiology of obesity is complex and dependent on more than just biology. For example, increased energy intake and/or decreased energy expenditure is commonly entwined with environmental factors, across multiple settings(Reference Dietz and Gortmaker4).
There has been a call for community-based obesity interventions as a strategy for prevention of childhood obesity(2). Community-based interventions recognise the depth of community understanding held by members and their knowledge of community resources and dynamics(Reference Economos and Irish-Hauser5). These provide the foundation when designing and delivering interventions, including choice of settings and strategies. Furthermore, individual behaviours are only sustained if they are carried out in an environment that supports healthy choices(Reference Economos and Irish-Hauser5). In the case of obesity, this means that individual behaviours associated with excess weight gain (including healthy eating and physical activity) must be addressed in the context of the environment and the societal and cultural factors relevant to the individual(Reference Caballero6).
Despite the recognition that multi-setting, multi-strategy community-based action should be the foundation of obesity prevention efforts(Reference Bell7), there are few published examples of such interventions(Reference Economos and Irish-Hauser5). The majority of childhood obesity prevention interventions are based in single settings, predominantly schools(Reference Summerbell, Waters and Edmunds8). Therefore, there is a clear need to develop an evidence base of effective community-based obesity prevention interventions. Historically, insufficient priority has been placed on appropriate evaluation designs or insufficient funding provided to allow rigorous evaluation(Reference Swinburn, Bell and King9). Further evaluation of community-based interventions is hindered by the complexity of communities(Reference Swinburn, Bell and King9), the inability to reproduce controlled environments often used in medical-based research and the lack of setting-specific tools appropriate to specific target groups that measure the outcome, process and impact of interventions.
The eat well be active (ewba) Community Programs in South Australia (SA) are community-based childhood obesity prevention interventions which address environmental and individual barriers to behaviour change through a portfolio of strategies in a range of settings (Fig. 1). The Programs are funded by SA Health (government) for 5 years, in response to the rising levels of childhood overweight and obesity and the need to obtain evidence of the effectiveness of community-based obesity prevention. The Programs aim to promote healthy weight in children and young people aged 0–18 years, and their families, through increasing healthy eating and physical activity behaviours. The ewba Community Programs have a rigorous evaluation framework that will contribute to the evidence regarding best practice for community-based childhood obesity prevention programmes. This framework includes quantitative and qualitative evaluation of process, impact and outcome elements and has a large scope across multiple population groups. The framework was designed to reflect the ewba interventions that were developed from a combination of best available evidence and extensive community consultation.
Fig. 1 Key messages, strategies and settings of the eat well be active Community Programs
The present paper focuses on the rationale, development and implementation of the baseline data collection in schools, a major component of the quantitative evaluation of the ewba Community Programs. It identifies a number of challenges experienced during this process and suggests some solutions.
Methods
Rationale
Structure of the ewba Evaluation Academic Team
The ewba Evaluation Academic Team has two internal members; the ewba Evaluation Coordinator (responsible for coordinating and compiling data collection and evaluation processes) and the ewba Manager who oversees this process; and three external academics who advise the team in nutrition, physical activity, childhood obesity and community development. Additional research staff (e.g. a PhD Candidate) are also members of this team. The internal members of the Evaluation Academic Team work closely with the Project Coordinators who implement the project; however these Project Coordinators are not part of the Evaluation Team.
Selection of intervention and comparison sites
The intervention including the metropolitan suburb of Morphett Vale in southern Adelaide and the Rural City of Murray Bridge were selected by SA Health and the community health services implementing the projects, in consultation with community stakeholders, based on their high levels of disadvantage and the presence of existing infrastructure and experience necessary to support the Programs(Reference Jones, Magarey and Dollman10). Two comparison communities (metropolitan suburbs of the Sea and Vines Department of Education and Children’s Services (DECS) district and the Port Pirie Regional Council Area) were selected to match the sociodemographic profiles of the intervention sites as closely as possible, including: number and age distribution of children, socio-economic status as measured by the Index of Relative Social Disadvantage (IRSD), educational levels, occupational and income distributions, family sizes, ethnic mix and rural–urban mix(Reference Jones, Magarey and Dollman10). The IRSD provides an indication of socio-economic disadvantage by ranking different geographic areas of Australia according to a ‘score’ that is created for the area based on characteristics of people, families and dwellings within that area(Reference Pink11).
Selection of samples for evaluation
Middle and upper primary-school children in school years five to seven (10–12-year-olds) were chosen as one litmus group for evaluation because: (i) there is a significant dose of ewba intervention delivered through the school setting; (ii) middle and upper primary-school children are cognitively able to complete simple written questionnaires and self-report dietary(Reference Baranowski and Domel12) and physical activity habits(Reference Welk, Corbin and Dale13); and (iii) the primary-school curriculum is reasonably able to accommodate the time required for data collection. School students, their parents, school principals, teachers, canteen and out-of-school-hours care (OSHC) managers at intervention and comparison schools were invited to participate in the evaluation.
Selection of anthropometric outcomes for measurement
Outcomes were chosen in consultation with the ewba Evaluation Team. Height and weight were measured and BMI was calculated. Waist circumference was taken as a surrogate measure of central abdominal adiposity based on the following rationale. First, waist circumference is an indirect measure of central adiposity, which is strongly correlated with risk for CVD in adults(Reference Ross, Fortier and Hudson14) and an adverse lipid profile and hyperinsulinaemia in children(Reference Freedman, Serdula and Srinivasan15). Second, children’s waist circumference correlates well with computerised tomography scan as a measure of subcutaneous abdominal adipose tissue (r = 0·93), and fairly well with intra-abdominal adipose tissue (r = 0·84)(Reference Goran, Gower and Treuth16). Third, waist circumference is easy to measure with simple, low-cost equipment, has low observer error, offers good reliability, validity and low measurement error(Reference Power, Lake and Cole17) and has been used as a measure of child central adiposity in similar obesity prevention projects in Australia(Reference Sanigorski, Bell and Kremer18).
Development of tools for quantitative ewba evaluation
The key messages and objectives of ewba (Fig. 1) informed the selection of the nutrition and physical activity outcomes to be measured. These outcomes included behaviours and also attitudes, knowledge and environments, which influence these behaviours. It was acknowledged that traditional methods of dietary and physical activity assessment (e.g. diet diaries and accelerometers) do not provide insight into attitudes, knowledge and environments but such information is invaluable to understand the obesity epidemic.
Due to the lack of tools in the published literature that addressed both the breadth of enquiry of ewba (Fig. 1) and the specific project goals, more contemporary evaluation questions and methods that encompassed these factors were specifically developed through consultation and review. Several unpublished questionnaires not yet tested for validity or reliability from similar interstate projects helped inform the content of these questionnaires. Self-report questionnaires were chosen based on cost and time effectiveness, lower respondent burden and the age range of the sample (10–12 years) being appropriate for self-reporting of dietary and physical activity behaviours(Reference Baranowski and Domel12, Reference Welk, Corbin and Dale13).
The seven questionnaires (Table 1) measure the obesogenicity of one or more of the home, school and community environments. This enables triangulation of data to test for agreement between reports (e.g. student and parent report). Four types of environments were investigated in all settings – political, physical, financial and sociocultural(Reference Swinburn, Egger and Raza19).
Table 1 The seven Program-specific eat well be active questionnaires for evaluation purposes in intervention and comparison sites
HE, healthy eating; PA, physical activity; OSHC, out-of-school-hours care.
*Physical, political, socio-cultural and financial environmentsReference Sanigorski, Bell and Kremer(18).
†Not all schools have these facilities.
All seven questionnaires were piloted with small convenience samples of the relevant groups, in areas external to the ewba sites, prior to their use. The child nutrition questionnaire has been shown to be valid and reliable(Reference Wilson, Magarey and Mastersson20) and can be accessed from http://www.ijbnpa.org/content/5/1/5. The child physical activity, teacher and parent questionnaires are currently being assessed for validity and reliability.
Implementation
Selecting and contacting schools
The process and timelines for contacting schools regarding participation in the ewba intervention and collection of baseline data are outlined in Fig. 2.
Fig. 2 Process used for contacting schools about the ewba intervention and baseline data collection
All government, catholic and independent primary schools in the intervention sites were invited to participate in the ewba intervention and evaluation. Those in comparison sites were invited to participate in the evaluation only. Schools in the intervention sites were offered the portfolio of ewba strategies that promoted healthy eating and physical activity and individually chose which strategies they each implemented throughout the intervention period. As a benefit of participation in data collection, both intervention and comparison schools were provided with aggregate information from all participating schools on students’ nutrition, physical activity and standardised BMI, and summarised policy and practice patterns.
Staff training
A team of fourteen staff were recruited and trained to collect the baseline data (Fig. 2). All staff attended a 1 d training session run by the ewba Evaluation Academic Team. This included training in body image sensitivity(Reference Gibbs, O’Connor and Waters21), measurement of weight, height and waist circumference, and description and practical run-through of the student questionnaires.
Staff were also trained to use three other resources including a standard preamble to introduce the data collection process and to provide instruction on completion of the student questionnaires, a poster depicting serve sizes of fruit and vegetables to assist students with portion size estimation in the nutrition questionnaire and a series of visual aids to assist students distinguish between organised and non-organised activities in the physical activity questionnaire.
Obtaining parental consent and child assent
All year five, six and seven students in participating schools were invited to participate in data collection and consent forms were sent to participating schools to distribute to parents (Fig. 2). Sample size calculations indicated that a response rate of 60 % would enable a 20 % change in prevalence of a range of nutrition and physical activity behaviours to be detected with 80 % power and alpha 0·05. Students returned consent forms with parental consent and child assent to their teachers and these were collected on the day of data collection. Students could consent to and participate in questionnaires but not anthropometric measures and vice versa, and they could withdraw from participation at any time. Verbal parental consent was accepted over the telephone on the day of measurement, provided the completed consent form was returned to ewba at a later date. Due to a low return rate of consent forms in the first eleven schools measured in School Term 3, all students returning a consent form (regardless of their consent to the measures) in the subsequent twenty-eight schools measured in School Term 4 were offered a small gift (a hacky sack), provided the school gave permission.
Baseline data collection
The process of baseline data collection is outlined in Fig. 3.
Fig. 3 Process of ewba baseline data collection in schools
Questionnaires. Prior to distribution of the questionnaires, the Measurement Team Leader read out the standard preamble to the whole student group. To avoid any systematic bias resulting from good student concentration during completion of the first survey and poorer concentration during completion of the second, the order in which the questionnaires were administered was varied between classes. Consequently, this effect (if present) was distributed across the two surveys. Students completed the questionnaires independently with the exception of two more complex questions in each questionnaire. These were led by the team leader with the whole student group, as piloting identified that students required extra assistance with these questions(Reference Wilson, Magarey and Mastersson20). Staff referred to posters depicting fruit and vegetable serve sizes, and organised and non-organised activities, when necessary.
Anthropometric measurements. The decision to take anthropometric measurements from children was carefully considered by the ewba Management Committee and Evaluation Team. In addition, informal consultation conducted with local stakeholder groups indicated majority support for the measurements to be taken whilst keeping in mind body image concerns.
A station for anthropometric measurements was set up with a set of scales (model HD319; Tanita, Tokyo, Japan), stadiometer (model PE087; Mentone Educational Centre, Melbourne, Australia) and tape measure for waist measurements (model W606PM; Lufkin, Albury, Australia). In the larger schools and when there was sufficient measurement staff, two stations were set up. If a separate room was not provided for anthropometric measures, the equipment was set up behind a screen in the same room as questionnaires were administered. In this case, students were out of view of other students when measured, but could still be in view of the observing teacher. One male and one female staff member were present to take measurements. Particular steps were taken to minimise body image concerns, based on the work of Gibbs et al.(Reference Gibbs, O’Connor and Waters21) (Table 2).
Table 2 Body Image Protocol used in the eat well be active Community Programs(Reference Wilson, Magarey and Mastersson20)
ewba = eat well be active.
Students’ names were called out and they were withdrawn temporarily from completing the questionnaires. Height, weight and waist circumference were measured without shoes using the protocols recommended by the International Society for the Advancement of Kinanthropometry (ISAK)(Reference Norton, Whittingham and Carter22). Waist circumference was measured at the level of the visible narrowing of the waist and at end-tidal expiration(Reference Jones, Magarey and Dollman10). All measurements were taken twice; a third was taken if the difference between the two measurements was too great (height: >5 mm; waist: if the difference exceeded 2 % of the lower of the two scores; weight: if there was a 1 % (or greater) difference between the first and second readings)(Reference Norton, Whittingham and Carter22). The mean of two and median of three measures were taken as the final score. All measurements were recorded on a standard record sheet. Inter-tester technical error of measurement (TEM) and intra-tester TEM were calculated for five of the measurers and shown to be well within acceptable ISAK standards(Reference Jones, Magarey and Dollman10, Reference Norton, Whittingham and Carter22).
Distribution of other questionnaires
Parent. On the consent form for child measurements, parents were asked to indicate if they were happy to complete a parent survey and if so to provide their home address. Questionnaires were sent by ewba home to parents who returned them directly to ewba via reply-paid envelopes. Parents who returned a completed questionnaire went into the draw to win one of twenty A$25 shopping vouchers.
School. Teacher, principal, OSHC and canteen manager questionnaires were sent to the schools prior to data collection (Fig. 2). Ideally, the completed questionnaires were collected on the measurement day (Fig. 3) and if they had not been completed, extra copies were provided with a request to return to ewba as soon as possible. The five schools with the highest return rate of teacher and student questionnaires were offered an A$100 voucher for sports equipment, a water cooler or fruit and vegetables.
Data entry and analysis
As cost prohibited all data to be entered twice and checked for agreement (double data entry), a random sample of 10 % of child nutrition, physical activity, teacher and parent questionnaires were checked by the ewba team. The scoring system used to assess validity and reliability of the child nutrition questionnaire(Reference Wilson, Magarey and Mastersson20) was used to analyse the data from this questionnaire, and a similar scoring system was developed for the physical activity questionnaire. Target scores were developed for the nutrition(Reference Wilson, Magarey and Mastersson20) and physical activity questionnaires and these were used to report meaningful frequency data at baseline. Frequency data on responses from parent, teacher, principal, canteen and OSHC questionnaires were also reported. SPSS version 12·0·1 was used to analyse data.
Ethics approvals
Ethics approval was granted from the SA Health Human Research, the DECS and the South Australian Aboriginal Health Research Ethics Committees.
Results
Consent and response rates
Of the forty-four primary schools in intervention and comparison sites, thirty-nine agreed to participate (89 % acceptance rate)(Reference Jones, Magarey and Dollman10). The five schools that declined to participate were in comparison sites.
Table 3 shows response, consent and completion rates for the student questionnaires and anthropometry; and parent, teacher, principal, OSHC and canteen questionnaires. For students, consent rate is different from completion rate due to absences on the day of survey. The number of students returning the consent form where either the student or parent did not assent/consent to either questionnaires and/or measurement was 262. One hundred and ten students who had consented were absent on the day of survey. Twenty-three parents provided verbal consent on the day of survey. Table 4 shows the difference in overall response, consent and completion rates in the schools that received a gift for return of surveys (twenty-eight schools) compared with those who did not (eleven schools). Response rates were significantly higher in schools receiving the gift (P < 0·001), and so were consent and completion rates (P < 0·05). One school chose not to take up the option of the gift.
Table 3 Rates of return for the student, principal, out-of-school-hours care, canteen, parent and teacher questionnaires and student participation in the anthropometric measurements for baseline data collection of the eat well be active Community Programs
PA, physical activity; OSHC, out-of-school-hours care.
*Not applicable: completion of questionnaire taken as consent – no extra consent form required.
†Returned a consent form, regardless of whether consented to questionnaires and/or anthropometric measures (could return a form and not consent).
‡Returned a consent form and consented to questionnaires and/or anthropometric measures.
§Questionnaires were completed and/or anthropometric measures were taken on the day of survey.
Table 4 Completion, consent and response rates to surveys and anthropometric measures by students receiving a small gift for return of consent form compared with those not receiving a gift
*P < 0·05; **P < 0·001.
Data entry
Ten per cent of the child nutrition, child physical activity and teacher questionnaires were checked for data entry errors (173, 173 and twenty-nine questionnaires respectively). The error rates (expressed as number of items with an error per total number of items in one questionnaire) were found to be 0·3 %, 0·5 % and 0·17 % respectively. Twelve per cent of parent questionnaires were checked (121 questionnaires). The error rate was found to be 0·69 % with three questions regarding higher errors than any others. Exclusion of these three questions reduced the error value to 0·48 %. These three questions were checked in all 983 questionnaires. The entire anthropometry data file was checked for errors and amended accordingly. Eleven students were excluded due to incomplete or missing data.
Discussion
The present paper describes the rationale, development and implementation of the quantitative baseline data collection in schools, which is one component of the evaluation of the ewba Community Programs. It focuses on the questionnaires and anthropometric measures taken in 10–12-year-old school students. This discussion will demonstrate the numerous logistical, practical and methodological challenges met during the data collection process, and will also consider solutions to these challenges, which provide learnings for other researchers involved in evaluation of similar programmes. To avoid biasing the results of the ewba evaluation, such adaptations will not be introduced in the present study at post-intervention data collection.
There were large variations between schools in terms of daily timetable, class structure and school dynamics, and it was necessary to ensure that data collection slotted into the schools’ schedules as seamlessly as possible. Similar projects should obtain school schedules as early as possible and ensure that their data collection process is flexible enough to fit in with differing school schedules.
Similarly, to ensure maximum return of consent rates and hence sample size, consent forms were not collected by ewba until the day of survey. Hence decisions that were based on the number of children consenting could not be made until the day of survey. For example, if consents were low, it was less disruptive to combine students from multiple classes with consent in a separate space to complete questionnaires, while when consents were high, it was more practical to visit each class separately and students without consent were kept busy with another task. It is important that researchers in similar studies allow enough time and flexibility in their schedule to allow such options.
Data collection was performed late in School Term 3 and early in Term 4 (September to November) 2006. Term 4 is recognised as a busy time for schools and this could have had an impact on their response to data collection. If possible, researchers should avoid data collection in Term 4, particularly towards the end. However, the short period over which data were collected was ideal as it limited any seasonal changes in nutrition and physical activity behaviours that were independent of the programme.
The ability of teachers to control student behaviour varied and in some cases ewba staff were required to assist with behaviour management, making it more difficult to complete other tasks. Some students had very low literacy levels and often required a staff member to guide them through the questionnaires. This was resource intensive and left fewer staff to assist the remainder of the class with queries. In the early stages of data collection, it became apparent that students in year five generally required more assistance to complete questionnaires than those in years six or seven. Consequently, larger teams (four to five people) were allocated to attend classes with year five students. It is important that researchers in similar studies assess whether behavior management and literacy problems are likely to be an issue and if so, provide sufficient staff members to handle such situations effectively.
Designation of a suitable space for data collection by the school often proved problematic. In particular, anthropometric measures had to be taken in an area that was not secluded, but private enough for children to feel comfortable and not vulnerable. It is important that schools receive clear guidelines about the types of spaces that are and are not appropriate for such measurements. In case a suitable space is not available, the research team needs to be prepared with a backup, such as a screen that can be used as a privacy shield.
It is important to allow sufficient time between recruitment of measurement staff and schools and the commencement of measures (Fig. 2). In terms of measurement staff, sufficient time ensures detailed training around the questionnaires, in particular how to respond to specific student queries, ensuring consistent responses across team members. However, an in-depth understanding of the types of questions asked by students only develops over time by undertaking the data collection process. This highlights the importance of extensively piloting questionnaires where possible. In the case of time between recruitment of schools and measures, in some cases, there was only 6–8 weeks between the first letter schools received from ewba and data collection (Fig. 2). This may explain why the target completion rate of 60 % was not reached. If possible, similar studies should allow more time between first contacting schools and data collection. This would allow greater flexibility with dates and times, provide opportunity for project staff to visit schools and identify suitable spaces for data collection, allow reminders/extra consent forms to be sent home and more time to collect information from schools including number of teachers, students and classes. Consent forms could also be collected prior to data collection, with additional consenters allowed on the day. However, such methods would require extra visits to the school, which would be more resource- and time-intensive.
The consent rates of students in the present study were similar to those in other studies. For example, 46 % of 10-year-olds in intervention and control schools consented to baseline measures in the Energize project(Reference Graham, Appleton and Rush23). Forty-four per cent of primary-school children in control sites and 58 % in intervention sites consented to measures in the Be Active Eat Well intervention(Reference Sanigorski, Bell and Kremer18).
The body image protocol, standard preamble and use of the small gift for students returning consent forms, all worked well during the baseline quantitative measures for ewba. These may also be beneficial strategies for similar studies. No specific issues around body image were raised, suggesting that the body image protocol was successful in this regard. The standard preamble was an effective introduction to the measuring team and questionnaires; no questions were consistently asked by students that indicated a lack of understanding of a certain question or instruction that should have been covered in the standard preamble. The use of the small gift for students returning consent forms (regardless of their consent to the measures) resulted in significantly higher response, consent and completion rates.
Conclusion
The present paper describes the rationale, development and implementation of one of the baseline quantitative data collection process as part of evaluation of a multi-setting, multi-strategy, community-based childhood obesity prevention programme. It demonstrates the complexity of developing a quantitative evaluation process in schools for a multi-strategy and multi-setting project, discusses some of the challenges associated with the data collection process and poses some solutions that may be considered by other similar projects. The experience of baseline data collection for the ewba Community Programs has provided a clearer understanding of the processes and potential difficulties involved with planning and implementing this type of evaluation.
Similar programmes can use: (i) the quantitative evaluation of the ewba Community Programs as an example of one part of a rigorous evaluation for a community-based intervention to inform their own quantitative data collection; and (ii) the challenges and potential solutions reported in the present paper as a form of practical advice to assist with planning and implementing quantitative evaluation of similar, multi-setting, multi-strategy programmes.
Acknowledgements
The eat well be active Community Programs are funded by the Government of South Australia, SA Health and implemented by Southern Primary Health, of Southern Adelaide Health Service, and Murray Mallee Community Health, of Country Health SA. The authors declare that they have no conflict of interest. A.M.M., J.D. and N.M., in collaboration with Fiona Verity, developed the evaluation framework and project tools. A.M.W. and N.M. assisted in data collection. M.J. collated sociodemographic and consent data and wrote the ewba Baseline Evaluation Report, in conjunction with A.M.M., J.D., N.M. and Emma Clover, which was used when writing the present paper. A.M.W. wrote the first draft of the manuscript and it was revised by all authors. The authors acknowledge Terri Lamoree, ewba Evaluation Coordinator at the time of baseline data collection, who provided input into the present paper; in particular comments that informed the writing of the discussion. The authors also acknowledge the team of staff who collected baseline measurements and administered surveys, the schools, students and parents participating in the ewba Community Programs, Lily Chan and Emma Clover from the Flinders University Department of Nutrition and Dietetics for data checking and data analysis, Jerry Moller for input into the initial stages of the evaluation plan and Associate Professor Fiona Verity from the School of Social Work at Flinders University for her role on the ewba Evaluation Academic Team.
