The obesity challenge

It has been recognized for several years that obesity has reached epidemic proportions (World Health Organization, 2000; Ebbeling et al. Reference Ebbeling, Rawlak and Ludwig2002; Lobstein et al. Reference Lobstein, Baur and Uauy2004) and that prevention and treatment of childhood obesity is key to preventing the continuing increase in rates (Swinburn et al. Reference Swinburn, Gill and Kumanyika2005). Obesity has been described as a public health crisis because it is recognized that if it is not addressed successfully healthcare systems will be overwhelmed. No country has yet developed and implemented a coherent action plan to prevent further weight gain in the population to manage its current obesity epidemic (Swinburn et al. Reference Swinburn, Gill and Kumanyika2005). Obesity has moved beyond the simple problem of advising individuals to eat a healthy diet and be active. Obesity has become a major public health problem and population health approaches are required to turn around the epidemic rise in fatness levels that is now affecting whole populations. Although there are just two key elements involved in obesity prevention, healthy eating and active living, achieving a shift in patterns of whole populations is very complex. This complexity is demonstrated by the continuous rise in obesity prevalence despite the myriad of programmes that have been implemented to combat obesity. Whole-population shifts require multiple integrated strategies that are multidisciplinary and multisector, ranging from targeting individuals to groups to sectors (Health Canada, 2001). A range of strategies that address issues at the level of the individual, create supportive environments and build healthy public policy are all required. However, there is a paucity of what could be called strong evidence, from a traditional sense, on which to base best practice for population health approaches. Nonetheless, to do nothing is not an option. There is movement within public health, health systems and health-policy literature to reconsider the way in which evidence is derived. The development of a common understanding of the terms ‘best practice’ and ‘best evidence’ is crucial for moving forward on population health in general and on obesity in particular. Agreement on how evidence is defined is fundamental to reaching the collective consensus required. Some researchers suggest that the term ‘best evidence’ should be replaced with ‘best-available evidence’ (Swinburn et al. Reference Swinburn, Gill and Kumanyika2005). Others identify ‘better practice’ rather than ‘best practice’ because this term reflects the continual evolution in context and knowledge that is generated from research.

Best practice and evidence

‘Best practice’ is a term that is widely used in business and health care. It implies that there are approaches or practices to prevent, treat or cure that are the ‘best’ or at least ‘better’ to use than others. In the earliest history of medicine and health care ‘best practices’ were approaches that an individual or group of individuals used because in their ‘experience’ such approaches worked best to treat, cure or care for an individual with a health problem. Their experience was their evidence for practice.

Evidence-based practice is often used synonymously for ‘best practice’, meaning there is evidence to support a particular practice or intervention. Evidence generally has meant knowledge gained about a particular subject from using scientific methods. Scientific methodology has a long history; the first controlled experiments were conducted by Francis Bacon in the late 16th century, with the first description of the approach to scientific methods credited to Descartes in the early 17th century (Wikipedia, Reference Wikipedia2006). Controlled experimentation is the centre piece of the scientific method. Despite its >500-year-old history, evidence-based medicine, which is the use of scientific methods for decision making, is a relatively recent phenomenon.

Campbell & Stanley (Reference Campbell and Stanley1963) were the first to suggest that there is hierarchy of study designs, identifying that some approaches are more consistent with scientific methods and thus provide more robust evidence than others (Rychetnik et al. Reference Rychetnik, Frommer, Hawe and Shiell2002). That premise was extended by the Canadian Task Force on the Periodic Health Examination (1979) with the idea of levels of evidence based on study design. It was further refined by the Centre for Reviews and Dissemination (Khan et al. Reference Khan, ter Riet, Popay, Nixon and Kleijnen2001), the Centre for Evidence Based Medicine (Phillips et al. Reference Phillips, Ball, Sackett, Badenoch, Straus, Haynes and Dawes2001) and others (Briss et al. Reference Briss, Zaza, Pappaioanou, Fielding, Wright-De Aguero, Truman, Hopkins, Mullen, Thompson and Woolf2000) and is relied upon for the development of clinical practice guidelines that are considered best practices. Levels I and II as shown in Table 1, considered to be the highest form or level of evidence, are versions of the randomized controlled trial with the first and most stringent trials having all personnel involved in the study blinded to knowledge of the group to which an individual has been randomized. Progressing down the hierarchy, lack of randomization lowers the level of evidence while having no control group on which to base comparisons is only a step above using opinion. Thus, through the establishment of a hierarchy and judging of evidence, the double-blind randomized placebo-controlled trial (RCT) is considered the most reliable source of evidence. The rationale for RCT as the most important form of evidence rests on the internal validity of the design (Des Jarlais et al. Reference Des Jarlais, Lyles and Crepaz2004). Internal validity in a study means that as much as possible bias and confounding have been taken care of along with other design considerations of adequate sample size and accurate and reliable measures. Randomization controls for known and unknown factors that may influence results, while blinding ensures that vested interests, such as those of investigators, subjects or stakeholders, do not influence study results. Internal validity ensures there is a high probability (usually 95%) or confidence that significant outcomes at the end of the study are the result of the intervention and not chance.

Table 1. Hierarchy of evidence (adapted from University of York NHS Centre for Reviews and Dissemination 2001; Khan et al. Reference Khan, ter Riet, Popay, Nixon and Kleijnen2001)

RCT, double-blind randomized placebo-controlled trial.

Hierarchies of evidence place systematic reviews of RCT at the pinnacle of defining evidence. However, RCT may not be available for review because of cost, practicality, feasibility or appropriateness for health promotion interventions. If RCT are considered the only valid design there is a risk that evidence could actually be distorted by relying on a research design that does not match the research question or evaluation objectives (Green & Tones, Reference Green and Tones1999). Nonetheless, consistent results from multiple RCT, identified using a systematic review process as described by The Cochrane Collaboration (The Cochrane Collaboration, 2005) and Oxman (Oxman et al. Reference Oxman, Guyatt, Singer, Goldsmith, Hutchison and Milner1991, Reference Oxman, Cook and Guyatt1994; Oxman, Reference Oxman1994), is considered the gold standard on which to base recommendations to change clinical medical practice. Systematic reviews of RCT consist of a carefully-constructed process of locating and critically appraising studies to answer very specific questions of very specific populations, usually about single interventions. In other words, they have the potential to answer very narrow questions very well.

While RCT have proven valuable for interventions focusing on individuals, for many areas of health practice, and in particular for health service delivery, there are few RCT and fewer systematic reviews of RCT to guide practice. Population health approaches in particular are not easily amenable to RCT designs for many reasons.

Limitations of double-blind randomized placebo-controlled trial designs for identifying evidence for population health

Interventions in public health may target individuals, groups, whole communities or populations (Green & Tones, Reference Green and Tones1999). The distinctions between clinical care and population health are many, starting with the objectives. The goal of population health is to shift the health of whole populations or communities. Evidence-based medicine approaches are designed for individuals (Green & Tones, Reference Green and Tones1999; Heller & Page, Reference Heller and Page2001) and generally short-term outcomes are considered as end points. Interventions to shift the health of whole populations take time, and thus for some public health practices surveillance data may provide better indicators of the success of multiple interventions than RCT (Heller & Page, Reference Heller and Page2001). In fact, some researchers would argue that it is unethical to include control groups who do not receive an intervention because of the length of time required to study the impact of interventions and the potential harm reduction that would be lost to control groups while waiting for results to become available.

According to Victora et al. (Reference Victora, Habicht and Bryce2004) RCT are useful when the causal chain between intervention and outcome is short and simple and results are easily extrapolated. On the other hand, public health interventions are usually large-scale, long-term and complex (Victora et al. Reference Victora, Habicht and Bryce2004; Swinburn et al. Reference Swinburn, Gill and Kumanyika2005) and may not be easily packaged in a form that is amenable to random allocation (Green & Tones, Reference Green and Tones1999). In addition, the costs of mounting such large-scale studies may be prohibitive. Generally, for an RCT to be successful a single or limited set of variables is manipulated, which can be too limiting for examining effects on a full population.

The focus of population health and public health is often on the determinants of health and their interactions (McKinlay, Reference McKinlay1992). Short- and medium-term outcomes as well as long-term outcomes are of interest. Thus, process and mediating factors (which may be on causal or contributing pathways) can be as important as long-term outcomes. Mediating factors of interest in public health are usually theory driven and often focused on human behaviours or environmental influences that are not easily amenable to experimental conditions. The appropriate methodology for gathering evidence on mediating factors might include qualitative approaches alone or in combination with more quantitative research designs (McKinlay, Reference McKinlay1992; Green & Tones, Reference Green and Tones1999). The situation in clinical medicine, in which ‘bench’ research identifies molecular and other biological pathways for disease development and potential cures, is in direct contrast to this approach.

In public health target populations are free-living in environments that are influenced by a myriad of complex factors (many of which are inter-related). Such populations are generally healthy and need to be enticed to ‘comply’ with interventions to improve health. It is difficult for RCT designs to change with the changing focus of a population and the popularity of activities that might serve as intervention strategies.

As population health approaches take place in the real world, control is often not possible to the extent required to address issues of confounding and bias (Green & Tones, Reference Green and Tones1999; Swinburn et al. Reference Swinburn, Gill and Kumanyika2005). Efficacy is determined from experimentation in a controlled environment with little variation in procedure, i.e. effectiveness under ideal conditions (Green & Tones, Reference Green and Tones1999; Victora et al. Reference Victora, Habicht and Bryce2004). The integrity of the setting, intervention and measurement are all maintained (Green & Tones, Reference Green and Tones1999; Victora et al. Reference Victora, Habicht and Bryce2004). For public health all the interventions are taking place in ‘real world’ conditions (Green & Tones, Reference Green and Tones1999; Victora et al. Reference Victora, Habicht and Bryce2004) and thus the results reflect the effectiveness, rather than the efficacy, of interventions.

As a result of the ‘real world’ nature of public health interventions context and process are crucial to understanding success. Success of an intervention depends not only on outcomes but also on the contextual factors that make up the environment in which a study takes place. In population health success is not just about outcomes, every stage of programme development is important, e.g. its popularity and the participant satisfaction with it.

External validity, or generalizability, appears to be a secondary concern in traditional approaches to critical appraisal that rely on RCT design (Mays et al. Reference Mays, Pope and Popay2005). In contrast, assessment of external validity is a critical component of population health approaches used in public health, as it takes into consideration the contextual factors that provide information allowing for judgments on applicability beyond the study population. Relative to this issue, tracking of integrity of interventions (i.e. faithfulness to design and implementation over entire period of intervention) is important and not always reported in research reports and publications (Green & Tones, Reference Green and Tones1999; Flynn et al. Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006).

In fact, the Transparent Reporting of Evaluation with Non Randomized Designs Statement published in 2004 (Des Jarlais et al. Reference Des Jarlais, Lyles and Crepaz2004) seeks to improve the reporting quality of non-randomized evaluations of interventions. The criteria in this statement include some of the traditional standards in the Consolidated Standards of Reporting Trials Statement (Begg et al. Reference Begg, Cho, Eastwood, Horton, Moher, Olkin, Rennie, Schulz, Simel and Stroup1996), such as a description of the planned study population, inclusion and exclusion criteria and flow of participants through each study phase (Des Jarlais et al. Reference Des Jarlais, Lyles and Crepaz2004). However, also included are factors such as barriers to implementing the intervention, how well the intervention stayed true to its original intent and compliance rates. These aspects, which can be critical for synthesis research endeavouring to develop best practice guidelines for population health approaches used in public health, are often missing in reports and publications.

Synthesis research

There are some researchers who consider the complexity of population health and health promotion interventions to be not amenable to traditional approaches of systematic review (Green & Tones, Reference Green and Tones1999; Victora et al. Reference Victora, Habicht and Bryce2004; Flynn et al. Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006), and there are others who might argue that levels of evidence are important but not sufficient or necessarily required in determining best practice (Swinburn et al. Reference Swinburn, Gill and Kumanyika2005). There is concern by still others that the pendulum will swing too far in reverse and clinical trials will not be deemed useful in informing public health practice (Rychetnik et al. Reference Rychetnik, Frommer, Hawe and Shiell2002). In fact, cluster randomized trials provide opportunities for randomization (Rychetnik et al. Reference Rychetnik, Frommer, Hawe and Shiell2002) and advanced statistical modelling has been successful in addressing the bias and confounding that plagues less-stringent designs (Heller & Page, Reference Heller and Page2001). All these viewpoints are part of a growing body of literature and evolving consensus on the need for developing new approaches to defining and interpreting evidence for public health practice (Green & Tones, Reference Green and Tones1999; Heller & Page, Reference Heller and Page2001; Victora et al. Reference Victora, Habicht and Bryce2004; Swinburn et al. Reference Swinburn, Gill and Kumanyika2005). A few published reports describe expanded approaches for conducting systematic reviews to identify best practice for population health approaches, which include the community guide to preventive services (Briss et al. Reference Briss, Zaza, Pappaioanou, Fielding, Wright-De Aguero, Truman, Hopkins, Mullen, Thompson and Woolf2000) and a review of methods used to synthesize evidence for effectiveness in health promotion (Jackson et al. Reference Jackson, Edwards, Kahan and Goodstadt2001). In addition, those researchers concerned with making systematic reviews more relevant to policy makers have proposed even broader approaches to the process of synthesizing evidence (Atwood et al. Reference Atwood, Colditz and Kawachi1997; Lomas, et al. Reference Lomas, Culyer, McCutcheon and McAuley2005; Mays et al. Reference Mays, Pope and Popay2005; Pawson et al. Reference Pawson, Greenhalgh, Harvey and Walshe2005), highlighting disparities in how researchers and policy makers define and describe evidence.

Reducing obesity in children: a synthesis

A recently-published synthesis research study, which has moved beyond the traditional approach of systematic reviews, provides a starting point on which to begin making judgments on best practices in population health (Flynn et al. Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006). This synthesis meets the criteria that are now being proposed to make systematic reviews and synthesis research useful to programme planners and policy makers. As a result of the number of programmes included and the broad approach used to critically appraise the research and evaluation designs, this synthesis can be used to begin to identify programmes that hold promise as examples of better practice in population health.

The methodology used in this synthesis included the following key innovations:

1. (1) using an international panel of experts and a concept model on population health (Hamilton & Bhatti, Reference Hamilton and Bhatti1996) to guide the process;

2. (2) not limiting the review to English language reports;

3. (3) including critical appraisal criteria in addition to the usual assessment of scientific rigour; the aim being to identify as much information as possible on best practice relevant to interventions on population health.

The search strategy was wide and comprehensive, and covered eighteen library databases (>13 000 hits), >250 internet sites, a hand search of the International Journal of Obesity, reference lists from articles identified in the search and key informant surveys. Unlike other systematic reviews on best practice a purposely broad set of inclusion criteria was used. For example, at least one reported outcome (e.g. indices of overweight) or process indicator (e.g. number invited to participate v. number completing the programme) and few exclusion criteria (e.g. children with health problems) were used. While non-English language reports are commonly excluded, sixty-four non-English-language articles and reports were identified in this synthesis. This approach resulted in a high number of reports being eventually selected for critical appraisal (500 studies or reports of >900 identified had at least one process or outcome indicator; see Table 2, which shows the steps in the synthesis review process.

Table 2. Steps in synthesis review (adapted with permission from Flynn et al. Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006)

Another key difference between this synthesis research and other systematic reviews of obesity intervention or prevention is that critical appraisal was not limited to assessment of scientific or methodological rigour in quantitative studies. Each programme also received a critical appraisal for qualitative methodological rigour (if applicable), programme development and evaluation, relevancy of population health principles and use of approaches to support immigrant health (it should be noted that immigrant health was a sub-objective of the review but will not be discussed in the context of the present paper). In order to synthesize findings and reduce the number of reports and articles to a more-manageable number, a criterion-based scoring system was developed and only those scoring in the top one-third of the scoring range in any of the four appraisal approaches used were included for the synthesis of findings. This process gave most programmes four opportunities to be included in the final synthesis based on each appraisal level. This selection of only top-scoring programmes reduced the number to a more-manageable 147 programmes (based on 158 reports) on which to base the synthesis.

Thus, the use of multiple critical-appraisal perspectives not only increases the number of programmes included in the review but it also adds a much broader perspective to the review of specific programmes. In the context of the Flynn et al. (Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006) synthesis review the addition of the appraisal of programme development resulted in the inclusion of more studies compared with previously-published systematic reviews focusing on childhood obesity and related factors, e.g. 147 compared with forty-two in a review by Jelalian & Saelens (Reference Jelalian and Saelens1999). The use of the additional appraisal criteria also, however, facilitated consideration of other pertinent factors in the assessment of better practice that are of interest in population health. For example, the appraisal of population health principles was based on the population health promotion model (Hamilton & Bhatti, Reference Hamilton and Bhatti1996) that served as the theoretical framework guiding this work. This assessment used scoring criteria that were based on the use of multidimensional strategies and the level of upstream investment. In fact, only one programme (Macaulay et al. Reference Macaulay, Paradis, Potvin, Cross, Saad-Haddad and McComber1997; Jimenez et al. Reference Jimenez, Receveur, Trifonopoulow, Kuhni, Paradis and Macaulay2003) was found to score highly on principles of population health (forty-eight programmes used some multidimensional approaches and forty included some ‘midstream’ approaches, which were primarily found in school-based settings). This dearth of population health approaches resulted despite the broad search strategy and inclusion criteria used (see Table 2 and Section 3.3 of Flynn et al. Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006). Clearly, this synthesis supports the contention of other researchers (Kumanyika, Reference Kumanyika2001; Swinburn & Egger, Reference Swinburn and Egger2002) that there is a lack of published and unpublished research and evaluation of population health approaches to address childhood obesity (Flynn et al. Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006).

A final key component of the synthesis was the development of a ‘features table’, which outlined the main characteristics for each of the top-ranking 147 programmes according to their settings (school, home, community, clinic etc.; Flynn et al. Reference Flynn, McNeil, Maloff, Wu, Mutasingwa, Ford and Tough2005). These characteristics facilitated an understanding of populations addressed, key programme intervention features and outcomes. The remainder of the synthesis process included the identification of gaps in populations studied and programme foci and, in a more limited number of programmes for which data was adequate, a summary of best practices in programme development and programme effectiveness (Flynn et al. Reference Flynn, McNeil, Maloff, Wu, Mutasingwa, Ford and Tough2005).

The components of this synthesis research, and in particular the ‘features table’, provides the information that is needed to begin to populate the promise table described by Swinburn et al. (Reference Swinburn, Gill and Kumanyika2005; see Table 3) that can be used by clinicians, health promotion specialists, community developers and policy makers to categorize the level of promise for potential interventions to address childhood obesity in their jurisdictions.

Table 3. Promise table (with permission, Swinburn et al. Reference Swinburn, Gill and Kumanyika2005)

Promising programmes

As part of the Prevention Group of the International Obesity Task Force, Swinburn et al. (Reference Swinburn, Gill and Kumanyika2005) describe Hawe & Shiell's (Reference Hawe and Shiell1995) portfolio approach to health promotion, which allows for selection of programmes or strategies based on best-available evidence while not excluding untried but promising interventions. This approach is designed to encourage a balance of high- and low-risk investments (programmes) to maximize returns (outcomes) much in the same way as individuals or corporations would choose financial investments. It is accomplished by examining intervention or programme effectiveness and the potential for population impact using a matrix to classify strategies from least to most promising (Swinburn et al. Reference Swinburn, Gill and Kumanyika2005).

The basic premise is that judgements on the risks of investments are made based on two components ‘certainty of effectiveness’ and ‘potential for population impact’ (see Table 4). ‘Certainty of effectiveness’ can be evaluated by identifying a level of certainty (high, medium and low) based on an assessment of quality of evidence in the traditional sense by judging internal validity. Internal validity is based on the design and conduct of the study or evaluation, i.e. its methodological or scientific rigour. In addition, programme outcomes are also examined on the basis of programme effectiveness, i.e. did it work and were there negative side effects?

Table 4. Assessment components for promise table rankings*

* For details of promise table, see Table 3.

Three factors can be considered in examining the ‘potential for population impact’ component in order to identify the level (high, medium and low) of the ‘potential for population impact’:

1. (1) programme logic, where the theoretical basis of the programme is considered, including whether it is logical and consistent with current knowledge of physiology, behaviour etc.;

2. (2) reach, which takes into account range and breadth of participation, including characteristics of the study population;

3. (3) uptake, which includes utility (such as evidence and extent of stakeholder involvement) and feasibility (such as practicality and costs etc. including rates of participation).

This latter component broadens consideration criteria to incorporate contextual and policy relevance, implementation and sustainability factors that require review when considering adoption of programmes to specific settings (Swinburn et al. Reference Swinburn, Gill and Kumanyika2005).

The judgments made for ‘certainty of effectiveness’ and ‘potential for population impact’ can transfer to the promise-table matrix to determine relative promise. Thus, programmes of high ‘certainty of effectiveness’ but low ‘potential for population impact’ (e.g. a clinical programme to treat obesity at a summer camp) would rate as promising. On the other hand, programmes of low ‘certainty of effectiveness’ and high ‘potential for population impact’ (e.g. the Pathways schools intervention programme from the USA that had positive qualitative results on reach, uptake and programme logic (Davis et al. Reference Davis, Going, Helitzer, Teufel, Snyder and Gittelsohn1999; Gittelsohn et al. Reference Gittelsohn, Evans, Story, Davis, Metcalfe, Helitzer and lay1999) but lack of hard positive outcomes (Caballero et al. Reference Caballero, Clay, Davis, Ethelbah, Rock, Lohman, Norman, Story, Stone, Stephenson and Stevens2003)) could also be rated as promising.

The final component of the promise-table exercise is consideration of adoption within a specific context, which requires investment from stakeholders that understand the specific conditions that will impact feasibility and sustainability in a particular setting. This process underscores the ‘real world’ nature of population health work; the need for understanding programme implications from a local perspective and the complex analysis required to identify useful evidence for practice. Two programme examples previously reviewed in the Flynn et al. (Reference Flynn, McNeil, Maloff, Mutasingwa, Wu, Ford and Tough2006) synthesis will be described to demonstrate use of the promise table; for example, if the focus of interest were on public health interventions to reduce chronic disease risk factors in children related to obesity.

Two examples for using the promise table

The features table from the web-site document (Tables 8.1–8.6 of Flynn et al. Reference Flynn, McNeil, Maloff, Wu, Mutasingwa, Ford and Tough2005) contains the content that is needed to identify ‘certainty of effectiveness’ and ‘potential for population impact’. The 147 programmes that were reviewed are all included in the features table arranged by the setting in which they took place and provide important contextual information on the population characteristics as well as description of the interventions and study or evaluation outcomes (for examples, see Table 5).

Table 5. Programme details for examples of studies (from Flynn et al. Reference Flynn, McNeil, Maloff, Wu, Mutasingwa, Ford and Tough2005)

MD, doctor; RD, registered dietitian; RN, registered nurse; ↑, increase; ↓, decrease; ↔, no change; ↑↓, P<0·05.