Background

The need for more effective and transparent ways to produce better and more relevant research results aimed at improving patients', consumers', policy makers', researchers' and ultimately citizens', decision-making has been the key message of the Evidence-Based Movement (EBM) over the last few decades (Guyatt et al. 2002). The debate on EBM, however, has more recently focused more on its limitations (namely its ability to fulfill the enormous expectations created among clinicians, patients and policy makers than in an honest and balanced reflection on the many substantial problems that affect the research environment (Ioannidis, Reference Ioannidis2005; Chalmers & Glatziou, Reference Chalmers and Glatziou2009).

The Cochrane Collaboration (CC) has, since its official launch in 1993, been an outstanding voice into the broader EBM world concentrating into a giant effort of synthesising the result of primary studies on the effects of health care interventions into rigorous systematic reviews. As documented in a recent survey by Moher et al., the CC has contributed so far to approximately one-quarter of the whole of published SRs and produced a relevant methodological spin-off in the field (Moher et al. Reference Moher, Tetzlaff, Tricco, Sampson and Altman2007).

The CC is thus the world's largest single organisation dedicated to the preparation and maintenance of systematic reviews of the effects of health care interventions. These reviews are published in full in the Cochrane Database of Systematic Reviews, which is part of The Cochrane Library, and is available on the internet (http://www.thecochranelibrary.com). Over the last few years, the number of systematic reviews in the health care literature has increased substantially.

Cochrane reviews have been found to be of higher quality on average than other systematic reviews (Jadad et al. Reference Jadad, Moher, Browman, Booker, Sigouin, Fuentes and Stevens2000; Olsen et al. Reference Olsen, Middleton, Ezzo, Goezsche, Hadhazy, Herxheimer, Kleijnen and McIntosh2001; Moja et al. Reference Moja, Telaro, D'Amico, Moschetti, Coe and Liberati2005; Moher et al. Reference Moher, Tetzlaff, Tricco, Sampson and Altman2007; Jensen et al. Reference Jensen, Katja, Maric, Tendal, Faurschou and Goezsche2008; Tricco et al. Reference Tricco, Pham, Brehaut, Tetroe, Cappelli, Hopewell, Lavis, Berlin and Moher2009) and decisions on their publication continue to be based more on their use of reliable methods (including the prior approval and publication of a protocol) rather than the size, statistical significance or newsworthiness of their results. However, there is less evidence about the relevance and ability of Cochrane Reviews to meet the needs of the different stakeholders. Some recent papers have pointed out limitations of Cochrane Reviews (Mandel et al. Reference Mandel, Littner, Mimouni and Lubetzky2006; Lang et al. Reference Lang, Edwards and Fleiszer2007; Pagliaro et al. Reference Pagliaro, Bruzzi and Bobbio2010), but the interpretation of these criticisms should be put in a proper perspective to avoid becoming a misleading charge of faults of the primary research rather than the systematic reviews that tries to summarise it (Liberati et al. Reference Liberati, Davoli, Filippini and Moja2010a).

As of 2009, there were more than 20 000 people in over 100 countries working together to achieve its aim of helping people making well-informed decisions about health care. Cochrane reviews cover a wide range of health care interventions, bringing together the eligible research for specific topics and presenting it in a standard, structured way. Within this structure is a section in which the authors of the review provide their conclusions, under two sub-headings: ‘Implications for practice’ and ‘Implications for research’.

The aim of this editorial is to identify and briefly discuss some broader issues that need to be considered when reflecting on the CC's role in and contribution to the complex world of actors of evidence-based medicine and evidence-based policy world, suggesting areas where the CC has primary responsibility to improve its governance, and make its products (mainly systematic reviews of primary studies more valid and relevant), and where the CC should be engaged in developing effective partnership with health care agencies and organisations.

Elsewhere in this issue of Epidemiology and Psychiatric Sciences specific contributions (Adams, Reference Adams2011; Davoli & Amato, Reference Davoli and Amato2011) analyse that from a different angle, making clear that there is no room for simple assumptions and straightforward indicators if one wants to discuss, the contribution that the CC has brought to the worldwide efforts that are needed to improve the scientific basis of health care.

Having said this, I will focus on the following points in this editorial: (a) How the impact of research in general can be assessed and where we are with the available methodologies; (b) why we should avoid falling into the simplistic trap of assuming that a good systematic review per se can change clinical practice overlooking the complexity of the ‘knowledge transfer path’: (c) how the Cochrane Collaboration is looking into itself when trying to measure the impact, (d) what is in the Collaboration's agenda to make it better than what it was made to achieve.

Challenges in assessing the impact of research

How to assess the impact (pay back) of research, be it primary or secondary, is an issue of growing interest worldwide and there have been several attempts to define it at conceptual and practical levels. In the past two decades, many theoretical frameworks and methodological approaches on how to measure research impact and returns have been developed (Banzi et al. Reference Banzi, Moja, Pistotti, Facchini and Liberati2011). A common feature of all these models is that they assess impact on many dimensions and try to integrate multidimensional categorisation of benefit (Hanney et al. Reference Hanney, Buxton and Green2007). A set of indicators and metric is generally associated with each category of impact. For example, bibliometric indicators are highly reported as a measure of the spread of research results, the so-called ‘knowledge advances’. These indicators, though to some extent accepted due to their directness, are only surrogate indicators of impact. Moving towards more hard metrics, such as those measuring the health status or the economic benefit of a population, is a complex task but is somehow essential (Williams et al. Reference Williams, Eiseman and Landree2009).

It is widely accepted that research is a crucial investment to improve patients care, foster innovation, knowledge advancement and social and economic development. For example, a straightforward knowledge gain is expected to arise from any biomedical and basic research. This output should then be translated, during a reasonable time lag (which is however open unpredictable) in a better health status for patients. When more applied types of health research, such as the comparative effectiveness research, health service research, health technology assessment are considered things are different. Health services, which are themselves often involved in the sponsorship of research activities, are interested in the promotion of innovation, in terms of selecting the more useful and required interventions to ameliorate the standard of care in this case the time lag of return should be shorter (Institute of Medicine Report, 2010; Liberati et al. Reference Liberati, Davoli, Filippini and Moja2010Reference Liberati, Traversa, Moja and Trottab).

By definition, research activities are risky and their returns highly unpredictable. So, any attempt to increase the system effectiveness and to assure and monitor quality is welcome by the whole scientific community and founders. Competition on limited resources and different funding modalities also raise additional concerns. From the little evidence available on the proportion of investment by research activity, it is quite clear that funding is unbalanced towards biomedical and basic research for which an impact and returns require a longer time frame (Chalmers & Glatziou, Reference Chalmers and Glatziou2009).

A priority setting exercise based on the capability to produce relevant, usable and transferable outputs is thus necessary.

Many of the methodological drawbacks that the overview of the current approaches has highlighted (Banzi et al. Reference Banzi, Moja, Pistotti, Facchini and Liberati2011) apply specifically to the difficulty of a full assessment of the work of the Collaboration because people may wrongly ‘… .shoot the messenger for the music it is bound to play given what the musicians produced..’

All the considerations above are quite true for primary research but are indeed even more relevant for the secondary one (i.e. research synthesis/systematic reviews) that is the typical Collaboration's activity. This is why great caution needs to be taken when asking the legitimate, yet often too simplistically put, question – ‘… …has the Cochrane collaboration made a difference in the health care arena…’

Understanding the complexity of the ‘knowledge translation path’

A shift from simplistic to much more complex conceptual models of the relationship between scientific knowledge and clinical practice has occurred in the last three decades (Lomas, 2007). It follows therefore that – aside from single case-studies where one could trace the specific impact of a given systematic review – it would be both wrong and inappropriate to look for evidence of the impact of the work of the Cochrane Collaboration to play undue emphasis on the rapid transfer of the results of a single or a series of reviews, ignoring the complex pathway that goes from evidence synthesis, to its incorporation into practice guidelines, to their active implementation and appropriate knowledge translations actions and, eventually, to a valid measurement of all the above steps.

Nevertheless, for an organisation such as the Cochrane Collaboration, is somewhat unavoidable to look into itself wondering about the impact of its product(s). While being a legitimate question it is crucially important that the question is framed in an appropriate way to avoid overoptimistic and selective examples, or, conversely, to reach inappropriate negative conclusions. It is therefore important to refer to the general debate about the impact of research findings into clinical and policy practice (Lomas, 2007).

Diffusion, dissemination and implementation are terms sometimes used interchangeably to denote the idea that information must be a communication process before it is available as an input for decision-making. Diffusion, however, is a quite passive concept as it is largely unplanned and uncontrolled. Those who receive the message were, highly likely, those already open to and out to seek the message. Dissemination is a more active concept; it in fact also implies targeting and tailoring the information to a more or less well-defined target audience. Secondary sources of information, such as typically are, and should be, Cochrane reviews, practice guidelines, consensus statements, guidance documents, etc. are active vehicles for this type of activity.

Implementation means that the goal of the communication is to do more than generally increasing awareness. Not only should the message(s) be well presented to the target audience(s) but its implications must be highlighted for the specific target audiences. Implementation always implies an effort to identify and overcome barriers that may prevent the proper application of knowledge. It is, by and large, a local process of communication in which appreciation of the research findings is a necessary but not sufficient condition to bring about changes in decision-making that reflect the message from research.

Therefore, the three terms – i.e. diffusion, dissemination and implementation – that should be considered when thinking about the impact of Cochrane reviews are not interchangeable terms, but rather phases in a process of increasingly active and more focused intents where, by and large, each phase depends on the success of the precedent.

How to do a better job in synthesising research

Synthesising research should bring about better future research by identifying where relevant questions have not been properly addressed (orphan areas), how most common methodological pitfalls occur, and how wastes that take places at different stages of the research process can be avoided (Chalmers & Glatziou, Reference Chalmers and Glatziou2009). But, once again, the availability of good reviews per se does not make it happen without coherent and consistent policies that see systematic reviews as key components of explicit research governance policies.

There is little doubt, that the collaboration has been, the most outstanding organised attempt to face, in a constructive fashion, the EBM challenge, namely ‘… …synthesizing the best evidence about the effects of health care interventions while pushing the science of research synthesis in order to produce the relevant information that is needed to assist patients and doctors when making decisions about the health care they receive and provide, and policy makers when they have to make coverage decisions… .’

Research teams that prepare Cochrane Reviews are made up almost entirely of volunteers. The task is onerous, perhaps more so than people who come to this for the first time had imagined, and over time it has become more challenging, as methods aimed at improving internal and external validity advance. Therefore, it is interesting to wonder what incentives there are for these teams, who continue, nonetheless, to complete or update over 500 reviews annually for publication in The Cochrane Library.

Firstly, it is almost certain that no other journal provides such extensive support to its would-be contributors. However, journals traditionally base their decision on whether to accept or reject full article ‘completed’ manuscripts. Cochrane Reviews are registered at the outset and developed in conjunction with a network of support. This includes training provided by members of the Collaboration and Cochrane Centres, guidance made available in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, Reference Higgins and Green2008) technology support via the information management system, and the contribution and assistance provided by Cochrane Review Groups through their search and editorial teams, and peer reviewers. As a consequence of these support systems, many reviews that would perhaps have been rejected by another journal are nurtured through to achieving the rigorous quality requirements of published Cochrane Reviews.

However, while this access to support might influence the choice of Cochrane as a recipient of a given review, it is likely only a small part of the incentive for a researcher – which must be to achieve the highest possible visibility for the proposed review and to impact on clinical practice and health policy. Therefore, reviews being used by national and international agencies and guidelines groups such as the World Health Organization, being used by patients, health professionals and consumers to inform decision-making, or being incorporated within decision support applications and e-textbooks, all should be considered under the “impact” category.

More narrowly, one can also measure impact by usage, and figures are encouraging – showing a 19% on year increase in visits to The Cochrane Library on the Wiley Online Library, over 1 million hits per month globally, and a search every 1 s, with an abstract view every 2 s and a full-text view every 3 s.

Finally, in terms of the bibliometric category there is much discussion on the ‘impact factor’: The Cochrane Database of Systematic Reviews (CDSR), published within The Cochrane Library, has just received its third impact factor – 5.653. So on average each Cochrane Review is cited over five times by researchers around the globe within two years of publication. The 2009 impact factor continues a positive growth trend for CDSR, whose first impact factor was 4.654 in 2007, and then 5.182 in 2008. While this trend is encouraging, the Cochrane research ‘footprint’ arguably is underestimated by the impact factor measure, given the size of the impact factor ‘denominator’, and its single focus on citations, which do not directly reflect the Cochrane goals of informing and improving healthcare decision-making.

Individuals from over 100 countries contribute to creating Cochrane Systematic Reviews. In 2009, authors from 85 different countries cited Cochrane Reviews in their published work. Comparing the reviews that receive the most citations with those most accessed demonstrates that there is little overlap between the two groups. This might reflect differences in the priorities between researchers and other Cochrane users such as health professionals, consumers and policy makers. In any case, it is a reminder that ‘impact factor’ is just a surrogate (Banzi et al. 2011) and just only one measure among many of overall impact. Given the international nature of the Collaboration, the range and geographical spread of the readers of The Cochrane Library, the need to actively identify those reviews that are most relevant to end users, in addition to those likely to be most cited, is very evident. A more inclusive approach to measuring impact, beyond simply the impact factor is essential if impact is to be one measure of the Collaboration's success. Find easy and valid indicators to measure the “impact on decision making” is a priority in this sense.

Another way of looking into a broader impact perspective is consider that Cochrane reviews are already been used in readdressing the imbalance in the research agenda and identifying where further and better investigations are needed (Clarke et al. Reference Clarke, Clarke and Clarke2007). A demonstration project (i.e. ‘disinvestment initiative’) is, for example currently underway at the National Institute for Clinical Excellence in the UK where interventions identified as ineffective by Cochrane reviews become the target of quality improvement initiatives aimed at progressively phasing them out from health services delivery (Pearson & Littlejonhs, Reference Pearson and Littlejonhs2007).

Having said that on the positive side, there is no question that further progresses need to be made if Cochrane reviews really want to provide the type of information that is needed to physicians and policy makers to take health care decisions'.

The Collaboration is hugely aware that it is urgent to act on it by finding mechanisms that would assure: (a) better coverage and prioritisation mechanisms; (b) minimum and uniform quality standards for the systematic reviews that are produced; (c) more attention to the harms of interventions and explicit criteria for inclusion of observational studies; and (d) improvement in the reporting format that is currently too burdensome even for interested researchers.

The 2008 Cochrane Collaboration Strategic Review

Before concluding this editorial, a short reference should be made to the Strategic Review that the CC has embarked to look at its future and sustainability in a positive and analytic fashion) (The Cochrane Collaboration Strategic Review, 2008). The Strategic Review was carried out in keeping with, and consistent to, the 10 principles of The Cochrane Collaboration. It was constructed as a deliberate conversation with the Collaboration's members, contributors, partners and funders, and carried out in a series of surveys and interviews, probing questions on the topics : (a) Purpose; (b) Brand and glue (external and internal coherence); (c) Competition (external environment); (d) Financial viability; (e) Accountability and decision-making; (f) Structures and processes; (g) Communication, advocacy, and engagement with external stakeholders.

Main findings from the review, which are now largely in their implementation phase, point to the need of : (1) Reaffirming that CC's primary purpose should be the production of systematic reviews and concentrate on training, methods development, and advocacy for evidence-based decision-making. (2) Implementing policy for minimal competencies for review author teams. (3) Engaging partners for mutual benefits, by developing communications strategy to promote external and internal awareness of the CC's values and achievements. (4) Improving the usability of The Cochrane Library and other products for diverse stakeholders and developing a partnership strategy to engage other systematic review producers and knowledge packagers in diverse activities such as a broad-based educational programme (‘Cochrane Education’); and a responsive review programme (‘Cochrane Response’) to be able to develop long-lasting partnership with relevant stakeholder organisations. (5) Undertaking a formal environmental scan every 2–3 years. (6) Review terms of reference and number and geographic spread of Cochrane entities to ensure efficient international representation.