Poll Results

The key poll results are shown in Supplementary Tables 1 and 2, which can be viewed online at www.journals.cambridge.org/thc2011026. The sample was not sufficient to identify any significant differences in ratings between the health professionals and non-health professionals. Although Germans rated the interestingness of the messages more highly in general, roughly the same reviews were regarded as interesting and uninteresting.

Very few statements were rated as significantly interesting (Reference Douw and Vondeling8 percent), while around half of all statements were significantly uninteresting (Supplementary Table 1). Based on the planned subset analysis, reviews without enough evidence to draw a conclusion were less likely to be significantly interesting (Reference Noorani, Husereau and Boudreau12 percent versus Reference Dickersin, Ssemanda, Marshall and Rennie7 percent). However, the rigid application of the criteria would have resulted in two of seven significantly interesting reviews being excluded.

A post-hoc analysis was conducted because there appeared to be a discriminative criterion for “uninteresting” statements: the combination of “not enough evidence” and a condition that is not common. That analysis showed that of fifteen reviews in uncommon conditions where there was not enough evidence to support a conclusive statement, all but one were significantly uninteresting (93 percent) (Supplementary Table 2). All of the significantly interesting statements in the subset from Issue 1 of 2005 were for common conditions (100 percent), and all but two had enough evidence for a conclusive statement (71 percent). The poll confirmed the general value of the eligibility criterion “enough evidence was found to support a conclusive statement,” with exceptions in practice for common conditions or commonly used interventions.

Theoretical Framework and Potential Data Sources

The framework for considering consumer-oriented priorities (Table 1) includes three major dimensions: (i) Healthcare evidence (both about interventions or technologies and the process of communication itself); (ii) Editorial considerations (including the potential for adverse effects); and (iii) Patients’/consumers’ interests.

Table 1. Theoretical Framework for Priority-Setting Criteria

The potential for adverse effects of publishing information requires further methodological and practical development, but is an explicit and important criterion for IQWiG. Our ultimate goal is to empower people, including providing support for them to set and achieve their own health goals. However, we also need to be concerned with adverse effects such as disempowering or debilitating people, causing despair, and worsening people's health or well-being. Some messages could have unintended effects or potentially predictable adverse effects. For example, the potential adverse effects of communicating a beneficial effect of continuing to smoke (some weight control) had to be taken into consideration in the editorial decision on whether or not to communicate all or some of the results of a systematic review on weight gain and smoking cessation (Reference Parsons, Shraim, Inglis, Aveyard and Hajek16). This was more than a theoretical concern, as there was some suggestion that this information could contribute to decreased motivation to quit smoking (Reference Parsons, Shraim, Inglis, Aveyard and Hajek16). We published the information, trying to reduce the risk of harm while informing comprehensively by including particular emphasis on the value of quitting smoking. We do not know whether we were successful here, as we have yet to develop methods for determining the effects in our readers of what we do (Reference de Joncheere, Gartlehner, Gollogly, Mustajoki and Permanand5).

The process of developing the framework resulted in the construct of “interestingness” to/for patients/consumers becoming increasingly explicit and differentiated. Patients’ and consumers’ interests in the context of information are both what is interesting to them and what it might be in their interests to know. Therefore, the term interests here incorporates interesting to and importance for. Rather than being a single item capable of measurement or determination by a single method, the concept consists of multiple potentially measurable elements. Table 2 fleshes out aspects of these elements, by indicating potential ways that they could be assessed.

Table 2. Potential Data Sources for Determinants of Topic Interest

The processes ultimately chosen for assessing interesting to patients/consumers at IQWiG include literature searches for qualitative research addressing patients’ concerns and what affects their decision making. This is critical for our product development in areas where significant resources are invested or where we are particularly concerned about the potential effects of what we communicate or how we communicate. Published qualitative research can also be used to help inform our understanding of what topics should be addressed within a priority condition (such as stroke), for example.

Information seeking on the Internet (both within the search engine of our own Web site, and search query data reported by Google) as well as at the call centers operated by Germany's statutory health insurers have emerged as vital sources for overall topic planning for the Institute. We plan to use a deeper analysis of these data to map out the major conditions which we must address to have a reasonably comprehensive information portfolio.

IQWiG's Routine Topic Selection Process

Figure 1 provides an overview of the process we use for assessing a proposed topic: Figure 2 provides more detail of the assessment steps for relevance and evidence. This process could be triggered by one of two things: either beginning with a suggested topic, or a topic could be triggered by the publication of an eligible systematic review or HTA.

Figure 1. An overview of the process used for assessing a proposed topic.

Figure 2. More detail of the assessment steps for relevance and evidence.

The Department of Health Information monitored the publication of new and updated systematic reviews and HTAs published in both English and German, primarily through the Cochrane Database of Systematic Reviews, the Database of Reviews of Effects, the INAHTA (International Network of Agencies for Health Technology Assessment) database of HTAs, and the McMaster Online Rating of Evidence system (Reference Husereau, Boucher and Noorani10). The template for summarizing the topic assessment process is included as Supplementary Table 3, which can be viewed online at www.journals.cambridge.org/thc2011026. Routine sources for rapidly assessing priority included national routine data collections on reasons for doctor visits, hospitals and sickness-related work absence, data on enquiries to statutory health insurance call centers (supplied in confidence by major insurers), the McMaster ratings of clinical relevance (Reference Haynes, Cotoi and Holland10), and national prescribing data (Reference Schwabe and Paffrath19). Prevalence and burden of disease information was obtained from the evidence sources, supplemented by data available in German clinical practice guidelines and the BMJ publication, Clinical Evidence.

Monitoring Effectiveness of Topic Selection and Determining Whether to Disinvest in Topics

Monitoring effectiveness of our methods for determining what is likely to be interesting is critical for the Web site's impact. Any priority setting before production is essentially predictive and presumptive. Whether the resulting information actually proves to be interesting to the intended audience and whether it attracts readers requires evaluation, including monitoring behaviors that indicate interest. Assessing actual rather than predicted interest could contribute to improved topic selection methods. In addition, being able to determine actual interest would enable good decisions about whether to continue to invest in keeping a topic up-to-date, or to choose instead to disinvest and archive (or remove) a topic.

With an estimated 1,000 topics as the Institute's goal for its information portfolio, errors in selection of topics are to be viewed critically. Decisions that have been supported by excellent information and great expertise are not necessarily going to get it right. Informed expert opinion has been shown to be poorly predictive of which technologies will become important (Reference Douw and Vondeling8), for example. Medical journal editors’ decisions, too, on which items of research will interest journalists are far from universally right (Reference Bartlett, Sterne and Egger3). Yet, despite the widespread use of priority setting in health, the issue of how to determine its effectiveness has received much less attention than mechanisms for choosing priorities. Each inadvertently unimportant/uninteresting topic selected by IQWiG not only has an initial opportunity cost: it continues to consume resources by means of the updating process. Correcting for topic selection errors, or changes in interest over time, could maximize the potential impact of the Web site.