Health technology assessment (HTA) has the objective of providing a basis for health care that is more evidence-based and for more efficient use of scarce resources. The desired end result is improved health for patients and the general population. However, improved health is a product of many interrelated factors, HTAs being only one. Separating the effects of HTAs from other variables is difficult. Thus, following changes in clinical practice after the publication of HTA reports is a more realistic approach. Another option is to determine whether HTAs have been used as a basis for clinical guidelines or policymaking, such as legislation, reimbursement for pharmaceutical expenses or research in a specific area.
Various approaches have been used to assess the impact of HTAs, e.g. narrative or case studies (Reference Hailey1–Reference Britton, Asplund and Brorsson5) and before–after studies (Reference Axelsson, Helgason, Lund and Adolfsson6;Reference Brorsson and Arvidsson7) or time series analysis (Reference Sheldon, Cullum and Dawson8). Several studies focus on the impact on public policy (Reference Hailey1–Reference Hailey, Corabian, Harstall and Schneider3;Reference Schumacher and Zechmeister9). One study concluded that seventeen of twenty-six HTAs appeared to have influenced policy (Reference Hailey1), while a study on rapid HTAs indicated that fourteen of twenty had influenced policy (Reference Hailey, Corabian, Harstall and Schneider3). Schumacher and Zechmeister found that HTA reports have increasingly been used for investment and reimbursement decisions; the economic impact of HTA recommendations was reflected by reduced expenditures (Reference Schumacher and Zechmeister9). Gerhardus et al. (Reference Gerhardus, Dorendorf, Röttingen, Santamera, Garrido, Kristensen, Nielsen Palmhöj and Busse10) presented examples of the impact of HTAs, broken down into six steps: awareness, acceptance, policy process, policy decisions, clinical practice, and outcome.
Decisions concerning reimbursement for pharmaceutical expenses are easier to implement due to the major economic consequences involved. Recommendations and guidelines, on the other hand, may or may not be implemented. Trying to convince individual physicians to change their behavior in the absence of decision-making power is an even tougher task.
Before-after studies (Reference Axelsson, Helgason, Lund and Adolfsson6;Reference Brorsson and Arvidsson7) and time series analyses (Reference Sheldon, Cullum and Dawson8) have the advantages of showing actual changes in clinical practice. Sheldon et al. used time series analyses, audits of case notes and interviews to determine whether NICE guidance had been implemented (Reference Sheldon, Cullum and Dawson8). They concluded that guidance was more likely to have been implemented when there was strong professional support, as well as a stable and convincing evidence base, as opposed to increased costs. These conclusions are fully in line with the findings of implementation research (Reference Grol and Grimshaw11;Reference Rosén and Jansson12).
Trying to estimate the impact of HTA on clinical practice is confounded by the presence of so many other factors, including systematic reviews, articles in scientific journals, discussions at international conferences, regional guidelines, and public policy decisions. Furthermore, Sweden has a decentralized healthcare system assigned to twenty-one county councils, creating special challenges when it comes to the implementation of results from HTA reports.
SBU is not involved in developing guidelines or in making policy or reimbursement decisions, the only task is to produce HTAs. Thus, its effectiveness depends on its trustworthiness, ability to disseminate information and implementation strategies. External analysts have concluded that SBU reports are scientifically rigorous from an international point of view and follow basic HTA principles (Reference Poulsen13;Reference Neuman, Drummond and Jönsson14).
The aim of this article is to describe and analyze whether SBU's “yellow” HTA reports published between 2006 and 2010 have influenced decisions, guidelines, research priorities, and clinical practice in Sweden.
DATA AND METHODS
Although the ultimate objective of HTA is to improve health, we believe that demonstrating the causal chain from a report to such changes is too daunting a challenge. It is a long process and many factors influence the final results. A more important outcome is whether clinical practice changes in accordance with the results of an HTA report. Practice can change in different ways depending on whether there is strong or weak evidence for applying or abandoning a specific method. Data obtained from questionnaires or public health registers have been used when available. Secondary outcomes may include decisions made by national or regional authorities, research priorities set by research foundations or implementation of the results as part of national or regional guidelines.
Conceptual Model
Schumacher and Zechmeister introduced a conceptual framework with the impact categories of awareness, acceptance, policy process, policy decision, practice, final outcomes (economic), and enlightenment (Reference Schumacher and Zechmeister9). The framework is similar to the one used by Gerhardus et al. (Reference Gerhardus, Dorendorf, Röttingen, Santamera, Garrido, Kristensen, Nielsen Palmhöj and Busse10). We have used a similar framework for our review, but with some other influences as well. Our conceptual model (Figure 1) served as the basis of our efforts to assess the impact of HTA. Confidence in HTA reports is required before healthcare professionals will take action and change practice in accordance with new evidence. HTAs and systematic reviews can directly influence legislation, reimbursement for pharmaceutical expenses and other public policy decisions. Because Sweden has a decentralized health care system, county councils may implement, postpone, or discontinue diagnostic or treatment methods. We summarize these decisions under the heading of national or regional policy decisions.
Figure 1. Conceptual model of ways that HTA influences clinical practice.
HTAs or systematic reviews often form the basis of national or professional clinical guidelines. This is the second way that HTA can influence health care. The third type of impact occurs when the HTA report instils enough confidence in healthcare professionals that they change clinical practice. The choice of treatment options is also influenced by patients even though they may not be knowledgeable about the results of HTA. HTAs often present insufficient evidence as to whether the interventions assessed have proven to be effective or not. In such cases, HTA has identified knowledge gaps and can encourage more research that will eventually improve health care. It goes without saying that other measures or information from additional interested parties contribute to change in clinical practice. For example, manufacturers exert heavy influence in the process of marketing their products. Awareness of all these avenues is necessary to determine whether HTA has contributed to changes in clinical practice.
METHODS
SBU has four different report series whereby the “yellow” reports on assessments of disease areas or established methods are assessed in this study.Footnote 1
The analysis in this study includes all twenty-six yellow SBU reports published between 2006 and 2010. For each project, we searched publications and documentation reflecting decisions, guidelines, research, or changes in clinical practice that may have stemmed from the results. Written documentation, before–after surveys and time series register data were used when available. Because most of the documentation is in Swedish, only a limited number of people can verify it. A table summarizing the twenty-six reports specifies the type of influence, our subjective estimate of the impact and the main results.
In some cases, the Swedish Patient Register (Reference Ludvigsson, Andersson and Ekbom15) and the Prescribed Drug Register (Reference Wettermark, Hammar and Fored16) were used to illustrate changes in clinical practice. These registers cover the entire Swedish population. The first register includes everyone who is hospitalized, while the second register includes everyone who has received a prescription and dispensed it at the pharmacy. Non-participation rates are very low.
We classified the cases according to whether or not they have influenced decisions, guidelines, research or clinical practice. As a basis for determining the confidence that leading healthcare professionals have in SBU, we will start by presenting a survey conducted by Statistics Sweden in 2010.
Confidence in HTA and SBU
To reinforce the chain of evidence, we will present the results of an independent survey conducted by Statistics Sweden in 2010. The respondents were physicians, heads of county councils and clinical departments, and public policy makers (17). The samples were randomly selected among the various categories. The response rate was 63 percent. There were a total of 1,137 respondents. All of them held leading positions in the Swedish healthcare systems, and 90 percent of other professionals knew of SBU.
Approximately 90 percent of the respondents considered the scientific evidence presented by SBU to be very or fairly reliable. Nearly all of those who usually influence clinical practice (chief medical officers at hospitals and heads of clinical departments) had confidence in information from SBU (Table 1). Virtually all of these medical officers also stated that they had practical use for the results of the SBU reports (17). Confidence was much higher in government agencies than in professional organizations, drug therapy committees, or the private sector. SBU was seen as a more reliable source of information than other government agencies or professional organizations. The results of the survey shows that leading healthcare professionals regard SBU as a highly trustworthy and reliable source.
Table 1. Confidence in SBU
Note. Survey of a random sample among each category of leading Swedish healthcare professionals. N = 1,137 responents (63% response rate). Source: Statistics Sweden, 2010
National and Regional Policy Decisions
To the best of our knowledge, SBU reports have rarely influenced national legislation. HTA focuses primarily on clinical methods rather than national policies. However, a report entitled “Rehabilitation of Patients with Chronic Pain” (Case 4) influenced Government decisions. The report concluded that multimodal rehabilitation facilitates return to work and reduces sick leave. Treatment strategies that include physical activity are more cost-effective than ordinary care alone. The report served as a basis for the Ministry of Health and Social Affairs to launch the rehabilitation guarantee. The findings also provided data for the design of several local and regional healthcare programs.
A report entitled “Vaccinations for Children – Protective Effects and Adverse Events” (Reference Örtqvist, Blennow and Carlsson18) was spurred by a trend toward fewer vaccinations throughout the country (Case 3). The report was used by the NBHW in preparing its informational material about child vaccination and was frequently cited in popular magazines. Child health centers made widespread use of the report when providing information to uneasy parents. The report also stirred great interest internationally.
Elderly care, including nursing for patients with dementia, has become a priority area. The SBU report on dementia disorders (Case 1) in 2006 created much interest. A project by the Swedish Association of Local Authorities and Regions (SALAR) to improve the training of municipal caregivers relied heavily on the report. The report has been downloaded from SBU's website more than 180,000 times, an all-time record.
Several interest groups and pediatricians wanted the NBHW and the National Food Administration to adopt mandatory fortification of flour to reduce the risk of neural tube defects in newborns (Case 2). The agencies asked SBU to perform a systematic review of the topic. SBU concluded that fortification reduces the occurrence of neural tube defects but could not rule out the possibility that folates play an important role in cell production and theoretically stimulate the growth of existing tumors. Increased risk could not be excluded. The NBHW and National Food Administration decided not to adopt mandatory fortification of flour but to provide more information for women of childbearing age.
National and Professional Clinical Guidelines
The NBHW always uses SBU reports as the primary source for developing its guidelines whenever they are available. Many SBU dental projects have served this purpose. The NBHW used “Methods of Diagnosis and Treatment of Caries” (Case 8), “Endodontics” (Case 9) and “Prosthetic Rehabilitation of Partially Dentate or Edentulous Patients” (Case 10) to develop national guidelines for adult dental care (Reference af Geijerstam, Oredsson and Britton19). A review of the scientific basis for the NBHW guidelines shows that the three reports were the most cited and influential sources (20).
In the process of developing its guidelines, the NBHW asked SBU to perform HTAs concerning four particularly thorny areas of diabetes: “Intensive Glucose-lowering Therapy in Diabetics” (Case 14), “Patient Education in Managing Diabetes” (Case 6), “Self-Monitoring of Blood Glucose (SMBG) in Noninsulin Treated Diabetes” (Case 13) and “Dietary Treatment of Diabetes” (Case 11).
The NBHW used all four reports when designing its guidelines for diabetes care. The guidelines explicitly mention the reports as the primary scientific basis for the guidelines concerning intensive glucose-lowering therapy, patient education, monitoring of blood glucose, and dietary treatment of diabetes (21). SBU reports are the most cited sources for these conditions under the diabetes guidelines. SBU reports on caries, periodontitis, anti-tobacco advice, and chronic pain are also mentioned as the scientific source for NBHW guidelines.
There are several areas for which the NBHW does not issue national guidelines. In many such cases, SBU has helped professional associations develop guidelines of their own. The Swedish Society for Vascular Surgery used a report entitled “Peripheral Arterial Disease – Diagnosis and Treatment” to design its guidelines (Case 5). The report has also been used for training purposes, for designing health care programs and for the pharmaceutical handbook of the Medical Products Agency.
The Ear, Nose, and Throat Association, the Swedish Association of General Practice, and the Association of Hearing Therapists developed guidelines based on an SBU report entitled “Tympanostomy Tube Insertion for Otitis Media in Children” (Case 15). The Swedish Ophthalmological Society and Swedish Glaucoma Society developed practical recommendations and guidelines based on the report “Open Angle Glaucoma– Diagnosis, Follow-up and Treatment” (Case 7).
Changes to Clinical Practice
The SBU report on methods of early prenatal diagnosis (Case 19), which was published in 2006, found that a combined test of ultrasound nuchal translucency measurement and maternal serum biochemistry (biochemical screening) is the best method for estimating the probability of chromosomal abnormalities. The patient is subsequently offered guidance in deciding whether she wants to take the second step and undergo an invasive test. The approach should reduce the risk of miscarriage.
Three years later a review in the weekly journal entitled Dagens Medicin found that fifteen of twenty-one county councils were offering the combined test to one extent or another (22). According to a survey, half of all midwives believed that the information they had previously given to patients was insufficient and that the SBU report had filled that gap.
In 2007, SBU published a report on dyspepsia and gastro-esophageal reflux (Case 16). The report concluded that proton pump inhibitors relieve the symptoms of undiagnosed gastro-esophageal reflux more effectively than histamine type-2 receptor antagonists (H2RAs). Eradication of Helicobacter pylori is more successful in preventing new gastric ulcer bleeding than prophylactic proton pump inhibitors unless the patient is taking a non-steroidal anti-inflammatory drug (NSAID). Long-term treatment of gastro-esophageal reflux with proton pump inhibitors is also justified in younger patients and is as effective as surgery while causing fewer adverse effects.
The report is one of the documents that the NBHW has used to develop “quality indicators for drug therapy in the elderly” in collaboration with local pharmaceutical committees throughout the country. The committees have also used the conclusions of the report to explain their lists of recommendations and have attached them to recommendations for specific medications.
The National Patient Register, which covers all surgical procedures, shows a steady rate of surgical procedures for gastro-esophageal reflux per 100,000 Swedes before the publication of the SBU report and a substantial decrease from 2007 to 2011 (Figure 2), strongly suggesting that the report influenced clinical practice.
Figure 2. Number of surgical procedures for gastro-oesophageal reflux (Gastro) and snoring and sleep-apnoea (ENC30, ENC40, DHB40, DHB50, DJD20) per 100,000 inhabitants in Sweden between 2005 and 2011. Men and women, all ages. Source: Patient Register, Swedish National Board of Health and Welfare.
In collaboration with other Nordic HTA organizations, SBU published a 2007 report on obstructive sleep apnea syndrome (Case 18). The report showed that continuous positive airway pressure (CPAP) is effective in reducing obstructive sleep apnea syndrome and daytime sleepiness. There was, however, insufficient scientific evidence to assess the effectiveness of surgery, which is associated with certain risks. Before the report, Norway used surgery more often than the other Nordic countries. Following intense public debate, the frequency of surgical procedures in Norway declined dramatically. Although Sweden had already been resorting to surgery less frequently, there has been a considerable decrease in surgical procedures there as well (Figure 2).
An SBU report on mild head injury (2000) concluded that the evidence was insufficient to determine whether in-hospital observation or a new strategy of CT scans and early return home was more effective in treatment of mild head injury (Case 21). A randomized multicentre study launched by SBU and published in 2006 (Reference af Geijerstam, Oredsson and Britton19;Reference Norlund, Marké, af Geijerstam, Oredsson and Britton23) showed that the two options were equally effective. But SBU's updated report found that the new strategy could reduce total costs by one-third, thereby freeing up resources for other healthcare needs (Reference Norlund, Marké, af Geijerstam, Oredsson and Britton23).
Several local, regional, and national health care programs have been redesigned with respect to procedures for emergency treatment of head injury patients. The case is more thoroughly discussed in a separate study (Reference Carlsson and af Geijerstam24), which shows a substantial decrease in both admission rates and mean hospital stay for mild head injury. The HTA report and the involvement of thirty-nine emergency departments in Sweden in the randomized multicentre study have most likely contributed to the influence on clinical practice.
SBU published a report in 2007 entitled “Methods for Promoting Physical Activity” (Case 20). One of the findings was that counseling supplemented by prescribed physical activity, diaries, pedometers, informational brochures, etc., increases activity by 15–50 percent. The number of prescriptions of physical activity in Sweden increased from 17,000 in 2007 to 28,000 in 2008, and 49,000 in 2010 (Reference Kallings25). The National Healthcare Network for Physical Activity on Prescription has actively disseminated information about the report, and the NBHW included prescriptions of physical activity in its 2011 guidelines for preventing disease.
The project on triage methods at emergency departments increased awareness, as the result of which eighteen additional departments introduced some kind of method or patient flow process (Reference Farrokhnia and Göransson26;27).
Initiating Research
In accordance with a commission from the Government, SBU has been developing a database of knowledge gaps in health and dental care since 2011. One objective of the database is to promote coordinated research in these areas. For example, SBU reports on tooth loss, endodontics (root canal therapy) and methods of preventing mental illness in children found insufficient evidence in many areas, leading to stepped-up research. Knowledge gaps in odontology have spurred collaborative research at universities, and the Swedish Research Council has supported clinical research schools in the field of odontology.
Along with projects by the Royal Swedish Academy of Sciences, the SBU report on methods of preventing mental illness in children spurred invitations by several research councils that led to grants of approximately 30 million euros (Case 22).
SBU and the Swedish Research Council together with the NBHW, SALAR and the Swedish Dental Association arranged national workshops and coalitions for addressing research questions related to knowledge gaps identified in SBU reports on dental care (Case 20, 21).
In each of the past three years, the Swedish Research Council has invited grant applications concerning areas for which there are gaps with respect to our knowledge of health care. The council asks applicants to look at the SBU website to identify such gaps. Only applications that focus on gaps in either the SBU database or the UK Database of Uncertainties about the Effects of Treatment (DUET) are considered.
Summary of the Results
The twenty-six reports published between 2006 and 2010 are summarized in Table 2. We have subjectively estimated whether the impact of each report was low, moderate, or high. Documentation was inadequate for the last three reports (Case 24–26) and the impact was assessed as low.
Table 2. SBU (“Yellow”) Reports Published between 2006 and 2010: Type of Influence, Estimated Impact and Results
Note. All 26 reports are available at www.sbu.se in full Swedish text and English summaries
SBU reports influence decisions to some degree. In the case of mandatory fortification of flour with folic acid, the report has had a high impact. Policymakers requested the report and followed its conclusions to the letter. The report on dementia has been used extensively for educational purposes, but we do not know whether it will change clinical practice. Adoption of the government rehabilitation guarantee suggests that the report on rehabilitation of patients with chronic pain has had a moderate impact.
The influence of SBU reports on national guidelines has been very high in every case. The documentation makes it eminently clear that the NBHW has followed the conclusions of the reports and relied on them as its most important source when drawing up its guidelines.
The next and more important question is whether HTAs influence clinical practice such that it becomes more evidence-based. The SBU report on dyspepsia and gastro-esophageal reflux is the single most impressive example. The decline in the number of operations immediately after the publication of the report convincingly demonstrates that the HTA played a role. In many of the other cases, the trend had already started when the report was published and the greater likelihood is that it reinforced the growth of evidence-based medicine. The case of mild head injury is also quite persuasive, involving a long process beginning in 2000. SBU took the initiative for a randomized controlled study and the results were published in BMJ and in a revised SBU report (2006). Given that thirty-nine Swedish emergency departments participated, professional awareness was high from the start. As a result, the number of admissions and bed-days for mild head injury continued to decline.
DISCUSSION
Changes to clinical practice are often the product of many interrelated factors and it is evidently difficult to identify the independent effect of results from HTAs. Because randomized controlled trials are usually not an option there are several limitations in measuring the impact. We also admit the risk of bias due to self-assessment, but have tried to only rely on written documents and register data possible for others to reassess.
This article has presented twenty-six SBU reports that have likely contributed to changes in practice or decision-making. However, evaluating the benefits and results generated by an evidence-based organization like SBU is not an easy task. The probability that the changes to clinical practice have been influenced by a report certainly varies from one case to another. Thus, a few comments are in order.
Confidence in SBU is a necessary but not sufficient requirement for clinical practice to change. Nevertheless, the results of a survey conducted by an independent organization like Statistics Sweden reveals a high level of confidence in SBU's evidence and value among leading healthcare professionals.
SBU reports have influenced decisions at the national or regional level to one extent or another. When the reports have been used for clinical guidelines by the NBHW or professional associations, the impact is plain to see.
There are several examples of immediate changes to clinical practice after publication of an SBU report. No national guidelines or decisions were adopted by other organizations in such cases. The changes were frequently identified by means of the National Patient Register or the National Prescribed Drug Register, which cover the entire Swedish population. The registers are reliable sources, and the risk of bias is mostly due to confounding factors. The steady number of surgical procedures for gastro-esophageal reflux followed by a substantial reduction immediately after publication of the SBU report offered convincing evidence. The lower frequency of operations for obstructive sleep apnea and the rapid decrease in the number of admissions and bed-days for mild head injury point to a similar impact.
The reports presented in this article show that HTA can influence research. In such cases, the Swedish Research Council has specifically mentioned or collaborated with SBU. The HTA reports are clearly the single explanatory factor.
SBU uses different means for implementing its findings, the most important being involvement of experts in its project groups. Attending meetings for 2–3 years, reading and assessing relevant articles and reaching agreement on findings and conclusions all serve to create effective advocates for the final report. Healthcare professionals are more likely to change their behavior if their colleagues are convinced of the reliability of HTA findings.
Other means used to implement SBU findings include painstaking identification of interested parties and target groups for each topic, widespread dissemination of summaries for caregivers and the general public, a well-considered media strategy, arranging and participating in conferences and educational programs, circulating newsletters, developing interactive patient cases with medical associations and collaborating with county councils and other government agencies. Successful implementation requires meticulous, ongoing effort.
CONCLUSION
Despite the difficulty of unambiguously stating that changes to clinical practice or policymaking are due to HTA only, we would argue that all of the cases presented in this article point to its importance, often substantial and immediate. Our conclusion is that HTA contributes to making the healthcare system both more effective and more responsive to the needs of the general population.
CONTACT INFORMATION
Måns Rosén, PhD ([email protected]), Executive Director, Swedish Council on Health Technology Assessment (SBU), Sweden and professor of health technology assessment, Karolinska Institutet, Sweden Mailing address: SBU, Box 3657, SE-103 59 Stockholm, Sweden Sophie Werkö.
CONFLICTS OF INTEREST
Both authors are employed by SBU and have a professional interest in HTA. All cases have been documented by official statistics, surveys or other sources.
Open access
