This issue of the International Journal of Technology Assessment in Health Care (IJTAHC) brings to the readers a lively discussion on the possibility of benchmarking health technology assessment (HTA) organizations. Michael Drummond and colleagues present a paper discussing “the conceptual and methodological challenges associated with benchmarking” (1). They propose a set of criteria for comparing the quality of organizations that produce assessments on health technologies. Aiming at explicit and transparent comparisons, Drummond et al. also highlight the possible problems in their approach, and present their paper as a starting point for discussion.

Objectives: The aim of this study was to summarize and assess economic evaluations of poison centers (PCs) from the perspectives of society, the payer, and the healthcare system.

Methods: A systematic review was performed to identify complete economic evaluations regardless of the language or publication status. Two reviewers evaluated the abstracts for eligibility, extracted the data, and assessed the study quality using a standardized tool.

Results: In total, 422 non-duplicated studies were retrieved, but only nine met the eligibility criteria. Five of the eligible studies were published in the 1990s, and four were published in the 2000s. Six studies met at least seven of ten quality criteria. In all studies, the presence of PCs was compared with a scenario of their absence. Eight studies used cost–benefit analyses and one used a cost-effectiveness approach. The cost–benefit ratios ranged from 0.76 to 7.67, which indicates that each United States dollar (USD) spent on poison centers can save almost 8 USD on medical spending. A cost-effectiveness analysis showed that each successful outcome achieved by a PC avoids a minimum of 12,000 USD to 56,000 USD in other healthcare spending.

Conclusions: The data in our review show that PCs are economically viable. PCs improve the efficiency of healthcare expenditure and contribute to the sustainability of the healthcare system. An investment in PCs is a rational public health policy approach that contrasts the current trend of reducing spending on PCs.

A decision analytic model often comprises a significant part of a health technology assessment. As health technology assessment in the hospital setting evolves, there is an increased need for modeling methods that account for patient care pathways and interactions between patients and their environment. For example, an evaluation of a computed tomography (CT) scanner for a new indication would need to consider the current and increased demand of the machine and how that may affect service in other areas of the hospital. This problem solving approach views “problems” through a systems perspective.

Objectives: The aim of this study was to evaluate the cost-utility of Docetaxel with doxorubicin and cyclophosphamide (TAC) and 5-fluorouracil, doxorubicin, cyclophosphamide (FAC) in node-positive breast cancer patients in the south of Iran.

Methods: A double blind study was done on a cohort of 100 patients suffering from breast cancer with node-positive over 8 months in the radiotherapy center of Namazi hospital, Shiraz-Iran. Health-related quality of life was assessed using questionnaire (QLQ-C30) from European Organization for Research and Treatment of Cancer (EORTC). QLQ-C30 scale scores were mapped to 15D and EuroQol 5D utilities to measure the quality-adjusted life-years (QALYs).Third party payer point of view was applied to measure and value the cost of treatments. Cost data were extracted from hospital and health insurance organizations. Robustness of the results was checked through a two way sensitivity analysis.

Results: TAC was associated with higher deterioration in HRQoL during treatment and higher improvements over 4 months follow-up. On average, the cost of treatment per patient in TAC was 15 times higher than FAC (p < .001). In overall, TAC was resulted in lower QALYs and higher cost over study period.

Conclusions: FAC was a dominant option versus TAC in short-term. The higher improvement in HRQoL over follow-up in TAC may not compensate the more intensive deterioration caused during treatment in short-term. The short time horizon of study may limit the generalizability of our findings and, hence, there is a need to conduct long-term economic evaluation studies whenever data is available to inform decision making.

Objectives: The aim of this study is to describe and illustrate a method to obtain early estimates of the effectiveness of a new version of a medical device.

Methods: In the absence of empirical data, expert opinion may be elicited on the expected difference between the conventional and modified devices. Bayesian Mixed Treatment Comparison (MTC) meta-analysis can then be used to combine this expert opinion with existing trial data on earlier versions of the device. We illustrate this approach for a new four-pole implantable cardioverter defibrillator (ICD) compared with conventional ICDs, Class III anti-arrhythmic drugs, and conventional drug therapy for the prevention of sudden cardiac death in high risk patients. Existing RCTs were identified from a published systematic review, and we elicited opinion on the difference between four-pole and conventional ICDs from experts recruited at a cardiology conference.

Results: Twelve randomized controlled trials were identified. Seven experts provided valid probability distributions for the new ICDs compared with current devices. The MTC model resulted in estimated relative risks of mortality of 0.74 (0.60–0.89) (predictive relative risk [RR] = 0.77 [0.41–1.26]) and 0.83 (0.70–0.97) (predictive RR = 0.84 [0.55–1.22]) with the new ICD therapy compared to Class III anti-arrhythmic drug therapy and conventional drug therapy, respectively. These results showed negligible differences from the preliminary results for the existing ICDs.

Conclusions: The proposed method incorporating expert opinion to adjust for a modification made to an existing device may play a useful role in assisting decision makers to make early informed judgments on the effectiveness of frequently modified healthcare technologies.

Objectives: The aim of this study was to promote approaches to health technology assessment (HTA) that are both evidence-based and values-based. We conducted a systematic review of published studies describing formal methods to consider equity in the context of cost-effectiveness analysis (CEA).

Methods: Candidate studies were identified through an unrestricted search of the Pub Med and EMBASE databases. The search closed on January 20, 2011. We identified additional studies by consulting experts and checking article bibliographies. Two authors independently reviewed each candidate study to determine inclusion and extracted data from studies retained for review. In addition to documenting methods, data extraction identified implicit and explicit notions of fairness. Data were synthesized in narrative form. Study quality was not assessed.

Results: Of the 695 candidate articles, 51 were retained for review. We identified three broad methods to facilitate quantitative consideration of equity concerns in economic evaluation: integration of distributional concerns through equity weights and social welfare functions, exploration of the opportunity costs of alternative policy options through mathematical programming, and multi-criteria decision analysis.

Conclusions: Several viable techniques to integrate equity concerns within CEA now exist, ranging from descriptive approaches to the quantitative methods studied in this review. Two obstacles at the normative level have impeded their use in decision making to date: the multiplicity of concepts and values discussed under the rubric of equity, and the lack of a widely accepted normative source on which to ground controversial value choices. Clarification of equity concepts and attention to procedural fairness may strengthen use of these techniques in HTA decision making.

Objectives: The aim of this study is to determine the relative value and contribution of searching different sources to identify adverse effects data.

Methods: The process of updating a systematic review and meta-analysis of thiazolidinedione-related fractures in patients with type 2 diabetes mellitus was used as a case study. For each source searched, a record was made for each relevant reference included in the review noting whether it was retrieved with the search strategy used and whether it was available but not retrieved. The sensitivity, precision, and number needed to read from searching each source and from different combinations of sources were also calculated.

Results: There were 58 relevant references which presented sufficient numerical data to be included in a meta-analysis of fractures and bone mineral density. The highest number of relevant references were retrieved from Science Citation Index (SCI) (35), followed by BIOSIS Previews (27) and EMBASE (24). The precision of the searches varied from 0.88% (Scirus) to 41.67% (CENTRAL). With the search strategies used, the minimum combination of sources required to retrieve all the relevant references was; the GlaxoSmithKline (GSK) website, Science Citation Index (SCI), EMBASE, BIOSIS Previews, British Library Direct, Medscape DrugInfo, handsearching and reference checking, AHFS First, and Thomson Reuters Integrity or Conference Papers Index (CPI).

Conclusions: In order to identify all the relevant references for this case study a number of different sources needed to be searched. The minimum combination of sources required to identify all the relevant references did not include MEDLINE

Objectives: The English language is generally perceived to be the universal language of science. However, the exclusive reliance on English-language studies may not represent all of the evidence. Excluding languages other than English (LOE) may introduce a language bias and lead to erroneous conclusions.

Study Design and Setting: We conducted a comprehensive literature search using bibliographic databases and grey literature sources. Studies were eligible for inclusion if they measured the effect of excluding randomized controlled trials (RCTs) reported in LOE from systematic review-based meta-analyses (SR/MA) for one or more outcomes.

Results: None of the included studies found major differences between summary treatment effects in English-language restricted meta-analyses and LOE-inclusive meta-analyses. Findings differed about the methodological and reporting quality of trials reported in LOE. The precision of pooled estimates improved with the inclusion of LOE trials.

Conclusions: Overall, we found no evidence of a systematic bias from the use of language restrictions in systematic review-based meta-analyses in conventional medicine. Further research is needed to determine the impact of language restriction on systematic reviews in particular fields of medicine.

Objectives: The aim of this study was to identify and characterize studies that have used quality-adjusted life-years (QALYs) based on measurements of patients’ health-related quality of life (HRQoL) as an indicator of effectiveness of screening programs.

Methods: Systematic search of the literature until March 2010, using several electronic databases. Initial screening of articles based on abstracts, and evaluation of full-text articles were done by at least two of the authors.

Results: The search identified 1,610 articles. Based on review of abstracts, 431 full-text articles were obtained for closer inspection and, of these, 81 reported QALYs based on patient-derived data using a valid HRQoL assessment. The most frequently used method to assess HRQoL was Time Trade-Off (55 percent) followed by EQ-5D (26 percent). The most frequently studied medical conditions were malignant diseases (23 percent) followed by cardiovascular diseases (19 percent). All studies employed some kind of modeling with the Markov model being the most prevalent type (65 percent). Majority of the articles (59 percent) concluded that the screening program studied was cost-effective.

Conclusions: The use of QALYs in the evaluation of screening programs has expanded during the last few years. However, only a minority of studies have used HRQoL data derived from patients, using direct or indirect valuation. Further investigation and harmonization of the methodology in evaluation of screening programs is needed to ensure better comparability across different screening programs.

Objectives: In 2000, the first “Dutch Manual for Costing: Methods and Reference Prices for Economic Evaluations in Healthcare” was published, followed by an updated version in 2004. The purpose of the Manual is to facilitate the implementation and assessment of costing studies in economic evaluations. New developments necessitated the publication of a thoroughly updated version of the Manual in 2010. The present study aims to describe the main changes of the 2010 Manual compared with earlier editions of the Manual.

Methods: New and updated topics of the Manual were identified. The recommendations of the Manual were compared with the health economic guidelines of other countries, eliciting strengths and limitations of alternative methods.

Results: New topics in the Manual concern medical costs in life-years gained, the database of the Diagnosis Treatment Combination (DBC) casemix System, reference prices for the mental healthcare sector and the costs borne by informal care-givers. Updated topics relate to the friction cost method, discounting future effects and options for transferring cost results from international studies to the Dutch situation.

Conclusions: The Action Plan is quite similar to many health economic guidelines in healthcare. However, the recommendations on particular aspects may differ between national guidelines in some respects. Although the Manual may serve as an example to countries intending to develop a manual of this kind, it should always be kept in mind that preferred methods predominantly depend on a country's specific context.

Objectives: In recent years, there has been growth in the use of health technology assessment (HTA) for making decisions about the reimbursement, coverage, or guidance on the use of health technologies. Given this greater emphasis on the use of HTA, it is important to develop standards of good practice and to benchmark the various HTA organizations against these standards.

Methods: This study discusses the conceptual and methodological challenges associated with benchmarking HTA organizations and proposes a series of audit questions based on a previously published set of principles of good practice.

Results and Conclusions: It is concluded that a benchmarking exercise would be feasible and useful, although the question of who should do the benchmarking requires further discussion. Key issues for further research are the alternative methods for weighting the various principles and for generating an overall score, or summary statement of adherence to the principles. Any weighting system, if developed, would need to be explored in different jurisdictions to assess the extent to which the relative importance of the principles is perceived to vary. Finally, the development and precise wording of the audit questions requires further study, with a view to making the questions as unambiguous as possible, and the reproducibility of the assessments as high as possible.

Health technology assessment (HTA) is now 35 years old (1). During this time, the science and use of HTA has evolved enormously, adapting to changing healthcare and policy environments. Today, the HTA community has strong methods and procedures to produce HTA (5–7;10), and its results are increasingly used worldwide for decision making. It seems therefore that the time has arrived to set up best practices and to benchmark HTA processes and organizations. The work initiated by Drummond et al. is highly valued, because it opens a new area of needed work following the HTA community's previous emphasis on methodological and process development.

The paper by Drummond et al. in this issue of the Journal discusses how the principles that have been proposed in previous papers by these authors might be used as the basis for benchmarking health technology assessment (HTA) organizations. This raises a number of important issues, some acknowledged and discussed by the authors, and some not. The commentary by Sampietro provides an analysis of many of these. Two issues strike me as fundamental in this debate: the need for an agreed and objective approach to describing health system decision-making systems; and the need for an agreed and objective approach to the assessment of the quality of HTA reports to support specific decisions.

Since the origin of technology assessment (TA) in the 1960s, including TAs of certain medical interventions, and formal recognition and institutionalization of health technology assessment (HTA) in the 1970s, the field of HTA has evolved in programs large and small in the public and private sectors. It has diffused into an increasing number of nations in a range of policy-making and decision-support contexts. Since its inception, the development and diffusion of HTA have been strongly characterized by international collaboration involving sharing of expertise and practical experience, and joint efforts to advance the state of the art. Also during that time, there have been periodic collaborations to identify or develop standard frameworks, good practices, guiding principles, checklists, and other normative, “how-to” documentation for the field. Recognizing the diversity and ongoing evolution of HTA, these efforts have generally sought to balance the appeal of standardization with that of flexibility to accommodate the ranges of HTA program remits and contexts.

Objectives: A new decision-making process was set up by the Austrian Ministry of Health to regulate coverage of new proposed Extra Medical Services (EMS; German: Medizinische Einzel-Leistung [MEL]) in 2008. As part of the annual decision-making process an independent academic institution (LBI-HTA) is evaluating relevant evidence on these new technologies and provides HTAs, including evidence-based recommendations for decision makers.

Methods: About ten EMS assessments are performed annually by the LBI-HTA simultaneously between January and March. Each peer-reviewed report consists of a systematic literature review and critical appraisal of evidence using the GRADE methodology. The generation of numerous reports of good quality standards within the short timeframe is achieved by a standardized workflow with predefined assignment of tasks for all participants.

Results: In total, the LBI-HTA performed twenty-five EMS assessments on thirty-three different interventions in the last three years. Coverage was recommended with limitation for eleven (33%) interventions, and not recommended for twenty-two (66%) interventions. The federal health commission decided on acceptance or preliminary acceptance of coverage in seven (22%) cases, rejection in eighteen (55%) cases and changed the status to “subject to approval” in seven (24%) cases.

Conclusions: Pre-coverage assessment of new hospital interventions was implemented successfully in Austria. It has proved to be a useful tool to support decision makers with objective evidence when deciding whether or not to reimburse medical services.

Introduction: HTA agencies, especially in developing countries, are under resourced and unable to conduct the desired amount of assessments. Adapting HTA reports (HTAs) from other jurisdictions is an alternative for saving resources.

Objectives: To explore HTA transferability experiences in Latin-America and Caribbean (LAC): are decision makers (DMs) using HTAs from other jurisdictions? Are researchers adapting HTAs when developing local reports? How useful is the information found in HTAs from other jurisdictions?

Methods: Web-based survey sent to 13031 HTA researchers and DMs.

Results: We received 671 responses from 19 countries. DMs reported using HTAs from other jurisdictions to guide decisions in the majority of the situations: 52.6 percent HTAs from outside LAC (e.g., Europe), 23.1 percent from other LAC countries, and only 24.3 percent HTAs from their own countries. 63 percent of researchers reported using HTAs from other jurisdictions. Usefulness scored significantly higher for HTAs from other jurisdictions as compared to local HTAs (7.1 versus 6.0 in a 1–10 scale; p < .01). Both DMs and researchers considered the information regarding safety and effectiveness more applicable than the information on social aspects, or economic evaluation. Barriers that limit transferability had significantly different scores for HTAs from other LAC countries as compared to those from regions outside LAC (i.e., poor methodological quality 6.7 versus 5.3, different epidemiological context 6.0 versus 7.4; all p < .01).

Conclusions: HTAs from outside the region are commonly used. However, DMs and researchers agreed that HTAs from LAC had the greatest potential for transferability, provided that barriers such as poor methodological quality could be overcome.

Objectives: Pharmaceutical reimbursement agencies’ processes and methods of appraisal vary across countries. The objective of this study was to examine the contribution of formal health economic analysis in a process using such analysis in Scotland in comparison to a process not routinely using such analysis in France.

Methods: A framework for classifying reimbursement systems was used to analyze the two systems. A typology of recommendation was defined and a qualitative analysis of decisions on a sample of medicines appraised by both reimbursement agencies was conducted. Reasons for differences in recommendations were analyzed and case studies selected to illustrate the common reasons.

Results: Thirty-nine common medicines appraised by both agencies were identified between 2005 and 2010, treating a variety of diseases for which the Scottish Medicines Consortium tended to provide more restrictive, or did not recommend, listing. Similarities in clinical evidence submitted to the respective reimbursement committees were observed. Differences in recommendation can be explained by a combination of the manufacturer's freedom to set price and the incentives provided by the consideration of health economic analysis and quality of life, alongside differences in relevant comparators, relevant outcomes, treatment guidelines, and the propensity to use network meta-analysis, in decision making.

Conclusions: This study provides some explanations and hypotheses for the differences observed in recommendations for a selected sample of medicines with regards to differences in appraisal processes and methods adopted. Further research using larger datasets may allow stakeholders to assess the impact of such differences on the efficient use of health resources.

In their article “Different methods for ethical analysis in health technology assessment: An empirical study” published in International Journal of Technology Assessment in Health Care, Volume 27 Number 4, Samuli I. Saarni, Annette Braunack-Mayer, Bjørn Hofmann, and Gert Jan van der Wilt present an empirical study about different methods for ethical analysis in health technology assessment (HTA). They claim that ethical analysis is an important issue which can highlight values inherent in a technology and value-decisions underlying a HTA process (4). Although their study is rich in detail; gives an interesting overview of casuistry, principlism, and axiological ethic frameworks; and make ethical evaluations within HTA more transparent, we suppose, one important point needs further serious discussion: the clarification of what is meant by “value.”

Kathrin Dengler and Uta Bittner demand a full-fledged philosophy of values in our empirical study of various methods for ethical analysis in health technology assessment (HTA). This may be like putting the classification of disease on hold until the concept of disease is clarified, or postponing the development of health care until the term “health” is clarified. As Dengler and Bittner rightly point out, the term value has many meanings, and as they properly recognize: “[P]hilosophically, the definition of what is meant by ‘a good life’ or ‘well-being’ is a very challenging project.” Hence, it may be a bit over the top to crave that we solve eternal issues in an empirical article on methodology.