Objectives: The prime objective of this study was to investigate whether sponsorship by the pharmaceutical industry affected the results of full economic evaluations (FEE) based on modeling. In particular, we focused on the flourishing literature based on Markov models, by far the most widely exploited tool for estimating lifetime costs and benefits.

Methods: We made a literature search of the international database PubMed to find all the studies on pharmacological treatments based on Markov models published in English in the period January 1, 2004 to June 30, 2009. We selected the FEEs focused on single drugs only, specifically cost-effectiveness and cost-utility analyses. Two hundred articles including FEEs based on Markov models were considered eligible. For the analysis, we classified the FEEs into two groups according to whether or not they had financial backing from the pharmaceutical industry. We then assessed the main conclusions, which were classified as (i) “favorable,” (ii) “doubtful,” and (iii) “unfavorable.”

Results: Of the 200 articles, 138 (69 percent) were sponsored and 162 (81 percent) reached favorable conclusions. Sponsored studies were much more likely to report favorable conclusions than nonsponsored ones (95 percent and 50 percent, p < .001), the former even omitting unfavorable conclusions.

Conclusions: The review found a substantial share of studies supported by the pharmaceutical industry, almost all concluding in favor of the drug studied, without any unfavorable conclusions at all. These results confirm also in the field of pharmacoeconomic studies that the best way of limiting confounding factors is by clearly distinguishing assessors from manufacturers and marketers of any new technology.

In the article “Are key principles for improved health technology assessment supported and used by health technology assessment organizations?” by Neumann et al. (3), the authors set out to evaluate the quality of processes used by health technology assessment (HTA) programs based on a set of principles they published in this journal previously (2). We believe that portions of the current article, as they relate to our own organization, are misleading or inaccurate, and welcome this opportunity to correct the inaccuracies.

Objectives: The aim of this study was to get insight in what criteria as presented in Health technology assessment (HTA) studies are important for decision makers in healthcare priority setting.

Methods: We performed a discrete choice experiment among Dutch healthcare professionals (policy makers, HTA experts, advanced HTA students). In twenty-seven choice sets, we asked respondents to elect reimbursement of one of two different healthcare interventions, which represented unlabeled, curative treatments. Both treatments were incrementally compared with usual care. The results of the interventions were normal outputs of HTA studies with a societal perspective. Results were analyzed using a multinomial logistic regression model. Upon completion of the questionnaire, we discussed the exercise with policy makers.

Results: Severity of disease, costs per quality-adjusted life-year gained, individual health gain, and the budget impact were the most decisive decision criteria. A program targeting more severe diseases increased the probability of reimbursement dramatically. Uncertainty related to cost-effectiveness was also important. Respondents preferred health gains that include quality of life improvements over extension of life without improved quality of life. Savings in productivity costs were not crucial in decision making, although these are to be included in Dutch reimbursement dossiers for new drugs. Regarding subgroups, we found that policy makers attached relatively more weight to disease severity than others but less to uncertainty.

Conclusions: Dutch policy makers and other healthcare professionals seem to have reasonably well articulated preferences: six of seven attributes were significant. Disease severity, budget impact, and cost-effectiveness were very important. The results are comparable to international studies, but reveal a larger set of important decision criteria.

Objectives: The aim of this study was to describe the current issues surrounding Coverage with Evidence Development (CED). CED is characterized by restricted coverage for a new technology in parallel with targeted research when the stated goal of the research or data collection is to provide definitive evidence for the clinical or cost-effectiveness impact of the new technology.

Methods: Presented here is information summarized and interpreted from presentations and discussions at the 2008 Health Technology Assessment International (HTAi) meeting and additional information from the medical literature. This study describes the differences between CED and other conditional coverage agreements, provides a brief history of CED, describes real-world examples of CED, describes the areas of consensus between the stakeholders, discusses the areas for future negotiation between stakeholders, and proposes criteria to assist stakeholders in determining when CED could be appropriate.

Results: Payers could interpret the evidence obtained from a CED program either positively or negatively, and a range of possible changes to the reimbursement status of the new technology may result. Striking an appropriate balance between the demands for prompt access to new technology and acknowledging that some degree of uncertainty will always exist is a critical challenge to the uptake of this innovative form of conditional coverage.

Conclusions: When used selectively for innovative procedures, pharmaceuticals, or devices in the appropriate disease areas, CED may provide patients access to promising medicines or technologies while data to minimize uncertainty are collected.

Objectives: The aim of this study was to test the feasibility of conducting rigorous, nonrandomized studies (NRSs) of healthcare interventions using existing clinical databases in terms of the following: recruiting a large representative sample of hospitals, identifying eligible cases, matching cases to controls to achieve similar baseline characteristics, making meaningful comparisons of outcomes, and carrying out subgroup analyses.

Methods: Data were extracted from the Intensive Care National Audit & Research Centre's Case Mix Programme Database to investigate the impact of management with a pulmonary artery catheter (PAC) in intensive care unit (ICU) patients. Participating ICUs were invited to collect additional data for the analysis. Patients managed with a PAC were matched to control patients on their propensity score. Hospital mortality was then compared between the two groups.

Results: Of 117 eligible ICUs, 68 (58 percent) agreed to participate, of which 57 (84 percent) collected additional data. Although a slightly higher proportion of larger ICUs in university hospitals participated, the patient case-mix was similar to that in nonparticipating ICUs. Almost all patients managed with a PAC (98 percent) were successfully matched to patients managed without a PAC. The two groups were similar for baseline characteristics. However, hospital mortality was worse for PAC patients than for non-PAC patients (odds ratio, 1.28; 95 percent confidence interval, 1.06–1.55). Subgroup analysis suggested that the impact of management with a PAC was modified by severity of illness.

Conclusions: Rigorous NRSs are feasible if they are based on data from high-quality clinical databases. However, the reliability of estimated treatment effects from such studies requires further investigation.

Thank you for the opportunity to respond to the letter by Gibson and Little. The authors raise several points that require a response. First, they claim that we inaccurately characterized the DERP/Washington State Medicaid agency, and did not appreciate that the DERP and the Washington State Medicaid agency have different missions. On the contrary, however, we fully recognized that DERP's mission is to conduct systematic reviews and not to make recommendations, whereas Washington State uses the reviews in making recommendations for their Medicaid enrollees. We specifically noted in the study that “Washington Medicaid is one of fourteen participants in the DERP. DERP researchers conduct health technology assessments for drug classes. Participants in the DERP, such as the Washington Medicaid program, retain local authority for interpreting DERP reports and for decision making regarding which drugs to pay for.” We chose to analyze DERP/Washington State as a single entity for our exercise because we were interested in analyzing the link between the HTA conducted and the decisions that follow them.

Objectives: Since 2002, the treatment with cholinesterase inhibitors (CHEIs) for Alzheimer's disease (AD) has been paid for by the public health system of the Brazilian Ministry of Health for any patient that fulfills clinical criteria established by an evidence-based guideline developed and published by the Ministry. The aim of this study was to evaluate compliance of prescription patterns to the national guideline for use of CHEIs’ in the southern Brazilian state of Rio Grande do Sul.

Methods: We created a regional expert-committee reference center to review all prescriptions of CHEIs and to send feedback to physicians whenever prescriptions without compliance to the guideline were noted. One thousand three hundred ninety-nine (1,399) CHEI prescriptions presented to the public health system from 2005 to 2007 were evaluated by an expert team of neurologists and psychiatrists. Clinical history, performance on mental status screening by Mini Mental State Examination (MMSE), Clinical Dementia Rating scale (CDR), laboratory results, and neuroimaging findings were evaluated in relation to the adherence to the national guideline's recommendations. If the prescription was rejected because of lack of adherence to the criteria of the guideline, a written response was sent by the expert committee to physicians concerning the request.

Results: The majority of the requests (n = 1,044; 75 percent) did not meet the AD guideline's criteria, either for diagnosis or for treatment, and were not granted. A diagnostic mistake was evident in 64.3 percent of cases. Findings of vascular or Parkinson's dementia or severe AD were the main reasons for rejection. Rivastigmine was the most prescribed cholinesterase inhibitor, used in 86 percent of cases. Of note was the reduction in the number of CHEIs prescriptions in the years following this intervention.

Conclusions: The public health strategy of using expert-review of prescriptions and their compliance to national guideline revealed a low rate of rational use of CHEIs for dementia. Such a strategy is relevant for protecting patients from unproven medical interventions and for reducing waste of resources.

There is increasing emphasis on providing patient-focused health care and ensuring patient involvement in the design of health services. As health technology assessment (HTA) is meant to be a multidisciplinary, wide-ranging policy analysis that informs decision making, it would be expected that patients’ views should be incorporated into the assessment. However, HTA is still driven by collection of quantitative evidence to determine the clinical and cost effectiveness of a health technology. Patients’ perspectives about their illness and the technology are rarely included, perhaps because they are seen as anecdotal, biased views. There are two distinct but complementary ways in which HTAs can be strengthened by: (i) gathering robust evidence about the patients’ perspectives, and (ii) ensuring effective engagement of patients in the HTA process from scoping, through evidence gathering, assessment of value, development of recommendations and dissemination of findings. Robust evidence eliciting patients’ perspectives can be obtained through social science research that is well conducted, critically appraised and carefully reported, either through meta-synthesis of existing studies or new primary research. Engagement with patients can occur at several levels and we propose that HTA should seek to support effective patient participation to create a fair deliberative process. This should allow two-way flow of information, so that the views of patients are obtained in a supportive way and fed into decision-making processes in a transparent manner.

Objectives: The aim of this study was to determine organizational characteristics evident in successful telemedicine programs. The study proposed four research questions to identify the key organizational characteristics for successful telemedicine programs.

Methods: Data were collected through an online survey using modified unified theory of acceptance and use of technology (UTAUT) and organizational change readiness (ORC) validated assessments.

Results: Using the UTAUT instrument and the ORC assessment, themes of successful telemedicine programs emerged. Overarching themes of successful programs included being formally structured, forward thinking, seeking to improve the quality of patient care, and being financially stable.

Conclusions: This research provides a data-driven understanding of organizational issues that serve to maximize a telemedicine program's ability to achieve the overall goals and mission of a healthcare organization.

Objectives: The aim of this study is to review briefly different methods for determining the optimal retrieval of studies for inclusion in a health technology assessment (HTA) report.

Methods: This study reviews the methodology literature related to specific methods for evaluating yield from literature searching strategies and for deciding whether to continue or desist in the searching process.

Results: Eight different methods were identified. These include using the Capture–recapture technique; obtaining Feedback from the commissioner of the HTA report; seeking the Disconfirming case; undertaking comparison against a known Gold standard; evaluating retrieval of Known items; recognizing the Law of diminishing returns, specifying a priori Stopping rules, and identifying a point of Theoretical saturation.

Conclusions: While this study identified a variety of possible methods, there has been very little formal evaluation of the specific strengths and weaknesses of the different techniques. The author proposes an evaluation agenda drawing on an examination of existing data together with exploration of the specific impact of missing relevant studies.

Objectives: The aim of this study was to describe a current practical approach of priority setting of health technology assessment (HTA) research that involves multi-criteria decision analysis and a deliberative process.

Methods: Criteria related to HTA prioritization were identified and grouped through a systematic review and consultation with a selection committee. Criteria were scored through a pair-wise comparison approach. Criteria were pruned based on the average weights obtained from consistent (consistency index < 0.2) responders and consensus. HTA proposals are ranked based on available information and a weighted criteria score. The rank, along with additional contextual information and discussion among committee members, is used to achieve consensus on HTA research priorities.

Results: Six of eleven criteria represented > 75 percent of the weight behind committee member decisions to conduct an HTA. These criteria were disease burden, clinical impact, alternatives, budget impact, economic impact, and available evidence. Since May 2006, committees have considered 102 proposals at sixteen biannual in-person advisory committee meetings. These have selected twenty-nine research priorities for the HTA program.

Conclusions: The approach works well and was easy to implement. Feedback from committee members has been positive. This approach may assist HTA and other research agencies in better priority setting by informing the selection of the most important and policy-relevant topics in the presence of a wide variety of research proposals. This may in turn lead to efficiently allocating resources available for HTA research.

Objectives: This study reviews the coverage and usefulness of a short-term register, established specifically for health technology assessment of a novel interventional procedure (minimally invasive repair of pectus excavatum, or the Nuss procedure).

Methods: Coverage of the register during 2004–07 was assessed by comparison with Hospital Episodes Statistics (HES) for England. Its usefulness was assessed by comparing safety and efficacy data with the published literature and by feedback from committee members who in 2009 were involved in reviewing NICE's original guidance from 2003.

Results: The register reported 260 cases from thirteen UK hospitals during nearly 9 years. During a coverage evaluation period of 3 years, there were 152 registered Nuss procedures. An additional 246 repairs of pectus excavatum were undertaken in twenty-six previously unidentified hospitals. Of the 246, 23 were Nuss procedures (from two hospitals), 140 were open procedures (from eleven hospitals), and 3 were coding errors. No details were available for eighty cases undertaken at ten hospitals. The quantity of published literature had increased substantially since publication of original guidance in 2003. It related mostly to technical and safety outcomes, whereas the register included patient reported outcomes. The literature and the register reported similar rates of major adverse events such as bar displacement (2–10 percent). Committee members considered that the Register made a useful contribution to guidance development.

Conclusions: This study shows that a register set up to support a health technology assessment process can produce useful data both about safety and about patient-reported outcomes. Coverage may be improved by active follow-up based on routine hospital statistics. Improvement in coding for new procedures is needed in the United Kingdom.

Objectives: The aim of this study was to identify and compare health technology assessments of the same new interventional procedures produced in different countries.

Methods: We selected five new interventional procedures and studied related assessments produced in different countries.

Results: There were twenty assessments (range, 3–5 per procedure) from nine countries—fourteen from Australia, Canada, and United Kingdom. The number of primary RCTs cited by the assessments ranged from 0 to 13. In the assessment reports, “headline” statements about the strength of evidence for efficacy (73 percent) were made more frequently than for safety (53 percent). These statements were scored for their apparent judgment of the strength of the evidence—1 (poor) to 5 (strong)—and received scores of 3 or less in all but four cases. Recommendations about additional research were included in 55 percent of the assessments. Statements in assessments about other aspects of use of the procedures were included more infrequently—in 35 percent for patient selection, in 20 percent for consent issues, and in 15 percent for types of clinical teams. Recommendations about appropriate healthcare settings, or about operator training, were included only in assessments produced by a single organization.

Conclusion: There was a only small number of assessments world-wide, for a range of new procedures with potentially high impact. Where available, assessments were produced on a relatively poor evidence base. International collaboration in evidence appraisal and review, and in the gathering of new data through research or registers, could improve the advice available to healthcare systems worldwide about the adoption of new interventional procedures.

Objectives: The English National Horizon Scanning Centre routinely scans thirty-five sources to identify new and emerging health technologies. The aim of the study was to develop and apply evaluation criteria and scores to assess the value of sources, and to identify a cutoff score below which sources would be recommended for removal from routine horizon scanning. Criteria to evaluate each source scanned could result in a more efficient approach in the selection process.

Methods: Evaluation criteria were developed following a review of the literature and discussions with horizon analysts. Proposed criteria were piloted on a random selection of six sources, and then applied to all thirty-five sources. The criteria were assessed using the Analytic Hierarchy Process.

Results: Eight criteria were identified as being most relevant for assessing the value of scanning sources. The three most important (primary) criteria were coverage (approximate percentage of relevant information), quality (reliable, accurate, objective), and efficiency (estimated time to identify one potentially significant health technology or other relevant information). Seven sources fell beneath the cutoff score and were recommended for removal from routine scanning.

Conclusions: The criteria were considered useful in the assessment of current sources, and have the potential to be used to assess new ones. These criteria may be useful for other horizon scanning centers to pilot and validate.

Objectives: The objectives of this study is to review the set of criteria of the Institute of Medicine (IOM) for priority-setting in research with addition of new criteria if necessary, and to develop and evaluate the reliability and validity of the final priority score.

Methods: Based on the evaluation of 199 research topics, forty-five experts identified additional criteria for priority-setting, rated their relevance, and ranked and weighted them in a three-round modified Delphi technique. A final priority score was developed and evaluated. Internal consistency, test–retest and inter-rater reliability were assessed. Correlation with experts’ overall qualitative topic ratings were assessed as an approximation to validity.

Results: All seven original IOM criteria were considered relevant and two new criteria were added (“potential for translation into practice”, and “need for knowledge”). Final ranks and relative weights differed from those of the original IOM criteria: “research impact on health outcomes” was considered the most important criterion (4.23), as opposed to “burden of disease” (3.92). Cronbach's alpha (0.75) and test–retest stability (interclass correlation coefficient = 0.66) for the final set of criteria were acceptable. The area under the receiver operating characteristic curve for overall assessment of priority was 0.66.

Conclusions: A reliable instrument for prioritizing topics in clinical and health services research has been developed. Further evaluation of its validity and impact on selecting research topics is required.

Objectives and Methods: Health technology assessment (HTA) often requires the identification and review of economic evaluations and models. This study surveys the available specific and general resources to search to identify economic evaluations. It also provides information on efficient searching of those resources and comments on the current evidence-base.

Results: Published checklists recommend searching for economic evaluations in specific information resources which collect economic evaluations such as NHS EED and HEED, followed by top-up searches of large biomedical bibliographic databases (such as MEDLINE and EMBASE). Other resources such as the HTA and DARE databases can yield reports of economic evaluations. Searches within NHS EED and HEED can be made more efficient by using database-specific search options. Searches within large biomedical databases such as MEDLINE and EMBASE require the use of economic search terms called search filters. Search filters are highly sensitive, retrieving most economic evaluations, but suffer from low precision returning many irrelevant records which need to be assessed.

Conclusions: It is relatively easy to identify rapidly a high proportion of economic evaluations but more research is required to improve the efficiency of this process. There are key high yield resources to search but more evidence is required on their overlap and unique contribution to searches. The value of other resources, particularly those providing access to gray literature, should be explored. Research into efficient retrieval requires clear definitions of economic evaluations to allow comparison across studies.

We read the recently published study by Lo Scalzo et al. (4). The study is a short version of a more extensive report published on the Italian Ministry of Health Web site (1). A careful evaluation of both documents reveals several critical issues that cast doubts on the correctness of their conclusions. Usually, decisions about clinical practice should be based on levels of evidence in literature and on the strength of recommendations issued in the guidelines of international scientific societies, many of which rely on the Oxford system (Oxford Center for Evidence based Medicine) (http://cebm.jr2.ox.ac.uk/docs/level.html). As far as diagnostic tests are concerned, the Oxford system assigns the highest level of evidence to systematic reviews supported by homogeneous results. The same system assigns the second highest level of evidence to the “independent blind comparison of an appropriate spectrum of consecutive patients all of whom have undergone both the diagnostic test and the reference standard,” while the expert's opinion ranks at the lowest level. Surprisingly, the authors flatly dismissed all sequential studies that fulfill the criteria of independent blind comparison, based on the fact that they were not randomized. At the same time, they cited as source data the results of a questionnaire presented to the attendees of an Italian meeting, never published in literature, which is plainly an “expert opinion.”

Objectives: Nonpublication of results of clinical trials can contribute to inappropriate medical decisions. The primary aim of this systematic review was to investigate publication delays between conference abstracts and full journal publications from randomized controlled trial results of new anticancer agents for breast cancer. The review was restricted to anticancer agents previously, or due to be, appraised in the United Kingdom by the National Institute for Health and Clinical Excellence. A secondary objective was to identify whether there are any apparent biases in the publication and reporting of these trials.

Methods: We searched six electronic databases up to August 2007, including Medline and the Cochrane Library. Two reviewers independently selected studies, extracted and assessed the data.

Results: Six anticancer treatments were identified: docetaxel, paclitaxel, trastuzumab, gemcitabine, lapatinib, and bevacizumab. Of eighteen included trials, only four publications from three trials reported the same outcomes in both abstract and full publication. Time delays ranged from 5 to 19 months. Eleven trial abstracts were still without a full publication at the end of our searches, varying from 3 to 38 months since abstract publication. Observational analysis revealed no particular publishing biases.

Conclusions: Whereas delays in publication appear reasonable over a period of months, many were not published in full over a period of years and others would appear to be unlikely to ever be published. Further research should investigate the impact of publication delays on the availability of new drug treatments in clinical practice.

I was dismayed to find that the article on HTA in the United Kingdom in the recent Special Issus of IJTAHC on the History of HTA (1) makes so little mention of the main component of HTA in England and Wales, namely the NHS HTA Programme. That seems a puzzling oversight for a program that, since its inception in 1993, has spent 138 million GBP on commissioning nearly 850 in-depth HTA studies, sifted and prioritized from some 15,000 suggestions drawn from all quarters of UK health service, policy, and research worlds. From these, it has published nearly 500 full-length HTA monographs (with over 125 more currently in editorial review) in the series Health Technology Assessment, which has attracted over 15 million downloads as well as being distributed in hard copy to just over 128,000 recipients. Around 120 of these monographs resulted from research commissioned explicitly to advance the methodology of HTA—a contribution that is unique among the world's HTA agencies. This stream of HTA publications, which has also spawned countless summaries, local adaptations, translations, academic papers, and other spinoffs, has had an acknowledged impact throughout the world of HTA.

Background: Comprehensive health technology assessments (HTAs) include thorough reflections on ethical issues associated with health technologies, their use, and value-based decisions in the assessment process. As methods of information retrieval for effectiveness assessments are not applicable to information retrieval on ethical issues, a specific methodological approach is necessary.

Objectives: In the absence of existing adapted methods, our objective was to develop a methodological approach for the systematic retrieval of information on ethical issues related to health technologies.

Method and Results: A literature search was conducted to verify the non-existence of published comprehensive methodological approaches for the information retrieval on ethical issues for HTAs, and resulted in no hits. We, therefore, developed a step-by-step workflow following the workflow of information retrieval for effectiveness assessments: Step 1: Translation of the search question using the PICO scheme and additional components. Step 2: Concept building by modeling and linking search components. Step 3: Identification of synonyms in all relevant languages. Step 4: Selection of relevant information sources. Step 5: Design of search strategies for bibliographic databases. Step 6: Execution of search strategies and information seeking, including hand-searching. Step 7: Saving of retrieval results and standardized reporting of the process and results. Step 8: Final quality check and calculation of precision and recall.

Conclusions: Systematic searching for information on ethical issues related to health technologies can be performed following the common retrieval workflow for effectiveness assessments, but should be performed separately applying adapted procedures and search terms on ethical issues relevant to the research question.