Objectives: The aim of this study was to review and compare current health technology assessment (HTA) activities for medical devices across non-European Union HTA agencies.

Methods: HTA activities for medical devices were evaluated from three perspectives: organizational structure, processes, and methods. Agencies were primarily selected upon membership of existing HTA networks. The data collection was performed in two stages: stage 1–agency Web-site assessment using a standardized questionnaire, followed by review and validation of the collected data by a representative of the agency; and stage 2–semi-structured telephone interviews with key informants of a sub-sample of agencies.

Results: In total, thirty-six HTA agencies across twenty non-EU countries assessing medical devices were included. Twenty-seven of thirty-six (75 percent) agencies were judged at stage 1 to have adopted HTA-specific approaches for medical devices (MD-specific agencies) that were largely organizational or procedural. There appeared to be few differences in the organization, process and methods between MD-specific and non–MD-specific agencies. Although the majority (69 percent) of both categories of agency had specific methods guidance or policy for evidence submission, only one MD-specific agency had developed methodological guidelines specific to medical devices. In stage 2, many MD-specific agencies cited insufficient resources (budget, skilled employees), lack of coordination (between regulator and reimbursement bodies), and the inability to generalize findings from evidence synthesis to be key challenges in the HTA of medical devices.

Conclusions: The lack of evidence for differentiation in scientific methods for HTA of devices raises the question of whether HTA needs to develop new methods for medical devices but rather adapt existing methodological approaches. In contrast, organizational and/or procedural adaptation of existing HTA agency frameworks to accommodate medical devices appear relatively commonplace.

Objectives: Clinical research data are often collected on paper and later inputted onto an electronic database. This method is time consuming and potentially introduces errors. Therefore, to make primary data collection more efficient and less error prone we aimed to develop a touch-screen application for data collection in a psoriatic arthritis research clinic and compared it with the pre-existing paper-based system.

Methods: We developed a Web application using Java and optimized it for the iPad®. It highlights missing fields for physicians in real time, and only permits submission of data collection form after corrections are made. For its evaluation, seven physicians participated, and before each patient visit they were randomly assigned paper or iPad® data entry. Number of errors, length of visit, and time between clinic visit and completion of data entry were measured.

Results: A total of 106 patients seen in the clinic who agreed to participate were randomly assigned to be evaluated by clinic physicians using the iPad® (fifty-three patients) or a paper protocol (fifty-three patients). On average, 3.34 omissions were found per paper form, of which 2.24 would have been detected on the iPad®. The iPad® increased the mean patient encounter time from 37.2 minutes to 46.5 minutes, but eliminated delay between a clinic visit and its data entry.

Conclusions: Entering data using the iPad® application makes the patient encounter slightly longer, but reduces “missing fields.” It also eliminates the delay between clinic visit and data entry thus improving the efficiency of clinical data capture in a research setting.

Objectives: Colloquial evidence (CE) has been described as the informal evidence that helps provide context to other forms of evidence in guidance development. Despite challenges around quality, and the potential biases, the use of CE is becoming increasingly important in assessments where scientific literature is sparse and to also capture the experience of all stakeholders in discussions, including that of experts and patients. We aimed to ascertain how CE was being used at the National Institute for Health and Care Excellence (NICE).

Methods: Relevant data corresponding to the use of CE was extracted from all NICE technical and process manuals by two reviewers and quality assured and analyzed by a third reviewer. This was considered in light of the results of a focused literature review and a combined checklist for quality assessment was developed.

Results: At NICE, CE is utilised across all guidance producing programmes and at all stages of development. CE could range from information from experts and patient/carers, grey literature (including evidence from websites and policy reports) and testimony from stakeholders through consultation. Six tools for critical appraisal of CE were available from the literature and a combined best practice checklist has been proposed.

Conclusions: As decisions often need to be made in areas where there is a lack of published scientific evidence, CE is employed. Therefore to ensure its appropriateness the development of a validated CE data quality check-list to assist decision makers is essential and further research in this area is a priority.

Objectives: Economic evaluations, although not formally used in purchasing decisions for medical devices in Canada, are still being conducted and published. The aim of this study was to examine the way that prices have been included in Canadian economic evaluations of medical devices.

Methods: We conducted a review of the economic concepts and implications of methods used for economic evaluations of the eleven most implanted medical devices from the Canadian perspective.

Results: We found Canadian economic studies for five of the eleven medical devices and identified nineteen Canadian studies. Overall, the device costs were important components of total procedure cost, with an average ratio of 44.1 %. Observational estimates of the device costs were obtained from buyers or sellers in 13 of the 19 studies. Although most of the devices last more than 1 year, standard costing methods for capital equipment was never used. In addition, only eight studies included a sensitivity analysis for the device cost. None of the sensitivity analyses were based on actual price distributions.

Conclusions: Economic evaluations are potentially important for policy making, but although they are being conducted, there is no standardized approach for incorporating medical device prices in economic analyses. Our review provides suggestions for improvements in how the prices are incorporated for economic evaluations of medical devices.

Objectives: Developing a search strategy for use in a systematic review is a time-consuming process requiring construction of detailed search strings using complicated syntax, followed by iterative fine-tuning and trial-and-error testing of these strings in online biomedical search engines.

Methods: Building upon limitations of existing online-only search builders, a user-friendly computer-based tool was created to expedite search strategy development as part of production of a systematic review.

Results: Search Builder 1.0 is a Microsoft Excel®-based tool that automatically assembles search strategy text strings for PubMed (www.pubmed.com) and Embase (www.embase.com), based on a list of user-defined search terms and preferences. With the click of a button, Search Builder 1.0 automatically populates the syntax needed for functional search strings, and copies the string to the clipboard for pasting into Pubmed or Embase. The offline file-based interface of Search Builder 1.0 also allows for searches to be easily shared and saved for future reference.

Conclusions: This novel, user-friendly tool can save considerable time and streamline a cumbersome step in the systematic review process.

Objectives: A rapid scoping review was performed to support the development of a new clinical technology platform. An iterative sifting approach was adopted to address the challenges posed by the nature of the review question and the extremely large volume of search results to be sifted within the timescales of the review.

Methods: This study describes the iterative sifting approach applied in the scoping review and a preliminary validation of the methods applied.

Results: The searches performed for the rapid scoping review retrieved 27,198 records. This was the full set of records subjected to the staged, iterative sifting approach and the subsequent validation process. The iterative sifting approach involved the screening for relevance of 17,354 (i.e., 63.8 percent) of the 27,198 records. A list of fifty-three potential biomarker names was generated as a result of this iterative sifting method, of which nineteen were selected by clinical specialists for further scrutiny. The preliminary validation involved the exhaustive sifting of the remaining 9,844 previously unsifted records. The validation process identified sixteen additional potential biomarker names not identified by the iterative sifting process. The clinical specialists subsequently concluded that none were of further clinical interest.

Conclusions: This study describes an approach to the screening of search records that can be successfully applied in appropriate review and decision problems to allow the prioritization of the most relevant search records and achieve time savings. Following further refinement and standardization, this iterative sifting method may have potential for further applications in reviews and other decision problems.

Objectives: The aim of this study was to explore stakeholders’ points of views regarding the applicability and relevance of a framework for user involvement in health technology assessment (HTA) at the local level. We tested this framework in the context of the assessment of alternative measures to restraint and seclusion among hospitalized adults and those living in long-term-care facilities.

Methods: Twenty stakeholders (health managers, user representatives, and clinicians) from seven regions of Quebec participated in a semi-structured interview. A thematic analysis of the transcribed interviews was performed.

Results: The findings highlighted the relevance and applicability of the framework to this specific HTA. According to interviewees, direct participation of users in the HTA process allows them to be part of the decision-making process. User consultation makes it possible to consider the views of a wide variety of people, such as marginalized and vulnerable groups, who do not necessarily meet the requirements for participating in HTA committees. However, some user representatives emphasized that user consultation should be integrated into a more holistic and participatory perspective. The most frequent barrier associated with user involvement in HTA was the top-down health system, which takes little account of the user's perspective.

Conclusions: The proposed framework was seen as a reference tool for making practitioners and health managers aware of the different mechanisms of user involvement in HTA and providing a structured way to classify and describe strategies. However, there is a need for more concrete instruments to guide practice and support decision making on specific strategies for user involvement in HTA at the local level.

Objectives: The aim of this study was to present the updated methodological guidelines for economic evaluations of healthcare interventions (drugs, medical devices, and other interventions) in Belgium.

Methods: The update of the guidelines was performed by three Belgian health economists following feedback from users of the former guidelines and personal experience. The updated guidelines were discussed with a multidisciplinary team consisting of other health economists, assessors of reimbursement request files, representatives of Belgian databases and representatives of the drugs and medical devices industry. The final document was validated by three external validators that were not involved in the previous discussions.

Results: The guidelines give methodological guidance for the following components of an economic evaluation: literature review, perspective of the evaluation, definition of the target population, choice of the comparator, analytic technique and study design, calculation of costs, valuation of outcomes, definition of the time horizon, modeling, handling uncertainty and discounting. We present a reference case that can be considered as the minimal requirement for Belgian economic evaluations of health interventions.

Conclusions: These guidelines will improve the methodological quality, transparency and uniformity of the economic evaluations performed in Belgium. The guidelines will also provide support to the researchers and assessors performing or evaluating economic evaluations.

Introduction: Evidence requirements may differ across HTA bodies, and so pharmaceutical companies must plan to synergize their evidence generation strategy, across global regulatory and HTA bodies. Until recently, companies had no official platform to discuss the clinical development of a drug with HTA bodies; however, this is changing.

Objectives: To achieve broad usage in the EU, products must achieve both regulatory and reimbursement approval, the latter of which is based on HTA appraisal in many markets. The objective of this study is to present and evaluate the different options available for early HTA consultation (during drug development/Phase III) in the major European markets from the industry perspective.

Methods: An exploratory (nonsystematic) literature review was performed to identify the European markets offering early HTA consultations, and each process was analyzed using a set of predefined metrics that are relevant to industry (the ability to consult with the regulatory body in parallel, consultation fees, length of consultation meeting, language of consultation meeting, maximum number of pharmaceutical company employees attending, procedural timelines, nature of data for which consultative advice can be sought, the output of the process, and the ability to involve external experts).

Results: Four different types of early HTA consultation processes were identified across the major European HTA markets. The nature of these processes varied in terms of the types and number of questions that can be addressed, the length of the meeting, the reporting output, and the ability to involve external experts.

Conclusions: The availability of various options for early HTA consultation may help to avoid a mismatch between the evidence generated by means of a product's clinical development program, and the evidence expected by HTA bodies and payers, which can facilitate the pricing and reimbursement process upon a product's market authorization.

Background: Evidence-based policy making is increasingly used for better resource allocation. Queensland Department of Health has developed a new model to introduce innovative health technologies through a health technology assessment (HTA) program.

Structure: A state-wide committee and several sub-committees at health service district level were established to oversee the HTA program and to monitor the uptake of technologies. The committees are supported by a multidisciplinary secretariat comprising staff with key HTA skills.

Process: The process starts with HTA applications, which are then shortlisted according to prespecified criteria. A due diligence process adopting a rapid evidence assessment approach is used to evaluate the applications. Based on the assessment, recommendations are made using a deliberative decision-making process guided by well-recognized tools. With positive recommendation, a technology is piloted in constrained local setting before its system-wide diffusion.

Outcome: The HTA program has assisted health administrators in prioritizing their health technology agendas. It has gained trust and wide support from policy makers and is increasingly used to support funding allocations, indicating the increasing awareness of and confidence in the program.

Conclusions: The HTA program is a valuable process to assist evidence-based policy development and to guide better resource allocation.

Objectives: Legal analysis can highlight important issues that are relevant when deciding whether a medical technology should be implemented or reimbursed. Literature and studies show that even though the law is an acknowledged part of health technology assessment (HTA), legal issues are rarely considered in practice. One reason for this may be the lack of knowledge about the diversity of legal issues that are relevant for HTA. Therefore, this contribution aims primarily to identify and then explain the relevant legal issues in HTA. This study offers a framework for identifying the legal issues in HTAs in different jurisdictions and provides a basis for further research.

Methods: After extensive literature search, the authors review Swiss health law to identify legal issues that are relevant to HTA. The authors then categorize these legal issues using a framework with an inside and outside perspective. Finally, they explain a selection of these legal issues with several examples.

Results: This study reveals numerous legal issues that are relevant for HTA and underlines the necessity of incorporating legal analysis in HTAs. The suggested perspectival framework in this study provides a basis to structure the legal analysis. The identified legal issues are relevant in other countries and the perspectival framework is transferable to other jurisdictions.

Conclusions: The article underlines the importance of in-depth discussion about the role of law in HTA. It provides a structured overview of the legal issues in HTA and suggests a development of more concrete instruments toward a standardized legal technology assessment.

Background: Although value issues are increasingly addressed in health technology assessment (HTA) reports, HTA is still seen as a scientific endeavor and sometimes contrasted with value judgments, which are considered arbitrary and unscientific. This article aims at illustrating how numerous value judgments are at play in the HTA process, and why it is important to acknowledge and address value judgments.

Methods: A panel of experts involved in HTA, including ethicists, scrutinized the HTA process with regard to implicit value judgments. It was analyzed whether these value judgments undermine the accountability of HTA results. The final results were obtained after several rounds of deliberation.

Results: Value judgments are identified before the assessment when identifying and selecting health technologies to assess, and as part of assessment. They are at play in the processes of deciding on how to select, frame, present, summarize or synthesize information in systematic reviews. Also, in economic analysis, value judgments are ubiquitous. Addressing the ethical, legal, and social issues of a given health technology involves moral, legal, and social value judgments by definition. So do the appraisal and the decision-making process.

Conclusions: HTA by and large is a process of value judgments. However, the preponderance of value judgments does not render HTA biased or flawed. On the contrary they are basic elements of the HTA process. Acknowledging and explicitly addressing value judgments may improve the accountability of HTA.

Background: Timely provision of information on the cost-effectiveness of innovations in health care becomes more and more important, resulting in increasing pressure on researchers to provide proof of cost-effectiveness in a short time frame. However, most of these innovations require considerable time and effort to optimally implement leading to a biased “steady state” cost-effectiveness outcome. As decision makers in health care predominantly have a short-term focus, the discrepancy between short-term study outcomes and long-term cost-effectiveness may very well lead to misguided decisions about the adoption of innovations in health care.

Methods: Factors such as learning effects, capacity constraints, and delayed time to benefit are all related to a short-run timeframe and result in inefficiencies during the implementation of an innovation. These factors and the mechanisms by which they influence the cost-effectiveness outcome are explained for three different types of healthcare innovations.

Results: As standard cost-effectiveness analysis assumes costs and effects to behave constant and representative for an innovation's entire economic lifetime, resulting cost-effectiveness outcomes might give a biased, and often overly pessimistic, reflection of the actual cost-effectiveness of an innovation. This is further amplified by the fact that short-run inefficiencies are most prevalent and impactful during an innovation's earliest stage of operation.

Conclusions: This study advocates to carefully take into account the different factors contributing to lag-time bias in the design and analysis of cost-effectiveness studies, and to communicate potential biases due to short-run inefficiencies to all stakeholders involved in the decision making process.

Objectives: The aim of this study was to identify and critically appraise the use of Value of Information (VOI) analyses undertaken as part of health technology assessment (HTA) reports in England and Wales.

Methods: A systematic review of National Institute for Health Research (NIHR) funded HTA reports published between 2004 and 2013 identified the use of VOI methods and key analytical details in terms of: (i) types of VOI methodology used; (ii) parameters and key assumptions; and (iii) conclusions drawn in terms of the need for further research.

Results: A total of 512 HTA reports were published during the relevant timeframe. Of these, 203 reported systematic review and economic modeling studies and 25 of these had used VOI method(s). Over half of the twenty-five studies (n = 13) conducted both EVPI (Expected Value of Perfect Information) and EVPPI (Expected Value of Partial Perfect Information) analyses. Eight studies conducted EVPI analysis, three studies conducted EVPI, EVPPI, and EVSI (Expected Value of Sampling Information) analyses and one study conducted EVSI analysis only. The level of detail reporting the methods used to conduct the VOI analyses varied.

Conclusions: This review has shown that the frequency of the use of VOI methods is increasing at a slower pace compared with the published volume of HTA reports. This review also suggests that analysts reporting VOI method(s) in HTA reports should aim to describe the method(s) in sufficient detail to enable and encourage decision-makers guiding research prioritization decisions to use the potentially valuable outputs from quantitative VOI analyses.

Objectives: The aim of this study was to determine the prevalence and methods of expert knowledge elicitation (EKE) for specifying input parameters in health economic decision models (HEDM).

Methods: We created two samples using the National Health System Economic Evaluations Database: (1) 100 randomly selected HEDM studies to determine prevalence of EKE and (2) sixty studies using a formal EKE process to determine methods used.

Results: Fifty-seven (57 percent) of the random sample included at least one EKE-derived parameter. Of these, six (10 percent) used a formal expert process. Thirty-four studies from our second sample of sixty studies (57 percent) described at least one aspect of the process (e.g., elicitation method) with reasonable clarity. In approximately two-thirds of studies the external experts estimated parameters de novo; the remainder confirmed or modified initial estimates provided by authors, or the method was unclear. The majority of elicitations obtained point estimates only, although a few studies asked experts to estimate ranges of parameter values.

Conclusions: The use of EKE for parameter estimation is common in HEDMs, although there is room for improvement in the methods used.

Objectives: The aim of this study was to compare Bayesian methods with the standard methods that are used for evidence-based policy making.

Methods: We performed a Bayesian reanalysis of the data underlying a reimbursement advice by the Dutch National Health Insurance Board (CVZ) regarding the anti-diabetic drug exenatide (an alternative to insulin). We synthesized evidence from various sources that was available when the CVZ advice was drafted: expert opinion (as elicited from internists), experimental data (from direct comparison studies), and observational data. Subsequently, the original frequentist results and the results from the Bayesian reanalysis were compared in terms of outcomes and interpretations. These results were presented in a meeting with staff from CVZ, whose opinions about the usefulness of a Bayesian approach were assessed using a questionnaire.

Results: The Bayesian approach yields outcomes that summarize different pieces of evidence, which would have been difficult to obtain otherwise. Moreover, there are conceptual differences, and the Bayesian approach allows for determining probabilities of clinically relevant differences. The staff at CVZ were fairly positive with respect to the use of Bayesian methods, although practical barriers were also seen as important.

Conclusions: The Bayesian outcomes are different and could be more suited to the informational needs of policy makers. The response from staff at CVZ provides some support for this statement, but more research at the interface of science and policy is needed.

Objectives: The aim of this study was to analyze the experience of the National Institute for Health and Care Excellence (NICE) in identifying new procedures entering the United Kingdom (UK) healthcare system, for assessment and publication of recommendations on their use. This system is designed to provide guidance in an area where regulation is lacking worldwide.

Methods: Retrospective analysis of all procedures notified to the Interventional Procedures Programme (NICE) between 2002 and 2012. Notifications were analyzed year by year for their source (who notified them), clinical specialties involved, and whether guidance was subsequently published.

Results: A total of 1,094 procedures were notified by clinicians (51 percent), and by others, including hospitals (6 percent), horizon scanners (5 percent), patients (4 percent), private health insurers (4 percent), and medical device manufacturers (3 percent). Guidance was published on 44 percent of procedures notified to the program. There was a decrease in the numbers of procedures notified during 2003–2012 (p = .049). There were notifications across all specialties, with the largest numbers in general surgery (125), urology (104), orthopedics (99), interventional radiology (93), cardiology (82), and obstetrics and gynecology (82).

Conclusions: The “open” NICE Web portal allows anyone to notify new procedures, aiming to maximize the opportunity of identifying all those procedures entering clinical practice. This has resulted in identification of large numbers of procedures from across the whole range of medical specialties. The fact that similar proportions of procedures notified from diverse sources have been selected for assessment and publication of practice recommendations suggests that this inclusive approach is worthwhile.

Objectives: Rare diseases are often heterogeneous in their progression and response to treatment, with only a small population for study. This provides challenges for evidence generation to support HTA, so novel research methods are required.

Methods: Discussion with an expert panel was augmented with references and case studies to explore robust approaches for HTA evidence generation for rare disease treatments.

Results: Traditional RCTs can be modified using sequential, three-stage or adaptive designs to gain more power from a small patient population or to focus trial design. However, such designs need to maintain important design aspects such as randomization and blinding and be analyzed to take account of the multiple analyses performed. N-of-1 trials use within-patient randomization to test repeat periods of treatment and control until a response is clear. Such trials could be particularly valuable for rare diseases and when prospectively planned across several patients and analyzed using Bayesian techniques, a population effect can be estimated that might be of value to HTA. When the optimal outcome is unclear in a rare disease, disease specific patient reported outcomes can elucidate impacts on patients’ functioning and wellbeing. Likewise, qualitative research can be used to elicit patients’ perspectives, with just a small number of patients.

Conclusions: International consensus is needed on ways to improve evidence collection and assessment of technologies for rare diseases, which recognize the value of novel study designs and analyses in a setting where the outcomes and effects of importance are yet to be agreed.

Rationale: Bayesian methods provide an interesting approach to assessing an implantable medical device (IMD) that has evolved through successive versions because they allow for explicit incorporation of prior knowledge into the analysis. However, the literature is sparse on the feasibility and reliability of elicitation in cases where expert beliefs are used to form priors.

Objectives: To develop an Internet-based method for eliciting experts’ beliefs about the success rate of an intracranial stenting procedure and to assess their impact on the estimated benefit of the latest version.

Study Design and Setting: The elicitation questionnaire was administered to a group of nineteen experts. Elicited experts’ beliefs were used to inform the prior distributions of a Bayesian hierarchical meta-analysis model, allowing for the estimation of the success rate of each version. RESULTS: Experts believed that the success rate of the latest version was slightly higher than that of the previous one (median: 80.8 percent versus 75.9 percent). When using noninformative priors in the model, the latest version was found to have a lower success rate (median: 83.1 percent versus 86.0 percent), while no difference between the two versions was detected with informative priors (median: 85.3 percent versus 85.6 percent).

Conclusions: We proposed a practical method to elicit experts’ beliefs on the success rates of successive IMD versions and to explicitly combine all available evidence in the evaluation of the latest one. Our results suggest that the experts were overoptimistic about this last version. Nevertheless, the proposed method should be simplified and assessed in larger, representative samples.

Objectives: A mapping exercise was undertaken to determine how HTA is being described and conducted across the International Network of Agencies for Health Technology Assessment (INAHTA), with the aim of harmonizing terminologies and approaches.

Methods: Three progressive surveys were undertaken. In 2010, INAHTA agencies were asked to provide details on all of their HTA products. In 2013, additional information was sought on key methodological characteristics of five of the most common HTA product types. Subsequently, final agreement was sought on three proposed product types.

Results: Forty-five HTA agencies responded to at least one of the surveys. In 2010, twenty-one agencies reported publishing over seventy named HTA products. Core domains associated with full HTA reports were reported by a third of agencies but were labeled differently, so products were classified according to product type (n = 17). Agencies producing short, tailored products increased between 2010 and 2013, with the publication of rapid reviews doubling from 33 percent to 66 percent. In 2013, half of the agencies adapted their common HTA products from documents produced by other agencies. A consensus (>70 percent) was achieved on definitions for HTA reports, mini-HTAs, and rapid reviews.

Conclusions: The product label for an HTA is not always indicative of its content. Terminology has, therefore, been agreed to make explicit the trade-off between rigor and timeliness in three common HTA product types. An INAHTA Product Type (IPT) Mark has been created to identify each of these. It is hoped this will further facilitate HTA adaptation between agencies and reduce duplication of effort.