Health apps are software programs that are designed to prevent, diagnose, monitor, or manage conditions. Inconsistent terminology for apps is used in research literature and bibliographic database subject headings. It can therefore be challenging to retrieve evidence about them in literature searches. Information specialists at the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed novel validated search filters to retrieve evidence about apps from MEDLINE and Embase (Ovid).

A selection of medical informatics journals was hand searched to identify a “gold standard” (GS) set of references about apps. The GS set was divided into a development and validation set. The filters’ search terms were derived from and tested against the development set. An external development set containing app references from published NICE products was also used to inform the development of the filters. The filters were then validated using the validation set. Target recall was >90 percent. The filters’ overall recall, specificity, and precision were calculated using all the references identified from the hand search.

Both filters achieved 98.6 percent recall against their validation sets. Overall, the MEDLINE filter had 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter had 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision.

The NICE health apps search filters retrieve evidence about apps from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about the interventions by information professionals, researchers, and clinicians.

The Spanish Network of Agencies for Assessing National Health System Technologies and Performance (RedETS) defined a patient involvement (PI) framework for health technology assessment (HTA) activities in 2016. The aim of this study is to evaluate the process and impact of those PI initiatives that were implemented in the first year following the publication of this new framework.

A survey was sent to those HTA researchers who implemented PI in RedETS projects. Responses were reviewed by two authors. An adapted thematic analysis was performed and the results were later discussed by all authors.

Six responses from six agencies/units were analyzed. The objectives of PI initiatives were the following: inclusion of patient perspectives, preferences and values; elicitation of important health outcomes measures; and barriers, facilitators, or suggestions for implementation. Different methods were used for PI: surveys, focus groups, in depth interviews, and participation in an expert panel. Five main themes emerged: (i) challenges with the recruitment process, (ii) needs identified, (iii) impact of PI, (iv) lessons learned, and (v) suggestions for the future.

PI initiatives within the RedETS framework were tailored to each HTA project, its specific goals and the individual needs and resources of each HTA agency. The results also pointed out how PI has a relevant impact that has enriched RedETS products providing key information on experiences, values, and preferences of patients, contributions that benefit the HTA and the process of drawing up recommendations. The main challenges were related to recruitment processes and capacity building.

The objectives of the study were to establish a benchmarking tool to collect metrics to enable increased clarity regarding the differences and similarities across health technology assessment (HTA) agencies, to assess performance within and across HTA agencies, identify areas in the HTA processes in which time is spent and to enable ongoing performance improvement.

Common steps and milestones in the HTA process were identified for meaningful benchmarking among agencies. A benchmarking tool consisting of eighty-six questions providing information on HTA agency organizational aspects and information on individual new medicine review timelines and outcomes was developed with the input of HTA agencies and validated in a pilot study. Data on 109 HTA reviews from five HTA agencies were analyzed to demonstrate the utility of this tool.

This study developed an HTA benchmarking methodology, comparative metrics showed considerable differences among the median timelines from assessment and appraisal to final HTA recommendation for the five agencies included in this analysis; these results were interpreted in conjunction with agency characteristics.

It is feasible to find consensus among HTA agencies regarding the common milestones of the review process to map jurisdiction-specific processes against agreed metrics. Data on characteristics of agencies such as their scope and remit enabled results to be interpreted in the appropriate local context. This benchmarking tool has promising potential utility to improve the transparency of the review process and to facilitate both quality assurance and performance improvement in HTA agencies.

To investigate the behavior of restricted mean survival time (RMST) and designs of a two-state Markov microsimulation model through a 2 × 4 × 2 full factorial experiment.

By projecting patient-wise 15-year-post-trial survival, we estimated life-year-gained between an intervention and a control group using data from the Cardiovascular Outcomes for People Using Anticoagulation Strategies Study (COMPASS). Projections considered either in-trial events or post-trial medications. They were compared based on three factors: (i) choice of probability of death, (ii) lengths of cycle, and (iii) usage of half-a-cycle age correction. Three-way analysis of variance and post-hoc Tukey's Honest Significant Difference test compared means among factors.

When both in-trial events and post-trial study medications were considered, monthly, quarterly, or semiannually were not different from one other in projected life-year-gained. However, the annual one was different from the others: mean and 95 percent confidence interval 252.2 (190.5–313.9) days monthly, 251.8 (192.0–311.6) quarterly, 249.1 (189.7–308.5) semiannually, and 240.8 (178.5–303.1) annually. The other two factors also impacted life-year-gained: background probability (269.1 [260.3–277.9] days projected with REACH-based-probabilities, 227.7 [212.6–242.8] with a USA life table); half-a-cycle age correction (245.5 [199.0–292] with correction and 251.4 [209.1–293.7] without correction). When not considering post-trial medications, only the choice of probability of death appeared to impact life-year-gained.

For a large trial or cohort, to optimally project life-year-gained, one should consider using (i) annual projections, (ii) life table probabilities, (iii) in-trial events, and (iv) post-trial medication use.

Economic models play a central role in the decision-making process of the National Institute for Health and Care Excellence (NICE). Inadequate validation methods allow for errors to be included in economic models. These errors may alter the final recommendations and have a significant impact on outcomes for stakeholders.

To describe the patterns of technical errors found in NICE submissions and to provide an insight into the validation exercises carried out by the companies prior to submission.

All forty-one single technology appraisals (STAs) completed in 2017 by NICE were reviewed and all were on medicines. The frequency of errors and information on their type, magnitude, and impact was extracted from publicly available NICE documentation along with the details of model validation methods used.

Two STAs (5 percent) had no reported errors, nineteen (46 percent) had between one and four errors, sixteen (39 percent) had between five and nine errors, and four (10 percent) had more than ten errors. The most common errors were transcription errors (29 percent), logic errors (29 percent), and computational errors (25 percent). All STAs went through at least one type of validation. Moreover, errors that were notable enough were reported in the final appraisal document (FAD) in eight (20 percent) of the STAs assessed but each of these eight STAs received positive recommendations.

Technical errors are common in the economic models submitted to NICE. Some errors were considered important enough to be reported in the FAD. Improvements are needed in the model development process to ensure technical errors are kept to a minimum.

To alert policy makers early about emerging health technologies that could significantly impact the healthcare system at the clinical, financial and organizational levels, the Agency for Care Effectiveness (ACE) in Singapore established a horizon scanning system (HSS) in 2019. This paper describes the development of the ACE HSS and showcases its application with cell and gene therapy products as the first example.

A literature review of existing HSS methods, including the processes of the EuroScan International Network and other overseas horizon scanning agencies, was done to inform the development of our horizon scanning framework. The framework was first applied to the new and emerging cell and gene therapies.

Identification sources, filtration and prioritization criteria, and horizon scanning outputs for the HSS were developed in alignment to international best practices, with recommendations for technology uptake represented by a traffic light system. For the first horizon scanning exercise on cell and gene therapies, forty therapies passed the filtration step, of which eight were prioritized for further assessment. The few early reports developed were used to inform and prepare the healthcare system for their potential introduction, particularly in terms of the need to develop health and funding policies.

Early assessment of prioritized topics has provided support for strategic efforts within the Ministry of Health. Given that ACE's horizon scanning program is still in its infancy, the framework will continue to evolve to ensure relevance to our stakeholders so that it remains fit for purpose for our healthcare system.

Despite the efforts of the European Union (EU) to promote voluntary cooperation among Health Technology Assessment (HTA) agencies, different reimbursement decisions for the same drug are made across European countries. The aim of this paper is to compare the agreement of cancer drug reimbursement decisions using inter-rater reliability measures.

This study is based on primary data on 161 cancer drug reimbursement decisions from nine European countries from 2002 to 2014. To achieve our goal, we use two measures to analyze agreement, in other words, congruency: (i) percentage of agreement and (ii) the κ score.

One main conclusion can be drawn from the analysis. There is a weak to medium agreement among cancer drug decisions in the European countries analyzed (based on the percentage of agreement and the κ score). England and Scotland show the highest consistency between the two measures, showing a medium agreement. These results are in line with previous literature on the congruency of HTA decisions.

This paper contributes to the HTA literature, by highlighting the extent of weak to medium agreement among cancer decisions in Europe.

The Pharmaceutical Benefits Scheme (PBS) provides timely, reliable, and affordable access to necessary medicines for Australians. We reviewed the Pharmaceutical Benefits Advisory Committee (PBAC) submissions and their related outcomes and timelines since 2010.

We examined the PBS Website to identify submissions and their related PBAC outcomes for new medicines, new indications, and new combination products that had been considered by the PBAC since 2010.

Thirty-five PBAC meetings were held during the study period, at which the Committee considered 781 submissions (1,074 medicine/patient population pairings). We saw an increase in the annual number of submissions (medicine/patient population parings). The recommendation rate for the study period was higher than the rejection rate. The annual mean value for the period from the date of initial PBAC recommendation to the date of PBS listing ranged from 357 to 644 days; the annual mean value for the period of the date of PBAC recommendation to the date of PBS listing ranged from 187 to 245 days. It took, on average, 1.70 submissions that included an economic evaluation to obtain a PBAC recommendation. It took more submissions to obtain a PBAC recommendation for a cost-effectiveness analysis submission than it did for a CMA submission. The PBAC was willing to recommend medicines for most acceptable base-case incremental cost-effectiveness ratio (ICER) bands, and the majority of the PBAC did not recommended any medicine in the study period that had a base-case ICER >AUD75,000.

The results of our analyses reveal a minor reduction in the period from the date of PBAC recommendation to the date of PBS listing. Several analyses were hampered by a high proportion of missing data.

Mental health problems can lead to costs and benefits in other sectors (e.g. in the education sector) in addition to the healthcare sector. These related costs and benefits are known as intersectoral costs and benefits (ICBs). Although some ICBs within the education sector have been identified previously, little is known about their extensiveness and transferability, which is crucial for their inclusion in health economics research.

The aim of this study was to identify ICBs in the education sector, to validate the list of ICBs in a broader European context, and to categorize the ICBs using mental health as a case study.

Previously identified ICBs in the education sector were used as a basis for this study. Additional ICBs were extracted from peer-reviewed literature in PubMed and grey literature from six European countries. A comprehensive list of unique items was developed based on the identified ICBs. The list was validated by surveying an international group of educational experts. The survey results were used to finalize the list, which was categorized according to the care atom.

Additional ICBs in the education sector were retrieved from ninety-six sources. Fourteen experts from six European countries assessed the list for completeness, clarity, and relevance. The final list contained twenty-four ICBs categorized into input, throughput, and output.

By providing a comprehensive list of ICBs in the education sector, this study laid further foundations for the inclusion of important societal costs in health economics research in the broader European context.

While involving patients in health technology assessment (HTA) has become increasingly common and important around the world, little is known about the optimal methods of evaluating patients’ involvement (PI) in HTA. This scoping review was undertaken to provide an overview of currently available methods for the evaluation of PI, specifically the impact of PI on HTA recommendations.

A literature search was conducted using nine databases as well as a grey literature search of the websites of 26 organizations related to the conduct, practice or research of HTA to identify articles, reports and abstracts related to the evaluation of PI impact in HTA.

We identified 1,248 unique citations, six of which met our eligibility criteria. These six records (five articles, and one report) were all published after 2012. Four assessed the impact of patient experience submissions on final HTA recommendations; one evaluated the impact of direct involvement on HTA committees, and one assessed impact of multiple forms of involvement. Methods of evaluation included quantitative analyses of reimbursement decisions, qualitative interviews with those directly involved in an assessment, surveys of patient groups and committee members, and the review of HTA reports.

Quantitative evaluation of PI based on associations with funding decisions may not be feasible or fully capture the relevant impact of PI in the assessment of health technologies. Rather, a combination of both qualitative and quantitative strategies may allow for the most comprehensive assessment of the impact of PI on HTA recommendations when possible.

Quality-adjusted life-years (QALYs) and disability-adjusted life-years (DALYs) are commonly used in cost-effectiveness analysis (CEA) to measure health benefits. We sought to quantify and explain differences between QALY- and DALY-based cost-effectiveness ratios, and explore whether using one versus the other would materially affect conclusions about an intervention's cost-effectiveness.

We identified CEAs using both QALYs and DALYs from the Tufts Medical Center CEA Registry and Global Health CEA Registry, with a supplemental search to ensure comprehensive literature coverage. We calculated absolute and relative differences between the QALY- and DALY-based ratios, and compared ratios to common benchmarks (e.g., 1× gross domestic product per capita). We converted reported costs into US dollars.

Among eleven published CEAs reporting both QALYs and DALYs, seven focused on pharmaceuticals and infectious disease, and five were conducted in high-income countries. Four studies concluded that the intervention was “dominant” (cost-saving). Among the QALY- and DALY-based ratios reported from the remaining seven studies, absolute differences ranged from approximately $2 to $15,000 per unit of benefit, and relative differences from 6–120 percent, but most differences were modest in comparison with the ratio value itself. The values assigned to utility and disability weights explained most observed differences. In comparison with cost-effectiveness thresholds, conclusions were consistent regardless of the ratio type in ten of eleven cases.

Our results suggest that although QALY- and DALY-based ratios for the same intervention can differ, differences tend to be modest and do not materially affect comparisons to common cost-effectiveness thresholds.

Traditionally, health technology assessment (HTA) focuses on assessing the impact of pharmaceutical technologies on health and care. Resources are scarce and policy makers aim to achieve effective, accessible health care. eHealth innovations are increasingly more integrated in all healthcare domains. However, how eHealth is assessed prior to its implementation in care practices is unclear. To support evidence-informed policy making, this study aimed to identify frameworks and methods for assessing eHealth's impact on health care.

The scientific literature in five bibliographical databases was systematically reviewed. Articles were included if the study was conducted in a clinical setting, used an HTA framework and assessed an eHealth service. A systematic qualitative narrative approach was applied for analysis and reporting.

Twenty-one HTA frameworks were identified in twenty-three articles. All frameworks addressed outcomes related to the technical performance and functionalities of eHealth service under assessment. The majority also addressed costs (n = 19), clinical outcomes (n = 14), organizational (n = 15) and system level aspects (n = 13). Most frameworks can be classified as dimensional (n = 13), followed by staged (n = 3), hybrid (n = 3), and business modeling frameworks (n = 2). Six frameworks specified assessment outcomes and methods.

HTA frameworks are available for a-priori impact assessment of eHealth services. The frameworks vary in assessment outcomes, methods, and specificity. Demonstrated applicability in practice is limited. Recommendations include standardization of: (i) reporting characteristics of eHealth services, and (ii) specifying assessment outcomes and methods following a stepped-approach tailored to the functional characteristics of eHealth services. Standardization might improve the quality and comparability of eHTA assessments.

We designed, developed, and implemented a new hospital-based health technology assessment (HB-HTA) program called Smart Innovation. Smart Innovation is a decision framework that reviews and makes technology adoption decisions. Smart Innovation was meant to replace the fragmented and complex process of procurement and adoption decisions at our institution. Because use of new medical technologies accounts for approximately 50 percent of the growth in healthcare spending, hospitals and integrated delivery systems are working to develop better processes and methods to sharpen their approach to adoption and management of high cost medical innovations.

The program has streamlined the decision-making process and added a robust evidence review for new medical technologies, aiming to balance efficiency with rigorous evidence standards. To promote system-wide adoption, the program engaged a broad representation of leaders, physicians, and administrators to gain support.

To date, Smart Innovation has conducted eleven HB-HTAs and made clinician-led adoption decisions that have resulted in over $5 million dollars in cost avoidance. These are comprised of five laboratory tests, three software-assisted systems, two surgical devices, and one capital purchase.

Smart Innovation has achieved cost savings, avoided uncertain or low-value technologies, and assisted in the implementation of new technologies that have strong evidence. The keys to its success have been the program's collaborative and efficient decision-making systems, partnerships with clinicians, executive support, and proactive role with vendors.

The terminology used to describe community participation in Health Technology Assessment (HTA) is contested and frequently confusing. The terms patients, consumers, public, lay members, customers, users, citizens, and others have been variously used, sometimes interchangeably. Clarity in the use of terms and goals for including the different groups is needed to mitigate existing inconsistencies in the application of patient and public involvement (PPI) across HTA processes around the world.

We drew from a range of literature sources in order to conceptualize (i) an operational definition for the “public” and other stakeholders in the context of HTA and (ii) possible goals for their involvement. Draft definitions were tested and refined in an iterative consensus-building process with stakeholders from around the world.

The goals, terminology, interests, and roles for PPI in HTA processes were clarified. The research provides rationales for why the role of the public should be distinguished from that of patients, their families, and caregivers. A definition for the public in the context of HTA was developed: A community member who holds the public interest and has no commercial, personal, or professional interest in the HTA process

There are two distinct aspects to the interests held by the public which should be explicitly included in the HTA process: the first lies in ensuring democratic accountability and the second in recognising the importance of including public values in decision making.

To synthetize the state of the art of methods for identifying candidate technologies for disinvestment and propose an evidence-based framework for executing this task.

An interpretative review was conducted. A systematic literature search was performed to identify secondary or tertiary research related to disinvestment initiatives and/or any type of research that specifically described one or more methods for identifying potential candidates technologies, services, or practices for disinvestment. An iterative and critical analysis of the methods described alongside the disinvestment initiatives was performed.

Seventeen systematic reviews on disinvestment or related terms (health technology reassessment or medical reversal) were retrieved and methods of 45 disinvestment initiatives were compared. On the basis of this evidence, we proposed a new framework for identifying these technologies based on the wide definition of evidence provided by Lomas et al. The framework comprises seven basic approaches, eleven triggers and thirteen methods for applying these triggers, which were grouped in embedded and ad hoc methods.

Although identification methods have been described in the literature and tested in different contexts, the proliferation of terms and concepts used to describe this process creates considerable confusion. The proposed framework is a rigorous and flexible tool that could guide the implementation of strategies for identifying potential candidates for disinvestment.

This study sought to explore main barriers and facilitators to implementing health technology assessment (HTA) in Kuwait from the perspective of key stakeholders.

Semi-structured qualitative interviews were conducted with ten key stakeholders: seven healthcare providers working at various departments of the Kuwaiti Ministry of Health (MOH), and three academics with substantial experience in teaching HTA or related fields. Interviews were conducted face-to-face, audio-recorded, and transcribed verbatim. Data were analyzed using an inductive thematic approach.

Participating stakeholders reported several factors that might act as a barrier to building HTA in Kuwait: minimal awareness of HTA, lack of institutional and human capacity, a fragmented healthcare system, poor communication between researchers and policy makers, the country's wealth, politics, as well as data quality, availability, and sharing. Institutionalizing HTA as a politically empowered body, enforcing its recommendation by law, and benefiting from neighboring countries' experiences were suggested as possible ways to move forward.

Studies exploring the unique challenges that high-income developing countries may face in implementing HTA are still scarce. The results of this study are consistent with evidence coming from other developing countries, while also suggesting that the abundance of financial resources in the country is a double-edged sword; it has the potential to facilitate the development of HTA capacity, but also hinders recognizing the need for it.

Consideration of ethical, legal, and social issues plus patient values (ELSI+) in health technology assessment (HTA) is challenging because of a lack of conceptual clarity and the multi-disciplinary nature of ELSI+. We used concept mapping to identify key concepts and inter-relationships in the ELSI+ domain and provide a conceptual framework for consideration of ELSI+ in HTA.

We conducted a scoping review (Medline and EMBASE, 2000–2016) to identify ELSI+ issues in the HTA literature. Items from the scoping review and an expert brainstorming session were consolidated into eighty ELSI+-related statements, which were entered into Concept Systems® Global MAX™ software. Participants (N = 38; 36 percent worked as researchers, 21 percent as academics; 42 percent self-identified as HTA experts) sorted the statements into thematic groups, and rated them on importance in making decisions about adopting technologies in Canada, from 1 (not at all important) to 5 (extremely important). We used Concept Systems® Global MAX™ software to create and analyze concept maps with four to sixteen clusters.

Our final ELSI+ map consisted of five clusters, with each cluster representing a different concept and the statements within each cluster representing the same concept. Based on the concepts, we named these clusters: patient preferences/experiences, patient quality of life/function, patient burden/harm, fairness, and organizational. The highest mean importance ratings were for the statements in the patient burden/harm (3.82) and organizational (3.92) clusters.

This study suggests an alternative approach to ELSI+, based on conceptual coherence rather than academic disciplines. This will provide a foundation for incorporating ELSI+ into HTA.

The aim of this study was to provide an overview of the methodological characteristics and compare the assessment methods applied in health technology assessments (HTAs) of public health interventions (PHIs).

We defined a PHI as a population-based intervention on health promotion or for primary prevention of chronic or nonchronic diseases. HTAs on PHIs were identified by systematically searching the Web pages of members of international HTA networks. We included only full HTA reports published between 2012 and 2016. Two reviewers extracted data on the methods used to assess effectiveness/safety, as well as on economic, social, cultural, ethical, and legal aspects using a-priori standardized tables.

We included ten HTAs provided by four different organizations. Of these, all reports assessed the effectiveness of the interventions and conducted economic evaluations, seven investigated social/cultural aspects, and four each considered legal and ethical aspects, respectively. Some reports addressed applicability, context/setting, and intervention fidelity issues in different ways. We found that most HTAs adapted their methods to some extent, for example, by including nonrandomized studies, expanding the search strategy, involving stakeholders, or applying a framework to guide the HTA process.

Our analysis provides a comprehensive overview of methods applied in HTAs on public health interventions. We found that a heterogeneous set of approaches is used to deal with the challenges of evaluating complex public health interventions.

Traditional decision rules have limitations when a new technology is less effective and less costly than a comparator. We propose a new probabilistic decision framework to examine non-inferiority in effectiveness and net monetary benefit (NMB) simultaneously. We illustrate this framework using the example of repetitive transcranial magnetic stimulation (rTMS) and electroconvulsive therapy (ECT) for treatment-resistant depression.

We modeled the quality-adjusted life-years (QALYs) associated with the new intervention (rTMS), an active control (ECT), and a placebo control, and we estimated the fraction of effectiveness preserved by the new intervention through probabilistic sensitivity analysis (PSA). We then assessed the probability of cost-effectiveness using a traditional cost-effectiveness acceptability curve (CEAC) and our new decision-making framework. In our new framework, we considered the new intervention cost-effective in each simulation of the PSA if it preserved at least 75 percent of the effectiveness of the active control (thus demonstrating non-inferiority) and had a positive NMB at a given willingness-to-pay threshold (WTP).

rTMS was less effective (i.e., associated with fewer QALYs) and less costly than ECT. The traditional CEAC approach showed that the probabilities of rTMS being cost-effective were 100 percent, 39 percent, and 14 percent at WTPs of $0, $50,000, and $100,000 per QALY gained, respectively. In the new decision framework, the probabilities of rTMS being cost-effective were reduced to 23 percent, 21 percent, and 13 percent at WTPs of $0, $50,000, and $100,000 per QALY, respectively.

This new framework provides a different perspective for decision making with considerations of both non-inferiority and WTP thresholds.

Ethics has been considered among the core domains of health technology assessment (HTA), but there are still disputes regarding ethical analysis. This study aimed to examine full final reports of the European Network for Health Technology Assessment (EUnetHTA) in terms of their compliance with the ethical methodology and ethical perspective of the HTA Core Model®.

The study examines seven full final HTA reports of EUnetHTA written based on the methodology proposed in the HTA Core Model®. The reports were analyzed using the following parameters: competency of the person/group who conducted ethical analysis, assessment elements, and the methodology of ethical analysis.

The results show that, although the HTA Core Model® helped to standardize the final reports of the assessment, there are still concerns regarding the competency of the ethical analysis team, the perspectives on the purpose of ethical analysis, data sources and viewpoints of various stakeholders, use of ethical analysis methodology, and the evaluation of the ethical appropriateness of the entire HTA process.

The HTA Core Model® helped to standardize the final reports on the HTA; however, not all issues with the content and outcomes were solved. The lack of expertise in ethics and insufficiency of the teams regarding ethical analysis are other existing problems. This study also demonstrated that stakeholder viewpoints in general and patient perspectives, in particular, have been overlooked in the HTA process.