Artificial intelligence (AI)-based health technologies (AIHTs) have already been applied in clinical practice. However, there is currently no standardized framework for evaluating them based on the principles of health technology assessment (HTA).

A two-round Delphi survey was distributed to a panel of experts to determine the significance of incorporating topics outlined in the EUnetHTA Core Model and twenty additional ones identified through literature reviews. Each panelist assigned scores to each topic. Topics were categorized as critical to include (scores 7–9), important but not critical (scores 4–6), and not important (scores 1–3). A 70 percent cutoff was used to determine high agreement.

Our panel of 46 experts indicated that 48 out of the 65 proposed topics are critical and should be included in an HTA framework for AIHTs. Among the ten most crucial topics, the following emerged: accuracy of the AI model (97.78 percent), patient safety (95.65 percent), benefit–harm balance evaluated from an ethical standpoint (95.56 percent), and bias in data (91.30 percent). Importantly, our findings highlight that the Core Model is insufficient in capturing all relevant topics for AI-based technologies, as 14 out of the additional 20 topics were identified as crucial.

It is imperative to determine the level of agreement on AI-relevant HTA topics to establish a robust assessment framework. This framework will play a foundational role in evaluating AI tools for the early diagnosis of dementia, which is the focus of the European project AI-Mind currently being developed.

The roles and potential value of patient preference (PP) data in health technology assessment (HTA) remain to be fully realized despite an expanding literature and various efforts to establish their utility. This article reports lessons learned through a series of collaborative workshops with HTA representatives, organized by the Health Technology Assessment International’s Patient Preferences Project Subcommittee.

Five online workshops were conducted between June 2022 and June 2023, seeking to facilitate collaborative learning and reflection on ways that PP data can be integrated into HTA. Participants included nine HTA representatives from the United States, Canada, Australia, England, and the Netherlands. Workshops were recorded, transcribed, and thematically analyzed.

Despite appreciating the value of PP data, participants were ambivalent about their use in HTA. Some felt that they were already getting the information they needed from the cost-effectiveness analysis or existing patient involvement processes. Others thought that PP data would be very helpful at the initial and final stage of the decision-making process and, particularly, in the following cases: (a) when technology has important non-health benefits; (b) when the clinical and/or cost-effectiveness evidence is marginal; and (c) when treatment is indicated for a large and heterogeneous population. Issues related to the validity and reliability of PP studies were frequently raised, with preference heterogeneity at the core of these concerns.

Collaborating with HTA representatives in the “co-creation” of PP research can help address their concerns and facilitate mutual learning about how PP data can be used in HTA.

International development agendas increasingly push for access to healthcare for all through universal healthcare coverage. Health economic evaluations and health technology assessment (HTA) could provide evidence to support this but do not routinely incorporate consideration of equitable access.

We undertook an international scoping review of health economic evaluation and HTA guidelines to examine how well issues of healthcare access and equity are represented, evidence recommendations, and gaps in current guidance to support evidence generation in this area. Guidelines were sourced from guideline repositories and websites of international agencies and organizations providing best practice methods guidance. Articles providing methods guidance for the conduct of HTA, or health economic evaluation, were included, except where they were not available in English and a suitable translation could not be obtained.

The search yielded forty-seven national, four international, and nine independent guidelines, along with eighty-six articles providing specific methods guidance. The inclusion of equity and access considerations in current guidance is extremely limited. Where they do feature, detail on specific methods for providing evidence on these issues is sparse.

Economic evaluation could be a valuable tool to provide evidence for the best healthcare strategies that not only maximize health but also ensure equitable access to care for all. Such evidence would be invaluable in supporting progress towards universal healthcare coverage. Clear guidance is required to ensure evaluations provide evidence on the best strategies to support equitable access to healthcare, but such guidance rarely exists in current best practice and guidance documents.

Health technology assessment (HTA) organizations generate guidelines to inform healthcare practices toward improved health outcomes. This review sought to identify and classify outcomes of guidelines from HTA organizations within published research.

We performed a systematic mixed studies review of empirical studies that (a) referred to a published guideline from an HTA organization and (b) reported an outcome resulting from a guideline. We searched the published literature in English or French within seven databases. Outcome types were classified within five dimensions of an existing framework for online health information (e.g., relevance, cognitive/affective impact, and use). Subdimensions were inductively developed. A two-phase sequential data synthesis was performed. Phase 1: a hybrid deductive–inductive thematic analysis identified the types of outcomes and displayed their relationships on a concept map. Phase 2: descriptive statistics were tabulated by the type of outcome.

A total of 6,719 records were retrieved through searches on 6 February 2023. After screening, we included 120 observational studies (twenty-one qualitative, ninety-four quantitative, and five mixed methods). Phase 1 identified twenty-nine types of outcomes. The most frequently reported outcomes were within the organizational dimension (reported in ninety-four studies). The most common subdimensions were “Referrals” (thirty-eight occurrences), the “Quality of Prescriptions” (fifteen occurrences), and the “Quality of Diagnosis” (eight occurrences). For Phase 2, we could only generate descriptive statistics on seventeen outcomes. These were almost equally distributed among positive, neutral, and negative effects. Our results contribute to knowledge about the outcomes of HTA guidelines and options for documenting and measuring them in future evaluations.

Health Technology Assessment (HTA) practitioners recognize the significance of qualitative methodologies that focus on how a technology is feasible, meaningfulness, acceptable, and equitable. This mapping aimed to delineate the frameworks employed to synthesize qualitative evidence and assess the quality of synthesis in HTA .

Mapping was conducted using Medline, LILACS, CINAHL, Embase, Web of Science, Scopus, PsycINFO, Cochrane Library, JBI, and ScienceDirect databases. Gray literature searches included PROQUEST, Open Grey, Canadian Agency for Drugs and Technologies in Health’s Grey Matters, Google Scholar, and HTA agency websites. The inclusion criteria were centered on global qualitative evidence synthesis frameworks. The data are presented in the tables.

Of the 2054 articles, 31 were included, mostly from Europe. Guide was the type of document more cited, and most authors are from HTA agencies and universities. Incorporating both patient and family perspectives is the most cited reason for include qualitative evidence. Regardless of the framework or tool, SPICE was the main acronym, and RETREAT was preferred for approach selection. Thematic synthesis dominated analytic methods, and CASP was the primary quality appraisal tool. GRADE-CERQual graded evidence synthesis, with ENTREQ as the top reporting guidance. The GRADE evidence-to-decision framework was mentioned for recommendations.

This mapping highlights the movement incorporate qualitative evidence in HTA employing specific frameworks. Despite the similarities among documents, most of them describe part of the process to synthesize qualitative evidence. Standardizing procedures to incorporate qualitative evidence into HTA can enhance decision-making. These findings offer essential considerations for HTA practice.

Evaluating the impact of patient involvement in health technology assessments (HTA) may help improve practices and avoid ineffective activities. Evaluation, however, continues to be infrequent, inconsistent, and often only relates to process quantity or quality. The Patient and Citizen Involvement in HTA Interest Group (PCIG) within Health Technology Assessment International set out to contextualize this impact to support evaluation.

Given the lack of established methodology to measure impact, the team performed a qualitative analysis of first-hand accounts about perceived changes in HTA due to involvement of patient stakeholders. A questionnaire was developed, piloted, and rolled out to collect personal perspectives from stakeholders with relevant experience. The stories were analyzed in the aggregate to identify themes in the data.

From January 2019 to September 2021, twenty-four responses were collected through PCIG’s network. Responses (including one joint industry-HTA body submission) came from patient representatives (12), HTA bodies (11), and industry representatives (2) from North America (5), South America (3), Europe (13), and Asia Pacific (3). Based on themes commonly reported, a three-domain framework for evaluating impact is proposed: impact on basis of HTA result or recommendation, impact on HTA body, and impact on patient participants. The framework includes components under each domain to support reporting.

Using the Three-Domain Impact Framework may be useful in identifying, evaluating, and communicating the value of patient involvement in HTA. Enhancing and increasing reporting practices may improve transparency and facilitate process improvements for meaningful integration of patient stakeholders into HTA appraisals across jurisdictions.

A conceptual framework, called Innovation of Health Technology Assessment Methods (IHTAM), has been developed to facilitate the understanding of how to innovate methods of health technology assessment (HTA). However, the framework applicability has not been evaluated in practice. Hence, we aimed to explore framework applicability in three cases of method innovation that are part of the HTx project and to develop a roadmap to improve framework applicability.

The IHTAM framework was applied to three cases of innovating HTA methods. We collected feedback from case study leaders and consortium members after a training session, an approximately 1-year follow-up of periodic case study meetings, and a general assembly meeting where innovation progresses of the three cases were reported through surveys and interviews. Feedback was then summarized using an open-coding technique.

According to feedback, the framework provided a structured way of deliberation and helped to improve collaboration among HTA stakeholders. However, framework applicability could be improved if it was complemented by a roadmap with a loop structure to provide tailored guidance for different cases, and with items to elaborate actions to be taken by stakeholders. Accordingly, a 48-item roadmap was developed.

The IHTAM framework was generally applicable to the three case studies. A roadmap, with loop structure and actionable items, could complement the framework, and may provide HTA stakeholders with tailored guidance on developing new methods. To further examine the framework applicability, we recommend stakeholders to apply the IHTAM framework and its roadmap in future practice.

The sandbox approach, developed in the financial technologies sector, creates an environment to collaboratively develop and test innovative new products, methods and regulatory approaches, separated from business as usual. It has been used in health care to encourage innovation in response to emerging challenges, but, until recently, has not been used in health technology assessment (HTA). This article summarizes our learnings from using the sandbox approach to address three challenges facing HTA organizations and to identify implications for the use of this approach in HTA.

We identified three challenging contemporary HTA-related topics to explore in a sandbox environment, away from the pressures and interests of “live” assessments. We convened a pool of 120 stakeholders and experts to participate in various sandbox activities and ultimately co-develop solutions to help HTA organizations respond to the identified challenges.

Important general learnings about the potential benefits and implementation of a sandbox approach in HTA were identified. Consequently, we developed recommendations to guide its use, including how to implement an HTA sandbox in an effective way and the types of challenges for which it may be best suited.

For many HTA organizations, it is difficult to carefully consider emerging challenges and innovate their processes due to risks associated with decision errors and resource limitations. The sandbox approach could reduce these barriers. The potential benefits of addressing HTA challenges in a collaborative “safe space” are considerable.

To assess the health technology assessment (HTA) process in Greece from its establishment in 2018 until 2023 in terms of timeliness and productivity.

Data were collected from the HTA Committee’s database and other publicly available sources. The overall study timeframe was divided into three periods: (i) July 2018–January 2020, (ii) January 2020–July 2021, and (iii) July 2021–February 2023.

During the study period, a total of 1,157 applications for medicinal products (MPs) (including 219 new active substances (NAS) and orphans) were submitted to the HTA Committee. The number of HTA recommendations increased from 60 (first period) to 641 (third period), while the backlog of MPs pending HTA and price negotiations decreased from 89 and 106 (January 2020) to 8 and 44 (February 2023), respectively. The median time intervals for all application types decreased significantly over time. In February 2023, the median time for clinical data assessment of NAS (excluding orphans) almost halved from 207 days in the first period to 114 days; median times for NAS and orphans from regulatory approval to HTA application were 420 and 457 days, and from HTA application to reimbursement 228 and 417 days, respectively.

The performance of the HTA process in Greece improved significantly over time, with increased MP appraisals, backlog reduction, and decreased timelines. Delays in reimbursement of NAS were mainly caused by the long gap between regulatory approval and HTA application. Overall, HTA review times in Greece are now on par with that of well-established European HTA systems.

This study investigated how patient representatives have experienced their involvement in medicines appraisal and reimbursement processes with the Council for Choices in Health Care in Finland (COHERE) and the Pharmaceuticals Pricing Board (PPB) and how authorities perceive the role of patient organizations’ input.

Semi-structured thematic individual and pair interviews were conducted in 2021 with representatives (n = 14) of patient organizations and government officials (n = 7) of the Ministry of Social Affairs and Health. The interview data were analyzed using qualitative content analysis.

Patient representatives expressed their appreciation for the PPB and the COHERE in creating consultation processes and systematic models that support involvement. However, there were many challenges: patient representatives were uncertain about how their submissions were utilized in official processes and whether their opinions had any significance in decision-making. Patients or patient organizations lack representation in appraisal and decision-making bodies, and patient representatives felt that decision-making lacked transparency. The importance of patient involvement was highlighted by the authorities, but they also emphasized that the patient organizations’ contributions were complementary to the other materials. Submissions regarding the medications used to treat rare diseases and those with limited research evidence were considered particularly valuable. However, the submissions may not necessarily have a direct impact on decisions.

The interviews provided relevant input for the development of involvement processes at the PPB and COHERE. The interviews confirmed the need for increased transparency in the medicines assessment, appraisal, and decision-making procedures in Finland.

In France, decisions for pricing and reimbursement for medicinal products are based on appraisals performed by the National authority for health (Haute Autorité de Santé (HAS)). During the appraisal process, additional real-world evidence can be requested as “Post-Registration Studies” (PRS) when there are uncertainties in evidence that could be resolved by additional data collection. To facilitate PRS planning, a retrospective exploratory analysis was conducted to identify the characteristics of medicinal products associated with a PRS request.

This analysis encompassed all appraisals finalized between January 1, 2016 and December 31, 2021 and compared products for which the appraisal led to a PRS request with those that did not.

Six hundred positive opinions for reimbursement were identified, with a PRS request present in 17 percent (n = 103) of cases. The independent characteristics associated with a PRS request were a mild or moderate clinical benefit score, a major to moderate or minor clinical added value score, previous availability under an early access program, and certain therapeutic areas (neurology, pulmonology, and endocrinology). These findings suggest two different profiles of PRS requests: (i) products for which there is uncertainty in the size of the clinical benefit and (ii) innovative products for which a substantial benefit is expected but uncertainties persist.

These results will assist health technology developers to better anticipate data generation to promptly address uncertainties identified by HAS. It may also help HAS and other assessment agencies to work together to improve postlaunch evidence generation according to the characteristics of the medicinal products.

Systematic priority setting is necessary for achieving high-quality healthcare using limited resources in low- and middle-income countries. Health technology assessment (HTA) is a tool that can be used for systematic priority setting. The objective of this study was to conduct a stakeholder and situational analysis of HTA in Zimbabwe.

We identified and analyzed stakeholders using the International Decision Support Initiative checklist. The identified stakeholders were invited to an HTA workshop convened at the University of Zimbabwe. We used an existing HTA situational analysis questionnaire to ask for participants’ views on the need, demand, and supply of HTA. A follow-up survey was done among representatives of stakeholder organizations that failed to attend the workshop. We reviewed two health policy documents relevant to the HTA. Qualitative data from the survey and document review were analyzed using thematic analysis.

Forty-eight organizations were identified as stakeholders for HTA in Zimbabwe. A total of 41 respondents from these stakeholder organizations participated in the survey. Respondents highlighted that the HTA was needed for transparent decision making. The demand for HTA-related evidence was high except for the health economic and ethics dimensions, perhaps reflecting a lack of awareness. Ministry of Health was listed as a major supplier of HTA data.

There is no formal HTA agency in the Zimbabwe healthcare system. Various institutions make decisions on prioritization, procurement, and coverage of health services. The activities undertaken by these organizations provide context for the institutionalization of HTA in Zimbabwe.

Our objective was to explore procedures and methods used at health technology assessment (HTA) agencies for assessing medical devices and the underlying views of HTA practitioners about appropriate methodology to identify challenges in adopting new methodologies for assessing devices. We focused on the role of normative commitments of HTA practitioners in the adoption of new methods.

An online survey, including questions on procedures, scoping, and assessments of medical devices, was sent to members of the International Network of Agencies for Health Technology Assessment. Interviews were conducted with survey respondents and HTA practitioners involved in assessments of transcatheter aortic valve implantation to gain an in-depth understanding of choices made and views about assessing medical devices. Survey and interview questions were inspired by the “values in doing assessments of health technologies” approach towards HTA, which states that HTA addresses value-laden questions and information.

The current practice of assessing medical devices at HTA agencies is predominantly based on procedures, methods, and epistemological principles developed for assessments of drugs. Both practical factors (available time, demands of decision-makers, existing legal frameworks, and HTA guidelines), as well as commitments of HTA practitioners to principles of evidence-based medicine, make the adoption of a new methodology difficult.

There is a broad recognition that assessments of medical devices may need changes in HTA methodology. In order to realize this, the HTA community may require both a discussion on the role, responsibility, and goals of HTA, and resulting changes in institutional context to adopt new methodologies.

The rising costs of drugs have necessitated the exploration of innovative payment methods in healthcare systems. Risk-sharing agreements (RSAs) have been implemented in many countries as a value-based payment mechanism to manage the uncertainty associated with expensive technologies. This study aimed to investigate stakeholder perspectives on value-based payment in the Singaporean context, providing insights for future directions in health technology assessment and financing.

This descriptive qualitative inquiry involved participant interviews conducted between October 2021 and April 2022. Thematic analysis was conducted in two phases to analyze the interview transcripts.

Seventeen respondents participated in the study, and five key themes emerged from the analysis. Stakeholders viewed RSAs as moderately positive, despite limited experience with them. They emphasized the importance of clearly defining objectives and establishing transparent criteria for implementing these schemes. The current data infrastructure was identified as both a barrier and facilitator, as RSAs impose administrative burdens. To successfully implement these payment mechanisms, capacity building, and effective stakeholder engagement that fosters mutual trust and cocreation are crucial.

This study confirms previously identified barriers and facilitators to successful RSA implementation while contextualizing them within the Singaporean setting. The findings suggest that value-based payment has the potential to address uncertainty and improve access to healthcare technologies, but these barriers must be addressed for the schemes to be effective.

Health technology assessment (HTA) plays a vital role in healthcare decision-making globally, necessitating the identification of key factors impacting evaluation outcomes due to the significant workload faced by HTA agencies.

The aim of this study was to predict the approval status of evaluations conducted by the Brazilian Committee for Health Technology Incorporation (CONITEC) using natural language processing (NLP).

Data encompassing CONITEC’s official report summaries from 2012 to 2022. Textual data was tokenized for NLP analysis. Least Absolute Shrinkage and Selection Operator, logistic regression, support vector machine, random forest, neural network, and extreme gradient boosting (XGBoost), were evaluated for accuracy, area under the receiver operating characteristic curve (ROC AUC) score, precision, and recall. Cluster analysis using the k-modes algorithm categorized entries into two clusters (approved, rejected).

The neural network model exhibited the highest accuracy metrics (precision at 0.815, accuracy at 0.769, ROC AUC at 0.871, and recall at 0.746), followed by XGBoost model. The lexical analysis uncovered linguistic markers, like references to international HTA agencies’ experiences and government as demandant, potentially influencing CONITEC’s decisions. Cluster and XGBoost analyses emphasized that approved evaluations mainly concerned drug assessments, often government-initiated, while non-approved ones frequently evaluated drugs, with the industry as the requester.

NLP model can predict health technology incorporation outcomes, opening avenues for future research using HTA reports from other agencies. This model has the potential to enhance HTA system efficiency by offering initial insights and decision-making criteria, thereby benefiting healthcare experts.

Traditional therapies are crucial in maintaining and improving human well-being. China’s healthcare policymakers are attempting to use health technology assessment (HTA) as a decision-making supportive tool. The value assessment framework for Chinese patent medicine (CPM) has been developed and is being adopted and validated widely by research institutions. Subsequently, the healthcare decision-makers particularly hanker for the value framework of traditional non-pharmacological therapies.

To construct a practical value framework for traditional non-pharmacological therapies, a scoping review methodology was adopted to identify the evaluation domains and obstacles. A search, screening, and analysis process was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR). Evidence was retrieved from scientific databases and HTA agencies’ websites.

The search strategy identified 5 guidelines records and 17 acupuncture HTA reports. By synthesizing the valuable reports of CPM and acupuncture evaluation in representative countries, this study found that Mainland China was promoting the comprehensive value assessment of CPM, whereas the United Kingdom, Singapore, Canada, the United States, and Malaysia had carried out the HTA evaluation of acupuncture for various conditions among which chronic pain was the most common. UK and Singapore applied the HTA results to support acupuncture reimbursement decisions. Three domains, including safety, effectiveness, and economy, were commonly adopted. The identified biggest challenge of evaluating traditional non-pharmacological therapies is the scarce high-quality clinical evidence.

This study identified value domains and issues of traditional therapies, and pointed out future research implications, to promote the development value framework of traditional therapies.