Artificial intelligence (AI)-based health technologies (AIHTs) have already been applied in clinical practice. However, there is currently no standardized framework for evaluating them based on the principles of health technology assessment (HTA).

A two-round Delphi survey was distributed to a panel of experts to determine the significance of incorporating topics outlined in the EUnetHTA Core Model and twenty additional ones identified through literature reviews. Each panelist assigned scores to each topic. Topics were categorized as critical to include (scores 7–9), important but not critical (scores 4–6), and not important (scores 1–3). A 70 percent cutoff was used to determine high agreement.

Our panel of 46 experts indicated that 48 out of the 65 proposed topics are critical and should be included in an HTA framework for AIHTs. Among the ten most crucial topics, the following emerged: accuracy of the AI model (97.78 percent), patient safety (95.65 percent), benefit–harm balance evaluated from an ethical standpoint (95.56 percent), and bias in data (91.30 percent). Importantly, our findings highlight that the Core Model is insufficient in capturing all relevant topics for AI-based technologies, as 14 out of the additional 20 topics were identified as crucial.

It is imperative to determine the level of agreement on AI-relevant HTA topics to establish a robust assessment framework. This framework will play a foundational role in evaluating AI tools for the early diagnosis of dementia, which is the focus of the European project AI-Mind currently being developed.

The roles and potential value of patient preference (PP) data in health technology assessment (HTA) remain to be fully realized despite an expanding literature and various efforts to establish their utility. This article reports lessons learned through a series of collaborative workshops with HTA representatives, organized by the Health Technology Assessment International’s Patient Preferences Project Subcommittee.

Five online workshops were conducted between June 2022 and June 2023, seeking to facilitate collaborative learning and reflection on ways that PP data can be integrated into HTA. Participants included nine HTA representatives from the United States, Canada, Australia, England, and the Netherlands. Workshops were recorded, transcribed, and thematically analyzed.

Despite appreciating the value of PP data, participants were ambivalent about their use in HTA. Some felt that they were already getting the information they needed from the cost-effectiveness analysis or existing patient involvement processes. Others thought that PP data would be very helpful at the initial and final stage of the decision-making process and, particularly, in the following cases: (a) when technology has important non-health benefits; (b) when the clinical and/or cost-effectiveness evidence is marginal; and (c) when treatment is indicated for a large and heterogeneous population. Issues related to the validity and reliability of PP studies were frequently raised, with preference heterogeneity at the core of these concerns.

Collaborating with HTA representatives in the “co-creation” of PP research can help address their concerns and facilitate mutual learning about how PP data can be used in HTA.

International development agendas increasingly push for access to healthcare for all through universal healthcare coverage. Health economic evaluations and health technology assessment (HTA) could provide evidence to support this but do not routinely incorporate consideration of equitable access.

We undertook an international scoping review of health economic evaluation and HTA guidelines to examine how well issues of healthcare access and equity are represented, evidence recommendations, and gaps in current guidance to support evidence generation in this area. Guidelines were sourced from guideline repositories and websites of international agencies and organizations providing best practice methods guidance. Articles providing methods guidance for the conduct of HTA, or health economic evaluation, were included, except where they were not available in English and a suitable translation could not be obtained.

The search yielded forty-seven national, four international, and nine independent guidelines, along with eighty-six articles providing specific methods guidance. The inclusion of equity and access considerations in current guidance is extremely limited. Where they do feature, detail on specific methods for providing evidence on these issues is sparse.

Economic evaluation could be a valuable tool to provide evidence for the best healthcare strategies that not only maximize health but also ensure equitable access to care for all. Such evidence would be invaluable in supporting progress towards universal healthcare coverage. Clear guidance is required to ensure evaluations provide evidence on the best strategies to support equitable access to healthcare, but such guidance rarely exists in current best practice and guidance documents.

To synthesize the results of cost-effectiveness studies of different triage tests in comparison to repeat cytology for women with atypical squamous cells of undetermined significance (ASC-US) or low-grade squamous intraepithelial lesions (LSIL) results.

Electronic databases (Medline/PubMed, Lilacs, Embase, The Cochrane Library, Scopus, Web of Science, Scielo, The NHS Economic Evaluation Database, Econlit, and CEA Registry) were searched for cost-effectiveness or cost-utility publications. Per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, two independent reviewers selected eligible publications based on the selection criteria and performed data extraction. Methodological quality was assessed using the Quality of Health Economic Studies tool.

Five cost-effectiveness analyses were included comparing HPV testing, immediate colposcopy, and liquid-based cytology with HPV testing reflex to repeat cytology. The main outcome adopted was cervical intraepithelial neoplasia level 2 or higher (CIN2+) cases detected. In pairwise comparisons, HPV testing was more frequently observed as the most cost-effective strategy. Incremental cost-effectiveness ratios were very sensitive to costs of test kit variation and accuracy estimates with some sensitivity analysis scenarios showing immediate colposcopy more cost-effective than HPV testing depending on the tests’ unitary costs and effectiveness.

This systematic review of economic evidence corroborates clinical evidence showing cytology is the least effective, although less costly, triage strategy. Cytology-based triage programs need to be updated to offer timely treatment to women diagnosed with ASC-US/LSIL and better resource allocation.

Healthcare disinvestment requires multi-level decision-making, and early stakeholder engagement is essential to facilitate implementation and acceptance. This study aimed to explore the perceptions of Malaysian healthcare stakeholders to disinvestment initiatives as well as identify disinvestment activities in the country.

A cross-sectional online survey was conducted from February to March 2023 among Malaysian healthcare stakeholders involved in resource allocation and decision-making at various levels of governance. Response frequencies were analyzed descriptively and cross-tabulation was performed for specific questions to compare the responses of different groups of stakeholders. For free-text replies, content analysis was used with each verbatim response examined and assigned a theme.

A total of 153 complete responses were analyzed and approximately 37 percent of participants had prior involvement in disinvestment initiatives. Clinical effectiveness and cost-effectiveness ranked as the most important criteria in assessment for disinvestment. Surprisingly, equity was rated the lowest priority despite its crucial role in healthcare decision-making. Almost 90 percent of the respondents concurred that a formal disinvestment framework is necessary and the importance of training for the program’s successful implementation. Key obstacles to the adoption of disinvestment include insufficient stakeholder support and political will as well as a lack of expertise in executing the process.

While disinvestment is perceived as a priority for efficient resource allocation in Malaysian healthcare, there is a lack of a systematic framework for its implementation. Future research should prioritize methodological analysis in healthcare disinvestment and strategies for integrating equity considerations in evaluating disinvestment candidates.

Health technology assessment (HTA) organizations generate guidelines to inform healthcare practices toward improved health outcomes. This review sought to identify and classify outcomes of guidelines from HTA organizations within published research.

We performed a systematic mixed studies review of empirical studies that (a) referred to a published guideline from an HTA organization and (b) reported an outcome resulting from a guideline. We searched the published literature in English or French within seven databases. Outcome types were classified within five dimensions of an existing framework for online health information (e.g., relevance, cognitive/affective impact, and use). Subdimensions were inductively developed. A two-phase sequential data synthesis was performed. Phase 1: a hybrid deductive–inductive thematic analysis identified the types of outcomes and displayed their relationships on a concept map. Phase 2: descriptive statistics were tabulated by the type of outcome.

A total of 6,719 records were retrieved through searches on 6 February 2023. After screening, we included 120 observational studies (twenty-one qualitative, ninety-four quantitative, and five mixed methods). Phase 1 identified twenty-nine types of outcomes. The most frequently reported outcomes were within the organizational dimension (reported in ninety-four studies). The most common subdimensions were “Referrals” (thirty-eight occurrences), the “Quality of Prescriptions” (fifteen occurrences), and the “Quality of Diagnosis” (eight occurrences). For Phase 2, we could only generate descriptive statistics on seventeen outcomes. These were almost equally distributed among positive, neutral, and negative effects. Our results contribute to knowledge about the outcomes of HTA guidelines and options for documenting and measuring them in future evaluations.

To describe the type of evidence and the clinical benefit of cancer medicines assessed for funding in Australia by the Pharmaceutical Benefits Advisory Committee (PBAC) and to assess it with the European Society of Medical Oncology Magnitude of Clinical Benefit Scale version 1.1 (ESMO-MCBS).

All data on applications submitted to PBAC between 2010 and 2020 were extracted from PBAC Public Summary Documents available online. ESMO-MCBS ratings were retrieved from the ESMO-MCBS website.

Then, 182 cancer indications for 100 cancer medicines were examined by PBAC, including 124 (68.1 percent) for solid tumors and 58 (31.9 percent) for hematological cancers. A total of 137 (75.3 percent) indications were recommended for PBS funding and 40 (21.9 percent) were rejected. Randomized clinical trials (RCTs) were the main source of evidence in 154 indications (84.6 percent), single-arm studies in 28 (15.4 percent) indications. Statistically significant improvement in overall survival (OS) was reported in 80 (44 percent) of the indications, with a median OS gain of 3.0 months (range 0.9–17.0) for solid tumors and 8.2 months (range 1–49.1) for hematological cancers when mature OS data were available. The ESMO-MCBS score was available for 99 solid tumor indications, of which 51 (51.5 percent) showed substantial clinical benefit according to ESMO-MCBS, including 40 (54.1 percent) of PBAC-recommended indications and 9 (42.9 percent) of PBAC-rejected indications. There was no association between the ESMO scoring and PBAC decision.

Most cancer medicines indications considered by PBAC were supported by RCTs. A minority showed a substantial improvement in OS.

The primary objectives of this umbrella review were to (a) quantify the relative importance, of “severity” and “rarity” criteria in health resource allocation; and (b) analyze the contextual factors influencing the relative importance. The secondary objective was to examine how “severity” and “rarity” criteria are defined.

Searches were carried out in PubMed and Embase to identify eligible systematic reviews. Quality appraisal of systematic reviews was undertaken. From identified systematic reviews, primary studies were extracted and further screened for eligibility. The inclusion of severity and rarity criteria and their respective weights in primary studies were examined. Descriptive and regression analyses were performed.

Twenty-nine systematic reviews were screened, of which nine met the inclusion criteria. Primary studies included in these systematic reviews were retrieved and screened, resulting in forty articles included in the final analysis. Disease severity was more frequently considered (n = 29/40) than disease rarity (n = 23/40) as an evaluation criterion. Out of all cases where both were included as evaluation criteria, disease severity was assigned higher weights 84 percent of the time (n = 21/25).

Our review found consistent evidence that disease severity is more relevant and preferred to rarity as a priority-setting criterion albeit constraints in statistical analysis imposed by limited sample size and data availability. Where funding for rare diseases is concerned, we advocate that decision-makers be explicit in clarifying the significance of disease severity and/or rarity as a value driver behind decisions. Our findings also reinforce the relevance of disease severity as a criterion in priority setting.

Health Technology Assessment (HTA) practitioners recognize the significance of qualitative methodologies that focus on how a technology is feasible, meaningfulness, acceptable, and equitable. This mapping aimed to delineate the frameworks employed to synthesize qualitative evidence and assess the quality of synthesis in HTA .

Mapping was conducted using Medline, LILACS, CINAHL, Embase, Web of Science, Scopus, PsycINFO, Cochrane Library, JBI, and ScienceDirect databases. Gray literature searches included PROQUEST, Open Grey, Canadian Agency for Drugs and Technologies in Health’s Grey Matters, Google Scholar, and HTA agency websites. The inclusion criteria were centered on global qualitative evidence synthesis frameworks. The data are presented in the tables.

Of the 2054 articles, 31 were included, mostly from Europe. Guide was the type of document more cited, and most authors are from HTA agencies and universities. Incorporating both patient and family perspectives is the most cited reason for include qualitative evidence. Regardless of the framework or tool, SPICE was the main acronym, and RETREAT was preferred for approach selection. Thematic synthesis dominated analytic methods, and CASP was the primary quality appraisal tool. GRADE-CERQual graded evidence synthesis, with ENTREQ as the top reporting guidance. The GRADE evidence-to-decision framework was mentioned for recommendations.

This mapping highlights the movement incorporate qualitative evidence in HTA employing specific frameworks. Despite the similarities among documents, most of them describe part of the process to synthesize qualitative evidence. Standardizing procedures to incorporate qualitative evidence into HTA can enhance decision-making. These findings offer essential considerations for HTA practice.

Evaluating the impact of patient involvement in health technology assessments (HTA) may help improve practices and avoid ineffective activities. Evaluation, however, continues to be infrequent, inconsistent, and often only relates to process quantity or quality. The Patient and Citizen Involvement in HTA Interest Group (PCIG) within Health Technology Assessment International set out to contextualize this impact to support evaluation.

Given the lack of established methodology to measure impact, the team performed a qualitative analysis of first-hand accounts about perceived changes in HTA due to involvement of patient stakeholders. A questionnaire was developed, piloted, and rolled out to collect personal perspectives from stakeholders with relevant experience. The stories were analyzed in the aggregate to identify themes in the data.

From January 2019 to September 2021, twenty-four responses were collected through PCIG’s network. Responses (including one joint industry-HTA body submission) came from patient representatives (12), HTA bodies (11), and industry representatives (2) from North America (5), South America (3), Europe (13), and Asia Pacific (3). Based on themes commonly reported, a three-domain framework for evaluating impact is proposed: impact on basis of HTA result or recommendation, impact on HTA body, and impact on patient participants. The framework includes components under each domain to support reporting.

Using the Three-Domain Impact Framework may be useful in identifying, evaluating, and communicating the value of patient involvement in HTA. Enhancing and increasing reporting practices may improve transparency and facilitate process improvements for meaningful integration of patient stakeholders into HTA appraisals across jurisdictions.

A conceptual framework, called Innovation of Health Technology Assessment Methods (IHTAM), has been developed to facilitate the understanding of how to innovate methods of health technology assessment (HTA). However, the framework applicability has not been evaluated in practice. Hence, we aimed to explore framework applicability in three cases of method innovation that are part of the HTx project and to develop a roadmap to improve framework applicability.

The IHTAM framework was applied to three cases of innovating HTA methods. We collected feedback from case study leaders and consortium members after a training session, an approximately 1-year follow-up of periodic case study meetings, and a general assembly meeting where innovation progresses of the three cases were reported through surveys and interviews. Feedback was then summarized using an open-coding technique.

According to feedback, the framework provided a structured way of deliberation and helped to improve collaboration among HTA stakeholders. However, framework applicability could be improved if it was complemented by a roadmap with a loop structure to provide tailored guidance for different cases, and with items to elaborate actions to be taken by stakeholders. Accordingly, a 48-item roadmap was developed.

The IHTAM framework was generally applicable to the three case studies. A roadmap, with loop structure and actionable items, could complement the framework, and may provide HTA stakeholders with tailored guidance on developing new methods. To further examine the framework applicability, we recommend stakeholders to apply the IHTAM framework and its roadmap in future practice.

The Health Technology Assessment International (HTAi) 2023 Annual Meeting included a novel “fishbowl” session intended to 1) probe the role of HTA in the emergence of generative pretrained transformer (GPT) large language models (LLMs) into health care and 2) demonstrate the semistructured, interactive fishbowl process applied to an emerging “hot topic” by diverse international participants.

The fishbowl process is a format for conducting medium-to-large group discussions. Participants are separated into an inner group and an outer group on the periphery. The inner group responds to a set of questions, whereas the outer group listens actively. During the session, participants voluntarily enter and leave the inner group. The questions for this fishbowl were: What are current and potential future applications of GPT LLMs in health care? How can HTA assess intended and unintended impacts of GPT LLM applications in health care? How might GPT be used to improve HTA methodology?

Participants offered approximately sixty responses across the three questions. Among the prominent themes were: improving operational efficiency, terminology and language, training and education, evidence synthesis, detecting and minimizing biases, stakeholder engagement, and recognizing and accounting for ethical, legal, and social implications.

The interactive fishbowl format enabled the sharing of real-time input on how GPT LLMs and related disruptive technologies will influence what technologies will be assessed, how they will be assessed, and how they might be used to improve HTA. It offers novel perspectives from the HTA community and aligns with certain aspects of ongoing HTA and evidence framework development.

We draw from the Health Technology Assessment (HTA) literature to propose how hospitals and local health networks can prepare the key components of early economic evaluations to support the development and management of health service interventions.

Using the case example of a proposed intervention for older people in the Emergency Department (ED), a conceptual logic model of a new health service intervention is articulated to inform the structuring and population of a decision-analytic model using observed data on the existing care comparator and structured elicitation exercise of initial stakeholder expectations of intervention effects.

The elicited patient pathway probabilities and lengths of stay quantities profile which of the existing types of patients are expected to avoid the ED and how this impacts the lengths of stay across the system. The exercise also quantifies the stakeholders’ uncertainty and disagreement, with qualitative insights into why. The elicitation exercise participants draw upon the rationale for how the intervention is expected to affect a change within the local context, as captured within the logic model, together with the descriptive analyses of the characteristics and utilization of their target population. Feedback indicates the methods are acceptably robust yet pragmatic enough for healthcare delivery settings.

As proposed in this paper, HTA methods can be used to capture how key stakeholders initially expect a service intervention to affect a change within their local context. The example results can be used in a decision-analytic model to guide the development and management of an intervention.

The costs of low back pain (LBP) are complex and difficult to estimate. This study aims to adapt the Cost for Patients Questionnaire (CoPaQ) for use in LBP populations.

In a cross-sectional qualitative study, we conducted cognitive interviews to assess the CoPaQ’s suitability for addressing costs related to LBP. Three groups of participants were included (n = 5 each): (i) persons with a history of LBP or primary caregiver, (ii) researchers with expertise in LBP, and (iii) primary care providers specialized in treating LBP. The interpretation, analysis, and summary of results used Knafl et al.’s qualitative content analysis method.

Persons with a history of LBP (n = 5), had a median age of 60 years (Interquartile Range (IQR): 26–71.5), and varying durations of LBP, the median duration of LBP 7 years (IQR: 4–32.5). Researchers (n = 5) had a median age of 33 years (IQR: 29–45). Primary care providers (n = 5) had a median age of 40 years (IQR: 37.5–65), and a background in chiropractic care (n = 3) and physiotherapy (n = 2). Content analysis of the interviews revealed sources of error with five pre-determined themes (clarity/comprehension, relevance, inadequate response definition, reference point, perspective modifiers) and one developed theme (organization). We modified the questionnaire for LBP populations based on the feedback.

Our study evaluated the content validity of a questionnaire that assesses the direct and indirect costs associated with LBP. Future studies should pilot this questionnaire with persons of varying LBP severity and compare it with cost diaries.

The sandbox approach, developed in the financial technologies sector, creates an environment to collaboratively develop and test innovative new products, methods and regulatory approaches, separated from business as usual. It has been used in health care to encourage innovation in response to emerging challenges, but, until recently, has not been used in health technology assessment (HTA). This article summarizes our learnings from using the sandbox approach to address three challenges facing HTA organizations and to identify implications for the use of this approach in HTA.

We identified three challenging contemporary HTA-related topics to explore in a sandbox environment, away from the pressures and interests of “live” assessments. We convened a pool of 120 stakeholders and experts to participate in various sandbox activities and ultimately co-develop solutions to help HTA organizations respond to the identified challenges.

Important general learnings about the potential benefits and implementation of a sandbox approach in HTA were identified. Consequently, we developed recommendations to guide its use, including how to implement an HTA sandbox in an effective way and the types of challenges for which it may be best suited.

For many HTA organizations, it is difficult to carefully consider emerging challenges and innovate their processes due to risks associated with decision errors and resource limitations. The sandbox approach could reduce these barriers. The potential benefits of addressing HTA challenges in a collaborative “safe space” are considerable.

The decision-making (DM) process in public administration is the subject of research from different perspectives and disciplines. Evidence-based policies, such as health technology assessment (HTA), are not the only support on which public policies are designed. During the COVID-19 pandemic WHO, national and subnational institutions developed HTA reports to guide DM. Despite this, inadequate variability was observed in the health technologies recommended and reimbursed by different provincial Health Ministries in a federally organized developing country like Argentina. The processes and results of DM on health technologies for COVID-19 in Health Ministries of Argentina were inquired.

A retrospective research design was developed, with triangulation of quantitative and qualitative methods. We retrieved information for the years 2020–2021 through document review of official webpages, surveys, and interviews with decision-makers of the 25 Argentinian Ministries of Health. We analyzed the recommendations and reimbursement policies of seven health technologies.

In contradiction with WHO’s policies, ivermectine, inhaled ibuprofen, convalescent plasma and equine serum were widely recommended by most of Argentina’s health ministries outside a clinical trial context, with risks for patients and a huge opportunity cost.

Despite an important HTA institutional capacity, the impact of HTA organizations and their technical reports was limited. Health Ministries with institutionalized HTA units had more adherence to WHO recommendations, but the influence of different technical and political criteria was identified. Power relations within and outside the administration, the pharmaceutical industry and academics, the media, social pressure, the judicial and legislative powers, and the political context strongly influenced DM.