The use of mobile applications in the treatment of health issues is more frequently becoming common practice. Apps are fast, versatile, and manageable tools that allow the empowerment of patients and professionals, and can reduce the possible stigmatization suffered by some patients, mainly in mental health. There are more than 325,000 health apps on the market, but their impact remains unclear. There are several initiatives to define how health applications should be assessed, however, all of them address only partial aspects of the evaluation. The theoretical frameworks existing to date highlight the need to develop new tools and methodologies to assess mobile applications whose objective is the management of specific pathologies.

The primary goal of the EvalDepApps project is to develop and pilot an assessment tool for mobile applications whose main objectives are the treatment, monitoring or social support of people suffering from depression. The project is inspired by the results and lessons learnt from a previous project, EVALAPPS, whose central aim was to develop a tool to assess health apps targeted toward the management of overweight and obesity. The first steps of the EvalDepApps project are: (i) to explore and characterize the current landscape of mobile applications available in the market to treat depression through a systematic appraisal, and (ii) to review the existing evidence about the effectiveness and safety of these applications through systematic research of the existing evidence.

Preliminary results show that all the depression management studies were by design based on cognitive-behavioral therapy (CBT) interventions (n=17) and the main management tools included in the services (web or apps) are psychoeducation and coaching (14), together with self-monitoring and feedback messaging (13).

Moreover, although health apps seem to be an interesting strategy to treat depression, there are very few apps available on the markets (30) and the supporting evidence is very limited. This result uncovers a need for further systematic and clinically oriented validation and testing of such apps.

Spain incorporated in 2020 changes in its health technology assessment (HTA), pricing, and reimbursement system for medicines including publishing reports, development of networks of experts, or consultation with stakeholders. Despite these changes, it is unclear how deliberative frameworks are applied and the process has been criticized for not being sufficiently transparent. This study analyses the level of implementation of deliberative processes in HTA for medicines in Spain.

We review the grey literature and summarize the Spanish HTA, pricing, and reimbursement process of medicines. We apply the deliberative processes for HTA checklist, developed to assess the overall context of the deliberative process, and identify the stakeholders involved and type of involvement following the framework for evidence-informed deliberative processes, a framework for benefit package design that aims to optimize the legitimacy of decision making.

In the Spanish HTA, pricing, and reimbursement process deliberation takes place in order to exchange viewpoints and reach common ground, mainly during the prioritization, assessment, and appraisal steps. It is closed to the public, not clearly summarized in published documents and limited to the Ministry of Health, the regulatory agency, other Ministries, and experts with mostly clinical and/or pharmaceutical background. The views of stakeholders are only represented through consultation. Communication is the most commonly used form of stakeholder engagement.

Despite improvements in transparency of the Spanish HTA process for evaluating medicines, aspects related to stakeholder involvement and implementation of deliberative frameworks need further attention in order to achieve further legitimacy of the process.

The aim of this study was to deliver insights from multiple stakeholders into actual and future collaboration for health technology assessment (HTA) in general and in oncology in particular.

Eighteen semi-structured interviews were conducted with experts from European HTA bodies (HTAbs), former board members of the European Network for Health Technology Assessment (EUnetHTA), and representatives from the pharmaceutical industry, a regulatory agency, academia, and patient organizations. The stakeholders were asked about their support of the EUnetHTA’s intent, about the general strengths and challenges of the EUnetHTA and its Joint Action 3 (JA 3), the strengths and challenges of the clinically oriented HTA collaboration in oncology during JA 3 across the technology life cycle, about future challenges to HTA in oncology with consequences for collaboration, and about collaboration in the economic domains of HTA. The transcribed interviews were analyzed qualitatively.

The participants perceived the intention and work quality of the EUnetHTA as positive. The experts described methodological, procedural, and capacity challenges in early dialogues (EDs) and rapid relative effectiveness assessments (REAs) meant to analyze clinical effectiveness in oncology. The majority attached increasing importance to collaboration in the future to cope with the uncertainty of HTA. Several stakeholders also proposed the incorporation of joint postlaunch evidence generation (PLEG) activities. Some gave sporadic suggestions for voluntary nonclinical collaboration as well.

Stakeholders’ continued readiness to discuss the remaining challenges to and sufficient resources for implementing HTA regulation, as well as further cooperative expansion along the technology life cycle, are necessary for improved HTA collaboration in Europe.

Evaluating the impact of patient involvement in health technology assessments (HTA) may help improve practices and avoid ineffective activities. Evaluation, however, continues to be infrequent, inconsistent, and often only relates to process quantity or quality. The Patient and Citizen Involvement in HTA Interest Group (PCIG) within Health Technology Assessment International set out to contextualize this impact to support evaluation.

Given the lack of established methodology to measure impact, the team performed a qualitative analysis of first-hand accounts about perceived changes in HTA due to involvement of patient stakeholders. A questionnaire was developed, piloted, and rolled out to collect personal perspectives from stakeholders with relevant experience. The stories were analyzed in the aggregate to identify themes in the data.

From January 2019 to September 2021, twenty-four responses were collected through PCIG’s network. Responses (including one joint industry-HTA body submission) came from patient representatives (12), HTA bodies (11), and industry representatives (2) from North America (5), South America (3), Europe (13), and Asia Pacific (3). Based on themes commonly reported, a three-domain framework for evaluating impact is proposed: impact on basis of HTA result or recommendation, impact on HTA body, and impact on patient participants. The framework includes components under each domain to support reporting.

Using the Three-Domain Impact Framework may be useful in identifying, evaluating, and communicating the value of patient involvement in HTA. Enhancing and increasing reporting practices may improve transparency and facilitate process improvements for meaningful integration of patient stakeholders into HTA appraisals across jurisdictions.

Patients with hematological malignancies are likely to develop hypogammaglobulinemia (HGG) and subsequent infections. Immunoglobulin (Ig) replacement is commonly given to prevent infections, but the total costs and cost effectiveness of its use are unknown.

A systematic review was conducted following PRISMA guidelines to assess evidence on the costs and cost effectiveness of Ig replacement, administered intravenously (IVIg) or subcutaneously (SCIg), in adult patients with hematological malignancies. This review was registered with PROSPERO (CRD42022321908).

Six studies were included out of a total of 3,612 citations. A narrative synthesis was conducted because of the high level of heterogeneity across the included studies. Two economic evaluations were identified: one cost-utility analysis (CUA) of IVIg versus no Ig and one comparing IVIg with SCIg. The quality of the evidence was low, with most studies having small patient numbers and a high risk of bias. Compared with no treatment, Ig replacement reduced the hospitalization rate in patients with hematological malignancies.

One study reported no change in hospitalization rates following a program to reduce IVIg use, and an observational study comparing IVIg with SCIg found more hospitalizations with SCIg but lower total costs per patient. The CUA comparing IVIg with no IVIg suggested that IVIg treatment was not cost effective, but this study was published in 1991 and had significant limitations. The other CUA found that home-based SCIg was more cost effective than IVIg, but model inputs were derived from unpublished data in a very small patient cohort with HGG and different malignancies.

Our review highlights key gaps in the literature. The cost effectiveness of Ig replacement in patients with hematological malignancies is still very uncertain. Despite the increasing use of Ig replacement there are limited data regarding its direct and indirect costs, and its optimal use and implications for healthcare resources remain unclear. Given the paucity of data on the cost and cost effectiveness of Ig treatment in this population, further health economic research is warranted.

Registration of trial protocols has become increasingly important in recent years. In the context of systematic reviews, published trial protocols facilitate the identification of studies. Data recorded in trials registers requires standardization to assist with ease of identification, and availability of the most current protocol version. Searching sources of trial protocols, for example trials registers, has issues relating to currency, coverage, functionality and indexing. An empirical study was conducted in the context of a funded systematic review, to establish; the proportion of trial protocols retrievable, the most effective retrieval methods, barriers to retrieving protocols, and whether the most easily retrieved protocol copy remains the document of record.

In a systematic review of primary care interventions for medically unexplained symptoms, seventy-four trials were identified as potential included studies. To search for the seventy-four trial protocols, multiple sources and methods were utilized to identify the the differential coverage of sources and the relative efficiency of retrieval methods. Retrieval methods included searching trials registers and bibliographic databases, internet searching, checking journal websites and contacting authors.

Results included; (i) number of trial protocols that were referenced in the corresponding study publication(s), (ii) percentage of protocols indexed in each checked source, including MEDLINE and various trials registers, (iii) number of authors that responded to email contact, (iv) number of authors that provided a reference to, or copy of, the protocol. Information on when the trial protocol was published, funding sources, and trial registration, was also recorded.

Conclusions are made regarding the coverage of different sources of trial protocols. This will enable Information Specialists to prioritize retrieval methods for identifying trial protocols to inform future search methods guidance. The main barriers to retrieving protocols are discussed together with recommendations for future empirical studies.

This study aims to develop a framework for establishing priorities in the regional health service of Murcia, Spain, to facilitate the creation of a comprehensive multiple criteria decision analysis (MCDA) framework. This framework will aid in decision-making processes related to the assessment, reimbursement, and utilization of high-impact health technologies.

Based on the results of a review of existing frameworks for MCDA of health technologies, a set of criteria was proposed to be used in the context of evaluating high-impact health technologies. Key stakeholders within regional healthcare services, including clinical leaders and management personnel, participated in a focus group (n = 11) to discuss the proposed criteria and select the final fifteen. To elicit the weights of the criteria, two surveys were administered, one to a small sample of healthcare professionals (n = 35) and another to a larger representative sample of the general population (n = 494).

The responses obtained from health professionals in the weighting procedure exhibited greater consistency compared to those provided by the general public. The criteria more highly weighted were “Need for intervention” and “Intervention outcomes.” The weights finally assigned to each item in the multicriteria framework were derived as the equal-weighted sum of the mean weights from the two samples.

A multi-attribute function capable of generating a composite measure (multicriteria) to assess the value of high-impact health interventions has been developed. Furthermore, it is recommended to pilot this procedure in a specific decision context to evaluate the efficacy, feasibility, usefulness, and reliability of the proposed tool.

The objective of this research was to evaluate managed access policy in England, drawing upon the expertise of a range of stakeholders involved in its implementation.

Seven focus groups were conducted with payer and health technology assessment representatives, clinicians, and representatives from industry and patient/carer organizations within England. Transcripts were analyzed using framework analysis to identify stakeholders’ views on the successes and challenges of managed access policy.

Stakeholders discussed the many aims of managed access within the National Health Service in England, and how competing aims had affected decision making. While stakeholders highlighted a number of priorities within eligibility criteria for managed access agreements (MAAs), stakeholders agreed that strict eligibility criteria would be challenging to implement due to the highly variable nature of innovative technologies and their indications. Participants highlighted challenges faced with implementing MAAs, including evidence generation, supporting patients during and after the end of MAAs, and agreeing and reinforcing contractual agreements with industry.

Managed access is one strategy that can be used by payers to resolve uncertainty for innovative technologies that present challenges for reimbursement and can also deliver earlier access to promising technologies for patients. However, participants cautioned that managed access is not a “silver bullet,” and there is a need for greater clarity about the aims of managed access and how these should be prioritized in decision making. Discussions between key stakeholders involved in managed access identified challenges with implementing MAAs and these experiences should be used to inform future managed access policy.

Based on a real-world collaboration with innovators in applying early health economic modeling, we aimed to offer practical steps that health technology assessment (HTA) researchers and innovators can follow and promote the usage of early HTA among research and development (R&D) communities.

The HTA researcher was approached by the innovator to carry out an early HTA ahead of the first clinical trial of the technology, a soft robotic sock for poststroke patients. Early health economic modeling was selected to understand the potential value of the technology and to help uncover the information gap. Threshold analysis was used to identify the target product profiles. Value-of-information analysis was conducted to understand the uncertainties and the need for further research.

Based on the expected price and clinical effectiveness by the innovator, the new technology was found to be cost-saving compared to the current practice. Risk reduction in deep vein thrombosis and ankle contracture, the incidence rate of ankle contracture, the compliance rate of the new technology, and utility scores were found to have high impacts on the value-for-money of the new technology. The value of information was low if the new technology can achieve the expected clinical effectiveness. A list of parameters was recommended for data collection in the impending clinical trial.

This work, based on a real-world collaboration, has illustrated that early health economic modeling can inform medical innovation development. We provided practical steps in order to achieve more efficient R&D investment in medical innovation moving forward.

Intention to treat (ITT) is a gold standard strategy to analyze the results of randomized controlled trials (RCTs). ITT analysis has been considered a methodological indicator of the quality of clinical trials. The extent to which the use of ITT is related to the treatment effects observed in RCTs has not been rigorously explored. Therefore, the main objective of this study was to determine the association between biases related to attrition and missing data and the use of intention to treat principle, and changes in effect size estimates in RCTs.

This was a meta-epidemiological study. A random sample of RCTs included in meta-analyses was identified. Data extraction including assessments of the use of intention to treat principle, missing data and drop-outs was conducted independently by two reviewers. To determine the association between biases related to attrition, missing data, and the use of intention to treat and effect sizes, a two-level analysis was conducted using a meta-meta-analytic approach.

Three-hundred and ninety-three trials included in 43 meta-analyses, analyzing 44,622 patients contributed to this study. From these, 134 trials (34.1%) used ITT and 218 (55.5%) did not use ITT. Trials which did not use the ITT principle, or which were assessed as having an inappropriate control of incomplete outcome data (based on the Cochrane risk of bias tool) tended to underestimate the treatment effect when compared with trials with adequate use of ITT (ES= -0.13; 95%CI -0.26, -0.01) or trials which were assessed as having an appropriate control of incomplete outcome (ES= -0.18; 95%CI -0.29, -0.08).

Our results suggest that when evaluating risk of bias of primary RCTs, systematic reviewers should pay attention to these biases since they could underestimate treatment effects. Systematic reviewers should perform sensitivity analysis including trials with low risk of bias in these domains.

To undertake a technical review of the search interface of the ISPOR Presentations Database. By technical review, we mean an evaluation of the technical aspects of the search interface and functionality, which a user must navigate to complete a search.

A validated checklist (Bethel and Rogers, 2014, Health Info Libr J, 31, 43-53) was used to identify where the interface performed well, where the interface was adequate, where the interface performed poorly, where functionality available in core biomedical bibliographic databases does not exist in the ISPOR database, and to establish a list of any issues arising during the review. Two researchers independently undertook the technical review in October 2021.

The ISPOR database scored 35 of a possible 165 (27/111 essential criteria and 8/54 desirable criteria). Two issues arising were identified, both of which will cause searchers to miss potentially eligible abstracts: (i) that search terms, which include * or ? as truncation or wildcard symbols should not be capitalized (e.g., cost* not Cost*; organi?ation not Organi?ation) and (ii) that quotation marks should be straight sided in phrase searching (e.g., “cost analyses” not “cost analyses”).

The ISPOR database is a promising and free database to identify abstracts/posters presented at ISPOR. We summarize two key issues arising, and we set out proposed changes to the search interface, including: adding the ability to export abstracts to a bibliographic tool, exporting search strategies, adding a researcher account, and updating the help guide. All suggestions will further improve this helpful database.

The hazard ratio (HR) is a commonly used summary statistic when comparing time to event (TTE) data between trial arms, but assumes the presence of proportional hazards (PH). Non-proportional hazards (NPH) are increasingly common in NICE technology appraisals (TAs) due to an abundance of novel cancer treatments, which have differing mechanisms of action compared with traditional chemotherapies. The goal of this study is to understand how pharmaceutical companies, evidence review groups (ERGs) and appraisal committees (ACs) test for PH and report clinical effectiveness in the context of NPH.

A thematic analysis of NICE TAs concerning novel cancer treatments published between 1 January 2020 and 31 December 2021 was undertaken. Data on PH testing and clinical effectiveness reporting for overall survival (OS) and progression-free survival (PFS) were obtained from company submissions, ERG reports, and final appraisal determinations (FADs).

NPH were present for OS or PFS in 28/40 appraisals, with log-cumulative hazard plots the most common testing methodology (40/40), supplemented by Schoenfeld residuals (20/40) and/or other statistical methods (6/40). In the context of NPH, the HR was ubiquitously reported by companies, inconsistently critiqued by ERGs (10/28), and commonly reported in FADs (23/28).

There is inconsistency in PH testing methodology used in TAs. ERGs are inconsistent in critiquing use of the HR in the context of NPH, and even when critiqued it remains a commonly reported outcome measure in FADs. Other measures of clinical effectiveness should be considered, along with guidance on clinical effectiveness reporting when NPH are present.

In model-based economic evaluations, the effectiveness parameter is often informed by studies with a limited duration of follow-up, requiring extrapolation of the treatment effect over a longer time horizon. Extrapolation from short-term data alone may not adequately capture uncertainty in that extrapolation. This study aimed to use structured expert elicitation to quantify uncertainty associated with extrapolation of the treatment effect observed in a clinical trial.

A structured expert elicitation exercise was conducted for an applied study of a podiatry intervention designed to reduce the rate of falls and fractures in the elderly. A bespoke web application was used to elicit experts’ beliefs about two outcomes (rate of falls and odds of fracture) as probability distributions (priors), for two treatment options (intervention and treatment as usual) at multiple time points. These priors were used to derive the temporal change in the treatment effect of the intervention, to extrapolate outcomes observed in a trial. The results were compared with extrapolation without experts’ priors.

The study recruited thirty-eight experts (geriatricians, general practitioners, physiotherapists, nurses, and academics) from England and Wales. The majority of experts (32/38) believed that the treatment effect would depreciate over time and expressed greater uncertainty than that extrapolated from a trial-based outcome alone. The between-expert variation in predicted outcomes was relatively small.

This study suggests that uncertainty in extrapolation can be informed using structured expert elicitation methods. Using structured elicitation to attach values to complex parameters requires key assumptions and simplifications to be considered.

Health technology assessment (HTA) agencies wish to ensure the impact of their HTAs. HTA impact assessment measures the influence of a HTA on decision-making and downstream to patient outcomes. Despite their potential to provide insights, the use of impact assessment frameworks by HTA agencies is limited. Understanding the underlying mechanisms of adopting HTA impact assessment frameworks is therefore important. Using a social cognitions lens, this study aims to provide insights into the enabling and hindering factors associated with the assessment of HTA impact by INAHTA members.

Using an interpretive description design, this cross-sectional study used semi-structured interviews of INAHTA members to gain insight into attitudes, social support, self-efficacy, barriers, and intentions towards HTA impact assessment. Transcriptions were analyzed using a social cognitions lens by two researchers using a constant comparative method to identify themes.

Twenty-six of forty-seven INAHTA members participated. Preliminary results showed that interviewees most often perceived support for assessing impact from their ministry of health or from agency staff. Most interviewees noted challenges to measuring impact at the right time and a lack of human resources, methods, and tools as internal barriers. A lack of transparency and a limited impact assessment culture were perceived as the main external barriers. Interviewees reported feeling fairly confident in overcoming internal barriers, but were less confident in overcoming external barriers. Providing feedback for improvement to HTA processes and making achievements visible were the most frequently reported advantages of assessing impact, whereas its time consuming nature was the biggest disadvantage.

This is the first study to use a social cognitions model to understand HTA impact assessment. Although the results of this convenience sample need to be interpreted with caution, they contribute knowledge on factors that facilitate and hinder agencies in the assessment of impact and illuminate opportunities for developing effective strategies to support HTA agencies in this area.

The fact that HTA is a value-laden process is recognized in the literature. This is one of the reasons for promoting a better integration of ethics in HTA processes. Although what is meant by value-judgment (VJ) and how it can be used in HTA is not clear for some authors; others have proposed the elicitation of implicit VJs, to make them more explicit, as one way for clarifying the role ethics may play in HTA. In order to clarify what a VJ is, a conceptual analysis is needed to distinguish it from a factual-judgment and see how they diverge on certain aspects and converge on others.

The distinction between VJs and factual-judgments was debated in the fifties. At the core of the philosophy of language was a distinction between factual-scientific assertions about facts, considered objective, and VJs on what is right/wrong-good/bad, considered subjective. In speech-act theory these distinctions were treated as two different operations: assertive and evaluative. A conceptual analysis of VJs, considering them as specific speech-acts, was used for clarifying/deciphering the role of VJs in HTA.

VJs are intrinsically embedded in decision-making since they are the reasons justifying decisions. This is why implicit VJs can be identified at every decision-step in the HTA process. Assessment is usually considered objective while appraisal seems subjective. Since VJs are entrenched in the decisions taken throughout the assessment process, the results are not completely objective. Ethical analysis also distinguishes two types of VJs, those based on normative criteria and those based on various degrees of value actualization. Furthermore, since evaluation requires criteria based on a rational process, VJs are not totally subjective.

Elicitation of VJs in HTA is one way of integrating ethics in HTA and offers decision-makers a more thorough picture of the ethical issues involved in their decision.

Differences between healthcare datasets in structure, content, and coding systems are widely recognized as significant barriers to generating robust evidence for regulatory and medical decision making. As a result, there is a growing interest in using common data models embedded within large data networks. By standardizing the structure, contents, and semantics of disparate healthcare databases, common data models like the Observational and Medical Outcomes Partnerships common data model (OMOP-CDM) enable multidatabase studies to be undertaken at speed and in a transparent way. To date, little attention has been given to their potential role in health technology assessment (HTA).

We identify the uses of observational data in generating evidence in HTA, some common analytical challenges faced in their estimation, and the infrastructural, technical, and data reusability constraints that limit its wider use. We discuss where and how the OMOP-CDM could overcome these barriers in relation to different types of evidence requirements.

The OMOP-CDM increases the interoperability of otherwise disparate datasets, allowing reliable evidence to be generated from multidatabase studies at speed and transparently. The current analytical tools are best suited for clinical characterization and population-level effect estimation. Further developments to these tools are required to support analyses common in HTA like parametric survival modeling. Differences in costing methods as well as the structure of healthcare delivery between countries may limit the feasibility and value of standardization.

The OMOP-CDM has the potential to support reliable and timely evidence generation in HTA. The analytical tools should be further developed to support common HTA use cases.