In low- and middle-income countries (LMICs) striving to achieve universal health coverage, the involvement of different stakeholders in formal or informal ways in health technology assessment (HTA) must be culturally and socially relevant and acceptable. Challenges may be different from those seen in high-income countries. In this article, we aimed to pilot a questionnaire for uncovering the context-related aspects of patient and citizen involvement (PCI) in LMICs, collecting experiences encountered with PCI, and identifying opportunities for patients and citizens toward contributing to local decision- and policy-making processes related to health technologies.

Through a collaborative, international multi-stakeholder initiative, a questionnaire was developed for describing each LMIC's healthcare system context and the emergence of opportunities for PCI relating to HTA. The questionnaire was piloted in the first set of countries (Brazil, Indonesia, Nigeria, and South Africa).

The questionnaire was successfully applied across four diverse LMICs, which are at different stages of using HTA to inform decision making. Only in Brazil, formal ways of PCI have been defined. In the other countries, there is informal influence that is contingent upon the engagement level of patient and citizen advocacy groups (PCAGs), usually strongest in areas such as HIV/AIDS, TB, oncology, or rare diseases.

The questionnaire can be used to analyze the options for patients and citizens to participate in HTA or healthcare decision making. It will be rolled out to more LMICs to describe the requirements and opportunities for PCI in the context of LMICs and to identify possible routes and methodologies for devising a more systematic and formalized PCI in LMICs.

Real-world evidence (RWE) is a useful supplement to a product's evidence base especially for medical devices, which are often unsuitable for randomized controlled trials. Generally, RWE is analyzed retrospectively (for example, healthcare records), which lack granularity for health-economic analysis. Prospective collection of RWE in hospitals can promote device-specific endpoint assessment. The advent of the General Data Protection Regulation (GDPR) requires a privacy-by-design approach. This work describes a workflow for a GDPR-compliant device-specific RWE collection as part of quality improvement initiatives (QII).

A literature review identifies relevant clinical and quality markers as endpoints to the investigated technology. A panel of experts grade these endpoints on their clinical significance, privacy sensitivity, analytic value, and feasibility for collection. Endpoints meeting a predefined cut-off are considered quality markers for the QII. Finally, an RWE data collection app is designed to collect the quality markers using either longitudinal, pseudonymized data or single time-point anonymized data to ensure data protection by design.

Using this approach relevant clinical markers were identified in a GDPR-compliant manner. The data collection app design ensured that patient data were protected, while maintaining minimum requirements on patient information and consent. The pilot QII collected data on over 5,000 procedures, which represents the largest single data set available for the tested technology. Due to its prospective nature this programme was the first to collect patient outcomes in sufficient quantity for analysis, while previous studies only recorded adverse events.

GDPR and RWE can co-exist in harmony. A design approach, which has data protection in mind from the start can combine high quality RWE collection of efficacy and safety data with maximum patient privacy.

To classify, analyze, and compare published guidelines for economic evaluation within health technology assessment (HTA) in European countries and highlight differences and similarities.

We performed a literature review to identify published guidance for the conduct and assessment of economic evaluation studies that are undertaken within the context of HTA processes in European countries. Organizations and working groups were identified via the ISPOR, INAHTA, and EUnetHTA databases. Following the identification of official documents, we performed a qualitative content analysis to highlight discrepancies or common practices under the following categories: comparator, perspective on costs/benefits, time horizon, economic evaluation method, instrument used for utility measurement, outcome measure, source for efficacy, modeling, sensitivity analysis, discounting, and incremental cost-effectiveness ratio.

A total of nineteen guidance documents were identified (in English) providing data for the analysis in nineteen countries. The comparative content analysis identified common practices in most countries regarding the approaches to the choice of comparator, source of data, the preferred economic evaluation method, the option for a lifetime analytical horizon, discounting, and the choice of key outcome measure—for which, most countries recommend the use of the EQ-5D instrument. Differences were mainly found in the choice of perspective, dealing with uncertainty and sensitivity analysis, the use of end points, and the required use of modeling.

The use of economic evaluation constitutes one of the key pillars of the HTA process in Europe. Although a methodological convergence has occurred during the last few years, notable differences still remain.

This paper aims to describe the development of a flowchart to guide the decisions of researchers in the Spanish Network for Health Technology Assessment of the National Health System (RedETS) regarding patient involvement (PI) in Health Technology Assessment (HTA). By doing so, it reflects on current methodological challenges in PI in the HTA field: how best to combine PI methods and what is the role of patient-based evidence.

A decisional flowchart for PI in HTA was developed between March and April 2019 following an iterative process, reviewed by the members of the PI Interest Group and other RedETS members and validated during an online deliberative meeting. The development of the flowchart was based on a previous methodological framework assessed in a pilot study.

The guidelines on how to involve patients in HTA in the RedETS were graphically represented in a flowchart. PI must be included in all HTA reports, except those that assess technologies with no relevant impact on patients’ experiences, values, and preferences. Patient organizations or expert patients related to the topic of the HTA report must be identified and invited. These patients can participate in protocol development, outcomes' identification, assessment process, and report review. When the technology assessed affects in a relevant way patient experiences, values, and preferences, patient-based evidence should be included through a systematic literature review or a primary study.

The decisional flowchart for PI in HTA contributes to the current methodological challenges by proposing a combination of direct involvement and patient-based evidence.

The use of surrogate outcomes in health technology assessment (HTA) is increasing and methods for validating surrogate relationships have been published. However, these may not be fully applicable to ultra-rare diseases due to challenges such as scarcity of evidence and heterogenous populations. This study reviews and summarizes the use of surrogate outcomes and committee's considerations in the evaluations within the National Institute for Health and Care Excellence's (NICE) Highly Specialised Technology (HST) programme, which was established in 2013 in response to the challenges associated with the assessment of ultra-rare diseases.

All HST evaluation documents published before November 2020 were reviewed. Data extracted included surrogate outcomes used, rationales, the committee's considerations on the validity and generalizability of the surrogate relationships, related uncertainties, and other factors considered in decision-making.

Seven out of the eighteen published HST topics used surrogate outcomes. The rationale for most of the surrogate relationships focused on biological plausibility. Common concerns raised by the committee included the generalizability of the surrogate relationship to the condition of interest, the lack of validation, and inability to prove or quantify the magnitude of benefits associated with the surrogate relationships. In some topics, other aspects of the evidence and clinical/patient expert's opinions were also considered by the committee.

The use of surrogate outcomes is common in NICE HST evaluations and the challenges in supporting surrogate relationships with more than biological plausibility are recognized. However, our review indicates that, the committee considers more than just biological plausibility and will take into account other related factors.

Literature searching for evidence on apps in bibliographic databases is challenging because they are often described with inconsistent terminology. Information Specialists from the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed validated search filters for retrieving evidence about apps from MEDLINE and Embase (Ovid) reliably.

Medical informatics journals were hand-searched to create a ‘gold standard’ set of app references. The gold standard set was divided into two sets. The development set provided the search terms for the filters. The filters were validated by calculating their recall against the validation set. Target recall was >90%.

A case study was then conducted to compare the number-needed-to-read (NNR) of the filters with previous non-validated MEDLINE and Embase app search strategies used for the ‘MIB214 myCOPD app’ NICE topic. NNR is the number of references screened to find each relevant reference.

The MEDLINE and Embase filters achieved 98.6 percent and 98.5 percent recall against the validation set, respectively. In the case study they achieved 100 percent recall, reducing NNR from 348 to 147 in MEDLINE and from 456 to 271 in Embase.

The novel NICE health apps search filters retrieve evidence on apps from MEDLINE and Embase effectively and more efficiently than previous non-validated search strategies used at NICE.

Despite a large number of mobile apps in the field of mental health, it is difficult to find a useful and reliable one, mainly due to the fact that the effectiveness of many apps has not been assessed scientifically. The present study aimed to assess the effects of mental health apps on managing the symptoms of stress, anxiety, and depression.

A comprehensive literature search was conducted in PubMed, Scopus, EMBASE, Cochrane, and Web of Science databases for the papers published from 2000 to 2019. Studies were included if they reviewed articles or mobile apps for their effectiveness in stress, anxiety, and depression. The reviews that had considered mobile apps or web-based mobile applications as an intervention or part of intervention were included, as well.

A total of 4,999 peer-reviewed articles were identified, out of which nine systematic reviews met the inclusion criteria. Seven systematic reviews measured depression outcomes, three measured stress, and five systematic reviews measured anxiety symptoms. The applications that used behavior change strategies, such as Cognitive Behavioral Therapy, Acceptance and Commitment Therapy, and Behavioral Activation, reported significant effects on depression, anxiety, and stress.

It seems that mental health apps can be promising media for reducing depressive symptoms. This field is an emerging area of mobile health, and further research should be done in future in order to reach conclusive evidence.

Human screening of title and abstracts in a systematic literature review (SLR) is labor intensive and time-consuming. In many instances, thousands of citations may be retrieved; the vast majority excluded upon screening. Text-mining semi-automates and accelerates screening by identifying patterns in relevant and irrelevant citations, as labelled by the screener. One such text-mining tool, Abstrackr, uses an algorithm within an active-learning framework to predict the likelihood of citations being relevant. The objective of this study was to assesses the performance of Abstrackr for title and abstract screening in an SLR of treatments for relapsed/refractory diffuse large B-cell lymphoma.

Citations identified from searches of electronic databases were imported to Abstrackr. An investigator-selected database of terms indicating relevance of title and abstract to the research question were uploaded. These terms were partly informed by the SLR inclusion/exclusion criteria. Citations deemed most relevant by Abstrackr were screened first (screening prioritization). Screening was carried out until a maximum prediction score of 0.4 or less, based on previous experience in the literature, was reached. Remaining citations were deemed unlikely to be relevant and did not undergo screening (screening truncation). Separately, a single-human screener screened all citations using Covidence.

A total of 7,723 citations and 154 initial terms were uploaded to Abstrackr. Of these citations, 2,572 (33 percent) were screened before a prediction score of 0.39 was reached. Compared to single-human screening (conducted on all citations), the workload saving associated with Abstrackr was 5 days. A total of 451 (6 percent) citations proceeded to full-text screening; ten (0.1 percent) were included in the final evidence base. No citations predicted to be irrelevant by Abstrackr were included in the final evidence base.

Text-mining tools such as Abstrackr have the potential to reduce workload associated with title and abstract screening, without missing relevant citations.

Increasing emphasis is given on involving patients in health technology assessment (HTA). While this is mainly done at the level of regional and national HTA agencies, this tendency is also emerging in local HTA units. In this study, we provide the results of a survey conducted in local HTA units in the province of Quebec, Canada. The aim of the survey was to provide an overview of local HTA unit practices to involve patients, users, caregivers, and citizens in their process, their interest in doing so, and their information needs for this.

The survey was conducted in 2017 with a response rate of eleven units over a possibility of twelve.

Three units out of eleven (27.3 percent) never involved patients or members of the public in their processes and all indicated that they will involve them in the next few years. The three most important needs for support identified in the HTA units were in: recruiting and selecting patients; integrating experiential knowledge; and knowing and implementing the best methods and practices for partnership.

Patient involvement in local HTA units is quickly evolving and that is why they urgently need tools to involve more effectively patients and members of the public in their process.

Drugs for relapsing-remitting multiple sclerosis (RRMS) are costly and not included in the National List of Essential Medicines of Thailand yet. This study aims to conduct an economic evaluation of high-cost drugs for RRMS.

The Markov model was used to estimate lifetime costs and quality-adjusted life years (QALYs) gained. The treatment options include Interferon beta-1a (IFN) and Teriflunomide (TERI) (first-line), Fingolimod (FIN) and Natalizumab (NATA) (second-line), and Alemtuzumab (ALEM) (third-line) compared with usual care. The effectiveness of drugs was retrieved by network meta-analysis. The probability of health state transition was obtained from primary data. Treatment-related costs were derived from the national database. Other costs and utilities were obtained from the study in Thai RRMS patients.

The lowest lifetime costs option was usual care (THB2 million) (USD65,808), while the highest QALY gained option was IFN-NATA-ALEM (8.6 QALY gained). All treatment options were not cost-effective compared with usual care at the threshold of THB160,000 (USD5,300) per QALY gained. However, the option of IFN-NATA-ALEM yielded the lowest incremental cost-effectiveness ratio (ICER), which was THB4.4 million (USD144,778) per QALY gained.

High-cost drugs were not cost-effective; nonetheless, the IFN-NATA-ALEM option could increase QALY gained with the lowest additional budget. The government should negotiate the price of IFN to decrease by eighty percent.

Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies.

Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions.

Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases.

Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective.

The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

Extrapolation methods are commonly used to model the cost-effectiveness of health technologies beyond observed data. Reassessing cost-effectiveness estimates using updated clinical trial data has the potential to reduce uncertainty and optimize decision-making. We present a case study based on percutaneous repair (PR) with the Mitraclip system, a technology to treat severe secondary mitral regurgitation (MR). For the study purpose, we considered the COAPT trial that evaluated the effectiveness of adding PR to medical treatment versus medical treatment alone.

We developed a time-varying Markov model to assess the cost-effectiveness of PR. Clinical inputs were based on reconstructed individual patient data from the COAPT trial results reported at 2 years, and at 3 years.

We developed parametric modeling for overall survival (OS) and heart failure hospitalizations (HFH) to obtain clinically plausible extrapolations beyond observed data. We adopted the French perspective and used a 30-year time horizon. We expressed incremental cost-effectiveness ratios (ICERs) as cost per quality-adjusted life year (QALY).

Based on 2 year-data, preferred parametric models for OS and HFH were exponential and log-logistic respectively, yielding an ICER of EUR21,918/QALY and >0.5 probability of PR being cost-effective (EUR50,000/QALY threshold).

Updated analyses at 3 years showed a change of OS trajectory for PR that justified the use of piecewise modelling, yielding an updated ICER that went up to EUR77,904/QALY (base-case), and to a minimum of EUR58,175/QALY (scenario analysis). Using data at 3 years, PR had <0.5 probability of being cost-effective.

In this case study, the availability of updated survival analyses of the main trial is likely to have some impact on decision-making and/or pricing discussion as part of health-technology assessment (HTA). We aim to provide further updated analyses as 4 years results of the COAPT study become available.

More broadly, original technology appraisals are frequently undertaken when mid/long-term follow-up trial data may be lacking. Our example suggests the need for continuous HTA review as new clinical data are released.

The idea of using extracorporeal cytokine adsorption therapy (ECAT) is to remove cytokines from the blood in order to restore a balanced immune response. Yet, it is unclear as to whether the use of ECAT improves patient-relevant outcomes. Hence, the aim of this article is to synthesize the currently available evidence with regard to a potential clinical benefit of ECAT used in cardiac surgery or sepsis.

We conducted an updated systematic review summarizing the body of evidence with regard to a potential clinical benefit of ECAT. The study followed the PRISMA statement and the European Network for Health Technology Assessment (EUnetHTA) guidelines. The quality of the individual studies and the strength of the available evidence was assessed using the Cochrane risk of bias tool (v.1) and the GRADE approach respectively. Mortality, organ function, length of stay in the intensive care unit and length of hospitalization, as well as adverse events, were defined as critical outcomes.

For the preventive treatment of ECAT in patients undergoing cardiac surgery, we found very low-quality inconclusive evidence for mortality (5 randomized controlled trials (RCTs), n = 163), length of stay in the intensive care unit (5 RCTs, n = 163), and length of hospitalization (3 RCTs, n = 101). In addition, very low-quality inconclusive evidence was found for (serious) adverse events (4 RCTs, n = 148). For the therapeutic treatment of ECAT in patients with sepsis/ septic shock, we found very low-quality inconclusive evidence for mortality up to 60-day follow-up (2 RCTs, n = 117), organ function (2 RCTs, n = 117) and length of stay in the intensive care unit (1 study, n = 20). Similarly, very low-quality inconclusive evidence was found for (serious) adverse events (2 RCTs, n = 117). There are currently eighteen ongoing RCTs on the use of ECAT.

There is a lack of reliable data on the clinical benefit of using ECAT as an add-on treatment preventively in cardiac surgery and therapeutically in patients with sepsis or septic shock. While theoretical advantages are anticipated, the current available evidence is inconclusive and was not able to establish the efficacy and safety of ECAT in combination with standard care in the investigated indications. In light of the available RCTs, we strongly recommend the consideration of studies with patient-relevant endpoints and adequate statistical power, instead of investing further research funds on small studies that may not shed more light onto the potential clinical benefit of ECAT. The results of ongoing RCTs are awaited to guide the decision on whether further research funds should be invested in ECAT research or to conclude that the intervention may not show clinical benefits for patients.

Integration of ethics into technology assessment in healthcare (HTA) reports is directly linked to the need of decision makers to provide rational grounds justifying their social choices. In a decision-making paradigm, facts and values are intertwined and the social role of HTA reports is to provide relevant information to decision makers. Since 2003, numerous surveys and discussions have addressed different aspects of the integration of ethics into HTA. This study aims to clarify how HTA professionals consider the integration of ethics into HTA, so an international survey was conducted in 2018 and the results are reported here.

A survey comprising twenty-two questions was designed and carried out from April 2018 to July 2018. Three hundred and twenty-eight HTA agencies from seventy-five countries were invited to participate in this survey.

Eighty-nine participants completed the survey, representing a participation rate of twenty-seven percent. As to how HTA reports should fulfill their social role, over 84 percent of respondents agreed upon the necessity to address this role for decision makers, patients, and citizens. At a lower level, the same was found regarding the necessity to make value-judgments explicit in different report sections, including ethical analysis. This contrasts with the response-variability obtained on the status of ethical analysis with the exception of the expertise required. Variability in stakeholder-participation usefulness was also observed.

This study reveals the importance of a three-phase approach, including assessment, contextual data, and recommendations, and highlights the necessity to make explicit value-judgments and have a systematic ethical analysis in order to fulfill HTA's social role in guiding decision makers.

The sequential use of alternative treatments for chronic conditions represents a complex, dynamic intervention pathway; previous treatment and patient characteristics affect both choice and effectiveness of subsequent treatments. Evidence synthesis methods that produce the least biased estimates of treatment-sequencing effects are required to inform reliable clinical and policy decision-making. A comprehensive review was conducted to establish what existing methods are available, outline the assumptions they make, and identify their shortcomings.

The review encompassed both meta-analytic techniques and decision-analytic modelling, any disease condition, and any type of treatment sequence, but not diagnostic tests, screening, or treatment monitoring. It focused on the estimation of clinical effectiveness and did not consider the impact of treatment sequencing on the estimation of costs or utility values.

The review included ninety-one studies. Treatment-sequencing is usually dealt with at the decision-modelling stage and is rarely addressed using evidence synthesis methodology for clinical effectiveness. Most meta-analyses are of discrete treatments, sometimes stratified by line of therapy. Prospective sequencing trials are scarce. In their absence, there is no single best way to evaluate treatment sequences, rather there is a range of approaches, each of which has advantages and disadvantages and is influenced by the evidence available and the decision problem. Due to the scarcity of data on sequential treatments, modelling studies generally apply simplifying assumptions to data on discrete treatments. A taxonomy for all possible assumptions was developed, providing a unique resource to aid the critique of decision-analytic models.

The evolution of network meta-analysis in HTA demonstrates that clinical and policy decision-making should account for the multiple treatments available for many chronic conditions. However, treatment-sequencing has yet to be accounted for within clinical evaluations. Economic modelling is often based on the simplifying assumption of treatment independence. This can lead to misrepresentation of the true level of uncertainty, potential bias in estimating the effectiveness and cost effectiveness of treatments and, eventually, the wrong decision.

Health technology assessment (HTA) can play a key role in evidence-based decision-making. However, HTA requires resources that might be lacking in low-income settings. To test the feasibility of adapting existing evidence as part of the HTA process, this project evaluated the effectiveness and economic impact of breast cancer screening programs for women over 40 years in the West Bank, where mammography screening is provided for free in governmental clinics.

We conducted a search for systematic reviews, HTAs and guidelines in electronic databases. We included the most recent global systematic review and meta-analysis that fulfilled our inclusion criteria. The European Network for Health Technology Assessment (EUnetHTA) adaptation toolkit was used to guide adaptation and undertake a budget impact analysis of the economic impact of mammography screening. We build capacity by working as a team of HTA experts and first-time HTA researchers. The results were disseminated to raise awareness for HTA.

The European Commission Guidelines on Breast Cancer Screening were identified as most recent global systematic review with meta-analyses, out of 2,365 references. The adapted evidence may inform policies on screening in the West Bank. Our experience is that adaption requires extensive skills and resources, including finding, assessing, and adapting relevant evidence. The EUnetHTA toolkit is useful, but also adds to the workload. Furthermore, local stakeholder engagement is important in topic selection, to access information, and to contextualize global evidence to the local setting.

This study is currently ongoing, but preliminary findings show that producing an HTA by adapting existing evidence in resource-limited settings is feasible. There is a need for nuanced guidance on transferability of evidence from other settings. Future studies should investigate innovative methods to optimize the adaption process. Capacity building in adaptation is important to ensure the production of quality HTA products. Inclusion of local team members and stakeholders is important for future development of HTA in the region.