The Ministry of Health in Québec requested the National Institute of Excellence in Health and Social Services to produce clinical and implementation recommendations for the prophylaxis, diagnosis, and treatment of Lyme disease.

(i) Describe the process of trialing different modalities of patient engagement as a means to integrate a diversity of patient perspectives and (ii) Describe the learning process of INESSS regarding the integration of the patient perspective.

All documents were analyzed, and a survey with all advisory committee members and semi-structured interviews with stakeholders were conducted. Each interview was transcribed verbatim and imported into QDA miner software for the purposes of analysis. Data analysis was carried out concurrently with data collection to allow for an iterative approach between data collection and analysis.

Five methods to integrate the perspectives of patients were used: (i) interviews with patients, (ii) inclusion of patient partners within the advisory committee, (iii) literature review, (iv) focus groups with one patient association, and (v) feedback from patient associations on recommendations intended for decision makers and other targeted stakeholders. The patient partners influenced decisions by sharing their experiential knowledge. The patient interviews and the literature review added an in-depth perspective on the disease and experience with the healthcare system. The patient association members shared their perspectives and helped disseminate the recommendation to sustain a practice change.

The combination of methods to collect and integrate patients’ knowledge and patient associations’ perspectives helped develop a comprehensive understanding of a controversial object of evaluation.

A national act (Order SSI/1356/2015) regulating Post-Launch Evidence Generation (PLEG) studies was set in Spain in 2015. These PLEG studies are to inform decisions about technologies already included in the Benefit Portfolio of the Spanish National Health System (SNHS) in order to confirm/exclude/modify their terms of use. Once a PLEG is established the selected hospitals provide the technology according to a common protocol and register outcomes until the required sample size is reached.

The PLEG studies are prospective, observational and single arm studies on safety, effectiveness and cost-effectiveness of a technology in real practice. The technology is selected because of the identification of an evidence gap, usually through a health technology assessment (HTA) report made by an agency of the Spanish Network of HTA Agencies (RedETS). The execution of a PLEG is assigned to one of the RedETS Agencies, which is responsible of delivering annual reports and a final report when the objectives are reached.

The following six PLEG studies, all of them on medical devices, have been launched in Spain so far, i) Endobronchial valve for patients with persistent air leak; ii) Biodegradable esophageal stent; iii) Percutaneous mitral valve repair system by clip; iv) Left Atrial Appendage Closure Device; v) Sensor-based glucose monitoring systems for children with type 1 diabetes mellitus; vi) Left ventricular assist devices for destination therapy. Five studies will finish their data collection by the end of 2020 or during 2021.

A new national procedure using PLEG has been made available in Spain facilitating the use of real-world evidence to inform national decision-making on the financing of selected technologies due to uncertainties about their effectiveness, safety, cost-effectiveness and organizational impact. The studies are requiring a high amount of coordination tasks, as they are involving an average of 21 hospitals each. The usefulness and suitability of this procedure to achieve its objectives must be evaluated once their results are available.

Clinical decision support systems (CDSS) are being developed to support evidence-based antibiotic prescribing and reduce the risk of inappropriate or over-prescribing; however, adoption of CDSS into the health system is rarely sustained. We aimed to understand the implementation challenges at a macro (policymakers), meso (organizational) and micro-level (individual practices) to identify the drivers of CDSS non-adoption.

We have adopted a mixed-method study design which comprised of: (i) systematic review and meta-analysis to assess the impact of CDSS on appropriate antibiotic prescribing, (ii) Online survey of clinicians in Australia from hospitals and primary care to identify drivers of CDSS adoption and (iii) in-depth interviews with policymakers to evaluate policy-level challenges and opportunities to CDSS implementation.

CDSS implementation can improve compliance with antibiotic prescribing guidelines, with a relative decrease in mortality, volume of antibiotic use and length of hospital stay. However, CDSS provision alone is not enough to achieve these benefits. Important predictors of clinicians’ perception regarding CDSS adoption include the seniority of clinical end-users (years), use of CDSS, and the care setting. Clinicians in primary care and those with significant clinical experience are less likely to use CDSS due to a lack of trust in the system, fear of comprising professional autonomy, and patients’ expectations. Lack of important policy considerations for CDSS integration into a multi-stakeholder healthcare system has limited the organizational capacity to foster change and align processes to support the innovation.

These results using multiple lines of evidence highlight the importance of a holistic approach when undertaking health technology management. There needs to be system-wide guidance that integrates individual, organizational and system-level factors when implementing CDSS so that effective antibiotic stewardship can be facilitated.

The Patient-Centered Outcomes Research Institute (PCORI) horizon scanning system is an early warning system for healthcare interventions in development that could disrupt standard care. We report preliminary findings from the patient engagement process.

The system involves broadly scanning many resources to identify and monitor interventions up to 3 years before anticipated entry into U.S. health care. Topic profiles are written on included interventions with late-phase trial data and circulated with a structured review form for stakeholder comment to determine disruption potential. Stakeholders include patients and caregivers recruited from credible community sources. They view an orientation video, comment on topic profiles, and take a survey about their experience.

As of March 2020, 312 monitored topics (some of which were archived) were derived from 3,500 information leads; 121 met the criteria for topic profile development and stakeholder comment. We invited fifty-four patients and caregivers to participate; thirty-nine reviewed at least one report. Their perspectives informed analyst nominations for fourteen topics in two 2019 High Potential Disruption Reports. Thirty-four patient stakeholders completed the user-experience survey. Most agreed (68 percent) or somewhat agreed (26 percent) that they were confident they could provide useful comments. Ninety-four percent would recommend others to participate.

The system has successfully engaged patients and caregivers, who contributed unique and important perspectives that informed the selection of topics deemed to have high potential to disrupt clinical care. Most participants would recommend others to participate in this process. More research is needed to inform optimal patient and caregiver stakeholder recruitment and engagement methods and reduce barriers to participation.

The clinical data is increasing at a considerably higher rate than the capacity of the healthcare system and clinicians to manage this data. Digital tools such as clinical decision support systems (CDSS) provide opportunities for evidence-based patient care by intelligently filtering and presenting the information required for clinical decision making at the point of care. Despite the success of pilot projects, CDSS have had limited implementation in broader health systems. We aimed to identify challenges faced by policymakers for CDSS implementation and to provide policy recommendations.

We conducted eleven semi-structured interviews with Australian policymakers from state and national committees involved in digital health activities. The data were analyzed using reflexive thematic analysis to identify policy priorities.

Our findings indicate that fragmentation of care processes and structures in the digital health ecosystem is one of the main impediments to delivering coordinated care using CDSS. Five themes for policy action were identified: (i) establishing a shared conceptual framework for user-centered design of CDSS that is aligned with stakeholders’ priorities, (ii) maintaining the right balance between the customization and standardization of systems, (iii) developing mutually agreed semantic interoperability standards at the local, state and national level, allowing generation and exchange of information across the health system without changing its context and meaning, (iv) reorienting organizational structures to build capacity to foster change, and (v) developing collaborative care models to avoid conflicting interests between stakeholders.

Findings highlight the importance of developing system-wide guidance to establish a clear vision for CDSS implementation and alignment of organizational processes across all levels of health care. There is a need to build a shared policy framework for modelling the innovative activities such as CDSS implementation across the digital health landscape which minimizes the operational and strategic fragmentation of different organizations.

The German health technology assessment (HTA) rejected additional benefit of alectinib for second line (2L) ALK+ NSCLC, citing possible biases from missing ECOG performance status data and unmeasured confounding in real-world evidence (RWE) for 2L ceritinib that was submitted as a comparator to the single arm alectinib trial. Alectinib was approved in the US and therefore US post-launch RWE can be used to evaluate this HTA decision.

We compared the real-world effectiveness of alectinib with ceritinib in 2L post-crizotinib ALK+ NSCLC using the nationwide Flatiron Health electronic health record (EHR)-derived de-identified database. Using quantitative bias analysis (QBA), we estimated the strength of (i) unmeasured confounding and (ii) deviation from missing-at-random (MAR) assumptions needed to nullify any overall survival (OS) benefit.

Alectinib had significantly longer median OS than ceritinib in complete case analysis. The estimated effect size (Hazard Ratio: 0.55) was robust to risk ratios of unmeasured confounder-outcome and confounder-exposure associations of <2.4.

Based on tipping point analysis, missing baseline ECOG performance status for ceritinib-treated patients (49% missing) would need to be more than 3.4-times worse than expected under MAR to nullify the OS benefit observed for alectinib.

Only implausible levels of bias reversed our conclusions. These methods could provide a framework to explore uncertainty and aid decision-making for HTAs to enable patient access to innovative therapies.

The COVID-19 pandemic led to a significant surge in clinical research activities in the search for effective and safe treatments. Attempting to disseminate early findings from clinical trials in a bid to accelerate patient access to promising treatments, a rise in the use of preprint repositories was observed. In the UK, NIHR Innovation Observatory (NIHRIO) provided primary horizon-scanning intelligence on global trials to a multi-agency initiative on COVID-19 therapeutics. This intelligence included signals from preliminary results to support the selection, prioritisation and access to promising medicines.

A semi-automated text mining tool in Python3 used trial IDs (identifiers) of ongoing and completed studies selected from major clinical trial registries according to pre-determined criteria. Two sources, BioRxiv and MedRxiv are searched using the IDs as search criteria. Weekly, the tool automatically searches, de-duplicates, excludes reviews, and extracts title, authors, publication date, URL and DOI. The output produced is verified by two reviewers that manually screen and exclude studies that do not report results.

A total of 36,771 publications were uploaded to BioRxiv and MedRxiv between March 3 and November 9 2020. Approximately 20–30 COVID-19 preprints per week were pre-selected by the tool. After manual screening and selection, a total of 123 preprints reporting clinical trial preliminary results were included. Additionally, 50 preprints that presented results of other study types on new vaccines and repurposed medicines for COVID-19 were also reported.

Using text mining for identification of clinical trial preliminary results proved an efficient approach to deal with the great volume of information. Semi-automation of searching increased efficiency allowing the reviewers to focus on relevant papers. More consistency in reporting of trial IDs would support automation. A comparison of accuracy of the tool on screening titles/abstract or full papers may help to support further refinement and increase efficiency gains.

This project is funded by the NIHR [(HSRIC-2016-10009)/Innovation Observatory]. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Health apps are software programs that are designed to prevent, diagnose, monitor, or manage conditions. Inconsistent terminology for apps is used in research literature and bibliographic database subject headings. It can therefore be challenging to retrieve evidence about them in literature searches. Information specialists at the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed novel validated search filters to retrieve evidence about apps from MEDLINE and Embase (Ovid).

A selection of medical informatics journals was hand searched to identify a “gold standard” (GS) set of references about apps. The GS set was divided into a development and validation set. The filters’ search terms were derived from and tested against the development set. An external development set containing app references from published NICE products was also used to inform the development of the filters. The filters were then validated using the validation set. Target recall was >90 percent. The filters’ overall recall, specificity, and precision were calculated using all the references identified from the hand search.

Both filters achieved 98.6 percent recall against their validation sets. Overall, the MEDLINE filter had 98.8 percent recall, 71.3 percent specificity, and 22.6 percent precision. The Embase filter had 98.6 percent recall, 74.9 percent specificity, and 24.5 percent precision.

The NICE health apps search filters retrieve evidence about apps from MEDLINE and Embase with high recall. They can be applied to literature searches to retrieve evidence about the interventions by information professionals, researchers, and clinicians.

Physician preference items (PPIs) are high-cost medical devices on which clinicians express firm preferences with respect to a particular manufacturer and a specific product. The aim of this research is to understand what are the most important factors, as well as their relative importance, in the choice of new PPIs (that is, hip or knee prosthesis) adoption on behalf of orthopaedic clinicians in Italy.

Based on a literature review and clinical experts’ opinions, we identified a number of key factors (for example, health technology assessment (HTA) recommendation) and their corresponding levels (for example positive HTA recommendation). We administered an online survey to hospital orthopaedists using two experimental techniques for preference elicitation (that is, discrete choice experiment (DCE) and case 1 best-worst scaling (BWS)). BWS data were analysed through descriptive statistics (that is, best-minus-worst score) and conditional logit model. A mixed logit model was applied to DCE data, and a willingness-to-pay (WTP) was estimated. All analyses were conducted using Stata 16.

A total of ninety orthopaedists (95% male; mean age: 52.8 years) were enrolled in the survey. In BWS, the most important factor was ‘clinical evidence’, followed by ‘quality of products’, ‘HTA recommendations’ and ‘previous experience’, while the least important was ‘cost’. DCE results suggested that orthopaedists prefer high-quality products with robust clinical evidence, positive HTA recommendation and affordable cost, and for which clinicians have a consolidated experience of use and a good relationship with the sales representative. The WTP for a high-quality product was estimated at EUR1,733, and for a good relationship at EUR2,843.

This is the first study aimed at analysing the multidimensionality of clinician's decision-making process in selecting new PPIs in orthopaedics in Italy. Despite the quality of products being declared as one of the most important dimensions in BWS, when other factors populate a hypothetical DCE scenario, physicians are not willing to accept quality at any cost (for example, high quality and very bad support from the producer or with uncertain clinical evidence).

Health technology assessment (HTA) can impact health inequities by informing healthcare priority-setting decisions. This paper presents a novel checklist to guide HTA practitioners looking to include equity considerations in their work: the equity checklist for HTA (ECHTA). The list is pragmatically organized according to the generic HTA phases and can be consulted at each step.

A first set of items was based on the framework for equity in HTA developed by Culyer and Bombard. After rewording and reorganizing according to five HTA phases, they were complemented by elements emerging from a literature search. Consultations with method experts, decision makers, and stakeholders further refined the items. Further feedback was sought during a presentation of the tool at an international HTA conference. Lastly, the checklist was piloted through all five stages of an HTA.

ECHTA proposes elements to be considered at each one of the five HTA phases: Scoping, Evaluation, Recommendations and Conclusions, Knowledge Translation and Implementation, and Reassessment. More than a simple checklist, the tool provides details and examples that guide the evaluators through an analysis in each phase. A pilot test is also presented, which demonstrates the ECHTA's usability and added value.

ECHTA provides guidance for HTA evaluators wishing to ensure that their conclusions do not contribute to inequalities in health. Several points to build upon the current checklist will be addressed by a working group of experts, and further feedback is welcome from evaluators who have used the tool.

Early identification of innovative medicines is crucial for timely health technology assessment (HTA) and efficient patient access. The National Institute for Health Research Innovation Observatory (NIHRIO) identifies, monitors and notifies key HTA stakeholders in England of ‘technologies’ (innovative medicines) within three to five years of regulatory approval. Increasing numbers of innovative medicines and significant uncertainties in clinical and regulatory pathways are major challenges in the monitoring and notification process. An active monitoring framework using pre-defined predictive criteria has previously been developed. This framework provides a standardized and consistent process, but is highly resource-intensive, requiring manual review of individual records.

Using the previous active monitoring framework, a scoring matrix was calculated and used to prioritize individual technologies using available data in the NIHRIO database: estimated regulatory timelines, regulatory awards/designations, innovative medicine type (for example gene therapies) and clinical trial phase, completion dates and results. A threshold for automatic and manual reviewing of technologies was developed and tested by NIHRIO analysts.

The scoring system identified approximately ninety percent of technologies meeting the threshold for semi-automated reviewing. The review period for these technologies are set automatically according to predefined criteria depending on data availability. The review periods are updated automatically until the record reaches the threshold that triggers manual reviewing. The remaining ten percent had estimated regulatory timelines necessitating the need for manual reviewing and early engagement with companies to verify regulatory timelines and/or notify HTA stakeholders.

Preliminary analysis indicates that each technology is routinely and automatically updated. The semi-automatic updating represents a significant improvement in the efficiency of the monitoring of the large volume of technologies on the NIHRIO database. Ongoing work is being undertaken to further refine, pilot and test the system.

This project is funded by the NIHR [(HSRIC-2016-10009)/Innovation Observatory]. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

There is increasing pressure to rapidly shape policies and inform decision-making where robust evidence is lacking. This work aimed to explore the value of soft-intelligence as a novel source of evidence. We deployed an artificial intelligence based natural language platform to identify and analyze a large collection of UK tweets relating to mental health during the COVID-19 pandemic.

A search strategy comprising a list of terms relating to mental health, COVID-19 and the lockdown was developed to prospectively identify relevant tweets via Twitter's advanced search application programming interface. We used a specialist text analytics platform to explore tweet frequency and sentiment across the UK and identify key topics of discussion for qualitative analysis. All collated tweets were anonymized.

We identified 380,728 tweets from 184,289 unique users in the UK from 30 April to 4 July 2020. The average sentiment score was fifty-two percent, suggesting overall positive sentiment. Tweets around mental health were polarizing, discussed with both positive and negative sentiment. For example, some people described how they were using the lockdown as a positive opportunity to work on their mental health, sharing helpful strategies to support others. However, many people expressed the damaging impact the pandemic (and resulting lockdown) was having on their mental health, including worsening anxiety, stress, depression, and loneliness.

The results suggest that soft-intelligence is potentially a useful source of evidence. The approach taken to identify and analyze this data may offer an efficient means of establishing key insights from the ‘public voice’ relating to critical health issues. However, there are still various limitations to consider concerning the technology and representativeness of the data. Future work to explore this type of evidence further, and how it might formally support decision-making processes, is recommended.

This project is funded by the NIHR [(HSRIC-2016-10009)/Innovation Observatory]. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

To evaluate the methodological and reporting characteristics of search methods of systematic reviews (SRs) using the Preferred Reporting of Systematic Reviews and Meta-Analyses (PRISMA) checklist and the Risk Of Bias In Systematic reviews (ROBIS) tool.

A sample of 505 SRs published in 2016 was taken from KSR Evidence, a database of SRs, and analyzed to assess compliance with Information sources and Search of the PRISMA checklist. Domain 2 (D2) (Identification and Selection of Studies) of the ROBIS tool was used to judge the risk of bias in search methods.

Regarding Information sources and Search of PRISMA, twenty percent of SRs which claimed to be PRISMA-compliant in their methods, were compliant; twenty-four percent of SRs published in journals that require PRISMA reporting were compliant; nineteen percent in total were found to be compliant. Twenty-eight percent of SRs were judged to be at a low risk of bias in D2 and so searched widely with an effective strategy and, finally, ten percent were both compliant with the reporting of Information sources and with Search of PRISMA and were judged to be at a low risk of bias in D2.

Ninety percent of SRs are failing to report search methods adequately and to conduct comprehensive searches using a wide range of resources. Editors of journals and peer reviewers need to ensure that they understand the requirements of PRISMA and that compliance is adhered to. Additionally, the comprehensiveness of search methods for SRs needs to be given more critical consideration.

Synthesis of clinical effectiveness is a well-established component of health technology assessment (HTA) combining data from multiple trials to obtain an overall pooled estimate of clinical effectiveness, which may inform an associated economic evaluation. Time-to-event outcomes are often synthesized using effect measures from Cox proportional hazards models assuming a constant hazard ratio over time. However, where treatment effects vary over time an assumption of proportional hazards is not always valid. Several methods have been proposed for synthesizing time-to-event outcomes in the presence of non-proportional hazards. However, guidance on choosing between these methods and the implications for HTA is lacking.

We applied five methods for estimating treatment effects from time-to-event outcomes, which relax the proportional hazards assumption to a network of melanoma trials, reporting overall survival: restricted mean survival time, an accelerated failure time generalized gamma model, piecewise exponential, fractional polynomial and Royston-Parmar models. We conducted a simulation study to compare these five methods. Simulated individual patient data was generated from a mixture Weibull distribution assuming a treatment-time interaction. Each simulated meta-analysis consisted of five trials with varying numbers of patients and length of follow-up across trials. For each model fitted to each dataset, we calculated the restricted mean survival time at the end of observed follow-up and following extrapolation to a 20-year time horizon.

All models fitted the melanoma data reasonably well with some variation in the treatment rankings and differences in the survival curves. The simulation study demonstrated the potential for different conclusions from different modelling approaches.

The restricted mean survival time, generalized gamma, piecewise exponential, fractional polynomial and Royston-Parmar models can all accommodate non-proportional hazards and differing lengths of trial follow-up within an evidence synthesis of time-to-event outcomes. Further work is needed in this area to extend the simulation study to the network meta-analysis setting and provide guidance on the key considerations for informing model choice for the purposes of HTA.

The Scottish Medicines Consortium (SMC) conducts early health technology assessment (HTA) of new medicines on behalf of NHSScotland. Evidence from patients and carers on end-of-life and orphan medicines is gathered during Patient and Clinician Engagement (PACE) meetings. The output is a consensus statement describing a medicine's added value from the perspective of patients/carers and clinicians, which is used by SMC committee members in decision-making. This study compared the importance of factors in the PACE statement to PACE participants and committee members.

A survey of ninety-eight PACE participants (consisting of forty-two patient group (PG) representatives and fifty-six clinicians) investigated the importance of quality of life (QoL) themes (family/carer impact, health benefits, tolerability, psychological benefit, hope, normal life, treatment choice and convenience) identified from an earlier thematic analysis of PACE statements. The findings from PG representatives and clinicians were compared, and the overall results were further compared with those from a previous survey of committee members (n = 26).

Among PACE participants who responded (twenty-six PG representatives and fourteen clinicians), 100 percent rated ‘health benefits’ and ‘ability to take part in normal life’ as important / very important. ‘Convenience of administration’ and ‘treatment choice’ received the lowest rating with fifteen percent and nineteen percent respectively of PG representatives versus seven percent of clinicians rating each as very important. ‘Hope for the future’ received the most diverse response with fifty-eight percent of PG representatives and fourteen of clinicians rating this as very important.

In general, PACE participants rated importance of QoL themes higher than committee members (n = 21) but the rank order was similar. Differences between the proportion of PACE participants and committee members who rated themes important/very important was greatest for ‘treatment choice’ (sixty-seven percent versus twenty percent respectively) and ‘hope for the future’ (eighty-two percent versus fifty-three percent).

The findings demonstrate some alignment between PACE participants’ and committee members’ responses, supporting the value of the PACE output in decision-making. Areas for further research are highlighted.