Co-production relates to patients and health professionals working in equal partnership with shared decision-making. Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) are increasingly being used to involve patients and measure healthcare quality. We set out to develop a set of universal experience questions for use across Wales. These will be used in various settings, including the national electronic PROMs and PREMs platform, which is already collecting outcome data across Wales and has received over 7,000 responses to date.

Patient experience leads and clinical leads were invited to a workshop to discuss standardized PREMs collection in Wales, with all health boards and trusts represented. It was agreed that quantitative patient experience data collection, while limited, would be a pragmatic way to collect responses from a large cohort. It was agreed that a previously developed set of PREMs questions could be adapted for use in all healthcare settings. Patient focus groups reduced the number of questions to a shortlist of those considered most important by patients. Wording was improved and an additional question was added.

In partnership with stakeholders we developed and agreed on a set of universal PREMs questions. These have been added to the national electronic platform, with collection commencing imminently. This will allow patients accessing secondary care in Wales to provide PREMs and PROMs responses.

Development of a standardized set of PREMs has allowed us to initiate collection on a national basis. Addition of PREMs to the national electronic platform provides a unique means of collecting large volumes of data consistently, allowing us to benchmark across and within organizations. It will also allow experience teams to target improvement initiatives and identify good practice. Together with outcomes responses, the data will be used to measure experience of care in Wales.

There is a lack of adoption and diffusion of health innovations needed to drive the implementation of important breakthroughs in value-based health care. To stimulate organizational changes, decision-makers need to see potential benefit at an early stage. The aim of the present study was to assess the potential effects of a conceptualized intention to provide digital home-based care and compare it to the current provision of such care. The new intervention aims to strengthen the municipality's care services by offering a digital communication platform to recipients of home-based health services and their dependents. The platform is designed to be implemented nationally and is in line with home service needs identified in several white papers.

An interdisciplinary team united to determine and quantify potential effects of the project. Effects of the digitalized service were distinguished in priced quantitative, unpriced quantitative and qualitative effects. A ten-year present value calculation with a calculation rate of four percent was used for the estimates. A risk analysis was also carried out.

The present value calculation resulted in estimated savings equal to EUR 25.8 million , with present value investments costs of EUR 5.5 million over ten years. This resulted in net present value per invested euro in the public sector equal to EUR 3.2. Overall assessment of uncertainty related to the intervention's socio-economic profitability was deemed average. Based on data quantified estimates from the conceptual phase, the project succeeded in the decision-making and funding needed to proceed into the next developmental phase of the project.

The present approach to early assessment may provide much desired decision support in an early innovation phase when data are still missing. Our experience is that early stakeholder involvement and the early assessment and quantification of value gains are of utmost importance to overcome the critical barriers to organization health innovations.

Mobile health (mHealth) is increasingly being seen as a promising solution to the multiple health system challenges that Vietnam faces, including inadequate health service provision. Nevertheless, there is limited published information describing the sustainability of such mHealth solutions in the Vietnamese context. This study reviewed the available projects and interventions to evaluate factors challenging sustainability of mHealth initiatives in Vietnam.

A multimodal scoping study was designed to collect data from various sources: published literature, government reports, unpublished literature, conference presentations, online documents, and key informant interviews. Relevant mHealth initiatives were identified and selected through electronic searches and informal discussions with key stakeholders. Collected data was charted and classified by thematic analysis. Challenges to the sustainability of mHealth initiatives were discussed in the context of the strengths, weaknesses, opportunities, and threats (SWOT).

Twenty mHealth initiatives with relevant information and available data were collected, covering the period from 2010 to 2017. Fifteen (75 percent) were primarily funded by external donors, one (5 percent) was government supported, and four (20 percent) were self-funded projects. Five (25 percent) were on-going, and fifteen (75 percent) were completed at the time of data collection. Four (20 percent) out of the completed initiatives were continuing to use materials, infrastructure, and technology to engage end-users.

The high percentage of external funding, complicated governmental bureaucracy and lack of government commitment, electronic medical record standardization, and legislation relating to mHealth are amongst the largest challenges to mHealth sustainability in Vietnam. In addition, findings demonstrate it is crucial for project managers of mHealth initiatives to build strong relationships with the Vietnam government and advocate for their mHealth initiatives in order to promote sustainability.

The adoption of electronic health records (EHR), which contain large volumes of aggregated longitudinal clinical data, guarantees substantial benefits, including better care, improved safety and decreased clinical risks; however, it is also associated with significant costs and large technical and organizational impacts. For these reasons, it is important to conduct a comprehensive evaluation of health care delivery outcomes. The purpose of the study is to gather evidence on the safety and overall effectiveness of EHR implementation at Bambino Gesù Children's Hospital.

A decision-oriented health technology assessment (HTA) method was applied to assess the technology on clinical, technical, organizational, economic, legal, ethical and safety domains. It is a new implementation of the EUnetHTA CoreModel integrated with the Analytic Hierarchy Process. The evaluation structure was a hierarchical decision tree filled with indicators of a technology's performance, where each indicator was weighted based on its relative impact on decision making. Finally, the alternatives’ ranking was defined. A subgroup of these indicators has been included in a checklist for the evaluation of six EHR implementation projects. This checklist was used as a tool by each involved professional during demo sessions.

The assessment took into consideration all the recommendations about the benefits and disadvantages of EHR. In particular, EHR seems to offer many benefits in terms of safety and clinical effectiveness, such as improved continuity, quality of care and accessibility of the data. Its implementation resulted in important organizational outcomes such as EHR configuration, learning curve and training; usability was the main technical characteristics of the technology taken into account. Finally, legal aspects on privacy and data security assumed a key role.

A detailed technology evaluation of EHR has permitted the hospital's decision-makers to knowingly assess its introduction in the hospital.

New medical technologies are an important part of delivering innovative healthcare, however, expanding use of medical technology is a major contributor to rising costs. The increase in medical spending is related to new technologies being rapidly developed, marketed and adopted; and often being incorporated into health systems with little evidence. They also come with higher prices when compared to existing technologies.

We describe how University of Washington (UW) Medicine has designed, and developed a new hospital-based health technology assessment (HB-HTA) program, Smart Innovation. Smart Innovation will replace a fragmented and complex set of purchasing and coverage-decision processes. The program will streamline the decision-making process for new medical technologies and balance quick turnaround with rigorous evidence standards. The program is also being developed in collaboration with UW Medicine's Value Analysis team, an evidence-based purchasing team and MedApproved, a new centralized software program for medical purchasing at UW Medicine.

Smart Innovation has been reviewing technologies during its first year and has received encouraging results. For example, by adopting a new liver ablation technology, UW Medicine has estimated improved patient outcomes by reducing the number of procedures and adverse events; as well as saving approximately USD 8,000 per patient. Additionally, The Smart Innovation program has achieved projected cost avoidance from deciding not to adopt uncertain or investigational technologies. For example, by not adopting a new bladder cancer screen, our models indicate UW Medicine will avoid spending USD 1.5 million per year.

Smart Innovation is proving to be an effective program for reviewing and making critical healthcare policy decisions that is showing significant fiscal and patient improvements for UW Medicine. As the program continues to grow and become embedded into UW Medicine, its impacts will become even more valuable and system-wide.

One of the main tools for Hospital-Based Health Technology Assessment (HB HTA) is the preparation of a mini-health technology assessment (HTA) report. Despite the high value of the results of mini-HTA reports for hospital decision-makers, the classical mini-HTA report does not allow a direct comparison of several health technologies among themselves.

Based on the analysis of international experience of using the principles of multiple-criteria decision analysis (MCDA) in the field of HB HTA, we created and approved our own managerial decision-making model which includes five standardized multiple criteria. The value (weight) of each criterion was defined as the arithmetic mean obtained as a result of interviewing hospital decision-makers and an HTA expert group.

Five standardized multiple criteria were included in the structure of our mini-HTA report. These criteria presented the main results of assessment of the viability of implementing new health technologies (HTs) in hospital practice and contain the following: i) Novelty/innovation; ii) Comparative clinical effectiveness and safety; iii) Relevance (demand); iv) Economic effectiveness; and, v) Payback period. We conducted the modeling of various options of HTA results by using multiple criteria, which allowed us to determine the threshold values of the evaluated HTs corresponding to their priority for implementation: i) High priority - HTs are recommended for implementation; ii) Medium priority - HTs can be recommended only if there are sufficient financial resources in hospital; and, iii) Low priority - HTs may be recommended only if there are strong reasons for their need.

Integration of the principles of MCDA in the structure of mini-HTA reports gives the opportunity to i) make comparative assessments of implementing new health technologies based on standardized criteria; ii) determine the priority for implementation of newly evaluated health technologies; iii) avoid the influence of subjective factors on the managerial decision-making in hospitals.

Innovation procurement is a key enabler to improve the quality and efficiency of public healthcare services by driving innovation from demand side to meet concrete public healthcare provider needs. Catalan Health Services (CatSalut) aims to optimize healthcare services through innovative solutions that encompass both innovative technologies and new processes of care. Answering this aim, the Hospital Clinic of Barcelona (HCB) is participating in an Innovative Pilot Program to optimize the efficiency in the management of Aortic Valve Stenosis (AVS) using an adaptation of the methods and knowledge from hospital-based health technology assessment (HB-HTA).

The first step was to identify unmet needs, main bottlenecks and problems in the comprehensive management of aortic valve stenosis (AVS) (from primary care to hospital discharge). Innovative technologies, solutions and health care organizations were proactively scanned through literature review and professional expertise. Lists of solutions were proposed through an inclusive stakeholder participation process.

A new healthcare model was proposed to be evaluated in the next three years based on an integral, transversal and multidisciplinary management of AVS (named MITMEVA). For each new proposed solution, the management, work streams, expected impact and key performance indicators (based on stakeholder information demands) were defined. To test the potential of the proposal, a theoretical modeling of the economic, clinical and process impacts of implementation was performed based on available scientific evidence, local professional and economic data. This analysis shows more quality-adjusted life years, fewer adverse effects and lower cost with the new proposed model.

HB-HTA usually recommends for/against investments. In the era of value based procurement, HB-HTA can also help in developing a Public Procurement of innovative solutions (PPI) project and in testing proactively its potential impact in healthcare, which will be later tested in real life. Therefore, adapting HB-HTA to hospital innovative procurement is another way for health technology assessment to push for the implementation and testing of high value innovative technologies.

Patients are the people who, with their informed consent, receive medical interventions. It is important, therefore, that patients have an understanding of interventions and their potential as a treatment for their condition. Patients are becoming more informed about their health care and the treatments that are available to them. At a population level, the potential benefits and harms of treatments need to be regularly assessed. This is part of healthcare decision making at a policy level about what treatments are publically available. As technology develops and old methods are replaced by new and evidence-based interventions and procedures, healthcare payers look to streamline their payment schedules and disinvest in old technologies and procedures. Some users of health care are reluctant to let go of outmoded methods, so disinvestment is best achieved through transparent processes. Successful engagement with key stakeholders of health care, engaging with payers, health service administrators, clinicians and patients, can facilitate implementation of disinvestment processes.

To assist in this process, Health Technology Assessment International (HTAi) Interest Groups and EuroScan have come together to develop the following key points to consider in the involvement and engagement of clinicians, patients, and the public in the disinvestment of services and technologies.

The best time to involve clinicians and patient representatives is right at the beginning of the process. Clinicians and patients can make valuable contributions as advisory committee members. The disinvestment processes may be led by clinicians, payers, or independent organizations. This will likely influence commitment of clinicians to the process.

Broader consultation with clinicians, patients and the public in the development and consideration of draft reports and recommendations can increase the transparency of the disinvestment process. Consultation is an important means of obtaining buy in. Feedback needs to be seen as taken seriously, and explanations given for any changes made or not made to the report and its recommendations.

As science advances the number of newly developed health technologies increases, but the lifecycles of health technologies becomes shorter. Thus, the importance of horizon scanning systems for identifying promising new health technologies and evaluating their potential impact is increasing. Engaging and collecting opinions from various stakeholders in this search process is very important. The purpose of this study was to develop a strategy for involving various stakeholders in all steps of the horizon scanning system in Korea.

The horizon scanning system consists of five steps: identification, filtration, prioritization, assessment, and dissemination. We identified the stakeholders to be considered at each stage, and examined who would be involved and how. In addition, we planned how to synthesize and apply stakeholder opinions and to test the feasibility of these methods by using them in a horizon scanning system.

In the identification stage, developers, health professionals, and consumers suggested new and emerging health technologies to investigate. In the filtration stage, the person in charge of licensing judged the technologies based on appropriateness, innovativeness, and potential of market entry. In the prioritization phase, experts from eight to ten related fields (clinical, health technology and drugs, policy, methodology, patient organizations, etc.) participated and judged the technologies according to seven criteria (burden of disease, clinical impact, innovativeness, economic impact, acceptability, social impact, and evidence). In the assessment stage, between one and four clinical and methodological experts assessed the potential impact of the selected promising health technologies using seven evaluation items (unmet needs, improved patient health, health equity, change in medical behaviors, acceptability with respect to the patient and clinical condition, change in medical costs, and social, ethical, political, and cultural aspects). Before its dissemination, the final report was delivered to relevant industries for feedback (with particular emphasis on accuracy of data on the technology).

There are many stakeholders in the horizon scanning system for new and emerging health technologies, depending on the healthcare system, policy, environment, etc. This study confirmed that stakeholder opinions on new technologies can vary. In addition, standards of social value judgment may change over time. It is therefore very important for horizon scanning systems to engage various stakeholders, collect their opinions, and make rational scientific decisions.

All health systems are challenged by finite resources to address unlimited demand for services. In many countries priority-setting and resource-allocation decision-making has been inconsistent and unstructured. In these cases, the lack of coherence between limitless promise and limited resources leads to implicit and covert rationing through waiting lines, low quality, inequities, and other mechanisms. Over the past decades, different countries have established specialized health technology assessment (HTA) organizations aimed at better informing health care policies and clinical practice. Although the first technology assessment institution, although not exclusively health related, was the Office for Technology Assessment (OTA) in the U.S. in the 1970s, HTA is not yet current nationwide practice. Nevertheless, there are more than fifty agencies in operation in over thirty countries to assist systematic priority setting, especially in high income countries. The cases of Ukraine, Colombia and U.S. represent different features of the need for systematic priority setting. Ukraine is moving from National essential medicines lists (EML) to more dynamic HTA use to update its publicly funded benefits package; Colombia established a few years ago nationwide HTA, but is currently attempting to use HTA for Pricing and Reimbursement since healthcare coverage is so heavily contested by judicialization. Nevertheless, even in countries where formal HTA activities are ongoing, and in most low and middle income countries, rationing still occurs as an ad hoc, haphazard series of non-transparent choices that reflect the competing interests of governments, payers and other stakeholders. Henceforth, there is the opportunity to closely review why the state of development for HTA varies so much according to setting.

Retrospective policy analysis considering common motivators for the implementation of HTA; the agenda setting model of the three streams (problems, policy and politics) for policy action ; and qualitative approaches for the inception of HTA are being used in these three cases.

Through a qualitative approach, ten “drivers” previously emerged with the ability to help or hinder HTA development in Colombia were used to assess the difference of HTA development in the USA and Ukraine (i.e. availability and quality of data, implementation strategy, cultural aspects, local capacity, financial support, policy/political support, globalization, stakeholder pressure, health system context, and usefulness perception). Policy/political and financial support, stakeholder pressure, cultural aspects and health system context were the most prominent drivers to induce or prevent institutional development of HTA in different countries.

Common motivators, similar drivers and context specific characteristics are all influential for the implementation of HTA at the national level. Policy/political and financial support, stakeholder pressure, cultural aspects and health system context preliminarily seemed the most prominent drivers to induce or prevent institutional development of HTA in different countries. Henceforth, methods and processes matter, as well as the political economy for HTA. Further research is needed to test these preliminary findings.

Health system reform is considered a tough issue worldwide. Great efforts have been made toward health system building and strengthening. However, it is still unclear which health system is appropriate for different countries. This study aimed to systematically compare the characteristics of the establishment periods between eighty-eight counties of National Health Service (NHS) and Social Health Insurance (SHI).

Forty-eight NHS countries and forty SHI countries with data availability were selected. The establishment years of current health systems and other eighteen indicators in economics, society, population and health during establishment periods were collected. Comparison between NHS and SHI was conducted by descriptive analysis of every indicator.

Most NHS countries were established during the cold war, while SHI had been set up since the cold war ended. The median of gross domestic product (GDP) per capita, urbanization rate and aging rate of SHI were USD 1535 in current dollars, 58.2 percent and 9.8 percent, respectively; compared with USD 1387, 41.2 percent and 4.7 percent, respectively of NHS. NHS countries had a smaller total population, lower mortality rate and elderly dependency ratio, while the birth rate and children's dependency ratio were higher. SHI countries showed a higher life expectancy and lower mortality rate in infants and children. NHS countries spent less in total health expenditure and a lower proportion of GDP. The median health expenditure per capita of SHI and NHS were USD 188 and USD 131 in current dollars, respectively. There was little difference among maternal mortality rates, and public and private health expenditure proportions.

NHS and SHI countries had different characteristics during the health system establishment periods. NHS was established earlier than SHI overall, so that SHI revealed higher levels in economic and social development. Health outcomes of NHS countries were slightly lower than SHI ones, while health expenditure was more in SHI countries. Specific social, economic, demographic and health conditions should be considered when countries are building their own health systems.

The National Institute for Health and Care Excellence (NICE) strategic review of its public involvement offer included a survey with stakeholders to explore how NICE can continue to deliver high quality, meaningful public involvement in a rapidly-changing environment.

NICE staff, committee lay members, and an external academic ran the project and designed an online survey. The survey was open for two weeks. A purposive sample, recruited through various communication channels, was invited to participate. The sample comprised: (i) external individuals involved in NICE work, (ii) NICE committee and Board members, (iii) NICE staff. The survey included qualitative and quantitative questions, covering the ‘who’, ‘when’, ‘how’ and ‘what’ of NICE's public involvement approaches.

The survey yielded 684 responses, which were stratified by stakeholder type. Overall the responses indicated that: (i) the suggested stages for involvement are all important, but on a sliding scale: ‘defining outcomes guidance should consider’ is most important, and ‘helping committee chair recruitment’ is least important; (ii) different perspectives are needed such as individual treatment or care decisions should incorporate views of directly affected people, and population-based public health decisions need the views of citizens. Quality improvement suggestions included: (i) seeking feedback on people's experiences of care, using clear, structured approaches including focus groups, interviews, surveys, social media; (ii) increasing communications about NICE's work, specifically about involvement opportunities and use of patient evidence; (iii) using data on people's experiences equally with academic evidence; (iv) providing education and training on involvement to NICE staff and the general public; and, (v) partnership working with other organizations to enhance engagement. A focus group with key stakeholders used the survey findings to shape the subsequent public consultation document.

There was consensus that public involvement is necessary throughout guidance development; however, the type of person involved and nature of participation should vary across the development stages. Project challenges included managing diametrically opposing views, and the associated implications for engagement.

In Brazil, the “Sistema Unico de Saúde” (SUS) is a public health system that has universal coverage, comprehensive care, and principles like community participation. The incorporation, update or exclusion of new health technologies is done by the National Committee for Technology Incorporation (CONITEC), which issues reports on the incorporation of technologies and submits them to public consultations, which is the main mechanism of public involvement and an opportunity to influence the decision to access and coverage to new health technologies. Our study aimed to investigate a typology of social representations on the contributions from 2012 to the CONITEC's public consultations to the incorporation of Trastuzumab for the treatment of initial breast cancer in Brazil.

Our study deployed a mixed-methods approach to semi-quantitatively analyze the social representativeness and corpus composition of all the public consultation contributions for the recommendation of the Trastuzumab's incorporation for treatment of initial breast cancer within SUS, as well as the authors' qualitative analysis of the IRAMUTEQ software as a potential effective and efficient tool to semi-qualitatively analyze such public consultations. All contributions were included (127 contributions, from several Brazilian states) and organized into a single corpus, which was submitted to 5 types of analyzes (classical lexical analysis, analysis of group specificities, descending hierarchical classification; similitude analysis and word cloud).

The general corpus consisted of 114 texts, separated into 685 text segments (TS), with use of 79.12 percent of total TS (684). The analyzed content was categorized into four classes: Class 1 – Patient Representations/ Advocacy (186 ST-34.3 percent); Class 2: Pharmaceutical Industry/ Advocacy (181 ST-33.4 percent); Class 3: Health Professionals (81 ST-14.9 percent); and Class 4: Individual Contributions (94 -17.3 percent). Class 1 corpus consisted mostly of contributions made from a breast cancer patient association/ advocacy report, which focused mainly on lay expertise terminology. We observed a proximity in corpus between Classes 2 and 3, showing a potential approximation between the pharmaceutical industry and health professionals' contributions, to whom the main word occurrences related to health technologies. Class 4 corpus focused on improvement and individual need, as well as in corpus referring to SUS.

From our findings, we observed: (i) a potential similarity in contributions of health professionals and pharmaceutical industry; (ii) how lay expertise might affect the contributions of patients individually and within advocacy and patient organizations; and (iii) the uses and limitations of IRAMUTEQ as potentially effective and efficient tool to semi-qualitatively analyze health technology assessment public consultation contributions.

Increasingly, health technology assessment (HTA) organizations have instituted mechanisms for involving patients in assessment and review processes. The reasons are obvious—to understand the “patient experience” with a disease and to ensure that patient perspectives are considered during deliberations about the value of new treatments. More recently there have been efforts to engage the public in HTAs and HTA-informed decision-making processes. However, the goals of these efforts have not been well articulated. This may be attributable to the lack of a shared definition of “the public”. The objective of this study was to develop a common understanding of the term “the public” within the context of HTA.

The following were conducted: a survey of HTA organizations; a systematic review; consultation with Health Technology Assessment international's Special Interest Group on Patient and Citizen Involvement; and a workshop comprising representatives from patient organizations, industry, and HTA bodies in Canada.

In many HTA processes, the terms “public” and “patients” are synonymous. Definitions found in scholarly articles vary and depend on the rationale for involving the public in a particular issue. Through consultations it became clear that, in the context of HTA, the definition depends on understanding what is missing from current deliberations around the value of new health technologies. There was consensus among workshop participants that: (i) “patients” and “the public” are not the same; (ii) the role of the public may be to ensure societal values are reflected in HTAs and HTA-informed decision-making processes (e.g. serving an audit function); and (iii) a legitimate definition of “the public” could be: “A non-aligned community member with no commercial or professional interest in the HTA process who is not a patient or member of a stakeholder group”.

Consensus on the use of the terms “patient” and “public” will support rigorous, evidence-based public and patient engagement in HTA. The proposed definition indicates a way forward in this debate.

Cervical cancer screening (CCS) is conducted through multiple testing modalities including Papanicolaou smears and more recently, HPV Testing. Participation in CCS is influenced by a multitude of barriers and facilitators governed by the preferences, values, and beliefs of women. This presentation will discuss the findings from a Patient Perspectives and Experiences review for a CADTH Health Technology Assessment on HPV Testing for Primary Cervical Cancer Screening.

A systematic literature search yielded 4864 citations published from 1 January 2002 to 1 November 2017. One hundred and six eligible studies were analyzed using the qualitative meta-synthesis methodology.

The social location, circumstances and resources available to women significantly influence how they negotiate the factors that influence their CCS participation. Some of the factors we identified are Emotions, Understanding Personal Risk, Logistics, and Multiple Roles of Women. In this presentation, we will discuss how these factors interact with a woman's social location to influence women's choices and preferences about engaging in cervical cancer screening. Specifically, we describe an analysis that conceptualizes social location as a balancing fulcrum, which changes the force exerted by factors acting as incentives and disincentives. Women who experience social and material deprivation may find that disincentives are harder to overcome than women who have access to ample social and material resources. More incentives in quantity and strength would tip the balance in favor of incentives and increase CCS participation. This presentation will also describe how incentives and disincentives were operationalized in the context of a patient perspectives and experiences review for a health technology assessment.

Women's decisions to participate in CCS are influenced by many factors. The way women negotiate these factors is closely related to their personal circumstances and the availability of social, material, and financial resources.

The growth of the evidence-based policy movement sought to determine how to better assess and incorporate qualitative evidence in clinical practice and policy development. The question engendered was not whether qualitative research is valuable but how researchers can enhance its rigor. From this discussion arose over one hundred appraisal tools for the quality appraisal process of qualitative studies. For those without a deep familiarity with the qualitative research paradigm, navigating through the breadth of tools to find the most suitable tool for the task is a cumbersome process. This presentation will review the descriptive characteristics of available quality appraisal tools for assessing the quality of primary qualitative studies in qualitative evidence syntheses (QES). This presentation will also offer a critical discussion on the use of reflexivity as a de facto quality criterion, and how methodological reporting may influence the application of quality criteria in QES.

We conducted a systematic search to identify quality appraisal tools of qualitative research designed for use in QES. This search built upon the work of Santiago-Delefosse and colleagues by extending their search to 2016.

We identified eight appraisal tools intended for use in the quality appraisal process of a QES. We provide a description of the structure, content, objectives, and philosophies of tools followed by considerations concerning their historical antecedents, common patterns regarding structure, content, and purpose, and the implications of these patterns on the QES process.

Quality appraisal of qualitative research is an important step in QES, and there have been a proliferation of tools for this purpose. By providing an overview of available tools detailing their intent and strengths, this presentation will assist those engaging in QES to choose an appropriate tool for their work.

Decision-makers are increasingly recognizing the usefulness of qualitative research to inform patient-centered policy decisions, and are accordingly increasingly demanding qualitative evidence as part of health technology assessment (HTA). In the context of tight HTA timelines, a new form of evidence synthesis has emerged—rapid qualitative reviews. The need for rapidity requires either an increase in resources or, more commonly, a compromise in rigor, yet guidance on appropriate compromises for qualitative reviews is lacking.

In order to inform de novo guidance, we conducted a systematic scoping review to identify existing guidance and published examples of rapid qualitative reviews. We searched Medline and CINAHL using medical subject headings and keywords related to “rapid reviews” and “qualitative” research, and screened the 1,771 resultant citations independently in duplicate. Additionally, we searched the grey literature and solicited examples from our contacts and other evidence-synthesis organizations. We summarized included guidance and reviews using the Search, AppraisaL, Synthesis, Analysis (SALSA) framework to identify abbreviations in the review process.

We found no guidance documents specific to rapid qualitative reviews. We found one published peer-reviewed rapid qualitative review, and several more (>10; grey literature search in process) through our organizational contacts. While methods to abbreviate the process are poorly reported, an abbreviated literature search (years and databases searched) and the use of a single reviewer appear common.

A number of agencies are producing rapid qualitative reviews, however our review identifies the urgent need to develop and explore methods for the synthesis of qualitative research that balance rapidity and rigor.

There are many approaches to synthesis of qualitative studies. The GRADE-CERQual approach (Confidence in the Evidence from Reviews of Qualitative research) provides a transparent method for assessing the confidence of evidence from reviews of qualitative research. This presentation aims at giving examples of applying CERQual, presenting and discussing its strengths and limitations.

This presentation draws on practical experiences with the conduction of three qualitative systematic reviews using the CERQual approach. The reviews differ in aim and field of research.

The three CERQual reviews to be discussed in this presentation are: (i) Emergency departments and mental health patients - Purpose: Uncovering knowledge in a project on merging emergency departments to include both somatic and psychiatric patients. (ii) Parental responses to severe or lethal prenatal diagnosis – Purpose: Providing physicians with knowledge on a patient group from their daily clinical practice. (iii) Patients' experiences with home mechanical ventilation – Purpose: Disseminating important knowledge from a national project to an international audience.

CERQual strengths:

• • Presents complex and large amount of knowledge in a clear way

• • Pools knowledge from different studies into common outcome measures across studies

• • Presents an assessment of the quality and strength of outcome measures

• • The clear presentation makes it useful in decision making.

CERQual weaknesses:

• • Time consuming to conduct the reviews

• • Simplification of qualitative research, missing out on context and nuances.

CERQual represents a useful tool to facilitate the use of qualitative evidence in clinical and political decision making. CERQual is time-consuming to learn, but a useful tool to apply when learned. CERQual may encourage more uniform reporting of qualitative research, including assessment of confidence in findings. This may increase the impact of systematic reviews of qualitative studies.

The National Institute for Health Research (NIHR) is a major funder of health research in the United Kingdom. Selecting the most promising studies to fund is crucial, and external expert peer review is used to inform the funding boards. Our aim was to evaluate the influence of different kinds and numbers of peer review and reviewer scores on Board funding decisions, and how we might modify the process to reduce the workload for stakeholders.

Our mixed method study included i) retrospective cross sectional analysis of funding board and external reviewer scores for second stage applications for research funding, using Receiver Operator Characteristic (ROC) curves to quantify the influence of reviewer scores on funding decisions and ii) qualitative interviews with thirty stakeholders (funding board members, applicants, external peer reviewers and NIHR staff).

Analysis of ROC area for reviewers indicated that areas changed very little with increasing numbers of reviewers from four to seven or more. External reviewers with clinical, methodological or patient expertise all appeared to influence Board funding decisions to a similar extent. The stakeholders interviewed valued peer review but felt it was important to develop a more proportionate process, to better balance its benefit with the workload of obtaining, preparing, reading and responding to reviews. Reviews are of most value when they fill gaps in expertise on the Board. Less than four reviews was felt to be insufficient but more than six, excessive. Workload could be reduced by making reviews more focused on the strengths and weaknesses of applications and identifying flaws which are potentially “fixable”.

Stakeholders supported the need for peer review in evaluating funding applications. Our results suggest that four to six peer reviews per application is optimum, depending on the expertise needed to complement that of advisory boards.

National health research funders are accountable to the public with regard to the societal impact of the research, including health technology assessment (HTA), that they fund. Failing to do so can not only negatively affect public trust in the allocation of resources to funding agencies, but can also lead to public mistrust in science.

We present the results of reducing research waste to ensure societal responsible research, both at an international and national level. In the Netherlands, the National Organization for Health Research and Development (ZonMw) developed an analytical framework to assess its research programs, including the national HTA program.

An evaluation of 12 national funding agencies in Australia, Europe and North America demonstrated that certain processes (e.g. how research questions are prioritized or decided) are not transparent. At the international level, health funders believe that they have a joint responsibility not just to seek to advance knowledge, but also to advance the practices of health-related research and research funding. In the Netherlands, ZonMw (HTA) research programs perform well regarding addressing societal relevance (e.g. stakeholder participation) and reasonably well on scientific quality (e.g. international cooperation and knowledge sharing). Efficiency (e.g. encouraging use of existing data and systematic reviews) appears to be less well developed, while integrity (e.g. preventing publication bias) is underexposed.

Although ZonMw is doing reasonably well in terms of reducing research waste, it was concluded that more focus on societal impact assessment is needed. To do so funding agencies need to collaborate with all relevant stakeholders. This is especially relevant in the field of HTA where the ambition is to move from evidence to impact.