Highlights
-
• This report provides practical guidance for developing or updating health technology assessment (HTA) guidelines, especially in low- and middle-income countries where HTA is increasingly used as a priority-setting tool for healthcare resource allocation and achieving universal health coverage.
-
• We provide context-dependent recommendations across six domains for HTA guideline development: setting goals, building a team, defining the stakeholder engagement plan, developing content, putting in place institutional arrangements, and evaluating guideline success.
-
• Recommendations support the development and adoption of high-quality HTA guidelines that strengthen HTA institutionalization across different settings.
Introduction
Health technology assessment (HTA) can support many types of decisions, from establishing priorities for public investment and formulating healthcare reimbursement policies to evaluating social programs and setting health research strategies. Successful implementation of HTA depends mainly on three factors: having a systematic and best-practice approach to the assessment of health technologies over their lifecycle, creating a well-defined process for the appraisal of this assessment, and mapping a pathway for the incorporation of evidence-informed decisions into health policy or practice. HTA guidelines support successful implementation of HTA by creating consistency and transparency in concepts, methods, processes, and use, thereby enhancing the legitimacy of the decision-making process 1 , 2 and improving health outcomes through high-quality evidence-informed practices.Reference Bertram, Dhaene and Tan-Torres Edejer 3
A HTA guideline is a document that provides recommendations about standards, which may or may not be mandatory, relating to some or all of the following:
-
• methods for evidence generation, assessment, appraisal, and presentation of results;
-
• principles for reporting the peer review of evidence (e.g., HTA dossiers);
-
• steps of a decision-making process;
-
• criteria and/or rules for appraisal;
-
• criteria and/or rules for decision making;
-
• stakeholder roles and responsibilities;
-
• and/or governance (e.g., mandate and jurisdiction of the policymaking process; mechanisms for stakeholder participation; mechanisms by which to ensure transparency, manage conflicts of interest, or to capacitate stakeholders with an oversight function).
HTA guidelines may be broad or narrow in scope and may be referred to by a variety of names, such as operational or procedural manuals, assessment, or reimbursement guidelines. In certain cases, a law or ordinance might include the components of a HTA guideline. Some countries distinguish between HTA methods and process guidelines. However, there is no standard practice across countries, and definitions of methods and process guidelines differ. 4 It is beyond the scope of this report to define what should constitute a HTA guideline. Rather, we set out a general good practice that aims to be relevant across different guideline scopes and contexts.
The existence of HTA guidelines is often used as an indicator of the degree of HTA institutionalization in a country. 1 , Reference Kumar, Suharlim, Amaris Caruso, Gilmartin, Mehra and Castro 5 Many high-income countries have published HTA guidelines, including specific guidelines for economic evaluation, and there is an increasing number of low-income and middle-income countries developing their own HTA guidelines.Reference Adeagbo, Rattanavipapong, Guinness and Teerawattananon 6 –Reference Kristensen, Husereau and Huić 8 Studies have reported a variable quality for existing guidelines, and adherence to them is mixed.Reference Daccache, Karam, Rizk, Evers and Hiligsmann 9 –Reference Sharma, Aggarwal, Wilkinson, Isaranuwatchai, Chauhan and Prinja 12
To date, there has been no specific guidance for the development and successful adoption of high-quality HTA guidelines, aside from clinical guidelines.Reference Schünemann, Wiercioch and Etxeandia 13 In response, this report lays out good practices and practical recommendations for developing or updating a HTA guideline. It focuses on HTA guidelines that are developed at the national, subnational, or cross-country level. In particular, the report aims to highlight when different approaches may be appropriate for individual contexts and stages of HTA development. The primary audience for this report is those tasked with coordinating the development of HTA guidelines. However, it may also be relevant to a broader audience, including HTA researchers, private sector, patient groups, and organizations commissioning or providing technical or procedural assistance for guideline development.
Methods
This report was developed under a joint working collaboration between Health Technology Assessment International (HTAi), HTAsiaLink, and ISPOR – the Professional Society for Health Economics and Outcomes Research. To provide strategic advice on the development of this good practice report, a task force consisting of subject-matter experts and academics was established in early 2022. The first meeting of the task force was convened in July 2022. Membership of the task force was expanded in April 2023 to include experts with experience across diverse geographies. Members of the task force are listed in the Acknowledgment.
The task force held five teleconferences to develop the guidance report. A qualitative study, comprising interviews, and a nonsystematic literature review targeting eight countries was conducted to understand the factors influencing the success of HTA guidelines (pending manuscript). These countries – Canada, Indonesia, the Netherlands, South Africa, Tunisia, Vietnam, Brazil, and the Philippines – were purposively selected to represent diverse geographies and levels of HTA institutionalization. Preliminary findings were presented for stakeholder feedback during a side meeting at the eleventh HTAsiaLink Conference in Putrajaya, Malaysia in September 2023. Working groups comprising members of the task force then drafted each section of the guidance report based on findings from the qualitative study and published literature. Where existing evidence from the literature review and interviews was insufficient, recommendations were based on the expert opinion of working group members. The draft guidance report was reviewed and agreed upon by all task force members, with no conflicting opinions. A consultation was held with HTA researchers, policy makers, and academics at the fourteenth Conference of the International Society for Priorities in Health in Bangkok, Thailand during May 2024. An open invitation to review the draft recommendations was shared with ISPOR membership and participants of the International Society for Priorities in Health, alongside invited review by the HTAsiaLink Board Members and the HTAi Scientific Development Capacity Building Committee. The task force then made revisions. All task force members approved the final document. A summary of the key steps is illustrated in Fig. 1.
Figure 1. A summary of key steps and milestones of the task force. HTAi indicates Health Technology Assessment International; ISPH, International Society for Priorities in Health; ISPOR, ISPOR – The Professional Society for Health Economics and Outcomes Research; SDCB, Scientific Development and Capacity Building committee.
Good practice for HTA guideline development
This section outlines good practices for developing or updating guidelines across six domains. A summary of key recommendations is shown in Table 1 and several case examples can be found in Box 1.
Table 1. Overview of recommendations for Good Practice in HTA guideline development
HTA indicates health technology assessment.
Box 1. Case studies from around the world of good practice for developing or updating guidelines.
Case study 1: Setting objectives for development of the Health Technology Assessment Methods Guide 2022–7 in South Africa.
The HTA Methods Guide in South Africa was primarily developed to provide guidance on processes and methods to follow when considering inclusion and exclusion of medicines for the South African NEML. 109 There was also a broader objective to sensitize and align stakeholders across the country on methods to promote comparability across medicine assessments. 109 The guideline was developed by consultants with expertise in HTA and experience working with policy bodies in South Africa, with overall leadership from the Department of Health Essential Drugs Programme.
The objectives and scope were agreed by the guideline authors, NEMLC secretariat, Department of Health, and Ministerially Appointed Committees at the outset of the process, before the development of the first draft. This aligned expectations between the guideline authors and the policy users within the NEML process, by defining what would be covered by the guide.
Having a clear scope also facilitated the stakeholder engagement process. At the time of guideline development, there was significant stakeholder interest in HTA, following a draft National Health Insurance Bill proposing to use HTA to define the benefit package. A clear definition of the guideline objectives and scope of what the guideline would/would not cover helped to direct discussions with stakeholders during the extensive 1-year consultation process and served as a reference for deciding which comments to incorporate.
Case study 2: Development team for the 2016 Guideline for Economic Evaluations in Healthcare (The Netherlands).
In the Netherlands, the ZIN is responsible for implementing and revising the economic evaluation methodological guidelines, as part of their role to advise the Ministry of Health (MoH) on the benefit package.Reference van Lessen Kloeke and Castle 110 , Reference Enzing, Knies, Boer and Brouwer 111 The third revision of the guidelines in 2016 aimed to promote comparable and quality economic evaluation, expanding the scope of the guideline to cover all health technologies and not just pharmaceuticals. 112
During guideline development, there was an authorship team, consisting of ZIN staff members with knowledge of the benefit package recommendation process and researchers, as well as an expert panel of senior academic staff. The expert panel was consulted several times prior to the launch of the guidelines. The resulting guidelines were considered to be at the forefront of economic evaluation,Reference Garattini and Padula 113 and there was good adherence.Reference Gabrio 114 ZIN chose to follow the same model for the 2024 guideline revision.
Case study 3: Stakeholder engagement plan for developing the Tunisian methodological guidelines for clinical data, pharmacoeconomic analysis, and budget impact analysis (2021–2).
The 2021–2 guidelines were developed by the INEAS, 115 which is a national scientific authority, under the auspices of the Ministry of Health. 116 At the time of guideline development, the scope of services provided under public insurance schemes was defined by the CNAM, under the Ministry of Social Affairs, 117 , Reference Dahmani, Fradi, Achour and Toumi 118 and no formal link has been established yet to use HTA to define the benefit package. However, HTA has been defined in the 2030 national health policy as the essential tool to support decision making in terms of inclusion and exclusion of new technologies and interventions in the benefit package.
During the guideline development process, INEAS adopted a strategic approach to stakeholder engagement. Draft guidelines were disseminated for public consultation, to build stakeholder awareness of HTA and the role of INEAS. The guideline development team also undertook targeted discussions with CNAM, pharmaceutical manufacturers, and other stakeholders, to move toward structured use of HTA to inform pricing and coverage decisions. Extensive staff time was dedicated to consultations and addressing stakeholder comments transparently, including publishing responses to all comments.Reference Jameleddine, Harzallah, Chemli, Grati, Jebali and Hamouda 119 This engagement successfully built trust with stakeholders, and HTA is increasingly being used for value-based pricing, reimbursement, and other type of decisions related to novel technologies.Reference Dahmani, Fradi, Achour and Toumi 118
Case study 4: Leveraging international networks to develop the 2020 Philippine HTA process guide and methods guide.
Following the 2019 Universal Health Care Coverage Act in the Philippines, the HTA Division (then the HTA Unit) dedicated time and resources for the development of a process and methods guide to support HTA institutionalization. 120 The draft guidelines were based on international literature reviews, which were contextualized and adapted based on staff knowledge and exposure to some extent of the prior Philippine National Formulary processes. Beyond document review, the HTA Division also consulted on a regular basis with guideline developers and implementers in other countries, to learn from their challenges and learnings. These insights helped the team to develop practical guidelines that have been successfully launched with positive feedback from stakeholders.Reference Wong, Co and Modina 121 The Philippines shall soon release the revised HTA implementing guidelines and the second edition of its annexes to incorporate improvements based on actual experiences, stakeholder feedback, and global best practices.
Case study 5: Oversight function of the HEWG in Thailand.
In Thailand, HTA guidelines are not legally mandated but are required and endorsed by the Subcommittee for the Development of the NLEM. Compliance to the guidelines is overseen by a HEWG,Reference Yoongthong, Hu and Whitty 122 first appointed in 2007 by the NLEM Subcommittee. Members of the HEWG include clinical experts, public payers, and researchers with various expertise (e.g., epidemiology, health economics).Reference Teerawattananon, Tantivess, Yothasamut, Kingkaew and Chaisiri 123 Both the Subcommittee and HEWG for NLEM operate on a statutory basis with 3-year cycle terms. All appointed members of the subcommittee and HEWG must declare conflicts of interest and cannot be affiliated with political parties. 124
The HEWG commissions domestic research teams from the NLEM research network to conduct relevant studies.Reference Tanvejsilp, Taychakhoonavudh, Chaikledkaew, Chaiyakunapruk and Ngorsuraches 125 Research teams throughout Thailand can apply to become a part of the NLEM research network and the application is reviewed and approved by the HEWG. During the process of conducting research studies, at least two rounds of stakeholder consultations must be held (one for the proposal and one for the preliminary findings), and at least one member representing the HEWG must be included in all consultations. Upon study completion, quality and compliance with the guidelines are assessed by at least two peer reviewers: one from the HEWG and one external reviewer.Reference Tanvejsilp, Taychakhoonavudh, Chaikledkaew, Chaiyakunapruk and Ngorsuraches 125 The reviewers and HEWG may request revisions from the commissioned research team if deviations from the guidelines are not well-justified.
CNAM indicates National Health Insurance Agency; HEWG, Health Economic Working Group; HTA, health technology assessment; INEAS, Tunisian HTA agency; NEML, National Essential Medicines List; NEMLC, National Essential Medicines List Committee; NLEM, National List of Essential Medicines; ZIN, National Health Care Institute.
Setting objectives, scope, and principles of the guideline
Agreement on the objectives, scope, and principles at the start of HTA guideline development sets mutual expectations between relevant stakeholders. It determines the subsequent stages of the guideline development process, informing decisions around the target users and appropriate language, structure, and content of guidance, and the stakeholder engagement plan. At a minimum, the relevant stakeholders to be involved in this step are guideline authors, project managers, and the institution commissioning or funding HTA guideline development. Depending on the local setting, other stakeholders may also be important to include (for example, members of policy committees or healthcare consumers).
The objectives define what the HTA guidelines aim to achieve. Our analysis found that HTA guidelines are more likely to be successful if they aim to set out best practices for HTA beyond the policy process in question (pending manuscript). For example, the 2009 National Policy on Health Technology Management and accompanying methodological guidelines in Brazil were developed not only to inform processes for federal level decision making but also to set out principles for HTA institutionalization at all levels of the health system, including state, municipal, and hospital levels. 14 After guideline publication, a network was established of HTA researchers and practitioners, which continues to play a prominent role in HTA institutionalization,Reference Lima, de Brito and Andrade 15 , Reference Lessa and Ferraz 16 and there was a significant increase in the number of economic evaluation studies published, with quality improving over time.Reference Decimoni, Leandro and Rozman 17
At different stages of HTA institutionalization, HTA guidelines may have different objectives:
-
• For settings in which policies are not yet made through a systematic process, the objective may be to raise awareness of HTA and its relevance for policy making, to align stakeholders on how evidence-informed decisions should be made, and to instigate discussions around change to current practice.
-
• If a systematic process for policy making has been developed but is implemented on an ad hoc basis, the objective may be to improve consistency in the application of HTA and to uphold standards across decisions, while also promoting transparency in the process.
-
• Once HTA has been adopted as the basis for decision making, the objectives are often framed as improving responsiveness to specific decision-maker needs through a transparent and accountable process.
The scope determines the content that will be covered by the guidelines, whereas the principles lay out a set of guiding rules for how the guidelines will be written. Box 2 outlines specific considerations that may be relevant when defining the scope and principles.
Box 2. Questions to consider when setting the scope and principles for HTA guidelines (note that this list is intended as a starting point and is not exhaustive).
Scope
-
• Which type of decision questions will the guideline address?
For example: recommendations for medical devices or procedures to include in the benefit package, price negotiation for high-cost medicines on the national essential medicines list.
-
• Which components of the decision-making process will be covered by the guideline?
For example: methods for evidence assessment, topic selection process, governance structures, and conflict of interest management.
-
• Who is the primary end-user of the guideline? What is their level of HTA experience and competency?
-
• Which resources are available to support guideline development (in terms of budget, staff time, and access to subject-matter experts)? What is the timeline?
Principles
-
• Which existing legal frameworks may influence the guideline development process?
For example: regulations around engagement with private sector, legally defined roles of institutions, delegation of authority to local level.
-
• What level of detail and explanation will be included in the guidelines?
-
1 If the concepts are relatively new to the target audience, more explanation of the theory or an annotated bibliography may be preferred.
-
2 Limited institutional memory and high staff turnover may warrant a detailed description of each step, whereas settings with strong institutional memory and high level of staff expertise may provide greater autonomy to staff and instead concisely outline required or recommended steps.
-
-
• Will the guidelines prioritize best practices, feasibility for researchers, and/or approaches that are understandable to policy makers?
-
1 For settings with difficulties accessing data or low technical expertise, the guidelines may include two layers of recommendations: best practices and approaches that can be used when the best practices cannot be followed.
-
2 For policy processes in which a technical committee makes a recommendation to the decision maker, the guidelines may emphasize approaches that are understandable to the decision maker.
-
-
• To what extent will the guidelines leverage existing materials and processes from HTA and from the broader health decision ecosystem?Reference Schünemann, Reinap and Piggott 108
-
• How will the guideline developers decide which parts of the guideline are mandatory, preferred, or optional?
HTA indicates health technology assessment.
Good practices
-
• Before starting HTA guideline development, it is recommended that the guideline authors, project managers, and the commissioner/funder agree on the objectives, scope, and principles for the HTA guideline(s). A concise summary of what is agreed upon can be a point of reference throughout guideline development.
-
• HTA guidelines that aim to generally promote good practices for HTA are more likely to be successful in advancing the systematic use of HTA for decision making compared with those tied to a single-policy process (e.g., a guideline for the process to define medicines included in the national benefit package).
Building the team for a quality guideline
Developing a high-quality guideline normally requires both a core team responsible for developing the guidelines (namely, guideline authors and project managers) and an independent group with an advisory oversight function. Our analysis of factors contributing to high-quality guidelines (those that promote conducting HTA in adherence to international standards) suggests that having a dedicated team focused on developing and/or revising the guideline is critical. The team should ideally have project management skills and the capacity to manage stakeholders, as well as technical expertise in various aspects of HTA (e.g., clinical or epidemiological assessments, economic evaluation, governance, and stakeholder deliberation).Reference Jeffery, Chi and Stewart 18 , Reference Glassman, Giedion and Smith 19
When revising or updating a guideline, if the core team has prior experience implementing guidelines within the target policy process, this may assist with the incorporation of lessons learned, improve quality, and support successful implementation. Particularly in settings where HTA is used on an ad hoc basis to inform decision making, it is important that the team leading the guideline development is perceived as legitimate and credible by a wide range of stakeholders. In many settings, this may require the team to have some level of independence so that they are not perceived as solely representing the interests of the payer (e.g., the insurance agency or government office financing the universal health coverage program). It is beneficial to separate the technical writing team from an oversight group advising on guideline development. Although this may be challenging in resource-limited settings, readers are encouraged to follow practical governance principles (e.g., Greer et alReference Greer, Vasev and Jarman 20).
Depending on the capacity and expertise available, one of the two models could be used:
-
(1) the writing team should be subject-matter experts with oversight from stakeholders linked to the policy process, or
-
(2) the writing team is linked to the policy process, and an advisory board of subject-matter experts provides oversight.
Good practices
-
• HTA guideline development is best led by a core team with oversight from a separate advisory group. One of these groups should have expertise in the technical content of the guidelines, whereas the other has expert knowledge of the policymaking context.
Defining the stakeholder engagement plan
A stakeholder engagement plan seeks to balance available time and resources for guideline development with the level of participation required for stakeholders to view the guideline development process (and consequently the resulting guidelines) as legitimate. A good stakeholder engagement plan can not only improve guideline quality and adherence but also strengthen understanding of HTA and its use in policy making. This is especially important for initial versions of HTA guidelines that tend to reshape decision-making norms and rules significantly.
Identifying which stakeholders to engage
Relevant stakeholders may vary, depending on the health system and the way in which HTA is implemented in each country. In general, principles for stakeholder engagement should provide a fair opportunity for participation while mitigating against conflicts of interest and accounting for available resources.Reference Greer, Vasev and Jarman 20 When deciding the appropriate stakeholders to engage, it may help to consider the following four types of stakeholders.
-
(1) Direct users of the HTA guidelines: These may include researchers or analysts undertaking the assessment of health technologies, private sector preparing HTA dossiers for submission, or the secretariat of policy processes applying HTA.
-
(2) Users of HTA outputs: These may include policy makers, government agencies, appraisal committees, payers, or insurance agencies.
-
(3) Stakeholders affected by the policy process(es) covered by the guidelines: These may include clinicians and healthcare professionals, technology developers, patients and health consumers, marginalized or vulnerable groups, civil society, or the public.
-
(4) International experts, engaged for peer review to uphold standards or for sharing experience on guideline development and implementation.
In settings at an early stage of HTA development or where no HTA guideline currently exists, there may be better guideline implementation and adherence with extensive stakeholder engagement, in which all types of stakeholders are consulted (although there may be greater depth of engagement with the first two groups). Although extensive stakeholder engagement can be resource intensive, it facilitates transitions in the norms and rules for decision making to move toward structured, HTA-informed policymaking processes. This is important even in settings with high-level endorsement or laws for HTA. For such transitions to be successful, it is important that the engagement process is open, transparent, and receptive to change.
Fragmented and overlapping decision-making systems or political tension around policymaking processes may impede stakeholder collaboration and implementation of HTA guidelines. In these cases, HTA guidelines that have been successful in overcoming these barriers followed one of the two approaches.
Approach 1 involves bringing stakeholders together to establish areas of common agreement. The scope of the HTA guidelines covers only those areas of consensus and does not provide guidance on areas of contention. Stakeholders additionally agree on a guideline owner, who is responsible for overseeing implementation and making updates to the guideline. An example of this approach was the development of the first pan-Canadian pharmacoeconomic guidelines.Reference Menon, Schubert and Torrance 21 , Reference Torrance, Blaker and Detsky 22
Approach 2 involves creating an advisory board comprising influential stakeholders from other decision-making processes in the country (such as the Chair of the Essential Medicines List committee, National Immunization Technical Advisory Group, or provincial benefit package policy bodies, for example). This approach encourages learning and adaptation from existing policy processes, as well as building collaboration and mutual trust. An example of this approach was the development of the procedures for the Canada Drug Review.Reference Boothe 23
For resilience to political turnover, it may be important to involve independent influential stakeholders, such as professional bodies, in both of the above approaches. In certain cases, countries may have to consider statutory provisions for setting out the authority of the guideline owner (approach 1) or advisory body (approach 2).
Finally, in settings where HTA is fully accepted, updates to the guidelines may require consultation only with direct users of the guidelines, provided that revisions are not expected to be extensive. However, it can be beneficial to maintain channels for communication and feedback with other stakeholder groups to inform guideline development and periodic revisions. It will provide opportunities for these groups to contribute so that they do not feel excluded.
Mechanisms for stakeholder engagement and communication
Stakeholder engagement may usefully occur at all stages of the guideline development process. During the initial stages of guideline development, input from direct users of the guidelines and users of HTA outputs can help to identify appropriate objective(s) and scope, as well as communicating to end users that the guidelines are being developed or updated. Surveys and questionnaires can solicit input from a wide range of stakeholders, whereas targeted meetings with key stakeholders can provide greater depth.
Draft versions of the guidelines may be circulated to different stakeholder groups, ensuring that stakeholders have ample time to respond. Depending on the objectives of guideline development and the context of HTA, this may be to promote learning and discussion across stakeholders around the role of HTA in decision making, to improve quality of the guidelines, and/or to encourage implementation and adherence. If guidelines are circulated to nontechnical audiences, it is important to consider how to present the information in an understandable and accessible way that allows them to engage. For further discussion on effective stakeholder engagement, the reader may refer to existing literature.Reference Fung 24 –Reference Oortwijn, Husereau and Abelson 29
Once the HTA guidelines have been finalized, transparent communication of where they will be published, held, and archived can improve accessibility. Dissemination activities, such as workshops and seminars, may engage stakeholders directly and facilitate discussions on implementation. Other dissemination activities, such as publication of the guidelines in peer-reviewed journals or presentation during conferences can demonstrate that guidelines meet international standards, in addition to building awareness. Establishing feedback mechanisms, such as online surveys or feedback forms, allows stakeholders to provide further input on the guidelines, facilitating continuous improvement and adaptation.
Throughout all stages of guideline development, capacity-building activities, including training workshops, educational programs, mentorship, or creation of knowledge repositories, can support stakeholders to adhere to the guidelines. These activities not only help upskill individuals but can also serve as a platform for building HTA literacy, enabling stakeholders to contribute more effectively to the process and present an opportunity to collect feedback on the guidelines for future updates. Evaluation may be conducted to improve the effectiveness of stakeholder engagement, for example, using the PANELVIEW instrument.Reference Wiercioch, Akl and Santesso 30
Building trust and legitimacy of the process
To establish trust and legitimacy, it is important to have strong leadership from the HTA agency or equivalent authority to promote transparency and accountability of the process. Certain stakeholders may be perceived to have a strong conflict of interest. The team will have to decide how to best represent diverse stakeholder views while mitigating bias.Reference Greer, Vasev and Jarman 20 , Reference Petkovic, Magwood and Lytvyn 31 Ideally, this leadership should be reinforced by support from high-level policy makers and other influential parties, to promote a culture of transparency and openness to change that can improve the legitimacy of HTA.
Good practices
-
• Develop a stakeholder engagement plan adapted to the HTA guideline objective(s) and HTA context. As a minimum, end users of the guideline and policy makers using the HTA outputs should be consulted.
-
• There should be transparent communication of the mechanisms for stakeholder engagement, the feedback received, and how comments were addressed.
Developing content and utilizing available resources
Many resources are available to facilitate the development of a good-quality guideline. When leveraging these resources, it is important to align them with the objective(s), scope, and principles of the proposed HTA guideline, as well as the prevailing decision-making context. To the greatest extent possible, the content of a HTA guideline should harmonize with existing decision-making procedures and/or previously established guidelines. This not only facilitates compatibility and acceptability among stakeholders but also improves implementation because the guidelines’ standards build upon familiar operating procedures and techniques.Reference Cairney 32
Table 2 4 , Reference Adeagbo, Rattanavipapong, Guinness and Teerawattananon 6 , 33 – 38 provides a nonexhaustive list of references that either provide links to existing country guidelines or compare HTA guidelines across countries. Table 3 Reference Bertram, Dhaene and Tan-Torres Edejer 3 , Reference Jeffery, Chi and Stewart 18 , Reference Oortwijn, Husereau and Abelson 29 , Reference Petkovic, Magwood and Lytvyn 31 , Reference Schunemann, Brozek and Guyatt 39 – 102 refers to international best practices and resources relevant for HTA guideline development. While using these resources, content should be adapted to local governance requirements, such as mandatory timelines for evidence submission or processes for stakeholder engagement, as well as the technical capacity for implementation of the local team.
Table 2. Resources for accessing HTA guidelines from other countries or cross-country comparisons (resources may be general or domain-specific, and they are not exhaustive)
Table 3. Checklists and guidance for different types of evidence, including reference materials related to procedural and governance aspects of HTA
Note: these are examples of key resources; the list is not exhaustive.
HTAi indicates Health Technology Assessment International; LMIC, Low- or Middle-Income Country.
Conducting learning discussions with agencies that have already developed HTA guidelines, both nationally and internationally, can be valuable in identifying and addressing challenges. This is particularly relevant when integrating new techniques or applying practices from other settings. Networks such as HTAsiaLink, The Health Technology Assessment Network of the Americas(RedETSA), International Network of Agencies for Health Technology Assessment, HTAi, and ISPOR serve as platforms connecting focal points from various organizations and countries.
Good practices
-
• Where appropriate, contextualize guidelines from other settings to fit with existing decision-making structures and requirements.
-
• Leverage international HTA networks to draw insights from the experience of other countries, regional initiatives, and institutions, particularly when considering new approaches or techniques.
Putting in place appropriate institutional arrangements
Effective use of HTA guidelines and good adherence require an authoritative body with technical capabilities and appropriate resourcing to oversee implementation. One of the main barriers to the implementation of HTA guidelines, particularly in settings with ad hoc decision making, is that guideline developers have limited ability to influence change in decision-making processes or low capacity to uphold standards set out in the guidelines (pending manuscript). If an appropriate body for overseeing guideline implementation and revisions does not exist at the outset, a guideline owner should be identified, with agreement on their role with relevant stakeholders. Such a body may be new or an existing body, such as a clinical licensing institution, with additional roles. Support for this body from relevant professional associations and stakeholder groups, together with a degree of independence from the ruling political party, can help to build HTA as an institution that is both resilient to election cycles and political turnover and more likely to retain professional and broader stakeholder support.
In resource-constrained settings, it may be challenging to identify an owner with the necessary expertise or one that is insulated from political influence. To overcome capacity constraints, an advisory group of experts (e.g., academics) may be set up to conduct a peer review of evidence, whereas good governance mechanisms can provide protection from undue external influence.Reference Bertram, Dhaene and Tan-Torres Edejer 3 , Reference Greer, Vasev and Jarman 20 An explicit legal framework for the use of HTA in policy making is not mandatory for the successful implementation of HTA guidelines but having governance arrangements that are agreed upon by stakeholders, with clearly defined roles and responsibilities, can enhance effectiveness and foster compliance. Although many HTA functions may sit within government, it is recommended to establish appropriate oversight structures and mechanisms to manage conflicts of interest effectively, so as not to undermine guideline adherence and the overall effectiveness of the HTA system.
Good practices
-
• Assign an agency or institution with the appropriate level of authority, independence, and technical capacity to oversee guideline implementation.
Monitoring and evaluating guideline success
Guideline evaluation can inform future guideline revisions and increase impact. Evaluation may take the form of pilot testing before guideline release, continuous feedback mechanisms after guideline release, or impact evaluation 1 to 5 years after guideline publication. It may either be conducted by the guideline owner or by an external third party, depending on the purpose of the evaluation: Assessment by a third party can ensure objectivity and transparency, whereas internal assessment may play an important role in learning and capacity building of the guideline development team.
The success of HTA guidelines can be judged in a number of ways. One criterion is the extent to which they support the institutionalization of HTA within a country’s health system. For example, case studies might assess their impact on establishing or strengthening decision-making processes that are recognized by stakeholders for their legitimacy and effective evidence utilization. In 2016, an assessment of the Common Drug Review Process in Canada was carried out through document analysis and expert interviews.Reference Boothe 23 It revealed that, although recommendations followed a rigorous and transparent process, the Common Drug Review Process had only partly contributed to greater harmonization of recommendations at the provincial level, and difficulties remained covering pricing and common utilization of evidence for drug listing. For HTA guidelines, the emphasis of the analysis can be the impact of the guidelines on changes to HTA practice or its systematic use.
Another criterion for success is compliance, which involves assessing whether the guidelines are being followed. Possibilities include studies comparing evaluations before and after guideline implementation, reviewing HTA reports to evaluate adherence to the prescribed process, and interviews or observational studies conducted by independent groups. This evaluation can focus on a specific policy process or gauge wider impact of the guidelines by evaluating adherence across multiple policy processes or all published HTA studies in the country. For instance, a review conducted in Indonesia to assess the current methods, reporting practices, and the quality of evidence sources revealed that the methods guideline did not influence the standards applied in the eighty-four studies.Reference Chavarina, Faradiba, Sari, Wang and Teerawattananon 11 Similarly, an analysis of reports from Brazil’s National Health Technology Assessment Commission highlighted that only a subset of the required criteria for decision making were discussed in Brazil’s National Health Technology Assessment Commission reports.Reference Campolina, Yuba and de Soárez 103
For guidelines on evidence generation, a third criterion is the level of impact that guidelines have on the quality and validity of HTA studies after the guideline implementation. Unlike the point above, this criterion does not measure adherence to the guidelines themselves, but instead evaluates studies according to internationally recognized standards of quality, such as Criteria for Health Economic Quality Evaluation for economic evaluation studiesReference Kim, Do and Synnott 51 and Appraisal of Guidelines for Research and Evaluation (AGREE-II) for clinical practice guidelines.Reference Brouwers, Kho and Browman 104 A review of national economic evaluation guidelines, for example, assessed quality using Drummond’s checklist.Reference Sharma, Aggarwal, Wilkinson, Isaranuwatchai, Chauhan and Prinja 12
Ultimately, success should align with the stated objectives of the guidelines. In settings without systematic decision-making processes in place, for example, the influence of guidelines on HTA institutionalization may be more important than the adherence of HTA studies to international quality standards. Guideline objectives should be explicit, to ensure a relevant evaluation of their actual impact on healthcare decision-making processes and quality of evidence-based practices.
Good practices
-
• Monitoring and evaluation of HTA guidelines may include (1) the extent to which the guidelines strengthen systematic and legitimate decision-making processes, (2) adherence, and (3) improvement in HTA quality with guideline use. Appropriate success indicators will depend on the specific objective(s) of the HTA guidelines.
Discussion
In this report, we lay out a set of resources and context-dependent recommended practices for developing or updating HTA guidelines. These recommendations seek to overcome barriers toward developing a HTA guideline that successfully promotes the use of HTA in decision making, including lack of a clearly defined scope, poor stakeholder buy-in, inappropriate institutional arrangements, or ineffective governance mechanisms.Reference Bertram, Dhaene and Tan-Torres Edejer 3 , Reference Glassman, Giedion and Smith 19 A detailed checklist or prescriptive guide was not intended because international experience suggests that there is no one-size-fits-all solution. We have, however, provided an inventory of best-practice resources that may be used to inform the development or update of a HTA guideline.
Timing of the development and update of HTA guidelines will depend on the local HTA landscape and political judgments regarding the pace of change for further HTA institutionalization. Our recommendations highlight the importance of adapting the content of HTA guidelines dynamically as the HTA system evolves, as well as the need to align measurements of guideline success with the objectives of guideline development, which will vary across jurisdictions.
Regardless of specific context, some good practices should almost always be adopted. The report emphasizes the importance of mechanisms for transparency, building trust among stakeholders, and fostering a culture of ongoing learning and improvement. This may be achieved, for example, through separating the functions of the writing team and an oversight group, transparently soliciting and addressing feedback on the guidelines (even after publication), and assigning authority for guideline implementation and revisions to a mutually trusted body that has an interest to uphold best practices.
Although the report aims to be broadly relevant to HTA guidelines developed for different purposes, most of the evidence and expert opinion underlying the recommendations comes from experience developing HTA guidelines for national policymaking processes around the services and technologies to provide in the healthcare system. Given the growing interest in harmonizing HTA across countries 105 , Reference Gozzo, Paterson and Wong 106 and expanding the remit of HTA in many jurisdictions,Reference Trowman, Migliore and Ollendorf 107 we recommend continuing to evaluate and collect good practices for HTA guideline development in these areas.
Beyond the recommendations and resources laid out here, HTA networks can play a key role in facilitating cross-country learning and sharing of good practices. Moving forward, such forums may provide a platform to monitor the use of these recommendations and collect experiences of HTA guideline development across different contexts, to inform periodic revision of these recommendations, ensuring their ongoing relevance.
Acknowledgment
The authors would like to thank members of the task force for their contributions to this work. Task force members representing HTAi are Tracy Merlin, Adelaide Health Technology Assessment (AHTA), the University of Adelaide, Australia, Oresta Piniazhko, HTA Department, State Expert Centre of MOH, Ukraine, Gavin Surgey, Radboud University Medical Center, The Netherlands; members representing ISPOR are Manuel A. Espinoza, Pontificia Universidad Católica de Chile, Chile, Gihan Hamdy El-sisi, HTA office and Faculty of Pharmacy, Future University, Arab Academy for Science and Technology and Cairo University, Egypt, Jasmine Pwu, Fu Jen Catholic University and Taipei Medical University, Taiwan; members representing HTAsiaLink are Ying-Li Chen, Centre for Drug Evaluation, Taiwan, Izzuna Mudla Bt Mohamed Ghazali, the Malaysian Health Technology Assessment Section, Malaysia, Kun Zhao, Division of Health Policy Evaluation and Technology Assessment, CNHDRC, China; and independent experts are Christian Suharlim, Management Sciences for Health (MSH), United State, Edwine Barasa, KEMRI-Wellcome Trust Research Programme, Kenya, Somsak Chunharas, National Health Foundation (NHF), Thailand, Anthony J. Culyer, University of York, UK, Wija Oortwijn, Radboud University Medical Centre, the Netherlands, Shankar Prinja, Post Graduate Institute of Medical Education and Research (PGIMER), India, Yot Teerawattananon, Health Intervention and Technology Assessment Program (HITAP), Thailand and National University of Singapore (NUS), Singapore, Hugo Turner, Imperial College London, UK, Anna Vassall, London School of Hygiene and Tropical Medicine (LSHTM), UK. The authors are also grateful to the following HTA experts, who submitted written comments on draft versions of this report: Abeer A. Al-Rabayah, Sneh Aakanksha, Sohail Ahmad, Sinaa Al-Aqeel, Dragana Joan Emmanuelle Amato, Atanasijevic, Giselle Balaciano, Terry Cothran, Maicon Falavigna, Adeola Famuboni, Clara Fatoye, Bocheng (Daniel) Gu, Eman Hammad, Qiang Hao, Nabil Harzallah, Mitch Higashi, Wanrudee Isaranuwatchai, Manjunath K.N., Takako Kaneyasu, Sitanshu Kar, Finn Børlum Kristensen, Satyabrata Kundu, Aurelio Mejia, Nicole Mittmann, Elizabeth Molsen-David, Marjorie Morrison, C’laurel Oluchukwu Nwaorgu, Daniel Ollendorf, Feyisayo Oyolola, Leselle Pierre-Romain, Oresta Piniazhko, Laura Pizzi, Nan Qiao, K.V. Ramanath, Marisa Da Silva Santos, Roza Sarimin, Shweta Shah, Hsin-Yun (Iris) Yang, Philippines HTA Council, and the HTAi SDCB Committee. The authors are sincerely grateful to their project collaborators who provided coordination and contributed to this work. These include Kelly Lenahan (ISPOR), Antonio Migliore (HTAi), Robert Selby (ISPOR), and HTAsiaLink Secretariat team. Finally, a special thank you is extended to Kanchanok Sirison and Angela Kairu for their continued contributions to this endeavor.
Author contribution
Siobhan Botwright and Manit Sittimart contributed equally.
Concept and design: Botwright, Sittimart, Merlin, Surgey, Espinoza, Culyer, Oortwijn, Teerawattananon.
Acquisition of data: Botwright, Sittimart, Chavarina, Bayani.
Analysis and interpretations of data: Botwright, Sittimart, Chavarina, Bayani, Merlin, Surgey, Suharlim, Culyer.
Drafting of the manuscript: Botwright, Sittimart, Chavarina, Bayani, Surgey, Espinoza, Culyer, Oortwijn.
Critical revision of the paper for important intellectual content: Botwright, Sittimart, Chavarina, Merlin, Suharlim, Culyer, Oortwijn, Teerawattananon.
Provision of study materials or patients: Chavarina, Suharlim, Oortwijn.
Obtaining funding: Sittimart, Teerawattananon.
Administrative, technical, or logistic support: Botwright, Sittimart, Chavarina, Suharlim.
Supervision: Botwright, Espinoza, Teerawattananon.
Funding/support
This research has received funding support from the National Science, Research and Innovation Fund (NSRF) via the Program Management Unit for Human Resources and Institutional Development, Research and Innovation (grant number: B41G670025). The staff of HTAi and ISPOR supported ongoing meetings of the task force, as well as the 2-round peer-review process.
Role of the funder
The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Author disclosure
Author disclosure forms can be accessed below in the Supplemental Material section. None to declare.
Open access
