Why Deliberate?

Deliberation allows participants to exchange their views and perspectives. Participants may learn about other views and perspectives as a result of deliberation, leading to predeliberative opinions being updated or revised. Participation through deliberation also gives some degree of influence and ownership of the final output to those involved. Deliberation can lead to an expansion of what counts as evidence in the process. Well-designed and well-executed deliberative processes also have the potential to provide stakeholders with reasons to perceive the HTA process as (un)fair, based on its procedural characteristics (e.g., inclusivity of all relevant stakeholders), even if they disagree with the outcomes of the process (Reference Abelson, Forest and Eyles2;Reference Fearon and Elster10;Reference Kolotourou, Ermacora and Grosvenor1 Reference Lomas, Culyer, McCutcheon, McAuley and Tetroe3–Reference Kolotourou, Ermacora and Grosvenor1 Reference Dobrow, Chafe, Burchett and Culyer4). It may yield outputs that are more inclusive, better framed, more balanced, and more feasible. This will increase trust in institutions and their decisions (Reference Solomon and Abelson15).

What Are the Desired Outcomes of the Deliberative Process?

HTA bodies might consider several desired outcomes related to the goals of deliberation. These include (i) the sharing and potential shifting of different viewpoints among participants, (ii) reaching common ground that underpins a collective decision; (iii) an opportunity to provide reasons for and against outputs, and a possible revision of them following deliberation; and (iv) strengthening mutual respect and understanding among participants. Although agreement can also lead to a collective opinion (e.g., advice, recommendations, or decisions), this is not always the required output of a deliberative process (Reference Abelson, Forest and Eyles2;Reference Fearon and Elster10;14).

What Is the Scope of Deliberation?

Deliberation can inform a range of processes within an HTA body (Box 1), from formal discussions of an HTA organizational structure (deliberation about processes) to topics for assessment and the revisiting of an earlier decision on a specific health technology in the light of new evidence (deliberations within processes).

What Contextual Factors Are Relevant to the Deliberative Process?

Internal and external contextual factors influence the type and scope of deliberative processes that can be implemented in any setting (Reference Pichon-Riviere, Augustovski, García Martí, Alfie and Sampietro-Colom26). Internal factors include governance, leadership and organizational culture, financial constraints, availability of local information, availability of participants, the knowledge and skills of applying HTA methods, and the capacity or financial resources to conduct HTA and communicate HTA decisions (Reference Kapiriri, Baltussen and Oortwijn27). External factors include the extent to which countries have established mechanisms to incorporate evidence into decision making, the perceived role of HTA in decision making, and the mandate given by legal or policy authority (Reference Kapiriri, Baltussen and Oortwijn27).

Another important external contextual factor is influence by interest groups, including politicians, ministers of finance, industry, patients, and care providers. This has led to a wider range of stakeholders being involved in HTA processes and pressure to consider a broader range of criteria in decision making (Reference Pichon-Riviere, Soto, Augustovski and Sampietro-Colom28). Poorly implemented deliberative processes that do not manage these power dynamics can lead to distractions that dilute fruitful discussion (Reference Mohara, Youngkong and Velasco29). This creates ethical challenges for patients (Reference Vanstone, Abelson and Bidonde30) as well as risks of delay or poor decision making (see Box 4; (Reference Pichon-Riviere, Soto, Augustovski, García Martí and Sampietro-Colom31)).

There are also contextual factors specific to a decision about use of a health technology that is established during the deliberation about the technology itself (see Box 5).

Having an HTA infrastructure that supports transparency, participation of relevant stakeholders, and accountability of decision makers will be an important minimum requirement for enabling deliberation in any setting (Reference Kapiriri, Baltussen and Oortwijn27). We cannot assume that the implementation of deliberative processes automatically improves decision making across all jurisdictions equally. Variation in internal and external factors will shape and constrain implementation. Identifying factors that may hamper the effectiveness of a deliberative process enables planning to mitigate or overcome these barriers.

Ways of mitigating these barriers include policy statements on the willingness to use HTA in policy or practice, and development of transparent procedures for deliberations even if the outputs need to remain confidential, such as the communication of who is involved, their conflict of interest declarations, and how the process is undertaken.

There is an opportunity among countries with less-developed deliberative processes to start building them with features that take account of decision-making needs and environmental and societal constraints. Most low- and middle-income countries are in the early stages of developing HTA systems and can potentially incorporate learnings from the experiences of others (Reference Oortwijn, Jansen and Baltussen6). In contrast, most high-income countries have already established institutions and processes that may be more difficult to change (Reference Ten Have, Oortwijn, Broos and Nelissen33–Reference Boers34). This means that these countries will require approaches to evaluate their current deliberative processes and to identify where improvements might be made.

What Are the Guiding Principles?

Principles are intended to guide the design and conduct of deliberative processes. They can be substantive or procedural (Reference Clark and Weale35). Efficiency in HTA, for example, is an important principle where resources for HTA are lacking. Walton et al. (Reference Walton, O’Connor, Carroll, Claxton and Hodgson36) reported, for example, that the National Institute for Health and Care Excellence (NICE) judged the resources dedicated to discussing individual topics to be not efficient and unsustainable.

To be useful to decision making, principles should be consistent with the overarching goals of the HTA body or healthcare system that the deliberative processes are intended to support (e.g., equal access to care). Such goals are often multidimensional, complex, and value-laden. Values such as transparency, impartiality, and inclusivity were recently identified as important procedural principles related to deliberations about drug coverage decisions (Reference Bond, Stiffell and Ollendorf5). Other values may include timeliness, consistency, and verifiability (e.g., making recommendations that are timely and understandable) (14). Values can be differently applied to the various stages of an HTA process and may conflict. An example of the potential conflict between values of transparency and impartiality in developing drug funding recommendations is shown in Box 6.

The selection and application of principles and their related values may be affected by local contexts as different decision makers may have different goals, work in different sectors, and hold different attitudes toward (the need for) deliberation.

In some countries, technology producers implicitly determine the topics for assessment by submitting reimbursement decision dossiers to HTA bodies throughout the year, whereas in other topics are nominated and selected with the involvement of patients, payers, ministry of health officials, healthcare managers, and clinical experts (37).

What Should Be Documented and Communicated to the Public?

The need for a deliberative process, the guiding principles, and key elements for conducting and supporting a deliberative process as reflected in terms of reference, its outputs, and how they are used, as well as the monitoring and evaluation thereof (see sections below), need to be documented and communicated to the public. This documentation will also aid in sharing and studying what design features work best, and assist in promoting standards for HTA bodies in years to come.

Who Deliberates?

Participants in a deliberative process may include stakeholders, experts, or both. Some participants may be asked to play more than one role. For example, an industry representative is both a stakeholder affected by a decision, but may also have expertise in trial design and interpretation of evidence within a therapeutic context. The mix of participants and approaches to recruiting them should be carefully considered and will depend heavily on the tasks to be completed. Participation could take the form of membership of a standing committee where patient representatives have the task to represent their personal experiences or represent the wider experience of a group of patients. There might be ad hoc deliberative processes on selected topics in which a wider range of stakeholders (e.g., industry groups, healthcare payers, clinical experts, and patients) participate.

Other models of representative deliberation exist (e.g., consensus conference, citizen forum, and platform for representative deliberative democracy [G1000] (Reference Česnulaitytė38)), and may be more feasible or better for meeting the goals of the HTA process. In the Netherlands, a deliberative citizen forum has been used by the Zorginstituut Nederland (ZIN; the national HTA body) to obtain insight into the criteria-informed citizens would propose for public reimbursement (Reference Bijlmakers, Jansen and Boer39). Even within the same political context, participants involved in deliberation may vary according to what is being deliberated, and the goals and intended outcomes of deliberation (see Box 7).

The number of participants should be dictated by the goals and principles of a deliberative process. Although ten to fifteen participants have been suggested as an effective number for multidisciplinary advisory committees (Reference Moynihan, Oxman, Lavis and Paulsen40), others have stated, “there is no magic number that constitutes the ideal-sized group.” Trade-offs between quality and speed must be made (Reference Rothwell41). In the case of controversial HTA questions (e.g., when major ethical issues are raised by the health technology that is being investigated), a larger group—including all relevant perspectives—that takes more time to deliberate (and may have higher quality deliberation) may be more desirable for engendering trust and legitimacy.

In practice, HTA bodies operate with standing deliberative committees consisting of nine (ZIN in the Netherlands) to twenty-nine members (Haute Autorité de Santé [HAS] in France). HTA bodies should balance the need for representativeness with administrative burden and group effectiveness. The form and composition of the group and (if applicable) who has voting rights will also vary according to its mandate or legislation. Publicly stating who is participating, their interests and how conflicts will be reported will enhance the perceived legitimacy of the process. A common approach to support effective participation in deliberative processes of HTA bodies is asking all participants to declare their interest(s) prior to each meeting (Reference Wranik, Zielińska, Gambold and Sevgur42).

What Membership Arrangements Enable Effective Deliberation?

Although an ad hoc group may be suitable for the purposes of deliberating about HTA processes, deliberation within processes typically involves one (e.g., in Brazil and the Netherlands) or more (e.g., in Australia, France, and Thailand) standing committees. A hybrid form can also be used. For example, in Germany, patients and representatives appointed by the Conference of Health Ministers of the German States have discussion and petition rights on all agenda items of the Federal Joint Committee (standing committee) meetings. In addition, several associations have representation at meetings with participation rights on specific topics.

How Will Participants Be Selected?

Identifying and selecting participants (stakeholders and experts) can be accomplished through an open process and interviewing (as done in the Netherlands) or by targeting groups with informative perspectives (as done in Brazil). HTA bodies may use advertising or invitations to accomplish either approach.

How Are Perspectives to Be Represented?

Participants may represent the views of others or only express their individual views (Reference Fox and Shotts43). Depending on the topic(s) under discussion and the stage at which it is being deliberated, an individual stakeholder or expert could also be considered an active participant in deliberation, or alternatively, simply as a provider of information to support deliberation by others (i.e., through consultation). HTA bodies need to decide how to include harder-to-obtain or often underrepresented perspectives (e.g., from minority groups) and how these views inform the output(s) of deliberation (i.e., whose votes count and with what weight regarding the recommendation or decision).

Participation of patients as well as public/citizens in the HTA process is also increasingly being considered. Despite the differing perspectives offered by individuals in these roles (Reference Wortley, Tong and Howard44), there has been a tendency to consider patient and public/citizen roles together (Reference Werkö and Staniszewska45). There is also considerable uncertainty regarding how best to capture both perspectives, and efforts continue to establish best practices in this area (Reference Boothe13;Reference Abelson, Wagner and DeJean18;Reference Werkö and Staniszewska45). Challenges with patient involvement have been identified, and include ethical issues related to coercion and selection bias when only the most strongly motivated patient representatives participate. For example, a recent analysis of patient representation at the Canadian Agency for Drugs and Technologies in Health (CADTH) highlighted concerns about a lack of attention to preventing some patient voices from “dominating the process” over others (Reference Bidonde, Vanstone, Schwartz and Abelson46).

Public involvement may be an even more challenging perspective to represent. Some HTA bodies have used separate citizens’ juries or panels to inform expert discussion (Reference Bombard, Abelson, Simeonov and Gauvin47), whereas others have attempted to have the public perspective represented on advisory committees. For public involvement, factors identified as being critical ingredients to effective deliberation include (i) credibility, that is, an ability to contribute knowledge that is considered valid and relevant and that will result in mutual learning and generation of new solutions; (ii) legitimacy, that is, an ability to speak on behalf of others affected by healthcare decisions; and (iii) power, that is, an ability to influence healthcare choices (Reference Boivin, Lehoux, Burgers and Grol48). An exploration of public expectations in Australia revealed the public wanted HTA processes to “include a diversity of individuals, be independent and transparent, involve individuals early in the process, and ensure that public input is meaningful and useful to the process” (Reference Wortley, Tong and Howard44).

How Will Participants’ Identities Be Disclosed?

HTA bodies must decide if and how participants’ identities will be disclosed. Disclosing who is involved, their interests, and how conflicts will be decided and settled will enhance the perceived legitimacy of the process. HTA bodies should also consider that full transparency on these issues could lead to privacy concerns for some participants, including undesired media attention or pressure from interest groups (Reference Chwalisz49).

How Open Should the Deliberation Be?

HTA bodies must decide how open the record of deliberation should be. Deliberative processes can be open to the public (e.g., ZIN in the Netherlands and the Institute for Clinical and Economic Review in the United States), closed (e.g., HAS in France; prioritization working group organized by the Health Intervention and Technology Assessment Programme [HITAP] in Thailand), or a mix (e.g., the NICE in England). Although an open process will result in more transparency, HTA bodies should also consider that full transparency could lead to strategic behavior of some participants.

What Type and Length of Deliberation Is Needed?

The mode of deliberative processes may be virtual (e.g., in Brazil, Scotland, England, and the Netherlands), face-to-face, or both. Participants may exchange views at a single meeting, at several meetings, at the same time, or at different times. The value of face-to-face deliberation may improve communication among participants, although better communication must be balanced against the potential for undue influence by dominant participants to minimize poor decision making by groups (Reference Mendonça, Ercan and Asenbaum50). These can include nominal group techniques, consensus-building approaches, and expert elicitation techniques. However, these approaches are seldom used by HTA bodies (Reference Peel, Jenks and Choudhury51). HTA bodies should decide what level of rigor for deliberation is required and what is feasible given the available resources.

What Are the Rules of Deliberation?

Establishing rules of deliberation requires consideration of the goals of the process, who is participating, and how they will be engaged. Rules of deliberation can encompass both rules regarding how participants interact and share views, over what time frame, as well as rules for what specific information or topics can be discussed. The rules of deliberation are far more influential on affecting the outputs of deliberation than the participants involved or the information considered (Reference Janis52). The rules of deliberation may also be of particular importance when there are multiple deliberative forums and a perceived need for consistency in approaches.

For those deliberative processes intended to provide an opinion (e.g., advice or recommendation), rules are important regarding what procedure will be used to lead to the conclusion of deliberation. For example, for drug coverage decision making, consensus building (e.g., in the Netherlands), majority voting (e.g., in France and Scotland), or both (e.g., in Australia, Brazil, and England) are used by HTA bodies. Deliberative processes intended to result in an opinion may need to consider how disagreement is managed, which could include voting rules. In contrast, processes related to understanding (diverging) views, establishing expectations, creating narratives, and empowering patients and communities will more strongly emphasize participatory dialogue approaches.

How Will the Exchange of Viewpoints Be Facilitated during the Deliberative Process?

Power differences between participants, including gender, wealth, ethnicity, organizational seniority, and education, have been highlighted as challenges to effective deliberation (Reference Gibson, Martin and Singer53;Reference Shayo, Norheim and Mboera55). Addressing power differences requires, in part, an adequate exchange of views between participants. This is often done through a central agent (e.g., chair[s] or facilitator[s]) whose role is to identify preferences and underlying beliefs of all participants, confront others with these, and work toward a mutual understanding of the issue(s). Effective deliberation also requires participants to understand the process.

A significant concern is that participants may undergo social pressure to conform to the wishes of the group and leave the process unsatisfied (Reference Janis52). They may also be susceptible to cognitive or confirmation biases, that may be further subject to social influences (Reference Kao, Berdahl and Hartnett56). As decisions may be swayed by allowing one member of a group to dominate, the importance of rigorous and balanced facilitation or approach to exchange cannot be underestimated (Reference Kahan, Morton, Farris, Kominski and Donovan57).

What Information Will Be Made Available?

HTA bodies need to consider the amount and type of information made available to participants, when it will be made available, as well as how it is gathered, interpreted, and communicated. The information provided will depend on the HTA stage.

For example, for topic selection and prioritization at the NICE (England), all participants receive similar written information, and they are backed up by clear process documents or standard operating procedures (in addition to Terms of Reference). When contextualizing HTA, some countries provide evidence synthesis using structured guidance (e.g., the Grading of Recommendations Assessment, Development, and Evaluation (short GRADE), as used in Germany; https://www.iqwig.de/methoden/general-methods_version-6-0.pdf), whereas others provide a draft HTA report (e.g., Scotland and England), or provide a draft HTA report with the reactions and perceptions of stakeholders on the draft report clearly identified (e.g., in the Netherlands).

To ensure an appropriate amount of scrutiny, consideration should be given to presenting a complete package of information to all participants in a deliberation, rather than presenting redacted or interpreted summaries, which are subject to error and may reduce perceptions of legitimacy. This information could include original clinical studies in addition to a synthesis report (as done at the CADTH in Canada), an original economic model in addition to a summary report (as done at the NICE in England), or other relevant information (e.g., patient-based evidence gathered through interviews, clinical reviews, and assessment checklist results).

If specific criteria are used to guide a decision or recommendation, an HTA body may additionally provide standardized summaries or lists that address these. Checklists, similar to standards used in published research (Reference Simera, Moher, Hoey, Schulz and Altman58) but tailored to a specific jurisdictional context, may also be useful. To further reduce errors when interpreting or synthesizing information for participants, the use of standard analytic judgments (e.g., consensus guidance for analysis, such as in Australia) could be considered. Participants will also benefit from an overview of what information is missing as well as the limitations of interpreting available information.

What Information Will Be Considered by Participants and How Will Information Be Reviewed and Revised?

Just as allowing dominant viewpoints can be a threat to meaningful participation, so can the introduction of information that has not been equally considered by all participants. This can be remedied by creating enforcing rules regarding what information is allowable (i.e., not allowing participants to introduce new or modified, and potentially out-of-scope, information during deliberation).

How Will Information Be Presented?

Meaningful participation requires a clear and common understanding of available information and its interpretation (Reference Shayo, Norheim and Mboera55). Effective participant understanding of presented information, including scientific evidence, is facilitated by the use of appropriate media (written, verbal, or both), reporting structure(s), clarity and level of language used, and ensuring a mutual understanding of how the information is identified and synthesized.

In presenting complex, technical information, HTA bodies will need to consider levels of health literacy among participants and introduce appropriate supports (e.g., training or briefing sessions) to ensure exchange that is not hindered by avoidable misunderstanding. This could also include standardized approaches to producing lay summaries and allowing participants sufficient time to clarify misunderstandings prior to deliberation. In England, NICE’s Public Involvement Programme provides direct support and training to patient and carer consultee organizations, their representatives, and individual lay persons, patients, or carers. This enables them to contribute to the discussions during deliberative meetings (https://www.nice.org.uk/about/nice-communities/nice-and-the-public/public-involvement/public-involvement-programme).

What Specific Supports for Participants Are Available?

HTA bodies will need to consider if and what remuneration they might provide to participants in deliberation as this may lead to more willingness to participate and better engagement. They can follow governmental policies on financial compensation to serve at committees of public agencies, as done in Canada, Poland, the Netherlands, and England (Reference Wranik, Zielińska, Gambold and Sevgur42). Better engagement can also be facilitated through training and education to orient participants to the deliberative process as done by the National Committee Health Technology Incorporation in Brazil and the HITAP in Thailand.

Tools and checklists may aid effective deliberation. A checklist for HTA bodies to design their efforts for meaningful stakeholder participation has been developed (Reference Jansen, Baltussen and Bærøe12). Wale et al. (Reference Wale, Thomas, Hamerlijnck and Hollander59) also provide good practices more narrowly focused on patients and public perspectives and expertise in HTA deliberative processes. Examples include the participation of patient experts to answer questions during the contextualization process of the Scottish Medicines Consortium in Scotland. HTAi has also developed relevant tools and guidance to support patient organizations in completing a patient group submission template (https://htai.org/interest-groups/pcig/). In Australia, the International Association for Public Participation (60) and the Patient Voice Initiative have similarly developed standards and online tools for patient groups and communities when making submissions to the Pharmaceutical Benefits Advisory Committee’s (PBAC; https://www.patientvoiceinitiative.org/resource-library/). In the United States, the National Health Council has a range of resources to support patient input to “value assessment” including basic HTA training (https://nationalhealthcouncil.org/education/value-classroom/).

Are Comprehensive Terms of Reference in Place?

To enhance public trust and be transparent about the key aspects of (conducting) a deliberative process, comprehensive terms of reference should be created and be publicly available. A term of reference defines the purpose, scope, outcomes, and structures of a deliberative process. We recommend that questions labeled with an asterisk “*” in the checklist (Table 1) are addressed in such a document and reviewed periodically (e.g., every 3–5 years).

How Will the Process and Output(s) of a Deliberation Be Reported and Communicated?

Documenting the process and output(s) of deliberation is the responsibility of the organization that is establishing or managing HTA processes and can be facilitated through video (livestreaming or published recording) and by taking meeting minutes. Meetings can be reported back to all relevant stakeholders in entirety or as a summary, for example, by using a standardized format that includes an explicit description of the issue, the information used, and the views and argumentation that have been put forward. A rapid communication of the output(s) (e.g., recommendation) to an organization submitting a health technology dossier is an important part of processes where appeals or resubmissions are an option (see below). On the other hand, rapid, abbreviated reporting can lead to information loss of specific considerations (Reference Boivin, Lehoux, Burgers and Grol48). Providing a complete record can facilitate a better understanding of situation-specific considerations that may arise within deliberation for all stakeholders not involved in the deliberation.

How Will the Outcomes of Deliberation Be Reconsidered?

Possible mechanisms for reconsideration include appeal, revision, and additional deliberation. The scope and process related to any of these mechanisms may vary depending on legislative or regulatory frameworks. Some HTA bodies, such as the Health Insurance Review and Assessment Service [HIRA] in Korea (Reference Bae, Hong and Kwon61), the NICE in England, and the PBAC in Australia have established mechanisms or guidance for lodging an appeal (see Box 8 for the process at the NICE).

Appeal refers to a mechanism that gives stakeholders the possibility to apply for a revision of a recommendation, involving arguments offered by the individual or group making an appeal, and leading to a reasoned response (Reference Sibbald, Singer, Upshur and Martin62). In designing appeal mechanisms, it is important to be explicit about what constitutes grounds for appeal and/or revision, the characteristics of the body in charge of the appeal (e.g., whether it is independent and the members of the appeal committee) (Reference Bond, Stiffell and Ollendorf5), who can or cannot appeal, how the appeal will be considered and decided, and the timelines involved.

For deliberative processes that do not result in a decision, a review and revision of viewpoints, including additional deliberation, may be warranted, particularly if new information from external experts and stakeholders is developed and presented.

How Will the Final Output(s) of Deliberation Be Communicated?

A communication strategy should include what, when, and how the output(s) will be communicated and to whom, and be publicly available. Such a strategy could be aimed at communicating the output(s) of deliberation and how it was arrived at. Some suggestions for reporting are provided in Box 9.

The responsible organization should make efforts to ensure that output(s) of deliberation are well communicated and target all relevant stakeholders, using a variety of channels and approaches. These may include the use of official documents (e.g., an official journal), Web sites of relevant organizations, policy briefs, newsletters, and news items on social media to address the broader public.

Consideration may also need to be given to the kind of information and different levels of detail for different audiences. For example, a health technology developer may expect a greater level of detail related to the assessment of evidence and uncertainty, in order to plan for any resubmission or appeal, whereas a patient may be more interested in the implications of the decision for access and possible next steps in that direction.

Is the Desired Change(s) from Implementing a Deliberative Process Established and Does It Align with the Health System Values or Principles Established and/or with Those of the HTA Body?

HTA bodies should start with a comprehensive description of how a desired change is expected to happen from a deliberative process in their context, including the resources (inputs) needed along with the rationale for such change. HTA bodies may assess whether the health system values and guiding principles of deliberation are upheld. The outcomes of a deliberative process could be assessed to evaluate the extent to which the desired change has been established.

How Will the Desired Change(s) from Implementing a Deliberative Process Be Measured?

Questionnaires, interviews (Reference Mercer, Chambers and Mai63), document reviews, or live meetings held with participants are some mechanisms to evaluate deliberative processes. The HTA body can ask for feedback on the process of how the evidence report has been developed, and how experts and stakeholders perceived the process in terms of fairness, transparency, and timeliness, as well as the level and impact of participation (Reference Oortwijn, Determann, Schiffers, Tan and van der Tuin64).

HTA bodies may also consider assessing the outcomes of deliberation that are tied to the broader aims of HTA, such as distributional justice in health care or overall improved health (Reference Bond, Stiffell and Ollendorf5). It is important to note that certain outcomes may not be achieved immediately, not because the processes are ineffective but rather due to a time lag for measuring the impact after implementation. It must also be recognized that both process (inputs and activities) and outcome measures are interrelated, with the inputs and activities mainly informing short-term success, whereas the outcomes reflect longer-term impact.

How Will the Desired Change(s) from Implementing a Deliberative Process Be Assessed?

Monitoring and evaluation require considerations of who does the assessment, how it can be done, and what tools can be used. In addition to involving participants in deliberative processes, there are advantages to seeking an external party, such as an independent evaluator, who may provide an unbiased perspective to the evaluation. Who to involve largely depends on how monitoring and evaluation are done, whether monitoring is done on an ad hoc basis, or more routinely and embedded within a broader framework for evaluating other components of HTA. The monitoring and evaluation approach can be integrated into an existing monitoring and evaluation framework of the HTA body. The HTA body may consider developing or adapting established tools, such as checklists or questionnaires for eliciting feedback. Alternatively, using qualitative methods, such as focus groups and in-depth interviews with key stakeholders either directly or indirectly involved in deliberative processes may be considered for ad hoc assessments.

What Indicators Will Be Used to Monitor and Evaluate the Deliberative Process?

In Table 2, we suggest indicators, stakeholders to be involved, and methods for monitoring and evaluation of deliberative processes. These are by no means comprehensive. It is recommended that those leading the monitoring and evaluation adapt them and develop context-specific measures for each, which may be qualitative or quantitative. The process by which these are defined may also be done through deliberation.

Table 2. Monitoring and Evaluation of Deliberation in health technology assessment (HTA)

Notes. Researchers refer to those generating or synthesizing evidence for HTA. Evidence users are policy makers considering HTA evidence.

^a Stakeholders include those interested in the HTA process and/or HTA-informed policy decisions with or without direct involvement in the HTA process.

^b General public refer to members of society who have no special role in HTA, but may benefit from the outcomes of the processes.