The Cooperation Framework

JA3 was a voluntary network with time-limited project funding, joint HTA activities were undertaken by a work package made up of a group of agencies and each activity had an agency that was the activity lead and who was responsible for the final output from the activity. Five learnings related to the framework needed to underpin HTA cooperation were identified from the JA3 experience.

1. 1. An ability to act as its own entity. To function optimally the cooperation must be able to approve its outputs, be liable for its outcomes, own its infrastructure, and create official agreements with other organizations.

2. 2. A resourcing and commissioning model that guarantees projects and project teams. Voluntary participation provides flexibility around involvement but can affect the ability to constitute project teams, the ability to undertake prioritized work, the clarity about uptake of outputs, and the ability to implement templates and procedures.

3. 3. A set of guiding principles and processes applied across all activities. The work package structure provides flexibility in how joint HTA activities are undertaken but can allow differences in approach across different types of joint HTA activity to grow. Where differences have no clear rationale, this can affect the ease of engagement and perceived coherence of the cooperation.

4. 4. Ongoing stability. Time-limited project funding can lead to gaps between projects without resourcing. The absence of continuity creates staffing issues since staff leave toward the end of each project when their contracts expire. It can also create issues with stability of infrastructure hosting and maintenance when project responsibilities end and can influence planning and usability of outputs as all activities must be finished at the end of the project.

5. 5. Flexibility to evolve. In JA3, changes needed to be made to working practices to respond to demands arising from the COVID-19 pandemic (5). HTA cooperation must have the flexibility to respond to feedback from participants and changes in the external environment that affect the outputs and activities that will be valued by users.

Valued HTA Activities

Three main joint HTA activities were carried out in JA3.

1. 1. Early Dialogues (ED): that is, advice provided by multiple HTA agencies to technology developers about their evidence generation plans.

2. 2. Joint and Collaborative Assessments (JA/CA): that is, relative effectiveness assessments carried out jointly by multiple HTA agencies.

3. 3. Post-Launch Evidence Generation (PLEG): that is, HTA activities undertaken jointly after a health technology is launched to support further evidence generation to address uncertainties in the evidence at the time of initial launch.

The experience of JA3 is that each of these activities are valued activities that should be part of ongoing HTA cooperation. PLEG is a less established joint HTA activity compared with ED and JA/CA, but PLEG activities are becoming ever more visible and the profile of PLEG will continue to grow (Reference Moseley, Vamvakas, Berntgen, Cave, Kurz and Arlett6).

In addition to the activities above, JA3 partners also identified a need to expand the joint HTA activities undertaken to include horizon scanning and re-assessment.

1. 1. Horizon scanning systems identify emerging health technologies and future technological and methodological challenges. Horizon scanning helps HTA processes be prepared for and respond to methodological and technological innovation.

2. 2. Processes to update the outputs of joint HTA activities are needed to identify when an output might be out of date and need updating and to ensure that an HTA remains of value to decision makers after it is published.

Finally, JA3 identified that there should be explicit and transparent links between joint HTA activities on the same topic, so that activities can inform each other and capitalize on the joint HTA activities already undertaken.

The valued joint HTA activities are summarized in Figure 1.

Figure 1. Joint HTA activities and the links between activities along the life cycle of a health technology.

Principles of Involvement

In JA3, agencies who were users or producers of HTA were engaged as partners. Joint HTA activities were also supported by stakeholders, including industry, patient organizations, providers, and payers. Emerging from JA3 are a set of involvement principles that support engagement in HTA cooperation.

1. 1. Participation should be guided by underlying principles applied across all activities and a framework for engagement. Differences in approach to participation in different HTA activities should have a clear and justified rationale.

2. 2. Participants should be engaged in specific joint HTA activities but also need to be provided with general information updates and consultation in decisions that will affect them.

3. 3. The structures for collaboration need to be as simple as possible. Complicated and burdensome structures can act to decrease the possibilities of cooperation because of the challenges engaging with them.

4. 4. Anybody directly participating in a joint HTA activity (e.g., in the project team or as an expert) should provide up-to-date information about their interests. The declaration of interest for each participant should be assessed before any involvement starts. Normally, conflicts of interest should be avoided.

5. 5. When designing joint HTA activities and preparing supporting resources:

   • • Cooperation structures, methods, and tools across activities should be aligned where possible to make participation easier.

   • • All participants should have an overview of the role, responsibilities, and expectations of participation.

   • • Participant documents should be harmonized so that they are easier for participants to become familiar with, encouraging ease of use, understanding of activities, and avoidance of procedural errors.

   • • There should be publicly available guidance on involvement.

6. 6. Relying on a single method of communication such as an intranet may not reach all relevant participants. Multiple methods of communication should be used.

HTA Agency Participation in HTA Cooperation

HTA cooperation relies on the agencies in the cooperation to undertake activities and to use the outputs created. The number of HTA agencies involved in EUnetHTA has grown over time, starting with thirty-five in the EUnetHTA Project 2006–8 (Reference Kristensen, Mäkela, Neikter, Rehnqvist, Håheim and Mørland7) to eighty-one agencies in JA3. Over time, EUnetHTA has experienced how HTA responsibilities in a country change, new agencies are set up, and existing agencies change and move organizations as the health care and political environment changes. Having experienced the growth and change in EUnetHTA, there are four learnings from JA3:

1. 1. HTA cooperation needs to be able to allow more than one agency per country to participate. This is to reflect the variations in how HTA is undertaken in different countries and the need to ensure inclusivity and that all relevant perspectives are captured.

2. 2. A large cooperation will require significant time and resources to manage and coordinate it to ensure ongoing engagement of all participants.

3. 3. Participation in the cooperation needs to be underpinned by transparent criteria about who needs to be engaged, based on their responsibilities and remit rather than their organization title.

4. 4. A simple mechanism to add, remove, and change participants in “real-time” is necessary so that the cooperation continues to reach out, engage, and meet the needs of the agencies it serves.

Governance of the Cooperation

In JA3, operational decisions were made by an Executive Board made up of HTA agencies who were leads and co-leads of the work packages and elected JA3 partners. Part-way through JA3 changes to the Executive Board were needed to improve its ability to make decisions.

Arising from this JA3 experience are the following learnings:

1. 1. Bodies making decisions on behalf of the network, need to:

   • • follow a set of shared and transparent principles,

   • • have flexible membership that can be configured differently for different types of health technologies,

   • • have a membership that is representative of the members HTA approaches,

   • • have a process for assembling the decision-making body that is transparent and inclusive and open to all HTA bodies,

   • • have clear communication structures with information cascade to participants that are not part of the governance and decision-making group.

2. 2. The decision-making body cannot function alone and need to have access a range of resources and personnel to support decision making, including:

   • • a Secretariat to provide administrative and coordination support;

   • • a chair and vice chair to work with the Secretariat and to lead meetings;

   • • expert groups to work and advise the decision-making body on scientific and procedural issues;

   • • a comprehensive set of corporate governance policies with the rules and procedures guiding the cooperation.

3. 3. Governing and decision-making structures need to be subject to evaluation and have sufficient flexibility to change to ensure that they continue to function optimally as the cooperation and its work evolves.

A summary of needed elements of a governance structure is shown in Figure 2.

Figure 2. The support needed by the decision-making body to function optimally.

Project Teams

Different configurations of project teams have been trialed in different EUnetHTA joint actions and activities. At the beginning of the project, there were large teams of partners involved in each activity, by the end of JA3, project teams had become smaller and now have greater similarity across different HTA activities. Based on this evolution in project teams, the following learnings emerged.

1. 1. Each joint HTA activity should be guided by the following recruitment principles:

   • • The project team must have the relevant skills and experience for the activity.

   • • Selection criteria in terms of skills and experience should be transparent and available.

   • • The procedure for selecting the team should be transparent, so it is understood how the skills of team members are evaluated and by whom.

2. 2. Joint HTA activities do not need to be undertaken by a large project team. It is normally sufficient for a project team to include:

   • • an HTA agency that leads the activity;

   • • a second HTA agency that supports them;

   • • additional project team members who provide input and review;

   • • dedicated project management to manage the activity and the project team.

3. 3. Adequate activity management is critical when managing different teams of people undertaking different and multiple activities. Each joint HTA activity needs to have a project manager who is part of the project team and who acts as a dedicated point of contact. The project manager must have access to scientific experts to ensure that the scientific elements of activity management, for example, judging eligibility of requests, quality checking of the output, and managing differences of opinion are carried out appropriately.

4. 4. To support a high-quality joint HTA output, it is necessary that project teams are not working in isolation and are able to access:

   • • scientific and methodological expert advisory groups, for example, information retrieval experts and statistical specialists;

   • • a common set of scientific tools to complete the activity;

   • • a range of centralized support services, for example, publishing and editing, IT, and stakeholder engagement;

   • • scientific oversight provided by a standing group of experts to ensure consistency in approach within a joint HTA activity;

   • • independent conflict resolution;

   • • corporate governance support, for example, to manage conflicts of interest.

   • • input into the project plan from the target users.

Experts and services providing guidance to project teams should be involved early in the process from the planning stages so that issues that could affect approval are identified in a timely manner and can be resolved without creating process delays.

A summary of the needed elements to support project teams is shown in Figure 3.

Figure 3. The support needed for project teams to produce joint HTA outputs.

Engagement of External Actors

JA3 involved a variety of stakeholder groups but had a particular aim to improve the involvement of experts in joint HTA activities and the collaboration with the regulatory authority the European Medicines Agency (EMA).

The following are key lessons from JA3:

1. 1. HTA agencies vary in their approach to and understanding of stakeholder engagement and this is an area with divergent opinions. The opinions are particularly divergent in regard to the involvement of industry and payers and the involvement of stakeholders once the outputs from joint HTA activities have been drafted but are not yet finalized.

2. 2. Patients and healthcare professional input should be routinely included in an HTA activity.

3. 3. Stakeholder engagement in topic identification and project planning is important and appropriate.

4. 4. Stakeholder and expert contributions should be supported and recognized:

   • • The contribution that stakeholders and experts provide in a joint HTA activity must be transparent and documented in the output.

   • • Guidance must be given to stakeholders about the input expected.

   • • Guidance must be given to project teams about using and reporting the input.

5. 5. Collaboration with regulators such as the EMA adds value and is able to respect the differences in the remits and responsibilities of HTA agencies and the regulators.

6. 6. HTA cooperation must link to the environments with which it interacts and pro-actively and strategically work to collaborate with related organizations and networks.

Support Services

In JA3, some support services (e.g., stakeholder management, information technology (IT), editing, communications) required to carry out joint HTA activities and coordinate cooperation were provided centrally and others were decentralized and carried out by work packages or individual HTA agencies.

A learning from JA3 is that support services are fundamental to maximizing cooperation and producing high-quality outputs and that these are more effectively, efficiently, and consistently delivered if centralized. This is because of the need for a harmonized approach and/or specialized skills. Particular areas where greater centralization is beneficial include:

• • Stakeholder management and expert engagement with a particular focus on patient engagement.

• • Conflict of interest: a centralized committee to assess potential conflicts of interest and a central support function to manage declarations of interest made by project teams and experts involved in joint HTA activities.

• • Information services: the foundation of HTA is scientific literature, a centralized service to identify, procure, and share literature in accordance with copyright law supports scientific rigour.

• • Editing and publishing: centralized editing and publishing supports a consistent “house style” which is important for users of reports.

• • Activity coordination: for example, organization and planning of meetings, liaison with collaborating organizations, and submission and exchange of documents;

• • Scientific software: for example, software to support information retrieval, evidence synthesis, and statistical analysis. Tools are not necessarily interoperable and so where participants have to use their own software this can create a challenge to collaborate.

• • Quality Management: a consistent approach to quality management across activities improves engagement and perceived coherence (Reference Luhnen, Ormstad, Willemsen, Schreuder-Morel, Helmink and Ettinger8).

A summary of the support services required to support HTA cooperation is shown in Figure 4.

Figure 4. The support services needed for an optimal cooperation. IT, information technology.

Training and Development

The main aim of JA3 was to increase production of joint HTA outputs. This focus on increased production meant that less resource was given to methodological development and training compared with previous joint actions.

The experience of JA3 has been that there remains a need for ongoing support in three areas: training, science development, and capacity development and implementation support.

1. 1. Training: JA3 has shown that we are not yet at a point where HTA agencies have sufficient staffing for joint HTA activities to be fully sustainable. HTA cooperation must always ensure that the pool of staff available to be in project teams is sufficiently large. To support sustainability, there needs to be

   • • ongoing training to ensure that pools of expertise are expanded and maintained; and

   • • opportunities for observer status in joint HTA activities and on advisory bodies to support up-skilling of participants to take on greater responsibilities.

2. 2. Science development: An ongoing program of scientific development activity is needed to ensure that new areas requiring technical guidance are identified and guidance developed before they become an issue for joint HTA activities. Existing templates and guidance need to be kept up to date as methods change.

3. 3. Capacity development and implementation support: HTA is still establishing in some countries and these HTA agencies should be able to access support for capacity development and implementation to maximize the benefits they obtain from HTA cooperation.

Information Technology

The project nature of EUnetHTA has meant that IT has grown in a piecemeal fashion with IT developed in different project phases, by different agencies and using different software.

At the end of JA3, partners recognized that the piecemeal development of IT over the joint actions has affected usability and ease of engagement for participants and the ability of the HTA cooperation to maintain and develop its IT efficiently. IT is fundamental to supporting efficient and effective HTA cooperation. More specific learnings from JA3 were:

• • IT should be managed as a single centralized function that supports all IT needs of the cooperation.

• • Different IT tools should be integrated to support the smallest number of different tools required for necessary function. For example, a single tool used to manage all joint HTA activities.

• • To support efficient maintenance and development, IT tools need to be developed consistently using the same software, programming, and hosting.

• • IT tools should ideally all use the same access credentials so that participants do not need to maintain multiple user identities and passwords.

• • IT requires ongoing and stable maintenance and hosting to maintain participant engagement.

• • IT tools and infrastructure must be subject to a regular review and evaluation to help ensure that they continue to meet needs.