Defining and Prioritizing Operational Innovations

While establishing Network infrastructure, TIN leadership discussed the definition of operational innovation. The definition agreed upon was “A novel technology, method, process, or paradigm that can be used to improve the conduct of the clinical trial or study and/or its ultimate speed or likelihood of translation into practice.” The addition of the word operational is important as it makes operational innovation into an umbrella term that can include the full range of resources and activities that are developed to improve trial efficiency and function. This definition aligned thinking and served to underpin conversations about how best to develop and test operational innovations across the TICs and RIC.

At an in-person meeting of all TIC and RIC Principal Investigators (PIs) and Project Leads (PLs) in 2018, attendees were asked to identify the key activities important to improving the success of clinical trials. The conversation was captured, and the 39 responses provided were coded into seven categories: optimization of sites and site selection; acceleration of study startup; community involvement; development of technical resources; education and training; trial management; and trial design.

This began a focused discussion regarding priority innovations that continued in early 2019, with another in-person meeting that included a reminder of the categories identified at the 2018 meeting and a vote by PIs and PLs to prioritize related areas for innovation. The five areas chosen were as follows: Engaging Participants at the Operational Level of Clinical Trials; Study StartUp with Site Incentives and Gamification; Site Assessment; Pragmatic Trials with Limited Variables; and a Next Level Master Protocol.

Mapping and Sharing Operational Innovations

The 2019 meeting was also the start of a TIN Work Group that met to map operational innovations across the network in order to: position the network to review and consider ongoing work and plan future innovations; ensure innovative work was aligned across the TICs and RIC; allow the network to highlight innovations developed by the TIN internally and externally; and lay the groundwork for measuring the impact of innovations. The innovation development pipeline imagined by the Work Group is illustrated in Fig. 2.

Figure 2. The Trial Innovation Network (TIN) innovation development pipeline began with known roadblocks and barriers. Innovations were developed, implemented in multi-center clinical trials, and then disseminated via a variety of communication pathways. Implementation and dissemination phases of the pipeline are pictured with continuous improvement cycles to indicate iterative improvement incorporating lessons learned.

The mapping exercise culminated in a repository of operational innovations derived from the literature, past experiences of TIC and RIC institutions, and work done at the TICs and the RIC and developed and/or implemented within the TIN. This repository was built for internal use to enhance transparency and ensure work across the Network was aligned and complementary. An overview of multi-center study roadblocks and barriers, TIN activities to support operational excellence in these areas, innovation categories, and innovations within each category is listed in Table 1.

Table 1. An overview of multi-center clinical trial roadblocks and TIN activity and operational innovation to address these roadblocks

CTSA = Clinical and Translational Science Award; FDP = Federal Demonstration Partnership; TIN = Trial Innovation Network; REDCap = Research Electronic Data Capture.

To inform the development of the mapped information, a survey was distributed to 49 people including all TIN PIs, NCATS and Liaison Team Leadership, and all TIN Work Group PLs (response rate 90%). Respondents selected the term “Innovations Catalogue” to refer to the initiative, agreed on inclusion criteria and information to be contained within the catalogue, and suggested ways of incorporating the catalogue into existing TIN systems and practices. The qualitative themes that arose were: the importance of framing discussions to be inclusive of all innovations that are used to improve the conduct of clinical trials; support for doing a phased internal roll-out of the catalogue; balancing procedure and over-complexity; designing systems for implementation; focusing on TIN priority areas; and establishing a process to support innovation evaluation.

The Innovations Catalogue was implemented in a TIC- and RIC-wide accessible format in 2020 and presented to TIN leadership and PLs. In 2021, the Innovations Work Group presented again, this time on reframing the approach to innovation to be more inclusive of operational excellence. The operational innovation definition agreed to by TIN leadership was not changed; however, the Work Group wanted to be explicit in creating a more inclusive space for processes that might not be novel but had the potential to improve the conduct of clinical trials. The intention was to include valuable contributions (e.g., technologies, methods, processes, or paradigms) to clinical trial execution. While this broadened an already inclusive definition, operational innovations included in the catalogue were all focused on addressing specific barriers or inefficiencies and were tied to operational hypotheses that could be tested.

Implementing and Disseminating Operational Innovations

Operational innovations that moved from the idea stage through development and into implementation were defined as mature. Mature innovations were presented at cross-consortium Liaison Team meetings, shared with study teams in TIN consultations, published in manuscripts and white papers, and included in the publicly accessible TIN website Toolbox. A new method of identifying trials that included innovations was also developed. These trials were identified as Testing Operational Hypotheses trials and served to test innovations with the goal of improving clinical trial conduct and/or efficiency. Innovations were embedded in multi-center trials as an operational experiment, with expenses and required personnel covered by the TIC or RIC that developed the innovation.

Operational innovations were implemented either at the initial TIN consultation stage, when investigators had meetings with the TICs and RIC to discuss their support needs (e.g., Design Labs [Reference Palm, Selker, Cohen, Lerner, Palm and Concannon12]) or within TIN-supported trials (e.g., Consent Builder [Reference Sward, Enriquez and Burr13]). The Catalogue included the innovation stage – design, development, or implementation – and the trials within which the innovation had been implemented.

Examples of implementation included a survey used to aid site selection (#7 in Table 1), a program to streamline informed consent across multiple sites (#9), the ability to connect trial data collection with electronic health records (#36), and systems implemented to manage risk (#31). Across time, as innovations matured, the number of operational innovations implemented within and across trials increased. Fig. 3 shows the number of innovations implemented, excluding innovations that are implemented in ways disconnected from trial design and conduct, for example, courses and templates that cannot be tracked [Reference Cunningham-Erves, Kusnoor and Villalta-Gil14,Reference Stallings, Boyer and Joosten15].

Figure 3. Split panel depiction of Trial Innovation Network (TIN) innovation implementation during the first TIN funding cycle 2016-2023. The left-hand side (3a) shows the number of innovations implemented, with lines indicating implementation at different stages of the clinical trial life cycle. The right-hand side (3b) showcases the number of Institutional Review Board (IRB) innovations as operationalizing single IRB (SIRB) was an early identified TIN objective.

Operational innovations were implemented within more than 110 TIN trials and over 40 disease areas, with some trials including a single innovation (e.g., SIRB coordination) and others including multiple innovations (e.g., accelerated startup program, gaming and incentives, one-part vs. two-part consent documentation, and streamlining local context review). While some of the innovations were implemented at the design stage pre-funding, the majority were implemented following grant submission and successful receipt of funding. Funders included NIH institutes and centers (e.g., NCI, NHLBI, NIAID, NIDDK, NINDS, and NCATS), PCORI, DOD, industry, and foundation grants.

Evaluation is ongoing, and metrics have been published for some innovations, while others are still being tested [Reference Palm, Selker, Cohen, Lerner, Palm and Concannon12–Reference Kusnoor, Villalta-Gil and Michaels26]. A TIN Work Group established to develop metrics agreed upon 10 priority metrics and data elements to measure clinical research study performance across the Network. These metrics are related to study startup, regulatory approval, the contracting process, patient accrual, and data quality. While further details are outside of the scope of this brief report, it is worth noting that the TIN is in the early stages of collecting cross-network metrics, with a goal to evaluate effectiveness of the innovations developed and implemented.

Facilitators, Barriers, and Lessons Learned

The process of defining and mapping operational innovations was instructive. An expert network of clinical trialists from five different organizations, each with their own specific environments and operational processes, came together to agree on a definition and structure for categorizing and describing operational innovations. These spanned the clinical trial life cycle, from study design through recruitment, and study conduct. It became clear that significant work was being done in complementary areas, and that alignment among them was important.

An example was work being done to support informed consent. All organizations involved in the TICs and RIC were undertaking activities related to supporting the informed consent process. These were complementary, ranging from support for streamlining the technical process of creating and updating informed consent forms, to informed consent training methods, to the development of key information sheets that are understandable and actionable [Reference Sward, Enriquez and Burr13,Reference Markman, Weicker, Klein and Sege27]. The innovations in this area were developed into a toolkit that can be used flexibly in a fit-for-purpose way depending upon study needs [28].

The operational innovations developed and/or implemented within the TIN were designed to be broadly useful to multi-center clinical trial design and conduct. However, it is worth noting that there are elements of clinical trials that are determined by the funding source or sponsor and their requirements. This information should always be a reference point and may have an impact on the decision to implement an innovation.

Barriers to the implementation of innovations were time, cost, and concern about the innovation’s potential disruption of trial processes. The importance of early engagement with investigators to ensure their interest and support was a lesson learned. Efforts were made to guarantee operational innovations were included in a way that did not require additional PI time or expense and that innovations enhanced, rather than disrupted, trials.

The operational innovations prioritized in 2019 varied in the success of their implementation. Although study startup with site incentives/gamification and site assessment were both operationalized, implementation of a pragmatic trial with limited variables and Next Level Master Protocol has been a slower process. Engaging participants at the operational level of clinical trials was the most difficult innovation to implement given the upfront time and ongoing effort that would be required by trial investigators. Fewer foundational and more operational innovations found support and were likely to be incorporated into trials.

Future Directions

Developing and implementing innovations that address roadblocks and barriers to successful and informative clinical trials is essential to improving clinical trial quality and conduct. The development and implementation of innovations in the TIN have been extensive; however, there is still significant work to be done. Three areas for continued development are the implementation of foundational innovations, measurement of outcomes, and broad dissemination. The lessons learned in the first TIN funding cycle should inform future development in these areas.

A process for mapping innovations and tracking their implementation has been developed. These data were used to identify gaps and areas of complementarity and to prioritize future efforts. To date, implementation has been limited to operational innovations, while foundational innovations have been more difficult to implement. It may be useful to consider a separate pathway for foundational innovations that may not be easily embedded in a clinical trial. Work Groups that address the barriers and facilitators to implementation of foundational innovations, and that include stakeholders and policy makers, may help develop more successful pathways.

Establishing a national network, with aligned and efficient processes, is a significant effort. We developed processes for embedding innovations and for capturing priority metrics. However, we are only in the early stages of implementing and testing innovations at scale. We will need time and a focus on iterative process improvement to ensure that we capture the necessary information and develop appropriate statistical analysis plans.

Finally, although some pathways for dissemination have been developed, it is essential that they be expanded and that investigators are provided with support to implement the relevant innovations in their trials. This may mean an expansion of the TIN toolbox, continued work with partners at CTSA hubs to understand support needs, and internal and external meetings to highlight further areas of innovation. Dissemination and implementation methodologists could also be helpful in considering strategies for impactful and broad dissemination.