Rationale for Novel Curricular Approach
Despite increasing emphasis on implementation research (IR), experts and traditional training opportunities are in short supply [Reference Ford, Rabin, Morrato and Glasgow1,2], highlighting the need for innovative training approaches. Bootstrapping refers to a self-starting process to better oneself in the face of limited resources. In a paradigm shift, four (4) Texas CTSAs used this novel approach to create capacity in the face of minimal internal expertise and limited IR resources. We sought to provide and evaluate a high-quality regional training program to increase IR scientific workforce capacity and enhance grant success. This paper discusses how generalizable best education practices and adult education methods in professional training were applied: document the educational need; establish competencies as learning outcomes; provide ongoing educational resources; use active instructional strategies; and determine impact using an evaluation model.
Unmet Need for Educational Gap
Although IR is not a formalized element of the CTSA Program, it is recognized to be essential in the translational science enterprise to move research into practice and enhance population health [Reference Liverman, Schultz, Terry and Leshner3,Reference Leppin, Mahoney and Stevens4] and is included in the National Center for Advancing Translational Sciences translational science spectrum [5]. As a nascent field, the nature of IR itself has been recast to ensure actionable and relevant findings [Reference Meissner, Glasgow and Vinson6]. This resulted in the emergence of new paradigms, theories, designs, and methods. This fact turns even accomplished researchers into IR novices who seek training in these new methods. Nationally, only half of CTSAs surveyed reported IR training efforts, funded resources, consultation, or research projects; barriers noted were lack of expertise, training, and tools/methods [Reference Dolor, Proctor, Stevens, Boone, Meissner and Baldwin7]. At the time of the Workshop, no Texas CTSAs housed IR cores/centers, although pockets of expertise existed.
Target Audience
The Workshop was designed for 30 faculty scientists and clinical partners from Texas CTSA hubs. To assure broad representation, faculty scientists from all four Texas CTSAs were invited. Participants registered in advance and completed pre- and post-Workshop activities. Interest was high and Workshop capacity was expanded. Sixty-three participants represented the four Texas CTSAs, multiple disciplines, institutional and military clinical partners, and research support administrators. Ninety-eight percent of participants held a doctoral degree.
Description of the Educational Method and Curricular Program
Because terminology in IR is not yet distinct, the planning group developed the following amalgamated definitions to promote common understanding for the initiative:
-
Implementation science (IS) is a specialized field that addresses uptake of evidence-based practices into everyday care to improve health and health care. This science is built through IR studies.
-
IR builds IS through the scientific investigation of methods and strategies that promote systematic application of research findings in routine clinical practice.
The Community Engagement core at the Institute for Integration of Medicine & Science at University of Texas (UT) Health San Antonio led the design, resourcing, and conduct of an educationally sound 2-day Workshop with its institutional and statewide CTSA partners. Perspectives from multiple disciplines were included in the planning committee.
Workshop planners combined the newly released National Institutes of Health/National Cancer Institute (NIH/NCI) OpenAccess IR training materials [2] with in-person internal and external IR experts to design interactive interprofessional learning activities for maximum impact. Using the NIH/NCI online program as the foundation ensured quality, currency, and generalizability of the Workshop. Funds were used to waive registration fees, support guest speakers travel and honoraria, print Workshop materials, and provide refreshments and lunch for working sessions.
The program was based on an established national curriculum [Reference Tabak, Padek and Kerner8], national consensus on dissemination and implementation (D&I) research competencies [Reference Padek, Colditz and Dobbins9], and recently published NIH/NCI OpenAccess training materials [2], offering a standardized frame. This approach provided two advantages: generalizability beyond the single training opportunity, and alignment with established IR standards to advance the field. The Workshop targeted basic-to-intermediate level competencies with these goals: 1) build interprofessional scientific capacity among teams of faculty and clinical partners to design and conduct relevant, rigorous, high priority studies; 2) enhance success in competing for NIH, Agency for Healthcare Research and Quality, Department of Defense, and Patient-Centered Outcomes Research Institute IR grants; and 3) position UT System research partners as IR leaders for healthcare improvement.
To engage faculty scientists, we employed dissemination strategies [Reference Glasgow, Vinson, Chambers, Khoury, Kaplan and Hunter10] and marketing approaches. A logo “branded” the Workshop made materials quickly identifiable and created a shared mental model of Workshop purpose, with “adoption” as the core idea. Initial logo designs were vetted by the planning group. The final logo (Fig. 1) contained built-in meaning, depicting “adoption.” [Reference Rogers11]
Fig. 1. Workshop logo.
Measurable learning objectives (Table 1) aligned with the NIH/NCI curriculum and guided selection of content, topics, and speakers (see details at https://iims.uthscsa.edu/sites/iims/files/Comm_Engagement/Workshop%20Report%20FINAL.pdf).
Table 1. Learning objectives guiding the training Workshop curriculum
We applied proven instructional design principles [Reference Moore, Chappell, Sherman and Vinayaga-Pavan12,Reference Hew and Lo13] to achieve the learning objectives. Activities included presentations/Q&A, guided individual and group work, panel discussions, and online resources to accommodate adult learner preferences. Learning strategies included pre-Workshop assignments and in-person interactive group activities. The availability of previously developed online IR training resources enabled participants to prepare for the 2-day experience. Pre-Workshop activities (Table 2) and resources were emailed to participants, and they were encouraged to complete these activities as a condition for registration fee waiver.
Table 2. Pre-class assignments: “5 things to do before you come”
* As of December 2019. See NIH updates June 10, 2021.
In tandem with pre-Workshop assignments, in-person activities were employed. Because team-based learning (TBL) (an active, learner-centered but instructor-led strategy) is shown to enhance learning [Reference Reimschisel, Herring, Huang and Minor15,Reference Hrynchak and Batty16], we scheduled group activities throughout the Workshop. Groups applied a step-by-step research design guide [Reference Hull, Goulding and Khadjesari14] to co-develop study ideas. TBL strategies were used to expand and reinforce learning, promote interprofessional collaboration, and move toward the development of IR project ideas.
Participants received a comprehensive four-color hardcopy workbook which included the syllabus, schedule, speaker biographies, worksheets, IR resources, participant list, and evaluation link. An electronic file (PDF) of the workbook was provided to registrants before the Workshop. Additionally, speakers shared presentation slides, helpful supplemental resources, and readings for their respective session. These were saved in a shared Google Drive folder for availability during and after the Workshop.
Interspersed summaries and participant-panel discussions stimulated ideas for collaboration, culminating in recommendations for next steps. To conclude the Workshop and focus on long-range capacity building, one representative from each Texas CTSA hub moderated a panel discussion entitled Building Scientific Workforce Capacity and Opportunities for Collaboration. This brainstorming session provided opportunity for participants to generate ideas and recommendations for preferred strategies to continue expansion of IR across the CTSAs.
Methods of Evaluation
To assess near-term impact and long-range sustainment of IR capacity building among Texas CTSAs, a comprehensive evaluation plan was designed. The Kirkpatrick model [Reference Kirkpatrick and Kirkpatrick17], widely used in evaluating training programs, provided structure. Table 3 identifies the four levels associated with Workshop outcomes and types of data gathered.
Table 3. Evaluation plan
Results
Institutional Support
The formal value proposition stimulated leadership buy-in for faculty involvement in this new research focus and garnered sufficient funds to support the costs of the Workshop. The four-hub planning approach raised interest and catalyzed institutional support of faculty-scientist participation and travel from each hub.
Impact of Training
Results from Workshop quantitative and qualitative data were classified into each of the four Kirkpatrick levels and analyzed.
Level 1 Workshop Experience
Twenty-seven of the 63 participants (42%) completed both the pre- and post-Workshop self-assessment, which included evaluating whether the 11 learning objectives were met (Table 1). The highest rated objective was #2 Evolution of IR (96% agreement); the lowest was #11 Prospectus Plan (59% agreement). Event location was rated excellent (63%) or good (33%). Participants reported networking opportunities as excellent (52%), good (33%), and fair (7%). Survey comments indicated that the Workshop experience was viewed very favorably. Comments included, “Wonderful workshop, thank you for inviting exceptional speakers and planning content that could help us immediately!” Also, “This workshop has made me think about how we build teams and projects. In science we are oriented to strive for the PI role but serving as a co-investigator is just as important.”
Level 2 Knowledge Acquisition
In the pre-post self-assessment of IR competencies, participants reported improvement across all domains of IR competencies (IR definitions, background/rationale, theory/approaches, design/analysis, and practice-based considerations). After the training, fewer participants (48%) considered themselves “novice” than prior to the training (80%). Also, the proportion of “Intermediate/advanced” participants increased from 25% to 52%. Familiarity with the IR theories increased from 56% to 73%. Participant response to open-ended questions indicated that the Workshop had a positive impact on knowledge and attitudes. One example was, “This program and workshop have provided me huge takeaways.”
Level 3 Behavior
In the group activities, participants connected with colleagues with similar interests. Groups used the ImpRES Tool [Reference Hull, Goulding and Khadjesari14] to develop ideas for an IR study. A participant stated, “This workshop has made me rethink how to use mixed methods in Implementation Science.” Another stated, “This workshop has taught me that I should also map to a theory and model and consider the inner setting.” A third commented, “Going forward, how can we make people aware of each other to allow for the development of natural collaborations?”
Level 4 Results
Participants indicated interest in ongoing network interaction, resource availability, and capacity building including consultation. One person commented, “Overall, I consider this to be a successful workshop because we have laid groundwork for collaborations that could lead to possible grant opportunities.” Another participant offered, “A network of clinical sites in Texas could serve to build scientific capacity. Since IR requires multiple sites, the suggestion is to develop formal collaboration across the Texas CTSAs.”
Participants endorsed first steps toward establishing the statewide Texas Implementation Science Research Network (TEX-IS). Given that the Texas CTSAs routinely gather quarterly in regional meetings, this venue was suggested for continuing interaction. Unfortunately, meetings were suspended due to the pandemic. However, ongoing communication has been achieved through the monthly TEX-IS Research Network New Notes e-newsletter.
Progress subsequent to the Workshop
Progress 12 months following the Workshop includes:
-
Posting Workshop learning resources on the host institution’s website https://iims.uthscsa.edu/community/activities.html
-
Launching a D&I Consultation Core to support investigators and healthcare professionals in integrating D&I science in their research and practice. (https://reach.uthscsa.edu/services/di/). As a direct result of the show of interest in IR, new Implementation Research Consultation Core was established at one hub.
-
Offering a series of UT Health Houston IR webinars (https://sph.uth.edu/research/centers/chppr/workshops/tiis/conference).
-
Publishing monthly newsletters of emerging trends in IR, including new resources on equity in IS (e.g., 2). To date, the TEX-IS Research Network News Notes has published eight monthly issues.
-
Creating the TEX-IS Research Network Listserv and registration site.
Discussion
Workshop program design and curriculum were key deliverables of this project. A key element for success was using the NIH/NCI curriculum: Planners capitalized on the newly released NIH/NCI OpenAccess training resources, providing opportunity for ongoing and expanded training to additional faculty scientists across the four UT System CTSAs. Similarly, other notable online resources [Reference Ford, Rabin, Morrato and Glasgow1] that align with the national training curriculum can also be leveraged.
While this test case focused on IR, the educational principles we used are applicable to a wide range of capacity building needs in translational science. Developers of such programs can consider interdisciplinary planning, articulated program goals, measurable learning objectives, active instructional strategies, extended resources, ongoing interaction, and formal evaluation approach. The 2-day curriculum and learning activities provided a comprehensive foundation for extending IR training initiatives throughout our region.
Limitations arise from several sources. First, this training program was offered throughout the four CTSAs hubs in Texas, each with geographical differences. CTSA hub similarities likely arose from the overarching UT System (state-wide) context. Also, regional differences may result in different responses from single hub in faculty participants and populations impacted. Future work could include demographic composition of attendees to throw light on equity. Although measures are in place to promote multi-site IR studies, the reality of the intervening pandemic changed the intended dynamics of collaboration. Finally, although the program is educationally sound, it is a single test case of a novel approach to bootstrapping resources to boost IR workforce capacity.
The program incorporated a standardized foundation for IR training to advance our capacity in IR effectively and efficiently. Our model combines internal strengths with external resources (both in-person experts and online learning resources) and provides a comprehensive plan for building and evaluating workforce capacity, beginning with the 2-day Workshop.
The project demonstrated that, even with limited onsite expertise, it was possible to bootstrap resources and build IR capacity de novo in the CTSA community. Our approach could be useful to other institutions to expand IR capabilities in the face of limited local expertise.
Acknowledgements
We wish to thank the University of Texas Health Science Center institutional and departmental sponsors for their support of this project: Office of the Vice President for Research, Institute for Integration of Medicine & Science, School of Nursing, School of Health Professions, Center for Research to Advance Community Health, UT Population Health Sciences, Institute for Health Promotion, Mays Cancer Center – MD Anderson Cancer Center, Joe R. & Teresa Lozano Long School of Medicine, and School of Public Health – San Antonio Regional Campus.
The project described was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant UL1 TR002645. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Disclosures
The authors have no conflicts of interest to declare.
Open access
