Hostname: page-component-586b7cd67f-r5fsc Total loading time: 0 Render date: 2024-11-25T07:50:06.214Z Has data issue: false hasContentIssue false

Evaluation and lessons learned from the dissemination and implementation science scholars program in the national cancer prevention and control research network

Published online by Cambridge University Press:  29 October 2024

Daniela B. Friedman*
Affiliation:
Department of Health Promotion, Education, and Behavior, Arnold School of Public Health, University of South Carolina, Columbia, SC, USA
Cam Escoffery
Affiliation:
Department of Behavioral, Social, and Health Education Sciences, Rollins School of Public Health, Emory University, Atlanta, GA, USA
Elaine H. Morrato
Affiliation:
Parkinson School of Health Sciences and Public Health, Loyola University Chicago, Chicago, IL, USA
Cynthia A. Thomson
Affiliation:
Department of Health Promotion Sciences, Mel & Enid Zuckerman College of Public Health, University of Arizona, Tucson, AZ, USA
Courtney N. Petagna
Affiliation:
Department of Behavioral, Social, and Health Education Sciences, Rollins School of Public Health, Emory University, Atlanta, GA, USA
Freda Allyson Hucek
Affiliation:
Department of Health Promotion, Education, and Behavior, Arnold School of Public Health, University of South Carolina, Columbia, SC, USA
Mary Wangen
Affiliation:
Center for Health Promotion and Disease Prevention, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Aubrey Villalobos
Affiliation:
Division of Cancer Control and Population Sciences, National Cancer Institute, National Institutes of Health, Rockville, MD, USA
James R. Hebert
Affiliation:
Department of Epidemiology and Biostatistics & Cancer Prevention and Control Program, Arnold School of Public Health, University of South Carolina, Columbia, SC, USA
Samuel Noblet
Affiliation:
Envera Health, Richmond, VA, USA
Mayank Sakhuja
Affiliation:
Department of Health Promotion, Education, and Behavior, Arnold School of Public Health, University of South Carolina, Columbia, SC, USA UNC Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
David O. Garcia
Affiliation:
Department of Health Promotion Sciences, Mel & Enid Zuckerman College of Public Health, University of Arizona, Tucson, AZ, USA
Jennifer L. Cruz
Affiliation:
Department of Social and Behavioral Sciences, Harvard T.H. Chan School of Public Health, Boston, MA, USA
Stephanie B. Wheeler
Affiliation:
Center for Health Promotion and Disease Prevention, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA UNC Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA Department of Health Policy and Management, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
*
Corresponding author: D. B. Friedman; Email: [email protected]
Rights & Permissions [Opens in a new window]

Abstract

Background:

The Centers for Disease Control and Prevention (CDC)-funded Cancer Prevention and Control Research Network (CPCRN) has been a leader in cancer-related dissemination & implementation (D&I) science. Given increased demand for D&I research, the CPCRN Scholars Program launched in 2021 to expand the number of practitioners, researchers, and trainees proficient in cancer D&I science methods.

Methods:

The evaluation was informed by a logic model and data collected through electronic surveys. Through an application process (baseline survey), we assessed scholars’ competencies in D&I science domains/subdomains, collected demographic data, and asked scholars to share proposed project ideas. We distributed an exit survey one month after program completion to assess scholars’ experience and engagement with the program and changes in D&I competencies. A follow-up survey was administered to alumni nine months post-program to measure their continued network engagement, accomplishments, and skills.

Results:

Three cohorts completed the program, consisting of 20, 17, and 25 scholars in Years 1-3, respectively. There was a significant increase in the total D&I competency scores for all three cohorts for 4 overarching domains and 43 subdomains (MPre = 1.38 MPost = 1.89). Differences were greatest for the domain of Practice-Based Considerations (0.50 mean difference) and Theory & Analysis (0.47 mean difference). Alumni surveys revealed that scholars appreciated access to D&I-focused webinars, toolkits, and training resources. 80% remain engaged with CPCRN workgroups and investigators.

Conclusions:

Program evaluation with scholars and alumni helped with ongoing quality assurance, introspection, and iterative program adaptation to meet scholars’ needs. This approach is recommended for large-scale capacity-building training programs.

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2024. Published by Cambridge University Press on behalf of Association for Clinical and Translational Science

Introduction

Background

Need for dissemination and implementation science training in cancer prevention and control

Dissemination and implementation (D&I) science entails studying how evidence-based practices, interventions, and policies can be translated into real-world settings to improve public health outcomes [Reference Brownson, Jacob and Carothers1]. Cancer prevention and control training programs with a D&I focus have been developed by federal agencies and through federally funded grant programs [2,Reference Wangen, Escoffery and Fernandez3] to increase our nation’s D&I research capacity to translate health evidence more rapidly into practice. There has been D&I programming for practitioners and public health scientists [Reference Brownson, Jacob and Carothers1,Reference Gonzales, Handley, Ackerman and O.’Sullivan4Reference Vinson, Clyne, Cardoza and Emmons8]. A major motivation for the National Cancer Institute’s (NCI) interest in D&I research is the fact that, while tens of billions of dollars have been spent on basic research, epidemiologic, and clinical research combined, many potentially relevant findings have not been translated into clinical or public health action [Reference Khan, Chambers and Neta9Reference Tinkle, Kimball, Haozous, Shuster and Meize-Grochowski11]. A well-trained workforce is an important limiting factor in disseminating and implementing scientifically valid research findings [Reference Davis and D’Lima12]. Therefore, intentional efforts to increase training to address D&I needs while also focusing on growing a diverse workforce in D&I science is critical for improving cancer outcomes and reducing cancer-related health disparities [Reference Ford, Rabin, Morrato and Glasgow13,Reference Morrato, Rabin and Proctor14].

The Centers for Disease Control and Prevention (CDC)-funded Cancer Prevention and Control Research Network (CPCRN) has emerged as a leader in cancer-related D&I science [15,Reference Wheeler, Lee and Young16]. While the CPCRN has been in existence since 2002 (2002–2024), it is only in the past decade that the network has focused heavily on D&I science [Reference Wheeler, Lee and Young16]. This has coincided with a deliberate and decisive move into community-based participatory research (CBPR) [Reference Wheeler, Lee and Young16Reference Mathias, Nichols and Golden-Wright18]. CBPR complements D&I science because effective D&I requires community involvement and support, especially if programs are intended to be sustained over long periods of time [Reference Giangrande, White and East19Reference Ramanadhan, Aleman and Bradley21].

Given increased demand for D&I research, the CPCRN Scholars Program was developed in 2020 and launched in 2021 to expand the number of practitioners, researchers (including early-stage investigators), and students (e.g., graduate students and postdoctoral fellows) trained in cancer D&I methods [Reference Friedman, Escoffery, Noblet, Agnone and Flicker22]. Emphasis in training and education was placed on D&I methods as a way to address and prioritize cancer health equity research and practice [Reference Aschbrenner, Oh and Tabak23]. In addition, the program uses both a tailored and co-learning approach for researchers and practitioners [Reference Leppin, Baumann and Fernandez24]. It also emphasizes translating research into practice through collaborative partnerships and the application of CBPR principles and partner engagement [Reference Khan, Chambers and Neta9,Reference Friedman, Escoffery, Noblet, Agnone and Flicker22,Reference Escoffery, Petagna and Wangen25].

Overview of the CPCRN and scholars program

The CPCRN is comprised of eight funded academic centers across the United States. Network members include researchers, public health professionals, and federal and community partners. The members and affiliate members of the network apply evidence-based interventions and strategies to reduce the burden of cancer, especially those disproportionately affected [15]. To effectively do this, the CPCRN also focuses on developing a workforce of a broad array of professions in cancer prevention and control focused on D&I science as part of its mission. Recognizing the need to grow the pool of student, researcher, and practitioner scholars focused on D&I science, the CPCRN created an annual scholars training program beginning in 2020.

Program development

The CPCRN Scholars Program was developed using a collaborative workgroup approach. The Scholars Program Planning Workgroup was coordinated by University of North Carolina at Chapel Hill (UNC) Coordinating Center staff and lead investigators at Emory University and the University of South Carolina with collaborators at the University of Arizona, Colorado School of Public Health, Loyola University, and federal agency partners at the CDC and NCI. The group also engaged representative investigators from across the network sites to initiate the program development process and assist in the evaluation design. A logic model was used to guide the implementation and evaluation of all components of this training program [Reference Friedman, Escoffery, Noblet, Agnone and Flicker22].

Instructional design

Given that this is a national network with funded centers across eight states and affiliate members with widespread geographical representation, meetings, and trainings have occurred mostly via Zoom (other than annual in-person meetings). Program and curriculum development was guided by a prior CDC-funded student-focused scholars program on brain health [Reference Croff, Tang, Friedman, Balbim and Belza26,Reference Soltani, Kannaley and Tang27], an in-depth formative evaluation, and capacity-building expertise of network members [Reference Wangen, Escoffery and Fernandez3,Reference Friedman, Escoffery, Noblet, Agnone and Flicker22,Reference Escoffery, Petagna and Wangen25].

The program curriculum, consisting of readings and videos that were posted on a virtual platform (Trello) for each scholar, focused on evidence-based public health/cancer interventions and D&I scientific frameworks, strategies, and methods to advance knowledge and increase competencies related to implementation research and practice. As part of this, within a 12-month period (this was condensed to 9 months for the first program cohort), scholars had to complete one of two D&I curricula and the curricula chosen were based on what track the scholar chose to be in at the beginning of the program. The curricula were: (1) the Putting Public Health Evidence into Action Training developed by CPCRN investigators [Reference Wangen, Escoffery and Fernandez3,Reference Mainor, Decosimo and Escoffrey28] and (2) the NCI’s Training Institute for Dissemination and Implementation Research in Cancer (TIDIRC), open-access modules [2]. In addition, scholars were asked to develop or participate in a project that provided experiential learning in terms of D&I principles and practices.. Often these were cross-center CPCRN projects based within workgroups; scholars could also conduct projects at their local CPCRN site, and some student scholars considered their thesis/dissertation to be the CPCRN project requirement. The mentors the scholars identified at the beginning of the program assisted their scholar throughout the scholar’s project. Scholars typically had 1–2 mentors and a mentor may have had multiple scholars. The projects could have involved other scholars or CPCRN investigators, particularly if the project was developed in a CPCRN workgroup. The scholars presented their projects to the Scholars Planning Workgroup, Scholar Program leadership team, and the Coordinating Center during end-program activities.

Scholars across all tracks had the opportunity to interact and learn from each other during scheduled virtual discussions and meet-ups. Together, they also attended virtual webinars on topics such as D&I theories and frameworks, implementation in action, implementation in a global context, scoping and systematic reviews, and D&I in real-world settings. Scholars participated in local and CPCRN network-wide meetings, and meetings with mentors and other scholars [Reference Friedman, Escoffery, Noblet, Agnone and Flicker22,Reference Escoffery, Petagna and Wangen25]. We had a kickoff and closing meeting for each cohort. Following the program, scholars received certificates of completion.

Scholar recruitment and selection

The Scholars Program Planning Workgroup drafted an email calling for applications, which was sent to members of the CPCRN Steering Committee consisting of principal investigators, project directors, and workgroup leads. Additionally, the application was sent to the overall Network listserv with a request to distribute the email and link to an electronic application form. Network members distributed the email via student and researcher listservs at their home institutions; affiliates and federal agency partners (CDC and NCI) also shared the information. Individual investigators and the CPCRN Coordinating Center (UNC-Chapel Hill) shared the opportunity on social media. As the program progressed with new cohorts in each of the second and third years of the program, alumni of the program shared the call with their networks and assisted with new scholar recruitment and mentorship.

Two Scholar Program Planning Workgroup reviewers were assigned to each application received. Reviewers were selected based on their familiarity with the CPCRN structure and content expertise. This made it feasible to link scholars with interests with expertise residing across the CPCRN and within specific workgroups (e.g., Survivorship, Health Behaviors). A rubric was used to rate applications on individuals’ evidence of interest in cancer prevention and control and/or D&I science; a clear, concise description of a proposed project; a clear description of the proposal goals and activities; feasibility of completing the proposed project in the one-year timeframe; how proposed goals and activities contributed to the diversity of the training program; and how the proposed goals and project aligned with CPCRN efforts. Applicants were asked to recommend program mentors who were typically at their local CPCRN site or affiliate institution and/or workgroup members. However, this was not a strict requirement as some applicants were completely unaffiliated with any CPCRN center. Additional details about program development, recruitment, and review of applications can be found elsewhere [Reference Friedman, Escoffery, Noblet, Agnone and Flicker22,Reference Escoffery, Petagna and Wangen25].

Study objective

The objective of this paper is to summarize lessons learned from the first three training cohorts (N = 63 scholars) of the CPCRN Dissemination and Implementation Science Scholars Program (2020–2023). Detailed information about the early development of the CPCRN Scholars Program, including structures and processes, can be found in previously published articles [Reference Friedman, Escoffery, Noblet, Agnone and Flicker22,Reference Escoffery, Petagna and Wangen25]. In the following sections, we describe the program evaluation data and lessons learned for other research networks implementing similar scholars programs.

Materials and methods

Program evaluation

For the three cohorts, the scholars program application also served as the baseline survey, and it included demographic questions, names of current mentors, CPCRN workgroups of interest, a proposed project description, one to three professional development goals, related activities they had for participation, and how their proposed work aligned with the CPCRN Strategic Plan and/or Logic Model.

Additionally, applicants were asked to self-rate their competencies in D&I science (1 = beginner, 2 = intermediate, 3 = advanced) based on published descriptions of competencies and similar Likert scoring [Reference Padek, Mir and Jacob29]. Specifically, scholars self-reported their competencies focused on D&I background knowledge (10 subdomain items), theory and approaches (7 items), study design and analysis (14 items), and practice-based considerations (12 items). The comprehensive list of competencies was developed using a multi-phase approach that included expert consensus identification of individual competencies followed by card sorting, an acknowledged, effective approach to categorize knowledge [Reference Faiks and Hyland30], organize domains, and establish subdomain items for inclusion under each domain [Reference Padek, Colditz and Dobbins31]. These competencies have been implemented widely in the development of the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program at Washington University in St Louis and have been used in several other D&I-focused training programs [Reference Rakhra, Hooley and Fort32,Reference Salloum, LeLaurin and Nakkash33]. These self-rated competencies specifically include relevant practice-based competencies which are especially important for our CPCRN Scholars program given the critical involvement and our focus on recruitment of practitioners (in addition to academics) and the strong interest of students and faculty members in the applied, real-life implementation of evidence-based strategies and programs.

Following the completion of the program, scholars received a post-program survey that included self-rated competency questions similar to the pre-program assessment. Response rates varied for this survey by cohort, ranging from 88% (cohort 2) to 92% (cohort 3). After the first cohort completed the program, we also conducted interviews with scholars to understand more in-depth their satisfaction with and recommendations for the program. Findings from these qualitative interviews of the first cohort can be found elsewhere [Reference Escoffery, Petagna and Wangen25]. In addition, an alumni survey was distributed to scholars (from cohorts 1–2 thus far) nine months post-program. It consisted of 13 questions, of which five items allowed for open-ended responses specifically regarding their projects, continued network engagement, accomplishments, and skills. The alumni survey asked about current employer information/position title, graduation information (if the scholar was a student while pursuing the training program), current level of engagement with CPCRN workgroups and interest in remaining engaged with CPCRN in a scholar alumni role, peer-reviewed publications, conference presentations, and/or grants submitted or funded since their time in the program, type of continued engagement with D&I science and/or cancer prevention and control, interest in serving as a mentor for a future CPCRN scholar, and skills they learned and/or wished they learned from the program. Descriptive statistics (frequencies, percentages) and paired t-tests were calculated using SPSS v27.

Results

Program participation

To date, the program has been implemented with three cohorts. Year 1 had 20 scholars (24 applicants) and 30 mentors; Year 2 had 17 scholars (19 applicants) and 15 mentors, and Year 3 had 25 (2 had to drop the program; 47 applicants) and 38 mentors. Year 3 demonstrated a 135% increase in applications from the previous year. Scholars came from 21 schools and organizations within the United States and abroad (Switzerland). Scholars were diverse and had good representation across our categories of audiences (e.g., students, practitioners). Scholars’ disciplines have mainly been in public health (including health promotion, epidemiology, health policy, and management), health sciences, and behavioral and social sciences. Table 1 presents detailed demographics about participating scholars; Table 2 presents scholars by track for each year of the program; and Figure 1 presents the number of scholars by year, geographic location, and institution (CPCRN center or affiliate center, or other). The final year of the program for this specific CPCRN grant cycle (2019–2024) involved scholar alumni from all years of the program (See Lessons Learned section for additional details).

Figure 1. Distribution of scholars across CPCRN centers and other institutions.

Table 1. Scholar demographics and characteristics (N = 62)

Table 2. Number of scholar program participants by career track

Overall program evaluation and impact

We present baseline and post-program survey data from all three scholar cohorts, and alumni evaluation data from two cohorts, including qualitative data captured at the end of the first yearof the Scholars Program. Scholars to date (N = 62) were mostly students (45.2%) or researchers (42.0%) across the three cohorts (Tables 1 and 2). They were mostly female (80.6%) and racially diverse (45.2%).

Table 3 presents the merged baseline and post-program survey data combined for the three Scholars Program cohorts. Scholars rated greater overall experience with D&I science after the program (MPre = 1.38, MPost = 1.89). There were significant increases in the total D&I competency score for 100% of the 4 overarching domains (MPre = 1.49, MPost = 1.99) and 43 subdomains. For each of the specific competency categories (A. Definition, Background, and Rationale, B. Theory and Approaches, C. Design & Analysis, and D. Practice-Based Considerations), scholars reported significant improvements from baseline to post-program. These differences were greatest for the domain of Practice-Based Considerations (0.50 mean difference) followed by Theory & Analysis (0.47 mean difference).

Table 3. Pre-/Post-scholar self-ratings on D&I science competencies*

D&I = dissemination and implementation.

* p < 0.05 for paired samples means test.

** Scale ranged from 1-3 with 1=beginner, 2=intermediate, 3=advanced.

Scholar alumni data and reflections

Fifteen of the 37 scholars responded (40.5% response rate). From the surveys, over half of the respondents (n = 8, 53.3%) were in new positions since completing the scholars program; 20.0% (n = 3) had graduated from a degree program since participating in the program. Eighty percent remained engaged with a CPCRN workgroup and/or still collaborated with a CPCRN investigator(s) and 86.7% (n = 13) had completed their program projects. Forty percent of scholar alumni (n = 6) expressed interest in serving as a mentor for a future CPCRN scholar and 33.3% (n = 5) offered to help plan upcoming events and initiatives. From the alumni survey, one alumna of the program served as a mentor to other scholars in future years of the program; another alumnus is now serving as a multiple principal investigator of a CPCRN Collaborating Center; and a third alumnus proposed a new interest group focused on LGBTQ + Health as part of the Health Equity workgroup and has taken on a leadership role with the network.

When asked what new skills they received from the program, most alumni mentioned the benefits of learning about theoretical and conceptual frameworks, and they appreciated the access to D&I-focused webinars, toolkits, and training resources. They also benefited from networking with other professionals and students and collaborating with them on academic products. When asked what they wished they had received or done in the program, we received five comments from alumni requesting content about sustainability planning (a webinar is being planned for our all-cohort professional development year), jobs outside of the academic space, and cancer-specific funding mechanisms. One respondent wished they had been more active in reaching out to their mentors outside of their institution and hoped there would be more collaborations across workgroups.

The main themes that emerged from the open-ended questions on the alumni survey related to scholars reporting being able to apply theoretical and conceptual frameworks to their D&I initiatives. They also gained knowledge and skills regarding the implementation and evaluation of evidence-based programs in their cancer prevention and control work and for their dissertation or projects, if they were students. Table 4 presents representative quotes from scholars on how they continued to use the content and/or skills they learned in the program.

Table 4. Scholar alumni reflections on how they are using content and skills from the program

CPCRN = cancer prevention and control research network; TIDIRC = training institute for dissemination and implementation research in cancer; PPHEA = putting public health evidence into action training; FQHC = federally qualified health center; D&I = dissemination and implementation.

Examples of Scholars’ research dissemination activities

Scholars were co-authors on 13 of 21 papers published in a CPCRN special supplement of Cancer Causes & Control published in 2023. The manuscripts focused on one of these five thematic areas: (1) Addressing Equity Through CPCRN, (2) Capacity Building, (3) Partnership Engagement, (4) Rural Cancer Prevention and Control, and (5) Future Cancer Needs and Priorities. Thirteen scholars were involved with presentations at the 16th Annual Conference on the Science of D&I co-hosted by the National Institutes of Health and Academy Health; nine scholars presented their projects at a national CPCRN network grantee meeting in May 2023.

Discussion

The CPCRN Scholars program is grounded deeply in public health, community engagement, and health equity principles. The program has been highly effective in achieving enrollment goals and diversifying the D&I science expertise of public health researchers, practitioners, and students.

Senior mentors have been from a wide variety of academic backgrounds with skill sets and life experiences that were particularly attractive to scholars. For example, a scholar may have been in an epidemiology program; so, receiving guidance and new knowledge in CBPR methods from workgroup members and/or a mentor at another institution across the country was considered unique.

This is one of few programs that was established not only for advanced researchers but also for students, postdoctoral fellows, junior faculty members, and practitioners who may be new to D&I science. Related to this, the network has always been open to people’s project ideas – for workgroups, interest groups, and new projects. Applicants represented a wide range of characteristics, including scholars’ commitment to a particular project idea. As noted, some will use this as an opportunity to further develop their dissertation research related to cancer control or D&I; therefore, they would have a specific project idea. Others would have a general idea of where their interests might lie. These individuals often were “shopping” for training ideas and would often be directed to particular workgroups that would help them to realize these interests.

Scholars reported significant improvements in self-ratings of all competencies from baseline to post-program. These differences were greatest for items related to Practice-Based Considerations followed by Theory & Analysis. This was expected given the core competencies were linked to specific curricular content which emphasized Theory & Analysis and promoted engagement in individual D&I research projects that integrated hands-on experiential learning in CBPR and related practice-based competencies. The slightly lower improvements in basic D&I knowledge, as compared to these two areas, may reflect the higher baseline knowledge (overall median competency score 1.52) of theory, possibly a result of the program attracting scholars with some background awareness, knowledge, and vested interest in D&I knowledge and skill expansion.

Lessons learned with regard to recruitment, process and implementation, and continued engagement and sustainability

Recruitment

As an academic research network, CPCRN was successful with recruitment of graduate students and postdoctoral fellows, for whom there are few mentored D&I programs at this training stage, but recruitment of investigators new to D&I research and cancer prevention and control practitioners from clinical and public health settings was more challenging. After the NCI’s TIDIRC facilitated program transitioned to non-mentored open access only in 2022 2, the number of researchers participating in the Scholars Program increased. This provides evidence that the Scholars Program is filling a gap in desired mentored training for academic investigators.

For the second cohort, concerted outreach to networks of clinicians and public health professionals helped bolster the involvement of these key partners in the Scholars Program. D&I training for practitioners is critical to enhance their capacity to identify, adapt, implement, evaluate, and sustain evidence-based interventions in practice at a scale that will achieve positive population health outcomes. Further, the inclusion of practitioners from clinical and public health settings in D&I training increases their readiness to engage with D&I researchers in planning, conducting, and disseminating more relevant science. As argued by Smith and Wilkins [Reference Smith and Wilkins34], building a community of D&I-trained practitioner scholars who can lead practice-based research or collaborate in research-practice partnerships can help bridge the intractable gap between research and translation to practice. The inclusion of practitioners in this training helps meet a documented global gap in D&I training for these critical partners [Reference Davis and D’Lima12,Reference Viglione, Stadnick and Birenbaum35]. It remains challenging to recruit practitioners, however, we have made it a priority to collaborate with practitioners in program design and improvements to ensure we are meeting the needs of those in the field.

Process and implementation

The CPCRN Scholars Program, which was developed under the national CPCRN, is unique in identifying clinicians, practitioners, and researchers for training in D&I research. Recruitment was initially conducted across the United States and by the third cohort had expanded to include international applicants. Geographical dispersal made it essential to consider a remote and flexible approach to training to ensure the greatest opportunities for engagement in terms of time commitments and geographical reach. The program was designed to be self-paced to afford greater flexibility to allow working adults to participate and provide transparent expectations for deliverables and deadlines for task completion. The Trello board was selected as a project management tool wherein each scholar was made aware of tasks, dates, and deadlines. This ensured that key training communications and content were made available to all fellows in a timely manner. Real-time tracking aligned with competency acquisition also allowed early identification of any scholar needing additional training support.

The self-paced program placed the responsibility for project progress with the scholar. Individuals who, though competent and highly motivated, were generally faced with competing demands on their time. This challenge of time for training and project completion has been noted across D&I capacity-building programs [Reference Davis and D’Lima12]. For such capacity-building programs, mentor oversight in terms of regular meetings to evaluate progress and even written progress reports can help ensure timely project completion and greater time equity across scholar projects. Interestingly, a significant percentage of program scholars continued with their project beyond the 12-month training period, and many engaged in additional projects to further expand their training and mentoring opportunities in D&I science.

Evaluation data from exit interviews (with Cohort 1) and surveys (pre/post and alumni) as well as ongoing program feedback from scholars at network annual meetings suggested that some scholars were not receiving enough general professional development opportunities in their current professional setting. Among the interests voiced by scholars are training in grant budgeting, scientific writing, and how to prepare and deliver scientific presentations, and interview job talks. While professional development was beyond the scope of this D&I training program, many scholars received this guidance informally from their mentors and/or the program leadership and they identified the value of mentoring in professional development as a positive aspect of the training experience. To address this need without diluting the current focus, the planning workgroup developed an all-cohort professional development year for which they are offering virtual seminars and networking events guided by input provided by all three scholar cohorts via a training assessment survey. Some of the topics included a writing retreat for progressing on their papers or grant proposals, how to conduct systematic reviews and writing a D&I grant.

Evaluating the program using surveys with scholars and alumni and interviews with scholars and mentors was extremely helpful for ensuring ongoing quality assurance, introspection, and iterative program adaptation to meet scholars’ needs. Continuing to ensure data are collected from scholars and mentors and from planning workgroups moving forward will be helpful given that national workgroups engaged a number of scholars during this grant cycle and workgroup leads and investigators may have also served as informal mentors for scholars. We also recommend that training programs are creatively combined with data collection activities. Mixed methods evaluation with qualitative reflection and stories are very important for improvement and for the recruitment of future cohorts. Collecting such narratives in a meaningful way will help with the expansion and sustainability of the program. Of note, the program evaluation to date has largely focused on short-term impact as outlined in our logic model. Going forward the program will need to demonstrate greater intermediate and long-term outputs and impact to support its sustainability. In fact, a more comprehensive logic model-driven, outcomes-centric approach to evaluating self-competencies has been developed and shown to be effective in the training of clinical translational scientists [Reference Robinson, Moore, McTigue, Rubio and Kappor36Reference Lee, Pusek and McCormack39].

Building on the CPCRN was a critical asset for the Scholar’s program. Beyond the curriculum and webinars, scholars were able to attend the annual CPCRN virtual and in-person meetings as its own infrastructure. This allowed them to network, contribute to the efforts of workgroups, share their own science, and build their network of D&I researchers and practitioners in cancer control. Learning more about the CPCRN and continued participation in the network was a common evaluation finding [Reference Escoffery, Petagna and Wangen25].

Limitations with program implementation evaluation

There are important limitations to note with this research training program. First, we approached diversity broadly in how we defined and assessed it in the application rubric. The mentor/scholar match was done primarily based on content and topic expertise and the diversity of the research background and interests of the applicants. More intentional consideration of diversity for future scholar/mentor matching should be conducted as has been done in other research training programs [Reference Austin, Ingram, McCollum, Levkoff and Friedman40,Reference Johnson, Friedman and Ingram41]. Second, we did not examine all aspects of the program in our evaluations, despite the use of multiple assessments. For example, we only asked about CPCRN-relevant dissemination on the alumni survey. We also did not inquire about scholars’ satisfaction with specific curriculum content or evaluate knowledge and skills obtained beyond the self-related competency assessment on the pre- and post-program survey. Third, while the Trello board was a convenient platform for housing the program curriculum, in retrospect, leveraging this system to track training competency at the individual scholar level by aligning learning activities with knowledge acquisition and applications would have ensured all competencies were met over time by each scholar.

Fourth, while the self-rated competencies assessment is published and has been used in several D&I trainings, there are no validation studies. In addition, we recognize that other methods such as logic model competency identification or competency identification through competency-based educational frameworks can be used to establish cutoff points for competencies given the potential limitation of implementing a tertile ranking for this purpose [Reference Dilmore, Moore and Bjork37,Reference Rubio, Primack, Switzer, Bryce, Seltzer and Kapoor38]. Importantly, we did use overall domain scores to drive our interpretation of the evaluation results so as not to overinterpret our findings related to individual subdomain responses. Fifth, we recognize that with the cohort size, we were unable to perform additional meaningful statistics (subanalyses) that would be possible with a larger sample. We also had not administered the alumni survey with the third cohort of scholars at the time this paper was written. Finally, we conducted in-depth interviews only with the initial cohort of scholars. While these interviews helped inform future years of the program, it is recommended that a more consistent collection of qualitative data be implemented with scholars, mentors, and workgroups.

Conclusions

This paper has described evaluation data and lessons learned about a formal training program of a CDC-funded research network. While the focus of this program has been on cancer prevention and control and D&I science, the process of program development, implementation, and evaluation as well as recommendations can be applied to other mentored capacity-building initiatives that collaborate across multiple institutions. Sustaining such a training program is important and deserves the thoughtful attention of planners [Reference Proctor and Chambers42]. The scale and scope of such a training program may deserve its own dedicated funding support to allow for further touch points (in-person meetings) or programming. The infrastructure of having a collaborative network of centers with a strong coordinating center is critical for the continued success of the CPCRN Scholars Program.

We plan to keep up the momentum and enthusiasm for this program through continued engagement with scholar alumni and mentors. We will do this by asking scholars to help recruit future cohorts, serve in mentor roles after completing the program, and remain engaged through participation in panels, webinars, and conference presentations. Encouraging continued collaboration on projects through national workgroups will also keep scholars engaged with prior and new mentors and collaborators and allow them to be part of initiatives that benefit their career trajectories regarding academic deliverables and access to community and other partner groups. Ongoing scholar assessment and evaluation will allow for building in preferred training topics and components that will be valuable to their career trajectory and make a public health impact. Future training initiatives are still needed to build capacity in D&I competencies and science and continue to improve public health in communities.

Acknowledgements

The authors would like to acknowledge Jennifer Leeman, DrPH, MPH, MDiv, University of North Carolina at Chapel Hill, and Swann Arp Adams, PhD, MS, FACE, University of South Carolina, for their collaboration on conceptualization, development, and implementation of the CPCRN Scholars Program. The authors recognize all of the scholars and mentors who participated in this program and continue to be engaged with it.

Author contributions

DBF: conceptualization, data collection, manuscript writing and editing; CE: conceptualization, data collection and analysis, manuscript writing and editing; EHM: conceptualization, manuscript writing and editing; CAT: conceptualization, manuscript writing and editing; CNP: conceptualization, data collection and analysis, manuscript writing and editing; FAH: conceptualization, data collection and analysis, manuscript writing and editing; MW: conceptualization and manuscript editing; AV: conceptualization, manuscript writing and editing; JRH: conceptualization, manuscript writing and editing; SN: data collection, manuscript editing; MS: data collection and analysis, manuscript writing and editing; DOG: manuscript editing; JLC: manuscript editing; SBW: conceptualization, manuscript editing. All authors read and approved the final manuscript.

Funding statement

Work on this paper was funded by the Division of Cancer Prevention and Control, National Center for Chronic Disease Prevention. and Health Promotion of the Centers for Disease Control and Prevention (CDC), U.S. Department of Health and Human Services (HHS) under Cooperative Agreement Numbers: U48DP006399, U48DP006377, U48DP006400, U48DP006413, and U48DP006401. The findings and conclusions in this article are those of the authors and do not necessarily represent the official views of, nor an endorsement, by CDC/HHS, or the U.S. Government.

Competing interests

None.

References

Brownson, RC, Jacob, RR, Carothers, BJ, et al. Building the next generation of researchers: mentored training in dissemination and implementation science. Acad Med. 2021;96(1):8692. doi: 10.1097/ACM.0000000000003750.CrossRefGoogle ScholarPubMed
National Cancer Institute, Division of Cancer Control & Population Sciences. Training Institute for Dissemination and Implementation Research in Cancer (TIDIRC) OpenAccess. Secondary Training Institute for Dissemination and Implementation Research in Cancer (TIDIRC) OpenAccess ND. https://cancercontrol.cancer.gov/is/training-education/training-in-cancer/TIDIRC-open-access. Accessed April 1, 2024.Google Scholar
Wangen, M, Escoffery, C, Fernandez, ME, et al. Twenty years of capacity building across the cancer prevention and control research network. Cancer Causes Control. 2023;34(Suppl 1):4556. doi: 10.1007/s10552-023-01690-2.CrossRefGoogle ScholarPubMed
Gonzales, R, Handley, MA, Ackerman, S, O.’Sullivan, PS. A framework for training health professionals in implementation and dissemination science. Acad Med. 2012;87(3):271278. doi: 10.1097/ACM.0b013e3182449d33.CrossRefGoogle ScholarPubMed
Jacob, RR, Gacad, A, Padek, M, et al. Mentored training and its association with dissemination and implementation research output: a quasi-experimental evaluation. Implement Sci. 2020;15(1):30. doi: 10.1186/s13012-020-00994-0.CrossRefGoogle ScholarPubMed
Karlin, BE, Brown, GK, Jager-Hyman, S, et al. Dissemination and implementation of cognitive behavioral therapy for depression in the Kaiser permanente health care system: evaluation of initial training and clinical outcomes. Behav Ther. 2019;50(2):446458. doi: 10.1016/j.beth.2018.08.002.CrossRefGoogle Scholar
Meissner, HI, Glasgow, RE, Vinson, CA, et al. The U.S. training institute for dissemination and implementation research in health. Implement Sci. 2013;8(1):12. doi: 10.1186/1748-5908-8-12.CrossRefGoogle ScholarPubMed
Vinson, CA, Clyne, M, Cardoza, N, Emmons, KM. Building capacity: a cross-sectional evaluation of the US training institute for dissemination and implementation research in health. Implement Sci. 2019;14(1):97. doi: 10.1186/s13012-019-0947-6.CrossRefGoogle ScholarPubMed
Khan, S, Chambers, D, Neta, G. Revisiting time to translation: implementation of evidence-based practices (EBPs) in cancer control. Cancer Causes Control. 2021;32(3):221230. doi: 10.1007/s10552-020-01376-z.CrossRefGoogle ScholarPubMed
Neta, G, Sanchez, MA, Chambers, DA, et al. Implementation science in cancer prevention and control: a decade of grant funding by the national cancer institute and future directions. Implement Sci. 2015;10(1):4. doi: 10.1186/s13012-014-0200-2.CrossRefGoogle ScholarPubMed
Tinkle, M, Kimball, R, Haozous, EA, Shuster, G, Meize-Grochowski, R. Dissemination and implementation research funded by the US national institutes of health, 2005-. 2012. Nurs Res Pract. 2013;2013:909606–15. doi: 10.1155/2013/909606.Google Scholar
Davis, R, D’Lima, D. Building capacity in dissemination and implementation science: a systematic review of the academic literature on teaching and training initiatives. Implement Sci. 2020;15(1):97. doi: 10.1186/s13012-020-01051-6.CrossRefGoogle ScholarPubMed
Ford, B, Rabin, B, Morrato, EH, Glasgow, RE. Online resources for dissemination and implementation science: meeting demand and lessons learned. J Clin Transl Sci. 2018;2(5):259266. doi: 10.1017/cts.2018.337.CrossRefGoogle ScholarPubMed
Morrato, EH, Rabin, B, Proctor, J, et al. Bringing it home: expanding the local reach of dissemination and implementation training via a university-based workshop. Implement Sci. 2015;10(1):94. doi: 10.1186/s13012-015-0281-6.CrossRefGoogle Scholar
Cancer Prevention and Control Research Network. https://cpcrn.org/about-cpcrn. Accessed April 1, 2024.Google Scholar
Wheeler, SB, Lee, RJ, Young, AL, et al. The special sauce of the cancer prevention and control research network: 20 years of lessons learned in developing the evidence base, building community capacity, and translating research into practice. Cancer Causes Control. 2023;34(S1):217239. doi: 10.1007/s10552-023-01691-1.CrossRefGoogle Scholar
Friedman, DB, Brandt, HM, Freedman, DA, et al. Innovative and community-driven communication practices of the south carolina cancer prevention and control research network. Prev Chronic Dis. 2014;11:E127. doi: 10.5888/pcd11.140151.CrossRefGoogle ScholarPubMed
Mathias, W, Nichols, KA, Golden-Wright, J, et al. Implementation during a pandemic: findings, successes, and lessons learned from community grantees. J Cancer Educ. 2023;38(3):957962. doi: 10.1007/s13187-022-02213-4.CrossRefGoogle ScholarPubMed
Giangrande, N, White, RM, East, M, et al. A competency framework to assess and activate education for sustainable development: addressing the UN sustainable development Goals 4.7 Challenge. Sustainability-BASEL. 2019;11(10):16.Google Scholar
Pronk, N, Kleinman, DV, Goekler, SF, Ochiai, E, Blakey, C, Brewer, KH. Promoting health and well-being in healthy people 2030. J Public Health Manag Pract. 2021;27(Suppl 6):S242S48. doi: 10.1097/PHH.0000000000001254.CrossRefGoogle ScholarPubMed
Ramanadhan, S, Aleman, R, Bradley, CD, et al. Using participatory implementation science to advance health equity. Annu Rev Public Health. 2024;45(1):4767. doi: 10.1146/annurev-publhealth-060722-024251.CrossRefGoogle ScholarPubMed
Friedman, DB, Escoffery, C, Noblet, SB, Agnone, CM, Flicker, KJ. Building capacity in implementation science for cancer prevention and control through a research network scholars program. J Cancer Educ. 2022;37(6):19571966. doi: 10.1007/s13187-021-02066-3.CrossRefGoogle Scholar
Aschbrenner, KA, Oh, AY, Tabak, RG, et al. Integrating a focus on health equity in implementation science: case examples from the national cancer institute’s implementation science in cancer control centers (ISC(3)) network. J Clin Transl Sci. 2023;7(1):e226. doi: 10.1017/cts.2023.638.CrossRefGoogle Scholar
Leppin, AL, Baumann, AA, Fernandez, ME, et al. Teaching for implementation: a framework for building implementation research and practice capacity within the translational science workforce. J Clin Transl Sci. 2021;5(1):e147. doi: 10.1017/cts.2021.809.CrossRefGoogle Scholar
Escoffery, C, Petagna, CN, Wangen, M, et al. Mixed methods evaluation of the inaugural year of the cancer prevention and control research network’s (CPCRN) scholars program. Cancer Causes Control. 2023;34(Suppl 1):5773. doi: 10.1007/s10552-023-01702-1.CrossRefGoogle Scholar
Croff, R, Tang, W, Friedman, DB, Balbim, GM, Belza, B. Training the next generation of aging and cognitive health researchers. Gerontol Geriatr Educ. 2022;43(2):185201. doi: 10.1080/02701960.2020.CrossRefGoogle ScholarPubMed
Soltani, SN, Kannaley, K, Tang, W, et al. Evaluating community-academic partnerships of the south carolina healthy brain research network. Health Promot Pract. 2017;18(4):607614. doi: 10.1177/1524839917700086.CrossRefGoogle ScholarPubMed
Mainor, AG, Decosimo, K, Escoffrey, C, et al. Scaling up and tailoring the “Putting public health in action” training curriculum. Health Promot Pract. 2018;19(5):664672. doi: 10.1177/1524839917741486.CrossRefGoogle ScholarPubMed
Padek, M, Mir, N, Jacob, RR, et al. Training scholars in dissemination and implementation research for cancer prevention and control: a mentored approach. Implement Sci. 2018;13(1):18. doi: 10.1186/s13012-018-0711-3.CrossRefGoogle ScholarPubMed
Faiks, A, Hyland, N. Gaining user insight: a case study illustrating the card sort technique. Coll Res Libr. 2000;61(4):349357.CrossRefGoogle Scholar
Padek, M, Colditz, G, Dobbins, M, et al. Developing educational competencies for dissemination and implementation research training programs: an exploratory analysis using card sorts. Implement Sci. 2015;10(1):114. doi: 10.1186/s13012-015-0304-3.CrossRefGoogle ScholarPubMed
Rakhra, A, Hooley, C, Fort, M, et al. The WHO-TDR dissemination and implementation massive open online course (MOOC): evaluation and lessons learned from eight low-and middle-income countries. Res Sq. 2022. doi: 10.21203/rs.3.rs-1455034/v1.Google ScholarPubMed
Salloum, RG, LeLaurin, JH, Nakkash, R, et al. Developing capacity in dissemination and implementation research in the Eastern Mediterranean region: evaluation of a training workshop. Glob Implement Res Appl. 2022;2(4):340349. doi: 10.1007/s43477-022-00067-y.CrossRefGoogle ScholarPubMed
Smith, LS, Wilkins, N. Mind the gap: approaches to addressing the research-to-practice, practice-to-research chasm. J Public Health Manag Pract. 2018;24:S6S11. doi: 10.1097/PHH.0000000000000667.CrossRefGoogle Scholar
Viglione, C, Stadnick, NA, Birenbaum, B, et al. A systematic review of dissemination and implementation science capacity building programs around the globe. Implement Sci Commun. 2023;4(1):34. doi: 10.1186/s43058-023-00405-7.CrossRefGoogle ScholarPubMed
Robinson, GFWB, Moore, CG, McTigue, KM, Rubio, DM, Kappor, WN. Assessing competencies in a master of science in clinical research program: the comprehensive competency review. Clin Transl Sci. 2015;8(6):770775. doi: 10.1111/cts.12322.CrossRefGoogle Scholar
Dilmore, TC, Moore, DW, Bjork, Z. Developing a competency-based educational structure within clinical and translational science. Clin Transl Sci. 2013;6(2):98102. doi: 10.1111/cts.12030.CrossRefGoogle ScholarPubMed
Rubio, DM, Primack, BA, Switzer, GE, Bryce, CL, Seltzer, DL, Kapoor, WN. A comprehensive career-success model for physician-scientists. Acad Med. 2011;86(12):15711576. doi: 10.1097/ACM.0b013e31823592fd.CrossRefGoogle ScholarPubMed
Lee, LS, Pusek, SN, McCormack, WT, et al. Clinical and translational scientist career success: metrics for evaluation. Clin Transl Sci. 2012;5(5):400407. doi: 10.1111/j.1752-8062.2012.00422.x.CrossRefGoogle ScholarPubMed
Austin, ME, Ingram, LA, McCollum, Q, Levkoff, SE, Friedman, DB. A dual approach to addressing gaps in scholar diversity in aging research. Gerontol Geriatr Educ. 2024:110. doi: 10.1080/02701960.2024.2343876.CrossRefGoogle ScholarPubMed
Johnson, CL, Friedman, DB, Ingram, LA, et al. Reflections on mentorship from scientists and mentors in an Alzheimer’s disease focused research training program. J Appl Gerontol. 2022;41(11):23072315. doi: 10.1177/07334648221109514.CrossRefGoogle Scholar
Proctor, EK, Chambers, DA. Training in dissemination and implementation research: a field-wide perspective. Transl Behav Med. 2017;7(3):624635. doi: 10.1007/s13142-016-0406-8.CrossRefGoogle ScholarPubMed
Figure 0

Figure 1. Distribution of scholars across CPCRN centers and other institutions.

Figure 1

Table 1. Scholar demographics and characteristics (N = 62)

Figure 2

Table 2. Number of scholar program participants by career track

Figure 3

Table 3. Pre-/Post-scholar self-ratings on D&I science competencies*

Figure 4

Table 4. Scholar alumni reflections on how they are using content and skills from the program