Introduction
Traditional approaches to demonstrating the impact of scientific research have focused on quantitative measures of productivity, such as grant applications and funding, publications, and citations [Reference Luke, Sarli and Suiter1]. Despite its focus on moving science from discovery to impact, the field of implementation science has largely followed this traditional path [Reference Maddox, Phan and Byeon2]. Implementation science aims to close this gap between research and practical implementation, but current training methods place little emphasis on impact. As interest in implementation science impact has grown, so too has the need for increased investment in training tools and programs that integrate impact, which are currently few in number, frequently oversubscribed, and focused on traditional scholarly productivity metrics [Reference Chambers, Pintello and Juliano-Bult3–Reference Proctor and Chambers5].
Training implementation scientists to demonstrate impact
Demonstrating the impact of implementation science to the broader community is important, in part, to secure funding and support for implementation science research and training, but operationalizing downstream impacts is challenging. Traditional graduate research training has emphasized immediate scientific outputs rather than the longer-term health and societal benefits of research [Reference Luke, Sarli and Suiter1,Reference Kwan, Brownson, Glasgow, Morrato and Luke6]. Such impacts can take longer to accrue and are difficult to attribute to a specific project or researcher [Reference Luke, Sarli and Suiter1,Reference Morgan Jones, Castle-Clarke, Manville, Gunashekar and Grant7]. Unfortunately, spending time and resources demonstrating this downstream impact has not been incentivized. Scientists often lack methods for tracking the impact of their research and are not trained to communicate their impact beyond academia. Promotion and tenure decisions are still perceived as valuing quantitative metrics, such as publications and grants, more than demonstrated impact [Reference Maddox, Phan and Byeon2]. As a result, researchers are typically communicating research outcomes through different channels (e.g., academic journals and conferences) than those used by practitioners and policymakers (e.g., workshops and professional associations) [Reference Brownson, Fielding and Green8].
Implementation science research and training has followed this same trajectory, as training programs and funders have typically focused on demonstrating traditional metrics (e.g., resulting grants and publications after participation) rather than the broader impacts of participants’ research projects [Reference Baumann and Carothers9,Reference Proctor, Landsverk and Baumann10]. Some intermediate measures of impact, such as increasing collaborations, have also been demonstrated [Reference Luke, Baumann, Carothers, Landsverk and Proctor11], but impact on policy and practice impacts remains elusive. The lack of attention to demonstrating impact is especially problematic for training in implementation science, which as a field is under even more pressure to produce this evidence of impact than other fields less centrally focused on optimizing impact.
The Translational Science Benefits Model & Translating for Impact Toolkit
Researchers from the Institute of Clinical and Translational Sciences (ICTS) at the School of Medicine and the Brown School of Social Work at Washington University in St Louis developed a new approach for assessing and demonstrating the broader impact of implementation science studies. The Translational Science Benefits Model (TSBM) allows public health and clinical scientists to assess the longer-term impacts of their work on health and society. It provides 30 specific, observable, and measurable indicators that reflect and capture the benefits that accrue from clinical and translational research. As defined by the National Center for Advancing Translational Sciences, translational research is the endeavor to traverse a particular step of the translational process for a particular target or disease [Reference Austin12]. The TSBM is applicable to research across the translational pipeline, including implementation science research to promote the adoption of evidence-based interventions and policies in health care and public health settings [13]. As depicted in Figure 1, the TSBM operationalizes downstream translational benefits of scientific research activity into four domains that are particularly relevant to clinicians, decision-makers, funders, and the public [Reference Luke, Sarli and Suiter1]:
Figure 1. Translational Science Benefits Model (TSBM) benefits by domain. TSBM benefits span four domains: clinical and medical benefits (i.e., adoption and implementation of new tools and procedures in clinical settings as a result of clinical and translational research), community and public health benefits (i.e., enhancement of healthcare or community and population well-being as a result of clinical and translational research), economic benefits (i.e., economic, commercial, or financial improvements that result from clinical and translational research), and policy and legislative benefits (i.e., involvement with the policy-making process or formal adoption into organizational or public policies, legislation, or governmental standards based on clinical and translational research).
-
Clinical and Medical Benefits: Adoption and implementation of new tools and procedures in clinical settings as a result of clinical and translational research.
-
Community and Public Health Benefits: Enhancement of healthcare or community and population well-being as a result of clinical and translational research.
-
Economic Benefits: Economic, commercial, or financial improvements that result from clinical and translational research.
-
Policy and Legislative Benefits: Involvement with the policy-making process or formal adoption into organizational or public policies, legislation, or governmental standards based on clinical and translational research.
Accompanying the TSBM framework, the Translating for Impact Toolkit includes a set of nine tools to plan, track, and demonstrate impact within these domains, illustrated in Figure 2. The Translating for Impact Toolkit guides researchers through the development of translational products better suited to communicate policy and practice impacts, including impact case studies [Reference Baumann and Carothers9]. Such products are especially useful in communicating the impacts of implementation research, where the audience is often implementers, such as community organizations and decision-makers, rather than primarily academic researchers [Reference Geng, Peiris and Kruk14].
Figure 2. Translating for Impact Toolkit components.
In this article we: (i) present the rationale for assessing benefits of implementation science beyond scientific productivity; (ii) present results of a pilot project in which Implementation Research Institute (IRI) alumni completed case studies; (iii) describe a set of training components and translational tools for assessing and communicating impact; and (iv) illustrate the integration of the TSBM and its tools into a mentored training network for implementation scientists. Integrating impact into implementation science training will help to normalize consideration of translational impact and ultimately enhance the visibility and impact of implementation science.
Materials and methods
Study context
The project was conducted within the IRI, a 2-year national training institute in mental health implementation science supported by a National Institute of Mental Health (NIMH) training and capacity building grant (2R25MH080916) and supplemented by the National Institute on Drug Abuse and the Department of Veterans Affairs. Components of the IRI include attending an annual weeklong training institute at Washington University in St Louis, being matched with a research mentor, traveling for a site visit to observe still-in-the-field implementation projects, receiving funding to conduct a pilot study, and attending an implementation science conference [Reference Proctor, Landsverk and Baumann10]. A description of IRI is reported in more detail elsewhere [Reference Proctor, Landsverk and Baumann10]. IRI fellows are selected from a competitive applicant pool and must demonstrate an interest in implementation research in the context of their career research agenda [Reference Proctor, Landsverk and Baumann10]. Because mental health implementation research informs improved access to evidence-based interventions, better models of healthcare financing and delivery, more effective treatment models, and faster implementation of research-based innovations in community settings, the TSBM model aligns closely with the IRI training goals. Other implementation science training programs have been modeled after IRI (e.g., the HIV, Infectious Disease and Global Health Implementation Research Institute, School Mental Health Assessment, Research, and Training Center, and Mentored Training for Dissemination and Implementation Research in Cancer), increasing the utility of TSBM as a component of implementation science training more widely.
In February 2021, we invited 15 purposefully selected alumni of the Implementation Research Institute to participate in a pilot study of the TSBM training. Because the research training program had not yet incorporated impact training, we asked training program faculty to nominate alumni who might have research benefits to report. These names were vetted further by training program leadership based on publication and grant funding records. The sample of nominated alumni came from different cohorts and from relatively early cohorts to increase the likelihood that, over time, benefits would become evident. Participants in the pilot program were offered: the Case Study Builder tool, videos, and other supportive material; interim feedback on draft case studies through individualized consultation from the TSBM team; and funds of $1,500 per case study to be used to support a research assistant or effort of the IRI alumni. Application required submission of an abstract identifying the project(s) upon which the case study would be based, identification of the type of impact the case study would demonstrate, and participation in a publication describing the study. Twelve alumni responded and attended an introductory workshop in March 2021. Six of the twelve submitted an abstract for the pilot project and began participation in May 2021.
TSBM training pathway
TSBM training included several components: sessions to guide case study development, a dissemination workshop, and a debrief session to conclude the process and finalize their translational products. Beginning June 1, 2021, the TSBM team provided consultation to participants as they used the Case Study Builder tool [15]. Participants shared the completed tool with the TSBM research team by July 15, at which point they began meeting individually on a virtual conference platform with a TSBM expert to receive feedback. Discussion focused on selection of translational benefits using the TSBM framework, categorization of benefits as demonstrated (e.g., those that have been observed and are verifiable) or potential (e.g., those logically expected with moderate to high confidence) (detailed in Table 1), and documentation of demonstrated impact within the case study. Given the wide-ranging benefits selected across the project, the TSBM team and IRI alumni used a range of evidence to demonstrate impact and included peer-reviewed literature, published reports or committee proceedings, websites, email communication, and other sources. Full documentation for all demonstrated impacts can be found in the published case studies. Final case study drafts were reviewed again by the TSBM team and published on the TSBM website. The TSBM team also completed impact profiles in collaboration with the IRI alumni. These shorter, one-page impact summaries are expected to be ideal for sharing with audiences who do not have the time for or interest in reading the full case study, such as policymakers.
Table 1. Demonstrated and potential translational benefits of Implementation Research Institute research
Summary of Translational Science Benefits Model benefits identified by the Implementation Research Institute training participants for their research projects across four benefit domains: clinical, community, economic, and policy. Each benefit is indicated as demonstrated (those that have been observed and are verifiable) or potential (those logically expected with moderate to high confidence).
For the next training step, participants worked with the TSBM team to develop dissemination plans for their case studies, considering with whom they wanted to share their case study (the audience) and how they would share their case study (the channels). For additional guidance on dissemination, participants were instructed to review the TSBM Dissemination Planner tool [16]. The TSBM team also encouraged the participants to share their case studies as widely as possible through their own networks (e.g., project teams, study participants, organizational partners, division/department chair, university/school dean’s office and communications department, and funders) and channels (e.g., websites, social media accounts, and blogs).
Participants returned in September–October 2021 for virtual debrief sessions on the number of benefits they had collectively identified and the distribution of these benefits across the TSBM domains. They also answered questions about the case study development process via virtual polling and subsequent discussion, including:
-
1. Has the case study process helped you to think more broadly about potential benefits of your research?
-
2. Has the case study process helped you to identify additional benefits of your research?
-
3. The time it took to develop my case study was too long, too short, or just right.
Participants were asked to develop a Lessons Learned section for their impact case study describing key takeaways from conducting their research. Potential lessons learned could be the importance of having a strong community partner, a critical discovery or innovation, or a key process or method that led to impact.
Calculation of results
Benefits indicated in the case studies were aggregated to look at trends in domain of impact across the alumni of the IRI mentored training network. Summary statistics are shared below. Results of the virtual polling were tabulated and summarized below.
Results
The IRI alumni participating in this pilot study developed six impact case studies and six impact profiles documenting the demonstrated and potential impact of their implementation science projects (shown in Figure 3). Participants were Caucasian and included five women and one man; demographics were similar to the 15 nominated and invited. The case studies covered a wide range of implementation and mental health topics, including increasing access to preventive telehealth services for women veterans, disseminating evidence about adverse childhood experiences to policymakers and the public, developing communication strategies to reduce addiction stigma, providing integrated support to treat parental substance use, improving the health of adults with serious mental illness, and preventing youth suicide and injury by implementing a secure firearm storage program. The research studies described in the case studies were funded by the National Institutes of Health, the National Institute on Drug Abuse, and the Veterans’ Administration. Though researchers were not required to indicate alignment in the TSBM case study application process, these impacts align closely with the objectives of Goal 4, “Advancing Mental Health Services to Strengthen Public Health” in the NIMH Strategic Plan, which aims to “improve the efficiency, effectiveness, and reach of mental health services through research (objective 4.1); expedite adoption, sustained implementation, and continuous improvement of evidence-based mental health services (objective 4.2); and develop innovative service delivery models to dramatically improve the outcomes of mental health services in diverse communities (objective 4.3) [17].” Consistent with researchers attending a training program, projects also were at varying stages of the research process – some had yet to begin active implementation while others had completed pilot testing phases and were focused on adapting or scaling the intervention. The impact case studies identified 5–12 translational benefits each, with an average of 8 benefits per case study. All case studies identified benefits in more than one TSBM domain, with an average of 3 domains per case study. A description of the impact of case studies is included in Table 2.
Figure 3. Implementation research institute impact case studies and impact profile. The example products illustrate Translational Science Benefits Model (TSBM) benefits across four domains: clinical and medical benefits (i.e., adoption and implementation of new tools and procedures in clinical settings as a result of clinical and translational research), community and public health benefits (i.e., enhancement of healthcare or community and population well-being as a result of clinical and translational research), economic benefits (i.e., economic, commercial, or financial improvements that result from clinical and translational research), and policy and legislative benefits (i.e., involvement with the policy-making process or formal adoption into organizational or public policies, legislation, or governmental standards based on clinical and translational research).
Table 2. Implementation Research Institute (IRI) impact case studies
Summary of six IRI case studies completed by training participants. Translational Science Benefits Model domains in which the case studies identified potential or demonstrated benefits are noted.
Together the six case studies identified 40 translational benefits of implementation science research and training. Of the 40 benefits, 22 were demonstrated benefits and 18 were potential benefits. Benefits included 21 community benefits (e.g., disease prevention/reduction, improved healthcare accessibility, and health education resources/tools), 11 policy benefits (e.g., scientific research reports and content for government legislation), five economic benefits (e.g., cost-effectiveness), and three clinical benefits (e.g., development of practice guidelines), as depicted in Figure 4. Although the clinical benefits were fewer in number, all clinical benefits had been demonstrated. Most of the (few) economic benefits were potential. A complete list of case study benefits is included in Table 1.
Figure 4. Translational benefits of implementation research by Implementation Research Institute (IRI) alumni. IRI research resulted in three clinical benefits (all demonstrated), 21 community benefits (12 demonstrated and nine potential), five economic benefits (one demonstrated and four potential), and 11 policy benefits (six demonstrated and five potential).
Participant feedback results
Participants reported that TSBM training helped them conceptualize their impact and develop a language for talking about impact with nonacademic audiences. They responded that the Case Study Builder tool in particular helped them select benefits and prompted thoughtful reflection. All (100%) reported that the case study process helped them think more broadly about the potential benefits of their research. Most (67%) reported that impact case studies were an effective way to disseminate their work, while one-third (33%) reported having no opinion.
Although the Case Study Builder did help participants identify benefits, some reported that it was challenging to select benefits for early-stage projects. Specifically, it was hard to anticipate the effects of these projects on downstream health outcomes. They also reported that some indicator definitions were unclear and that they were unsure how many benefits to include. Some participants described Portable Document Format (PDF) usability issues when completing the Case Study Builder tool, including difficulties entering text and saving progress.
The TSBM team published the six complete case studies on the TSBM website at translationalsciencebenefits.wustl.edu and developed and published IRI Impact Highlights 2021, a two-page PDF summary of all six case studies. The dissemination plan identified a primary audience of IRI and TSBM partners and a secondary audience of social science researchers and clinical and translational scientists. The dissemination goal was to demonstrate the use of the TSBM and the Case Study Builder and other Translating for Impact tools to highlight project impact in IRI research. The team’s active dissemination approach included:
-
Hosting the case studies on the TSBM website at translationalsciencebenefits.wustl.edu
-
Announcing the case studies via email to over 100 TSBM partners
-
Disseminating the case studies through: (1) IRI channels such as the Center for Mental Health Services Research newsletter; (2) Washington University ICTS channels, such as the ICTS website, monthly email digest, and Twitter feed; and (3) Brown School channels
-
Publishing an announcement in the Association of Schools & Programs of Public Health Friday Letter
The research team followed up with participants in late 2023 to see how they had disseminated their impact case studies. Five of six participants responded. All indicated that they had shared their TSBM case study with external audiences, most frequently with other researchers, clinical or public health practitioners, non-profits, or foundations. A few also shared their impact case study with priority populations and study participants. The most common method of sharing was via direct email, followed by presentations (to community partners or at scientific conferences).
Participants reported that creating a TSBM case study helped to increase the impact of their project in a number of ways. They noted that the case studies helped them translate their research for the public, demonstrate credibility to partners, and clearly present implications for practice.
“The TSBM case study has strengthened existing partnerships as partners have appreciated seeing the “big picture” impact of the project.”—IRI alumnus
Notably, even those working in earlier stages of their research found that the TSBM case study helped increase their impact, despite them having few demonstrated benefits to report at this stage. For projects at this juncture, the case study was a valuable planning tool.
“I have found that the TSBM is an important tool for helping to articulate planned impacts in the early stages of project and evaluation planning and pushing the conversation to support thinking in terms of direct, downstream impacts (rather than traditional upstream research outcomes). The case study has been helpful as a visual tool for supporting those conversations.”—IRI alumnus
Discussion
Results of the pilot demonstrated the value of TSBM for mental health implementation research training. The six case studies reflected a variety of projects and demonstrated a range of impacts of included mental health implementation research projects.
Implications for TSBM training. In their feedback, participants suggested several ways to enhance future TSBM training and tools. Selecting benefits was challenging, especially for projects in early development stages. To address this issue, TSBM team members updated and expanded indicator definitions. The TSBM team also added a place to create a benefits table directly into the Case Study Builder tool so that the tool more closely resembles the final impact case study format and allows for more space to describe benefits during case study development. To address these usability issues, the TSBM team expanded text box fields. Given reported PDF usability issues that complicated the development of impact products, in 2023 the TSBM team created a web-based version of the Translating for Impact Toolkit, which allows users to log in, enter responses directly into online forms, save their progress, and invite team members to collaboratively complete the tools. Future iterations of the TSBM tools will be developed in a web-based format to allow users to enter text, save, and return to their work later more easily.
Implications for implementation science training
The pilot program demonstrated the feasibility and value of training for impact mapping and reporting, even for early-stage researchers. The TSBM could supplement already well-established training in designing and conducting research with training in describing potential impacts and benefits that demonstrate the value of the research proposed. Accordingly, IRI leadership began including TSBM training for all IRI fellows earlier in the research process. Planning tools like the TSBM Roadmap to Impact [18], which was introduced to IRI alumni but not required in the pilot, can help researchers conceptualize impact before developing an impact case study. Developing impact tracking systems, especially for those benefits that are not readily monitored (e.g., expert testimony), could also help identify and demonstrate progress toward benefits, furthering the utility of the TSBM for demonstrating the impact of implementation science research and training.
The TSBM team and IRI leadership have further developed training methods and the suite of tools for building TSBM case studies, including the Case Study Builder. Even before conclusion of the pilot, and beginning with the 2019 cohort of 13 fellows, IRI directors adopted TSBM training into routine use, offering three types of support: (1) an introductory workshop on research impact during the annual training institute, (2) a second workshop on developing dissemination products to demonstrate impact, and (3) sessions to guide case study development, including interactive tools and one-on-one consultation with the TSBM research team to complete a set of dissemination products. Due to the COVID-19 pandemic, the workshops were held virtually. They involved didactic presentation and interactive components, including large group discussions and polling to engage fellows in conceptualizing how they could apply the TSBM model to their research. Fellows were also briefly introduced to the TSBM tools in the Translating for Impact Toolkit.
Limitations
In addition to the challenges with tool use and effects of the COVID-19 pandemic on TSBM training noted earlier in this section, findings in this paper are subject to a few important limitations. First, the study is a small pilot project with 15 invited IRI alumni resulting in 6 case studies. It is possible that in a larger sample, experiences using the TSBM may be more heterogeneous in terms of perceived benefits and challenges. IRI alumni who developed case studies were not randomly selected, as they were nominated and invited to participate (all who agreed were selected to complete case studies). Those who completed the case studies were likely to have been more confident in the impact of their work, which may have been more advanced than those who did not complete case studies. Most of those submitting completed case studies later became IRI faculty. They may have been more highly motivated or committed to implementation science training and/or demonstrating impact than the typical implementation scientist. Planning for generating and sustaining participation is a critical consideration when integrating the TSBM into future implementation science training models.
Conclusion
The intended goal of dissemination and implementation research is to bridge the gap between research and practice and build a knowledge base about how health interventions, clinical practices, and policies are integrated into public health and health care [19]. The TSBM model and training can help move implementation research toward achieving this goal by providing a language for talking about the impact of implementation science and creating dissemination products that help spread awareness of project innovations to audiences beyond academia, particularly those implementers who are most likely to play a role in achieving and sustaining downstream impacts (e.g., policymakers and practitioners) [Reference Kwan, Brownson, Glasgow, Morrato and Luke6,Reference Beidas, Dorsey and Lewis20]. These audiences prefer different formats and messaging to scientific publications [Reference Kwan, Brownson, Glasgow, Morrato and Luke6]. TSBM translational products can help meet this need by communicating benefits of implementation research in a narrative, plain language format that is free of jargon and highlights impacts relevant to these audiences’ interests. The case studies themselves also can be a training tool for other researchers to illustrate how impact was achieved and provide lessons learned for their translational activities. The case studies demonstrated impact of the individual projects but also the impact of the mentored training network itself, suggesting wider applications for the TSBM in demonstrating the impact of initiatives and programs. Program officers from the institutes funding the training were enthusiastic about the value of demonstrated downstream benefits. Case studies demonstrating return on research investment may bolster advocacy for implementation science training focused on translational impact.
Integrating impact into implementation science training is important to build researchers’ understanding of downstream impact, especially early in their careers. Earlier and greater focus on impact will help to normalize consideration of the broader clinical, community, economic, and policy benefits of implementation science, complementing traditional metrics such as publications and grants in the evaluation of individuals, projects, and training programs, including promotion and tenure metrics for implementation scientists. For example, at one of our home academic institutions (Washington University), impact is being formally recognized as part of the evidence supporting promotion and tenure, including specifically in the School of Medicine and a newly formed School of Public Health. At a time when the general public is increasingly skeptical about the academic enterprise, a focus on health and social impacts can only help us when we argue for the long-term importance of the clinical, translational, and implementation sciences.
Author contributions
Stephanie Andersen contributed to conception and design of the TSBM model, development of the TSBM tools, development of the case studies, data collection and analysis, and drafting of the manuscript. Ms. Andersen takes responsibility for the manuscript as a whole. Annalee Wilson contributed to development of the case studies, data collection and analysis, and drafting of the manuscript. Todd Combs contributed to conception and design of the TSBM model, development of the TSBM tools, data collection and analysis, and drafting of the manuscript. Laura Brossart contributed to conception and design of the TSBM model, development of the TSBM tools, interpretation of analysis, and drafting of the manuscript. Julie Heidbreder contributed to development of the case studies and data collection and analysis. Stacey McCrary contributed to conception and design of integration of the TSBM into the IRI training institute, development of the case studies, and drafting of the manuscript. Rinad S. Beidas contributed to development of the case studies, contribution of data, and drafting of the manuscript. Leopoldo J. Cabassa contributed to development of the case studies, contribution of data, and drafting of the manuscript. Erin P. Finley contributed to development of the case studies, contribution of data, and drafting of the manuscript. Emma E. McGinty contributed to development of the case studies, contribution of data, and drafting of the manuscript. Jonathan Purtle contributed to development of the case studies, contribution of data, and drafting of the manuscript. Lisa Saldana contributed to development of the case studies, contribution of data, and drafting of the manuscript. Enola Proctor contributed to conception and design of integration of the TSBM into the IRI training institute, development of the case studies, and drafting of the manuscript. Development of the case studies, contribution of data, and drafting of the manuscript. Douglas Luke contributed to conception and design of the TSBM model, development of the TSBM tools, and drafting of the manuscript.
Funding statement
This work was supported by the National Institute of Mental Health (grant number 1UH3DA050193-01).
Competing interests
Dr Beidas is principal at Implementation Science & Practice, LLC. She is currently an appointed member of the National Advisory Mental Health Council and the National Academies of Sciences, Engineering, and Medicine study, “Blueprint for a national prevention infrastructure for behavioral health disorders,” and serves on the scientific advisory board for AIM Youth Mental Health Foundation and the Klingenstein Third Generation Foundation. She has received consulting fees from United Behavioral Health and OptumLabs. She previously served on the scientific and advisory board for Optum Behavioral Health and has received royalties from Oxford University Press. All activities are outside of the submitted work. Dr Saldana is the developer of the Families Actively Improving Relationships – Families Actively Improving Relationships program, trademarked by the Oregon Social Learning Center. She currently is an appointed member of the National Academy of Science, Engineering, and Medicine study, “Blueprint for a national prevention infrastructure for behavioral health disorders.”
Open access
