Introduction
Engaging patients and other stakeholders in the continuum of biomedical research is a promising approach to creating more effective interventions and accelerating translation of effective interventions into practice [Reference Hoffman1–Reference Fleurence6]. Stakeholder engagement is a collaborative approach to research that values the unique perspectives and strengths of nontraditional research partners. It shares the spirit and goals of community-based participatory research (CBPR), but does not require the full model of CBPR as a starting point. Like CBPR, stakeholder-engaged research blends lived experiences of patients, expertise of service providers, power of policy makers, and rigorous science. The focus is on conducting relevant research, informing policy, and promoting successful implementation and long-term sustainability.
In parallel, comparative effectiveness research is the direct comparison of 2 or more interventions to determine which works best for which patients. When designed to evaluate the effectiveness of interventions in real-world practice with heterogeneous patient populations, the study is a pragmatic clinical trial [Reference Anderson, Califf and Sugarman7].
One of the first 5 large pragmatic clinical trials funded by the Patient-Centered Outcomes Research Institute (PCORI) is the Comprehensive Post-Acute Stroke Services (COMPASS) Study. COMPASS will randomize 41 hospitals (40 randomized units) and enroll a widely diverse group of 6000 patients across North Carolina (NC). Results of the trial will provide evidence to create national quality standards for post-acute stroke care delivery, which currently do not exist.
The purpose of the COMPASS Study is to evaluate the comparative effectiveness of a patient-centered, transitional care model that provides both structure and processes for post-acute care of stroke patients versus usual care. Our working hypothesis is that the patient-centered model will improve functional outcomes and reduce caregiver strain and hospital readmissions.
PCORI has established stakeholder engagement as a central tenet of their funded research portfolio. There is currently little guidance in the biomedical literature on how best to create and foster stakeholder engagement throughout the research process [Reference Pandi-Perumal8–Reference Concannon10]. The gaps include a lack of understanding of how to engage a broad range of critical decision makers across the health-care system; which methods and modes of stakeholder engagement are feasible during the earlier and later phases of research; how the views of stakeholders are synthesized and used to shape research design, implementation, and dissemination; and how to partner with stakeholders to increase intervention effects, reduce disparities, and/or sustain tested interventions [Reference Concannon10, Reference Wallerstein and Duran11]. Thus, the purpose of this article is to address gaps in the literature on stakeholder engagement in comparative effectiveness research by presenting the model that guides our stakeholder engagement efforts, and sharing the methods we used throughout the pre-award and post-award study design periods to inform best practice methods for engaging stakeholders in large pragmatic trials.
Methods
Overview
The COMPASS intervention is built on early supported discharge (ESD) studies in Canada and Europe [Reference Fearon and Langhorne12]. ESD is now standard of care in the United Kingdom [13] and Canada [14], but has not been used or evaluated in the US or rural settings. The COMPASS model of care combines ESD with Centers for Medicare and Medicaid Services (CMS) transitional care services provided by advanced practice providers (APP) and ESD services coordinated by the APPs and adapted for the US health-care system.
The COMPASS intervention provides both structure and processes to post-acute care for stroke patients. The innovative intervention combines services from a post-acute care coordinator (PAC) who is an registered nurse, and an APP (physician assistant (PA) or nurse practitioner (NP)) with linkages to community-based services to enhance continuity and coordination of stroke post-acute care. Patients are contacted at discharge, 2 days after discharge by phone, an in-person visit within 7–14 days after discharge, and phone calls at 30 and 60 days after discharge. During the clinic visit within 7–14 days of discharge, the PAC performs standardized assessments to assess the physical, social, and medical determinants of health and specifically screens for common post-stroke problems. The COMPASS intervention is unique in its focus on addressing post-discharge needs of both patients and family caregivers. An assessment of the patient’s functional status and related post-acute care needs, and the caregiver’s capacity for assisting the patient provide the basis for an individualized electronic care plan addressing specific needs; linking patients and caregivers to relevant community-based services (eg, caregiver support, medication management), and follow-up care. The PA or NP reviews the results of each assessment with the PAC and—incorporating input from the patient and caregiver—creates an individualized electronic care plan for each patient. This plan is shared with home health, outpatient therapists, primary care, and community service providers. Therefore, all providers—and the patient and caregiver—understand and have reviewed the plan of care. The effectiveness of the intervention is assessed via a 90-day outcome survey call to the patient, mail survey to the caregiver, and claims data.
Research reported in this manuscript was funded through a PCORI Award. PCORI had no role in the design of the study, data collection or interpretation, or writing or approving the finished manuscript.
Conceptual Model of Engagement
In designing our stakeholder engagement model and processes, we began with the engagement standards proposed by Curtis et al. [Reference Curtis15] in their report to PCORI’s Methodology Committee. Given the complexity of a large pragmatic trial, we expanded this rubric to include: (1) Patient and Other Stakeholder Identification and Selection; (2) Patient and Other Stakeholder Involvement Across the Spectrum of Research Activities; (3) Dedicated Resources for Patient and Other Stakeholder Involvement; (4) Support for Patient and Other Stakeholder Engagement Through Organizational Processes; (5) Communication with Patients and Other Stakeholders; (6) Transparent Involvement Processes; (7) Tracking of Engagement; and (8) Evaluation of Engagement. Below we describe each model component and how it was operationalized.
Patient and Other Stakeholder Identification and Selection
Our stakeholder partners are listed in Table 1. COMPASS team members have partnered for years with many groups conducting projects focused on hyper-acute and acute stroke care in NC. The NC Stroke Care Collaborative [Reference George16, Reference Rosamond17], originally funded as a Paul Coverdell National Acute Stroke Registry by the Centers for Disease Control and Prevention, was established in 2004 and provides the COMPASS Study infrastructure. The Collaborative had partnered with the American Heart Association/American Stroke Association’s Get With The Guidelines® Stroke program to improve quality of acute care. The academic centers that came together to create COMPASS had also worked with the NC Department of Health and Human Services and the Justus-Warren Heart Disease and Stroke Prevention Task Force on stroke research and improving systems of care. The current COMPASS project capitalized on extant systems and relationships around stroke systems of care in NC and expanded their focus from the acute care setting to the post-discharge setting.
Table 1 Stakeholder groups engaged in the Comprehensive Post-Acute Stroke Services (COMPASS) Study
SES, socio-economic status; NC, North Carolina; NCDHHS, NC Department of Health and Human Services; IT, information technology; eCare Plan, electronic care plan; PC, personal computer; NCSA, NC Stroke Association.
While preparing the PCORI proposal, we identified stakeholder partners from among those listed above, who actively participated in the study planning and grant writing. Still more stakeholders were added after funding was awarded, primarily from the study team’s professional networks. These key stakeholders have established working relationships with the study team. Importantly, they were enthusiastic about joining the project and did not require a lengthy pre-engagement period to agree to participate. Prompt participation of stakeholders was essential to meet the rapid turnaround time between PCORI’s acceptance of the COMPASS letter of intent and the deadline for the full proposal. We later expanded the number of partners during study design phase, seeking to include all stakeholder groups who would be affected by the study and could influence its success. COMPASS now includes all project-relevant stakeholder groups whose input has shaped intervention design and implementation plans, for the purpose of maximizing effectiveness and uptake.
The academic members of the COMPASS team represent 5 major universities in NC with expertise in neurology, nursing, rehabilitation, pharmacy, primary care, public health, health services research, quality improvement, epidemiology, and biostatistics. Importantly, they also have decades of experience engaging stakeholders in health services research, health system change, and policy change.
We included thought leaders in each stakeholder group, including state policy makers, to support dissemination and long-term sustainability. A previous systematic review of stakeholder engagement in research indicated that engagement with patients was frequent, whereas engagement with clinicians and other stakeholders like payers, product makers, and policy makers was not [Reference Concannon10]. In COMPASS, we have included a broad range of critical decision makers across the health-care system and community to maximally support patients’ recovery after stroke (see Table 1).
Importantly, we sought the participation of a racially diverse group of patient partners who share lived, but varied, experiences of stroke recovery. We also wanted to be guided by people with experiences in advocacy and hospital politics who understood the communities in rural eastern NC, where the risk for stroke is the highest in the nation. The lead patient partners for COMPASS are a female African-American stroke survivor known in NC as an effective advocate and patient leader in initiatives for stroke and heart disease prevention and a male Caucasian stroke survivor who, after his recovery, became involved in his local hospital and is now vice chair of a hospital board in eastern NC. These choices are important because African-Americans, women, and residents of the eastern part of the state are at greater risk of poor outcomes after a stroke and their voices needed to be incorporated into the COMPASS Study.
Patient and Other Stakeholder Involvement Across the Spectrum of Research Activities
We embedded patients and other stakeholders in the COMPASS Steering Committee and all study subgroups according to their self-reported interests and skills, so they have input and support decision making across the research process. Stakeholders experience the iterative nature of research and their input is incorporated into research decisions. Examples include educating data collectors to slowly read survey questions so stroke survivors can understand them, selecting outcome measures, and making the consent process understandable. This diffusion of stakeholder influence rests on shared leadership and facilitates management of stakeholder engagement for a trial of this size and complexity.
A challenge of engaging a large and diverse number of stakeholders across sectors and across a wide geographic area is that not all stakeholders interact with all investigators. However, by embedding stakeholder partners in all subgroups, researchers and stakeholders can maintain close working relationships around the topics of their stated preferences. Stakeholders’ self-reported interests, expertise, time, and communication requirements informed who would work on which tasks. We consciously allow the level and type of engagement to vary by stakeholder according to interest, skill, time, health status, project needs, and budget constraints. We anticipate that these strategies will leverage stakeholders’ strengths and interests and foster more meaningful and productive participation.
During the study planning/design phase we included stakeholders in 136 engagement activities, 18 of which occurred during proposal writing or before PCORI funding was awarded, 118 within the first 10 months of the grant. Using PCORI’s definition of engagement levels, of these 136 engagement activities, 66 (48%) were information sharing, 40 (30%) were consultation, 17 (12%) were collaboration, and 13 (10%) were stakeholder directed. Engagement also cut across levels of influence (Fig. 1).
Fig. 1 Stakeholder groups that participated in the planning of the Comprehensive Post-Acute Stroke Services (COMPASS) study.
To illustrate, multiple stakeholders from multiple groups defined intervention goals, objectives, and key messages; and developed intervention materials and activities based on the existing scientific evidence [Reference Fearon and Langhorne12, Reference Condon18, Reference Naylor19], payment structures, clinical workflow, and stroke patients’, caregivers’, and clinicians’ lived experiences. Multiple stakeholder groups also determined intervention-specific roles, and the most effective training methods for these various roles given existing time constraints. These groups also developed intervention training modules and materials and will train hospital-based and community-based providers. Patient and caregiver feedback was used to determine our selection of outcomes and data collection procedures.
Patients and caregivers also shaped our consent process for data collection. For example, our partners reviewed several iterations of the consent form to retain Institutional Review Board (IRB)-required content that will (1) foster comprehension among potential participants with a range of health literacy levels, and (2) not raise concerns among potential control group participants (receiving usual care) because they will not receive the intervention. The result is a consent protocol for pragmatic trials that meets IRB requirements, but will be easier to understand and be received favorably by potential participants.
Community-based service providers assisted with hospital and patient recruitment by leveraging their professional and personal connections to key decision makers within hospitals who made the decisions about enrolling their hospitals or hospital systems in the trial.
Hospital-based stakeholders included stroke program coordinators, stroke neurologists, physician emergency department directors, quality/performance improvement coordinators, nurse managers, patient education directors, and senior hospital administrators. These stakeholders made a key contribution to the study design by advocating for a delayed start design [Reference D’Agostino20], meaning that, at the end of the trial, all hospitals enrolled in the study eventually will receive the intervention. At baseline, hospitals are randomized to either administer the COMPASS intervention or to provide their usual standard of care (control). In phase 1, patients in these hospitals are followed for 90 days so that the effects of the intervention on outcomes can be observed. In phase 2, all hospitals implement the COMPASS intervention, and the data obtained during this phase will be used to evaluate how well the intervention hospitals sustained the intervention without grant support.
Our statistical collaborators agreed that this design provides the best statistical control to measure the intervention’s effects, and our hospital stakeholders agreed that this design would help ensure that all communities benefit from study participation, which was essential to generate buy-in from individual hospitals and larger health systems.
Although some stakeholder partners contribute to the COMPASS Study with limited financial compensation (patients, caregivers, Piedmont Triad Regional Council Area Agency on Aging), most do not. Several patients and caregivers decline payment because they value “giving back” to their communities and being involved in a state-wide study aimed at improving the way we deliver care to stroke patients. Other stakeholders recognize value in their participation in other ways. Examples include study participation facilitating entry into hospital systems to build previously inaccessible partnerships and access to study team members, who can advise stakeholder organizations on survey design or program design unrelated to stroke care. Building such relationships is time consuming, but making them a priority holds mutual benefits and sows the seeds of long-lasting partnerships.
Dedicated Resources for Patient and Other Stakeholder Involvement
The more stakeholders are engaged, the more feedback is received. This can lead to still more relationships, more feedback, and more management. In the COMPASS Study, a faculty “engagement officer” (25% effort) is charged with developing and executing a strategy to weave engagement through the trial and holding the study team accountable. A research coordinator (100% effort) is dedicated to engagement efforts. The oversight of engagement activities is centralized, but all faculty and staff team members (except biostatisticians) spend time leading engagement activities in study subgroups (eg, intervention, outcomes measurement). COMPASS project managers can also direct additional administrative staff to support engagement efforts when needed. The engagement and administrative teams meet weekly to discuss administrative support for engagement.
Support for Patient and Other Stakeholder Engagement Through Organizational Processes
Our initial orientation of stakeholders to the COMPASS Study includes a discussion of the overall importance of engaging stakeholders in research, and the various levels of and opportunities for engagement (and compensation) in COMPASS, in particular. Our goal is not to turn stakeholders into researchers but to allow their expertise to shape the study design, implementation, and dissemination. We clarify that we are seeking partners with complementary strengths and unique perspectives on the health-care system and that understanding their needs is critical to the success of the project.
During the pre-award period, the researchers developed a stakeholder management plan (“roadmap”) outlining where stakeholders could influence the study. The study team firmly believed that engagement had to be an iterative process in the post-funding phase because, if we truly partnered with stakeholders, their influence could certainly influence the study direction. Creating a stakeholder roadmap allowed us to target defined actions to engage stakeholders quickly, even before the study begun. Our stakeholder engagement plan focused on 4 key sets of activities: (1) study planning (including study design, intervention design and procedures, outcomes measurement, materials); (2) hospital/patient recruitment and retention; (3) study implementation; and (4) translation, including interpreting study findings and disseminating results back to participating communities and the public. Within each activity, we have a series of circumscribed projects defined (see Table 2).
Table 2 The Comprehensive Post-Acute Stroke Service (COMPASS) study stakeholder engagement roadmap
QI, quality improvement.
Given the rapid start-up phase, we wanted to be sure to capture any additional opportunities for engagement. Thus, we have designated and trained 1 researcher in each study subgroup to be the engagement advocate. This person notifies the faculty engagement officer either when engagement has occurred or when additional engagement of stakeholders will be needed as the study progresses. This information is tracked in the Research Electronic Data CAPture (REDCap) data system’s engagement tracker, which is described below.
Communication with Patients and Other Stakeholders
For stakeholders not continuously engaged in the COMPASS Study in the ways already noted, study updates are provided via conference calls, or by COMPASS team member updates at regional meetings that those stakeholders were already scheduled to attend. The team includes skilled facilitators with expertise in leading focus groups or group feedback sessions. In addition, the team includes experts in primary care practice and pharmacy quality improvement; physical therapy; and family caregiver needs. They engage with their respective communities for this study where their contacts and stature as opinion leaders are most effective. An important component of our stakeholder roadmap is explaining to partners how their feedback was incorporated and, if not, why not—and in a timely manner.
Face-to-face contact is invaluable for building relationships, and all study team members have spent significant time traveling to meet with stakeholders. Yet, limited time and resources for state-wide travel necessitate using regular conference calls, webinars, and email contacts. When patient and caregiver stakeholders do not have email access or cell phone minutes, we call them at their homes. Openness to special arrangements is important. For example, to connect with 1 highly engaged couple (each both a stroke survivor and a caregiver) in rural NC, we email their neighbor. This person prints out emails and drives them to our partners, who then call us from their house phone when they have time to talk. We also mail documents to our partners when that seems most practical.
Transparent Involvement Processes
Including patients, family caregivers, clinicians, and organizations dedicated to improving care for stroke patients in COMPASS subgroups, including the Steering Committee, allows for information exchange, keeps the patient perspective central, and fosters transparency. To make our engagement processes clear, we clarify roles, expectations, compensation, workload, timelines, deliverables, and decision-making processes.
Most of our stakeholder interactions are defined by the IRB as non-research activities (eg, committee work, team meetings), but some are defined as research and require IRB approval (eg, interviews, focus groups). This distinction is made explicit to our stakeholders.
Our written compensation policy has evolved to account for stakeholder feedback while meeting PCORI principles of reciprocal relationships, trust and transparency, and budget constraints. For example, we do not reimburse clinical or industry partners for their participation, but we do reimburse patient and caregiver participants. Because employees of our main community-based partner cannot accept consulting fees, we budgeted support to the organization in a subcontract. We make clear in advance when we cannot compensate individuals for their time in an upcoming activity. Some choose to participate and others decline. This flexible participation is an integral part of our model.
Tracking of Engagement
Central to our infrastructure is a system to capture engagement work and stakeholder feedback so that we can track what actions were ultimately taken based on their input and to better delineate the impact of a given strategy on outcomes. To do this, we developed an engagement database within the REDCap software system, a secure platform for research data collection and analyses developed within the Clinical and Translational Science Award network [Reference Harris21]. Our database was designed to (1) document all engagement activities systematically, (2) monitor the complexities of our engagement process, (3) facilitate data collection for the annual engagement report to PCORI, and (4) tie engagement activities to processes and eventual outcomes. Data include which stakeholder groups were involved; when, where, why, and how they were engaged; their level of engagement (using PCORI’s framework); how challenges to engagement were overcome, and how stakeholder feedback shaped the study. The database includes a dashboard to visually track each engagement activity and help the engagement team ensure that activities are on schedule (see Fig. 2).
Fig. 2 Screenshots from the Research Electronic Data CAPture engagement tracker.
Evaluation of Engagement
We will monitor our engagement process through proactive oversight of milestones and deliverables spelled out in our PCORI contract and engagement proposal.
We will perform quarterly check-ins via survey to ask all patient and stakeholder partners if they feel their perspectives are valued by the study team, if they are satisfied with their level of involvement, if they are involved in the activities they want to be involved in, and if the methods used to elicit their feedback match their preferred style of communication. These data will be used to continuously improve study processes, which signals to all stakeholders the power and value of their voice.
Conclusion
This paper describes the comprehensive methods used to incorporate stakeholders in the planning and design of a large pragmatic trial. To date, we have successfully included perspectives and input from multiple stakeholder groups in a timely fashion because our efforts were driven by a sound conceptual model and built on an existing state-wide infrastructure. This paper is intended to provide guidance to others planning or conducting stakeholder-engaged research as we collectively attempt to identify evidence-based approaches by which engagement can enhance research and close the gap from evidence to action.
Acknowledgments
The authors thank Lynette Staplefoote-Boynton, Betsy Vetter, Gladys Lundy, Laurie Mettam, and Mysha Sissine for their scientific contributions.
Author Contributions
S.B.G. led conceptualization and design of the work, and drafted and critically revised the article for important intellectual content. K.P.K. made substantial contributions to the conceptualization and design, and made substantial revisions to the article for important intellectual content. J.H., J.P.B., J.F., D.M.C., B.J.L., S.C., C.B., W.R. made substantial contributions to the conceptualization and design, and critically revised the article for important intellectual content. P.W.D. made substantial contributions to the conceptualization and design, critically revised the article for important intellectual content, obtained funding for the work, and provided overall supervision. All the authors approved the version submitted for review; all authors agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Financial Support
Research reported in this publication was funded through a PCORI Award (PCS-1403-14532). The statements presented in this publication are solely the responsibility of the author(s) and do not necessarily represent the views of the PCORI, its Board of Governors or Methodology Committee. The authors acknowledge the REDCap support of the Wake Forest Clinical and Translational Science Institute, which is supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant Award No. UL1TR001420 (Clinical Trials.gov ID No. NCT02588664).
Declaration of Interest
None.
Supplementary Material
To view supplementary material for this article, please visit https://doi.org/10.1017/cts.2016.26
Open access
