PUBLIC ENGAGEMENT AND THE ROLE OF PROFESSIONAL TRAINING

In recent decades, public engagement has moved from the fringes of discourse within the political science discipline to become a widely accepted part of the work of political scientists. One key early marker in this movement for greater public engagement was Robert Putnam’s 2002 American Political Science Association (APSA) presidential address. In his speech, Putnam argued that “an important and underappreciated part of our professional responsibility is to engage with our fellow citizens in deliberation about their political concerns, broadly defined,” adding that “[p]olitical science must have a greater public presence” (Putnam Reference Putnam2003, 249). In the subfield of international relations, prominent scholars issued similar calls (Jentleson Reference Jentleson2002; Nye Reference Nye2008; Walt Reference Walt2005). More recently, political scientists outlined different mechanisms through which scholars can connect with public audiences and policy communities (Busby Reference Busby2018; Campbell and Tama Reference Campbell, Tama and ThiesForthcoming; Murphy and Fulda Reference Murphy and Fulda2011); identified best practices for effective and responsible public and policy engagement (Barma and Goldgeier Reference Barma and Goldgeier2022; Berling and Bueger Reference Berling and Bueger2013; Levine Reference Levine2020, Reference Levine2021; Mikhael and Norman Reference Mikhael and Norman2021; Toft Reference Toft2018); and highlighted conditions that facilitate impactful links between research and policy (Avey et al. Reference Avey, Desch, Parajon, Peterson, Powers and Tierney2022; Desch Reference Desch2019; Maliniak et al. Reference Maliniak, Peterson, Powers and Tierney2020; Tama et al. Reference Tama, Barma, Durbin, Goldgeier and Jentleson2023).

APSA has supported several initiatives intended to foster public engagement by political scientists. In 2015, an APSA task force on public engagement issued recommendations for how the discipline can pursue public outreach more effectively (Lupia and Aldrich Reference Lupia and Aldrich2015). Between 2016 and 2018, APSA coordinated workshops that were designed to train scholars in communicating about their research in the public arena.Footnote ¹ In 2020, an APSA task force on new partnerships developed a series of programs to facilitate public scholarship and partnerships between political scientists and institutions outside of academia (Smith et al. Reference Smith, Rasmussen, Galston, Han, King-Meadows, Kirkpatrick, Levine, Lieberman, Mylonas, Rigger, Sinclair-Chapman, Shay, Van Vechten and Grigg2020). APSA’s new civic engagement section and Institute for Civically Engaged Research represent additional efforts within the discipline to connect with communities beyond academia (Bennion Reference Bennion2022; Dobbs et al. Reference Dobbs, Hess, Bullock and Udani2021).Footnote ² At the same time, many political scientists have demonstrated their interest in public engagement by writing for nonacademic outlets. As of 2019, almost 3,500 political scientists had published in the Monkey Cage (Farrell and Knight Reference Farrell and Knight2019).

Yet, political scientists continue to face challenges when it comes to public and policy outreach. Perhaps most important, the professional criteria for evaluating university-based political scientists still do not tend to give much weight to public engagement (Desch et al. Reference Desch, Goldgeier, Petrova and Peh2022; Kendrowski Reference Kendrowski2022; Lupia and Aldrich Reference Lupia and Aldrich2015; Maliniak, Peterson, and Tierney Reference Maliniak, Peterson and Tierney2019). Even in countries where university “impact” is explicitly evaluated by government agencies, promotion criteria for individual faculty members may not value public engagement (Williams and Grant Reference Williams and Grant2018). In addition, with a few exceptions—including the APSA programs noted previously and those run by Bridging the Gap (BTG) (see Jentleson Reference Jentleson2015)—political scientists typically are not trained in how to disseminate their work to nonacademic audiences or to engage with policy practitioners. There also do not seem to be any published studies evaluating the impact of programs designed to train political scientists in public and policy outreach.

However, there are empirical studies of public-engagement training in the natural sciences and across disciplines, and this research highlights the value of such training. Some of this work has been conducted in the United Kingdom, motivated in part by the adoption of the Research Excellence Framework, which requires universities to demonstrate impact beyond academia. Notably, studies have found that scholars in different disciplines are more likely to be willing to participate in public engagement if they have received training in it or if they feel confident or well equipped for it (Burchell, Sheppard, and Chambers Reference Burchell, Sheppard and Chambers2017; Hamlyn et al. Reference Hamlyn, Shanahan, Lewis, O’Donoghue, Hanson and Burchell2015; Seakins and Fitzsimmons Reference Seakins and Fitzsimmons2020). Moreover, the studies reveal that many scholars are reluctant to engage in public outreach because they feel that they lack the skills or expertise needed to do so (Stylinksi et al. Reference Stylinski, Storksdieck, Canzoneri, Klein and Johnson2018). One study of a program that trains natural scientists in public engagement found that the participants reported improvement in their engagement skills and an increase in the frequency of their engagement after undergoing the training (Stylinksi et al. Reference Stylinski, Storksdieck, Canzoneri, Klein and Johnson2018).

We built on these findings by conducting the first study of the impact of training in public and policy engagement for political scientists. We hypothesized that training in public and policy engagement will make political scientists (1) more knowledgeable about options for public and policy engagement, (2) more well prepared to conduct public and policy engagement, and (3) more confident in their public- and policy-engagement skills.

STUDY DESIGN

We conducted the study as part of the 2022 International Policy Summer Institute (IPSI), which is conducted by BTG, a multi-university initiative that aims to foster greater links between scholars of international policy issues and communities outside of academia. Admission to IPSI is based on a competitive application process. For the 2022 IPSI, BTG selected 24 faculty and postdoctoral scholars from various US and international universities to attend the four-day training program, which took place at American University in Washington, DC. Of the 24 participants, 21 were political scientists; the remaining three were trained in history, peace and conflict studies, and African studies. Of the 24 participants, 17 self-identified as female in our pre-workshop survey; the remaining seven self-identified as male (Tama, Rublee, and Urban Reference Tama, Rublee and Urban2023).

The program sessions were directed by six international relations professors affiliated with BTG. They included discussions of engaging with policy officials in the areas of peace building, international development, and security policy; communicating with public audiences through newspaper op-ed articles, blog posts, and podcasts; writing for policy-oriented journals and magazines; developing a media strategy; interacting with think tanks; engaging with congressional staff; and pursuing policy engagement in a responsible and ethical manner. Speakers included academics who have conducted various forms of public and policy engagement, as well as current and former government officials, think-tank experts, newspaper and magazine editors, and communications professionals. The participants also gained experience in being filmed for a mock television interview about their research—after which they received constructive criticism about their performance—and they workshopped their own draft op-ed articles and blog posts in small groups. (See the online appendix for the complete workshop agenda.)

To evaluate the effectiveness of IPSI, we designed pre- and post-survey questionnaires designed to measure changes prompted by the workshop. Used widely across academia, the pre-test/post-test design is useful because it permits researchers to determine to what extent a workshop, training session, or other intervention makes a difference in respondent answers. A positive change between the pre- and post-workshop answers indicates that the intervention was effective (Davis et al. Reference Davis, Baral, Strayer and Serrano2018). As Stratton (Reference Stratton2019, 573) noted, “An advantage of a pre-test and post-test study design is that there is a directionality of the research, meaning there is testing of a dependent variable (knowledge or attitude) before and after intervention with an independent variable (training or an information presentation session).” To ensure that any changes can be attributed to the intervention rather than other events, the post-test must be completed soon after the intervention.

We piloted the survey design with participants in a similar policy-engagement workshop, facilitated by BTG personnel, for Australian academics in September 2020 (Rublee Reference Rublee2023). Using the results from the pilot study, we designed the survey for IPSI participants to measure their perceptions of knowledge, confidence, and preparedness in policy engagement before and after the workshop. (The full surveys are available in the online appendix.) We asked questions related to three types of policy engagement: conducting media interviews, writing for nonacademic outlets, and engaging with policy officials and practitioners. The first question was related to knowledge about mechanisms and pathways for these three types of engagement, using a 5-point Likert scale: very knowledgeable, knowledgeable, somewhat knowledgeable, a little knowledgeable, and not at all knowledgeable. The second question asked about how well prepared respondents felt to conduct these three types of policy engagement, using a 5-point Likert scale: very well prepared, well prepared, somewhat prepared, a little prepared, and not at all prepared. The third question asked how confident participants felt in their skills for conducting the three types of policy engagement, using a 5-point Likert scale: very confident, confident, somewhat confident, a little confident, and not at all confident. Other questions in the surveys asked about the participants’ experience in policy engagement, whether the workshop met their expectations, and how the program could be improved in the future. The only demographic question asked was regarding gender. Because we knew that the cohort would be small, including other demographic questions would have allowed us to identify respondents, and our priority was to maintain anonymity for participants.

The surveys were anonymous and administered through Qualtrics by email, with ethics approval granted through one of our universities. The pre-workshop survey was emailed to all 24 members of the 2022 IPSI cohort on May 31; all participants responded by June 7. The post-workshop survey was emailed on the last day of IPSI (June 15), and we closed responses on June 30. Of the 24 respondents, 21 completed the post-workshop survey (i.e., 87.5%) and all of them completed the survey in its entirety.

FINDINGS

The survey results indicated that participation in the four-day IPSI greatly increased scholars’ knowledge, preparedness, and confidence regarding public and policy engagement. Across all three questions and types of policy engagement included in the questionnaire, the pre- and post-workshop surveys revealed large and statistically significant increases. These findings support our hypotheses that offering political scientists training in public and policy engagement will assist them in becoming more knowledgeable about pathways for engagement, better prepared to conduct engagement activities, and more confident in their public- and policy-engagement skills.

The following subsections report the results in two ways for each survey question. First, we report the percentage of respondents whose answers were in one of the top two categories (i.e., knowledgeable or very knowledgeable; well prepared or very well prepared; and confident or very confident). In doing so, we combined the top two categories into a single category (i.e., knowledgeable, prepared, or confident) to simplify the presentation of results. Second, we report the average respondent score using Likert-scale values, with 5 representing the top category of very knowledgeable, very well prepared, or very confident and 1 representing the bottom category of not at all knowledgeable, not at all prepared, or not at all confident. The results for each question reveal substantial shifts using both measures, and t-tests indicated that all of these increases were statistically significant (p<0.001).

Knowledge

Participant knowledge about public and policy engagement increased dramatically across the different domains of media interviews, nonacademic outlets, and engagement with policy officials and practitioners. (See Figure 1.) Before IPSI, none of the participants reported feeling knowledgeable about pathways or mechanisms toward conducting media interviews. When the responses were evaluated using Likert-scale values, the average score for this question was 1.71. In the post-workshop survey, the percentage of participants reporting knowledge of pathways or mechanisms toward conducting media interviews had increased to 81%, with an average score of 3.86. Similar increases were reported across other forms of policy engagement. The proportion of participants feeling knowledgeable about pathways or mechanisms to write for nonacademic outlets increased from 25% before IPSI to 95% after IPSI, with average scores increasing from 2.88 to 4.48. Only 4% of participants reported knowledge for pathways or mechanisms to engage policy officials and practitioners in the pre-workshop survey; afterwards, 67% did. The average Likert score increased from 2.12 to 3.81.

Figure 1 Knowledge of Forms of Public and Policy Engagement

Note: The bars show the percentages of survey responses in the top two categories for each question.

Preparedness

Participants also reported substantial increases in their preparedness for different types of public and policy engagement. (See Figure 2.) None of the participants felt prepared to conduct media interviews before participating in IPSI. By the end of the workshop, 52% reported being prepared to do so, with a corresponding increase in average Likert-scale values from 1.75 to 3.67. Only 25% of participants felt prepared to write for nonacademic outlets before IPSI, whereas a full 100% of respondents felt prepared after the workshop. Average scores for this question increased from 2.88 pre-IPSI to 4.57 post-IPSI. Finally, the proportion of participants who felt prepared to engage policy officials and practitioners increased from 4% to 76%, with a change in average scores from 2.33 to 3.9.

Figure 2 Preparedness for Forms of Public and Policy Engagement

Note: The bars show the percentages of survey responses in the top two categories for each question.

Confidence

The results were much the same for the participants’ confidence in their ability to engage with the public and policy makers. (See Figure 3.) The proportion of respondents who reported feeling confident in their ability to conduct media interviews increased from 0% pre-IPSI to 57% post-IPSI, and the average value of the Likert scores for this question increased from 1.83 to 3.48. Before IPSI, 29% of respondents reported feeling confident in their ability to write for nonacademic outlets, with an average score of 3.00. After IPSI, those figures had increased to 86% and 4.24, respectively. Whereas only 8% of participants were confident in their ability to engage policy officials and practitioners before IPSI, 71% reported confidence in this ability after the workshop. Average Likert-score values for this question increased from 2.5 to 3.81 during the same period.

Figure 3 Confidence in Public- and Policy-Engagement Ability

Note: The bars show the percentages of survey responses in the top two categories for each question.