WHAT DO POLITICAL SCIENTISTS THINK?

To provide a better understanding of political scientists’ attitudes toward ChatGPT and plans for addressing ChatGPT in the classroom, we developed and distributed a brief survey in March 2023 to a sample population of political science faculty in the United States (see online appendix A for the survey instrument).Footnote ⁵ The survey initially was sent via email to 100 political science faculty members, requesting each of these recipients to share the survey link with their colleagues.Footnote ⁶ Our 100 survey requests resulted in 109 completed survey responses from faculty members in 27 different states and more than 40 unique colleges and universities representing minority-serving institutions and private, public, and community colleges (Ardoin and Hicks Reference Ardoin and Hicks2024).

Our sample included more men than women: almost 60% identified as men, 39% identified as women, and the remaining 1% identified as other. Slightly more than 80% of our sample were faculty members who identified as white; approximately 7% identified as Latino, 4% as Black, 2% as Asian, and 4% as Middle Eastern/North African. The plurality of our sample included respondents who predominantly teach American politics (40%). However, about 17% teach comparative politics, 14% international relations, 8% public administration or policy, and 6% political theory. Approximately 40% of our sample included faculty members who teach in departments that grant PhDs; slightly more than 35% teach in departments whose highest degree granted is an MA, 20% in departments that grant only BA/BS degrees, and the remaining 2% in community colleges. Most of our sample included faculty who teach two or three classes per semester: 38% and 41%, respectively. Although precise data on political scientists are difficult to obtain, recent estimates from the American Political Science Association suggest that our sample population is reasonable with respect to many of these dimensions (Smith Reference Smith2023). Specific information regarding our sample is in online appendix A.

When respondents were asked to identify the primary challenges or threats created by ChatGPT or other AI chatbots to the teaching and learning experience in higher education, their survey responses generally reflected concerns noted in the literature. The two most common concerns were student cheating or plagiarism (37%) and the challenges that ChatGPT presents for assessing student work (31%). Respondents also noted concerns regarding the long-term impact that ChatGPT may have on students’ writing and research skills (26%). Surprisingly, only a few respondents noted any concerns with ChatGPT’s tendencies to fabricate information or hallucinate (6%).

The findings from our survey focus on two broad themes. First, we asked a set of specific questions to faculty members about what they currently are doing or plan to do to prevent cheating using ChatGPT or other AI. Second, we asked them their abstract opinions about what the advent of ChatGPT and other AI means for the future of teaching and learning. We explore and discuss faculty members’ opinions about these themes in this order.

We also were interested in learning why some answers to these questions might systematically vary. For example, why might some faculty members actively change critical features of their courses in light of ChatGPT, whereas others give the technological innovation the equivalent of a shoulder shrug? We view our research as somewhat exploratory with respect to this purpose. We encourage future researchers to collect more data and to evaluate similar and different covariates. That said, two covariates that we explored are rank (i.e., junior versus senior) and underlying concerns with cheating in general; for example, perhaps some faculty members simply are more concerned with cheating in any form and therefore are more activated by technologies such as ChatGPT.Footnote ⁷

We view these as reasonable covariates to explore. With respect to rank, it seems reasonable to assume that higher-ranking faculty members may be more antagonistic toward AI than lower-ranking faculty members because, on average, the former have spent more time developing and administering teaching and learning materials in a pre-AI/pre-ChatGPT world. With respect to cheating concerns, we assume for various reasons that some faculty members have deeper concerns about student conduct and cheating (whether these concerns are based in experience or personality traits, we do not know), and these concerns could lead to higher levels of concerns about new technologies that change the nature of cheating.

To measure respondents’ underlying concern for cheating, we combined their answers to two different questions. Respondents were asked how concerned they were about students: submitting another student’s work as their own and hiring or having someone write their papers. The choice of answers ranged from “none at all” (0) to “a great deal” (5). These two questions had a Cronbach’s alpha of 0.7996, which indicates that they measured the same construct. We viewed these questions as indicators that correlate with larger concerns about cheating in general. We first calculated the average value for each respondent across these two questions. We then divided respondents into two groups based on their average scores. Those whose mean scores were above average were classified as more concerned with cheating; those whose mean scores were at or below average were classified as less concerned. We reduced the number of categories in this covariate because our sample is small. (We explore an alternative conception in which we divided respondents into thirds in online appendix A.) Of course, we wanted to use the full richness of every covariate but our sample was sufficiently small as to make such an exercise more problematic.

To measure faculty rank, we asked faculty members how long they have taught at a college or university. We used this question to divide faculty into junior and senior groups. Junior-level faculty were grouped as respondents who said that they have taught at a university or college for seven or fewer years. Respondents who have taught longer than that were classified as senior faculty. We explore how our outcomes vary across a wider array of teaching experience in online appendix B. However, similar to cheating concerns, our sample was sufficiently small such that we were compelled to explore the smallest number of meaningful categories in order to maximize the sample size in each category.

Figure 3 plots our findings with respect to what faculty members were currently doing or planning to do in light of ChatGPT. One clear pattern revealed in figure 3 is that there were few differences based on faculty rank.Footnote ⁸ For example, t-tests revealed statistically significant differences in only two of eight questions. First, we found that junior faculty were more likely than senior faculty to require students to write essays in the classroom (p<0.1). Second, we found that senior faculty were more likely than junior faculty to allow students to use AI tools as long as they acknowledged their use (p<0.05). For lack of better terms, it appears from these two questions that junior faculty were more conservative and senior faculty were more liberal on the usage of ChatGPT.

Figure 3 Active Changes to Teaching and Learning in Light of ChatGPT by Faculty Rank

Note: Horizontal bars around the point estimates represent smoothed confidence intervals. As the interval nears 100, the shading lightens.

Despite these few differences, the point estimates also revealed interesting patterns. A supermajority of respondents (more than 80%) stated that they were actively changing the types of questions they ask and/or the types of assessments they assign in light of ChatGPT. We also found majorities among both senior and junior faculty members who claimed that they will use more questions about recent events in their assessments. Because AI often trains on data (e.g., websites) that are stored and become outdated quickly, very recent events are likely to be more challenging for AI to include. However, ChatGPT can utilize its knowledge of past events to generate responses concerning current events. For instance, ChatGPT drew on its understanding of past midterm elections to respond to a query in December 2022 regarding factors that influenced the 2022 midterm elections. Finally, we found a narrow majority of faculty members who use or plan to use software to detect the use of AI. Fewer faculty members overall were interested in using browser-blocking software or asking students to write essays in the classroom.

Figure 4 presents our findings but, in this case, faculty members were divided according to their underlying concerns about cheating. In this case, of the eight questions, t-tests revealed one statistically significant difference and one near-significant difference. Faculty members who were more concerned about cheating also were more likely to require students to write in the classroom (p<0.05). These concerned faculty members also were more likely to use software to detect AI (p≈0.14). Concerns about cheating in particular appeared to be a stronger motivation than rank for faculty members to take action in light of ChatGPT.

Figure 4 Changes to Teaching and Learning in Light of ChatGPT by Concern for Cheating

Note: Horizontal bars around the point estimates represent smoothed confidence intervals. As the interval nears 100, the shading lightens

Figure 5 presents our findings regarding the effects of ChatGPT and other AI on the future of teaching and learning. We were particularly interested in whether faculty view AI as a threat or an opportunity regarding teaching and learning. The proportion of faculty members who thought AI will fundamentally change higher education was almost 50%. It was slightly more than 50% for senior-level faculty and slightly less for junior-level faculty, although the differences were not statistically significant. That said, senior-level faculty were more likely to believe that AI and ChatGPT represent opportunities to re-envision learning relative to junior-level faculty and less likely to believe that they represent a threat to teaching and learning. The data also suggest that junior-level faculty were more skeptical than their senior-level colleagues about the role of AI in teaching and learning. However, the proportion of junior- and senior-level faculty who believe that ChatGPT and other AI represent a significant threat was lower than 50%.

Figure 5 ChatGPT and the Future of Teaching and Learning According to Seniority

Note: Horizontal bars around the point estimates represent smoothed confidence intervals. As the interval nears 100, the shading lightens.

Figure 6 illustrates how these proportions vary with respect to faculty concerns about cheating. Only one sharp difference was found: the extent to which ChatGPT and other AI represent a threat to teaching and learning. A majority of faculty who generally were more concerned with cheating believed AI to be a significant threat, whereas only a minority of those who were less concerned about cheating believed the same. A t-test reveals that the difference was statistically significant (p≈0.07). Also, faculty members with more concerns about cheating were much more likely to indicate that AI will fundamentally change higher education. It appears that some of the pessimism regarding AI is anchored in concerns about cheating.

Figure 6 ChatGPT and the Future of Teaching and Learning Divided by Cheating Concerns

Note: Horizontal bars around the point estimates represent smoothed confidence intervals. As the interval nears 100, the shading lightens.

ChatGPT IN THE CLASSROOM

It may not be possible to create assignments or exams that are ChatGPT-proof; however, faculty can develop questions and essay prompts that minimize the likelihood of students misusing ChatGPT. This section discusses some of the limitations of ChatGPT about its capacity to accurately respond to questions and includes recommendations for crafting questions that minimize the likelihood of students inappropriately using ChatGPT.

• • ChatGPT has problems with tasks that involve the application of knowledge. Therefore, questions that require students to apply course concepts and theories to contemporary political issues will limit their ability to use ChatGPT.

• • Assignments incorporating scaffolding such as outlines, multiple drafts, and opportunities for constructive feedback or peer review will limit students’ use of ChatGPT. These measures also align with best practices in writing pedagogy.

• • Requiring students to include in their research primary data from personal interviews, surveys, and experiments can restrict the use of ChatGPT. However, it is important to recognize that ChatGPT has the capability to generate fabricated interviews and survey results (Kelly Reference Kelly2023a).

• • ChatGPT training is not continuous and the latest version (i.e., ChatGPT-4) ceased in September 2021 (Wiggers Reference Wiggers2023); therefore, it struggles with answering questions concerning current events. As noted previously, faculty should recognize that ChatGPT can utilize its knowledge of past events to generate responses concerning current events.

• • The use of ChatGPT is limited when assignments or essay prompts require students to reflect on class discussions or activities because ChatGPT lacks knowledge of what transpired in these individual and unique settings.

• • ChatGPT struggles to accurately summarize and critique recently published scholarship not included as primary sources in its training data. ChatGPT will use secondary sources that may have referenced the recently published scholarship or previous scholarship by the same authors, leading to inaccurate summaries and critiques.

• • Faculty should dedicate an afternoon to personally exploring and experimenting with ChatGPT. Spending a few hours interacting with ChatGPT will provide a basic understanding of its potential impact and identify ways to minimize its potential negative effects on teaching.

• • There is no question or assignment that is “ChatGPT-proof.” To assess ChatGPT’s potential responses, faculty should submit assignments and questions and adjust as needed. Also, it is important to recognize ChatGPT’s answers to the same question vary each time it is submitted.<end bl>

Recognizing the challenges of creating assignments and questions that are ChatGPT-proof, we encourage faculty to embrace rather than ignore or avoid this emerging technology and to integrate it into their teaching. ChatGPT is becoming increasingly prevalent in the field of politics and likely will be an essential tool for political science students’ future careers in campaigning, government, and law. An ability to effectively use ChatGPT may not only improve students writing and research skills (Sakhno Reference Sakhno2023; Watkins Reference Watkins2022) but also help them to recognize abuses of ChatGPT. As noted by Kreps and Kriner (Reference Kreps and Kriner2023), ChatGPT may present a significant threat to democratic engagement and representation by nefarious individuals or interest groups who utilize AI software to misrepresent public opinion through orchestrated astroturf campaigns. Incorporating ChatGPT in the classroom can help students to better understand how the technology works as well as its potential biases, limitations, and threats to manipulate democratic political processes.

Following are suggestions for incorporating ChatGPT into courses and facilitating students’ learning, writing, and critical thinking about the technology.Footnote ⁹

SYLLABI EXAMPLES

Students may not fully understand that using ChatGPT is a form of academic dishonesty, mainly because ChatGPT and similar AI programs are very new and many universities have not provided or communicated to students any standard policy for the use of AI programs.Footnote ¹⁰ Following are a few examples of syllabi language for faculty members to potentially include in their syllabi.

Faculty should understand that the use of AI-generated text—even with AI software detection programs—is difficult to prove and, as ChatGPT improves, it will become even more difficult.Footnote ¹¹ With traditional incidences of plagiarism, faculty easily can confirm a student’s academic dishonesty by identifying the plagiarized source and comparing the student’s text to the source. This ability of faculty to compare potentially plagiarized student text to a published source does not exist with AI-generated text. Also, it is possible that a student’s suspected AI submission is simply an example of very generic writing that an AI detector may falsely identify as “written by AI” (PackBack.com).

In addition to the challenges of identifying AI-generated texts, it is difficult to determine what is and is not allowed. If students use ChatGPT to review their original texts for grammar and readability, is this an inappropriate use? Grammarly and other writing tools included in common writing software are simply more basic versions of AI software, and future updates of Microsoft Word are expected to include a form of ChatGPT to check for grammar and writing style (Kelly Reference Kelly2023b).

If faculty do not want students utilizing ChatGPT or similar AI programs in their courses, a clear statement in the syllabus (see the following example) should directly address the issue.

• • Syllabus Example 1: All written work submitted in this course must be originally produced by you. If you use an outside source, you must properly cite the source in the assignment. Using a ChatGPT or similar AI software and submitting the work as your own is a form of academic dishonesty.

If you want your students to use ChatGPT or other generative AI software in an academically honest way—similar to how they may use and cite other traditional sources—consider adding syllabus language that instructs them on the appropriate use of generative AI. Following are three potential syllabi examples:

• • Syllabus Example 2: AI-generated content (e.g., from ChatGPT) can be a useful tool for research and learning. All work submitted for this course must be your original work. Using AI-generated content without proper citation or submitting AI-generated content as your own is considered plagiarism.

• • Syllabus Example 3: AI tools, such as ChatGPT, can be valuable for research, brainstorming, and generating ideas. If you are using AI tools, it is crucial that you critically engage with the information and develop your own understanding and analysis. Directly submitting AI-generated content without proper citation, substantial modification, or critical analysis is considered plagiarism.

• • Syllabus Example 4: In this course, it is expected that students will properly cite and attribute all sources, including AI-generated content. If you choose to use AI-generated content (e.g., from ChatGPT) in your work, you must provide appropriate attribution, clearly indicating the source of the content. Failing to do so is considered plagiarism.