LITERATURE REVIEW

Relatively little has been written about teaching methods for graduate-level methodology courses in political science. The limited work in this area tends to focus on what—rather than how—graduate students are taught (e.g., Bennett, Barth, and Rutherford Reference Bennett, Barth and Rutherford2003). Although these discussions are useful in understanding methodological standards and expectations in the discipline, they are less beneficial for understanding the impact of various pedagogical choices. Another body of literature focuses on the challenges of teaching undergraduate methods (e.g., Andersen and Harsell Reference Andersen and Harsell2005; Bernstein and Allen Reference Bernstein and Allen2013; Bos and Schneider Reference Bos and Schneider2009; Brandon et al. Reference Brandon, Brown, Lawrence and Heerde2006; Brown, Blankenship, and Kramer Reference Brown, Blankenship and Kramer2007; Centellas Reference Centellas2011; Thies and Hogan Reference Thies and Hogan2005). This is a noteworthy body of work; however, to be successful in the discipline, graduate students must become knowledge generators and disseminators—a point not often at issue with undergraduates, who most commonly find methods to be irrelevant or, in regard to quantitative methods, mathematically overwhelming (e.g., Bos and Schneider Reference Bos and Schneider2009).

Almost every graduate student in political science, however, has read the ubiquitous Designing Social Inquiry (King, Keohane, and Verba Reference King, Keohane and Verba1994), often called “KKV” by disciplinary insiders. While KKV’s recommendations for qualitative research led to a wide-ranging debate, we highlight the following observation that would generate little controversy:

This perfunctory treatment of the research process aptly describes the perception of many graduate students who commonly begin their methodological training by reading the published, polished work of scholars who have erased all of the trials and tribulations from their end product. How, a graduate student may wonder, can such a level of scholarly perfection ever be reached? Of course, such a perspective does not reflect the reality of social science research. As KKV (1994, 182) noted, “[P]erfection is unattainable, perhaps even undefinable.” Instead, research often is messy and requires creativity more than perfection. Why, then, in graduate methods curricula does the discipline typically focus on the elegance of the final publication rather than on all of the mistakes it took to arrive at that point? Perhaps it is because most researchers are reluctant to reveal their gaffes or, perhaps—as we discovered—they simply have not been asked about these challenges. The goal of the seminar series described in this article was to uncover the missteps, the challenges, the dead ends, and the insights that lead to innovative research. We believe that such a narrative will better prepare graduate students for their own research journey.

To understand the potential benefits of such a pedagogical approach, we need to consider the body of literature on narrative-centered learning. The construction of a story is one of the most fundamental ways to create shared meaning, an activity central to all aspects of learning (Bruner Reference Bruner1986; Clark and Rossiter Reference Clark and Rossiter2008; Connelly and Clandinin Reference Connelly and Clandinin1990; Wells Reference Wells2009; White Reference White1980). A narrative-centered learning environment has two elements: transportation and performance (Mott and Lester Reference Mott and Lester2006). By transporting learners into a plot full of characters and ideas, this learning environment enables students to participate in active problem solving and to draw their own inferences (Mott and Lester Reference Mott and Lester2006, 676). Furthermore, the communication of a narrative plays a central role in memory by providing an organizing structure for new experiences and knowledge (Mandler Reference Mandler1984). As Schank and Berman (2002, 293) stated, “for communication, memory, and learning purposes, stories are likely to be richer, more compelling, and more memorable than the abstracted points we ultimately intend to convey or learn when we converse with others.”

According to Mott and Lester (Reference Mott and Lester2006), a narrative-centered learning environment is particularly suited to inquiry-based learning that emphasizes concept building, hypothesis formation, data collection, and testing. Mott et al. (Reference Mott, Callaway, Zettlemoyer, Lee and Lester1999) noted that “in the sciences, an inquiry-based curriculum featuring dynamic narratives of the highly nonlinear process of scientific discovery could foster an in-depth understanding of how real-world science plays out.” For graduate students who must learn how to conduct their own research in the disorderly reality of social science, such an endorsement seems particularly relevant. Mott et al. (Reference Mott, Callaway, Zettlemoyer, Lee and Lester1999) discussed three types of narrative-centered learning activities: co-construction, exploration, and reflection. In our seminar series, we focused primarily on reflection (i.e., activities that engage post-hoc analysis) and exploration (i.e., activities that consider how intentions affect actions in the evolving narrative). Co-construction (i.e., activities that allow participation in narrative creation) also may be a useful strategy in graduate-level instruction—for example, by creating alternative stories about how a potential research question might be addressed. However, this particular type of narrative-centered learning was beyond the scope of our seminar series, which asked scholars to relate their own version of events and thereby bring to life to the research process. In hearing and reflecting on the narratives of these scholars, we expected that attendees would be more cognizant and open about their own evolving research path.

BACKGROUND OF THE SEMINAR SERIES

In the fall of 2011, the Political Science Graduate Student Association at Purdue University convened a working group, which included a faculty member who regularly taught required methods courses. The purpose of the working group was to discuss issues in the current methodological training of graduate students as well as possible remedies. From the discussions, it became clear that the current methods curriculum did not offer graduate students a panoramic view of the research process; neither did it provide access to scholars with diverse research experiences. This gap was not being filled by research mentorships with faculty. As is the case in many departments, most students were teaching assistants, independent instructors, or on independent fellowships. Although some students had participated in limited projects with faculty (e.g., data collection), only a few had participated in the entire research process of any project. With limited resources, how could this deficiency be remedied?

We devised a lecture series to provide a more comprehensive “look under the hood” at social science research. The goal was to provide opportunities for students to examine the research process through actual experiences of established scholars. We did not aim for a conventional lecture series in which a researcher presents a work-in-progress for audience consideration and feedback. Instead, the focus of the “Look under the Hood” colloquium was to be on a piece that had already been published in a major peer-reviewed journal. Using the publication as the end point, authors presenting at sessions would be asked to describe their research process from the beginning, giving particular attention to the conceptual genesis of the project, the rationale for key methodological choices, and the misjudgments made along the way. Such a narrative meant that speakers covered material that went well beyond the traditional research presentation—and even beyond a typical class session in a graduate methodology seminar. Our expectation—which was confirmed when the series began—was that after authors had successfully published a piece in a visible scholarly outlet and had received all due professional recognition, they would be open to having a frank conversation about the “scaffolding,” “agony,” and “uncertainty” that surrounded the piece.

Given the unusual nature of these lectures, a rubric was prepared for the authors who were invited to present a piece. Under this rubric, each presentation was to be structured as a retrospective account of a specific research project. Speakers would begin by citing their source of inspiration, whether it was the result of—for example—a debate with a peer or an attempt to fill a gap in the existing literature. At this point, researchers could share the relevant literature; however, an extensive overview of the literature was discouraged. Instead, we encouraged them to move from a discussion of the formation of the research question to a detailed explanation of how the approach to analyzing it was chosen. Here, speakers would address method selection and application. Although the series was not structured as a class, we asked authors to discuss the virtues of the particular methods used in their article as well as how empirical analysis and interpretations changed in response to comments from colleagues, panel discussants at professional conferences, editors, and anonymous peer reviewers.

With these goals in mind, several examples of innovative work in the social sciences were selected. Care was taken to ensure that the group of articles represented a variety of different methods as well as different academic fields. Some of the methods (e.g., social-network analysis and experimental methods) are not regularly taught in our current curriculum, which increased the value of these seminars. The seven-part series was held in the spring of 2012 on the Purdue University campus. Almost all of the authors who were invited to present at a “Look under the Hood” colloquium agreed to come and present the back story for their already-published piece. ^{Footnote 1} Each session was scheduled for three hours, which included a brief reception for the presenter, the retrospective narrative, and a lengthy informal question-and-answer (Q&A) period. The series was well attended not only by graduate students but also by many faculty members.

ASSESSMENT OF THE SPEAKER SERIES

To assess the seminar series, attendees completed a brief anonymous survey after each lecture. The survey consisted of two parts. In the first part, respondents evaluated their general level of agreement with the following five statements (i.e., 1 = completely disagree; 10 = completely agree):

• S1 I perceive this seminar as useful for professional development.

• S2 This seminar is relevant to my research interests.

• S3 I enjoyed today’s presentation.

• S4 This presentation is appropriate for this seminar.

• S5 I feel that the methods used in this seminar will be useful for future personal research projects.

The second part of the survey solicited open-ended feedback. A total of 91 questionnaires were returned for the seven presentations. The mean scores for the five evaluative statements were generally positive, particularly with respect to professional development (i.e., M = 9.15); other averages were above 7.5 with relatively little variation in responses (figure 1). In the open-ended section of the survey, one student wrote that he or she attended to learn “how methods are applied in real research.”

Figure 1 Average Survey Response for Seminar Series by Statement

Note: There are 91 total responses for each statement, with the exception of statement 5, which includes 88 responses.

We note that the seminar taught more than research methods. During the series, speakers addressed professionalization, conference presentations, and—in some cases—the lengthy journey from research question to published article. Students expressed great interest in these topics. In the Q&A period, they frequently asked for more “insider” details on working with coauthors (e.g., “How were the tasks divided?”); the targeting of particular journal outlets (e.g., “Was there a reason you did not attempt to publish in _____?”); and challenges when revising for resubmission (e.g., “Was there a point when you considered pulling the manuscript, given the seemingly impossible-to-satisfy Reader 3?”). In one survey response, a graduate student emphasized that “it was valuable to hear from published authors and what they did in order to get a major publication through the review process.” Several faculty members who attended the seminar mentioned how reassuring it was to hear that their own challenges in publishing were not unique. One wrote in the assessment survey that the “Look under the Hood” series offered “encouragement and help” at his or her particular career stage.

DISCUSSION AND IMPLICATIONS

Overall, this series exposed students to the process of discovery in political science, “warts and all,” and provided opportunities to discuss methodological techniques that might not have arisen in our standard graduate training curriculum. As one graduate student remarked in the survey, “I have been struggling with a problem in my current research project and I believe that today’s method [i.e., bootstrapping] can help me fix this problem.”

Although some students believed that certain highlighted methods were not applicable to their own research program, the series provided useful information on professionalization and publishing. In some cases, survey responses indicated that the opportunity to learn methods was not the primary reason for attending the seminar—some attendees came to learn about submitting work for publication and others to find solutions to common problems in conducting research programs. One student wrote that he or she attended to “learn about problems with writing research papers, especially the publishing process.”

Educators throughout the academic disciplines agree that to improve learning outcomes, traditional “chalk-and-talk” instruction should be supplemented with “active learning”—that is, opportunities to learn while doing. When teaching graduate students about the principles of research design and causal inference, the selection of appropriate tools for empirical analysis, the write-up of results, and the challenges of publishing, the retrospective narratives in the “Look under the Hood” series constituted a form of vicarious active learning. Our experience demonstrates that authors of major scholarly works generally enjoy discussing the evolution of a piece and that these reflections offer valuable lessons for novice researchers.