THE BEST BREAKFAST IN TOWN: PROJECT-BASED METHODOLOGICAL LEARNING

The BBiT project is the cornerstone of my research methods course, encompassing the entire semester as a way to focus student learning on a subject in which they are interested. Its genesis was in the spring of 2010, when I noticed a sign in the window of a local restaurant proclaiming that it had “The Best Breakfast in Town”—but with no indication of the source of its claim. I tasked my students to figure out whether this place deserved the title it had claimed for itself. In seven applications of the project, it never, in fact, has won.

Each step of the project and its associated activities are briefly described in the following sections. Prompts, rubrics, and other detailed materials are available at the PS: Political Science & Politics supplementary-materials website.

STEP 1: DEFINING THE RESEARCH QUESTION

The first step in any research project is to clearly define the question that the scholar wants to answer. To that end, the project begins with a class discussion on potential topics for study—breakfast, tacos, museums, or whatever they choose. Students must fill in the blank for “Where is the best _______ in town?” The key is to retain some ambiguity in the topic, allowing for multiple definitions or measures. This is essential as a challenge to students in Step 5 (Measurement).

STEP 2: LITERATURE REVIEW

Although one benefit of the project is that students already have a wealth of information about the topic from their own lived experience, they still must learn why that experience cannot substitute for unbiased and scholarly sources. Additionally, working on a literature review is an excellent opportunity to learn about databases and search strategies. In this step, students are tasked with individual assignments (usually as homework) to find and annotate five sources that will identify possible breakfast establishments. They also must research how other studies defined “town” and criteria used to evaluate restaurant and food quality. Another task is to determine whether similar projects have been undertaken and who the winners were in those cases; this provides criterion validity checks and also informs the subsequent sampling process. Students also must describe and defend their chosen search strategies. Results are posted on the Canvas discussion board so students have access to all of the sources, which form the basis for the references used in their final paper (Step 12, Final Report and Evaluation).

STEP 3: RESEARCH DESIGN

The next step is to determine the research design for the project. This typically entails one of two avenues: (1) a mix of observation and evaluation, in which students generate a questionnaire, visit their assigned restaurant, and evaluate it according to a set of criteria and procedures; or (2) a survey of their fellow students or community. This is decided via class discussion following a lecture on different types of research design. Once a decision is made, it is locked in and cannot be changed—the first of several times this rule applies. In all cases, my class has chosen the observation and evaluation route because this entails that they do the eating rather than tracking down other people who ate.

STEP 4: ETHICS

At this stage, we have a class discussion about the ethics involved in the study. We discuss the Institutional Review Board (IRB) process and why any human subjects involved in our research require protection. If students choose to do a survey, they would be assigned at this stage to write the IRB application for review, with groups responsible for each section of the application. There is usually about a month between Step 4 and Step 9 (Data Gathering), which for my IRB is ample time for a decision. Depending on the timeline at other institutions, instructors may want to obtain a blanket IRB decision to encompass the project before the semester begins.

STEP 5: MEASUREMENT

This is usually the point at which students become highly engaged in the project. They are tasked with defining and operationalizing the three terms at stake in our research: “best,” “breakfast,” and “town.” Each term is ambiguous in some respect, and working through them helps students to understand how and why measurement matters. “Town,” for example, can be defined by official borders or as a metropolitan area and operationalized as a zip code or radius from a center point. “Breakfast” can be considered a meal at a certain time of day or as consisting of certain foods—and the arguments about which definition should be used can be vociferous. “Best” is how we define evaluation criteria that will be used in the questionnaire or survey, and typically consists of categories such as taste, quality, restaurant environment, service, operational ethics, and cost.

I assign students to one of three groups, one for each term. As a homework assignment, they must recommend a way to define their term and come to class prepared to make a case to other students. The “town” and “breakfast” groups also must operationalize their definitions. After receiving feedback on their presentations, students are given class time to revise their definitions and present again, after which they work together to find a definition that receives majority support. These definitions then are locked in. We operationalize “best” as we develop the survey instrument.

STEP 6: POPULATIONS, SAMPLING, AND ASSIGNMENT

With the operationalization of “breakfast” and “town” in place, the class is ready to determine the population to which the study applies. This is the first point at which students realize how their prior decisions can have a major impact: if they defined “town” too broadly, for example, they will have an overwhelming number of restaurants to identify, whereas if “breakfast” is too vague, it will be difficult to determine which restaurants qualify. Instructors should assist students only in keeping the project manageable; letting them make mistakes provides ample fodder for debriefing in Step 12.

In this stage, for those classes choosing observation and evaluation, students first discuss potential sampling frames they can use to identify the population of restaurants. Then they are individually assigned to various sampling frames and tasked to come up with an initial list, which must be posted on the discussion board. In the next class session, we look at the lists, noting the differences, and attempt to consolidate them, double check each restaurant, and find any missing population members. This may entail students calling each restaurant manager and asking questions to ensure that the establishment qualifies for our study.

Once we have a relatively complete list, we discuss whether we can cover the entire population or if we must sample. Depending on the size of the population and the size of the class, it may be possible to cover the entire list; however, this may mean that only one student visits each restaurant. Sampling sacrifices thoroughness but allows multiple students to visit each establishment. The discussion about the tradeoffs of sampling is an excellent way to continue discussions about reliability and validity that begin in the measurement activity. Of course, if students decide on a sample, then the next step is requiring them to make individual proposals on how to sample (e.g., purposive, random, or haphazard), sample size, and which restaurants would be in the sample according to their method. As with the sampling frames, this is done individually as homework, after which we discuss it as a class and make a decision.

If the class is doing a regular survey, then sampling frames and sampling discussions focus less on the restaurants and more on which population of people will be surveyed for their restaurant preferences. The same type of assignment can be used with a different unit of analysis: people rather than restaurants.

The final part of this stage is assignment. If students are visiting restaurants, they must be assigned. We discuss the merits of random assignment and then students are randomly assigned to a specific restaurant. If they are interviewing people, then we divide subjects among class members. At this point, every decision on the project so far is fixed with no changes allowed (except assignment, if an emergency arises).

STEP 7: SURVEY DESIGN

Students are assigned to groups, with each group responsible for one previously agreed-on criterion of “best.” For homework, groups must devise questions or Likert items to be included on the survey questionnaire. During class, each group trades questions with another group and critiques them. After a round of revisions, we produce a final questionnaire. Once we are in agreement about the questionnaire, no changes are allowed. This step can be completed prior to step 6 if the instructor chooses.

STEP 8: PROCEDURES

The final task before gathering data is to set procedures for maximizing inter-rater reliability. This is a class discussion in which we agree on whether there is a minimum amount of time students must spend in a particular restaurant, whether they have to order specific menu items, if other people are allowed to join them, what merits top and bottom ratings, and how much money they are allowed to spend. These procedures are posted on the Canvas discussion board and are included at the top of the finalized survey instrument.

STEP 9: DATA GATHERING

Students have three weeks to gather the data. This period typically coincides with midterm exams and the mid-semester break, but they also have a single class session they can use for site visits if necessary. During this time, students visit their assigned restaurants to complete their surveys or interviews.

STEP 10: CODING AND DATA ENTRY

When the surveys are returned, the responses must be entered into an Excel spreadsheet and any qualitative responses must be coded. Students can divide the surveys among themselves and do this as a homework assignment, storing the spreadsheet as a Google document or another collaborative document system. Alternatively, a teaching assistant or work-study student can enter the data.

STEP 11: DATA ANALYSIS

I teach students to analyze the data using Microsoft Excel. Following Clark (Reference Clark2011) and Jackson (Reference Jackson2013), Excel is perfectly adept at basic descriptive and inferential statistics with several advantages over SPSS, STATA, and other packages. The primary advantage is that most students have easy access to Excel on their own computer and at public libraries, which will continue after college. Because Excel is regularly used in the workplace, familiarity with it is a valuable skill in its own right. As this course focuses on research methods rather than quantitative analysis, Excel is sufficient for my purposes.

After a lesson on Excel basics, I give students their data and train them in descriptive statistics, building indexes, and basic data analysis. The class usually has various practice exercises using other datasets first and problem sets for homework so students can hone their skills before working with the BBiT data.

STEP 12: FINAL REPORT AND EVALUATION

The culmination of all of this work is writing a 15-page research paper with two parts. The part first is a full research paper in which students ultimately answer the question, “Where is the best breakfast in town?” This formal paper has separate sections for the research question, thesis, literature review, methodology, data analysis, conclusion, and references. Following Bos and Schneider (Reference Bos and Schneider2009) and Bernstein and Allen (Reference Bernstein and Allen2013), the paper components are completed throughout the course, then written up in the final paper. The second part of the paper requires students to assume the role and voice of a journal reviewer, tasked with evaluating the project and determining whether they would accept the first part of the paper for publication. Students must note how all of the decisions made early in the process affected their subsequent decisions; those who disagreed with previous choices have an opportunity to critique the process.

I allow students to make mistakes during the process, particularly in their definitions, questionnaire, and sampling design. The mistakes are never sufficient to derail the entire project—I want them to stay invested and have data to work with—but they sometimes are serious enough that a “reject” decision is the only acceptable one in the final paper review. I have found that this is how the best learning occurs: by realizing how early mistakes make subsequent tasks difficult and frustrating, students learn more about what they should have done than if I provide the correction in the moment. For example, the survey for one set of students did not use a consistent scale for the questions—meaning that every single student had to recode the data to ensure that “high” meant “high” and “low” meant “low.” In another case, the choice to use a restricted definition of “town” meant that not one restaurant found on the “best” list from the literature review ended up in their population.

The final research paper is due at the penultimate class session, and students must be prepared to discuss their choice and how their analysis led them to that conclusion. We determine the overall class winner; then, during the last class session, I treat the students to breakfast at the winning restaurant. I create an award certificate and we present it to the manager to celebrate and acknowledge both the restaurant itself and the work the students put in. It is at this shared breakfast that we debrief the entire project.

VARIATIONS AND ADAPTATIONS

A flexible feature of this project is that it can be used in its entirety or only in part. Rather than committing to the entire project, simply one or two activities can be used—for example, the component in which students work through developing conceptual and operational definitions or the sampling activity. Many assignments are easily adapted from discussions to out-of-class homework or from individual to groups.

Another variation is to use the project as an end-of-course summative assignment rather than a 16-week, course-defining theme. Students can use the final two weeks to apply their methodological skills to the problem of finding the best breakfast venue. Another alternative is to have multiple sections of the research methods course conduct the project simultaneously, comparing the various results at the end of the semester. This provides ample opportunity to discuss how the same research question—and even the same procedures—can result in different answers. ^{Footnote 2}

Finally, although the project is called “The Best Breakfast in Town,” instructors should feel free to adapt the subject to something else of local interest. My students have done this project as the “Best Coffee Shop in Town,” and one section fell one vote short of researching the “Best Donut in Town.” Desserts, parks, museums, pizza—as long as a subject has ambiguity regarding what is and is not included, it is appropriate for this project. Be creative and consider allowing students to choose the topic because doing so increases their motivation to find an answer. In a larger class, let students work in smaller groups, each working on their own topic and sharing their results at the end.

ASSESSMENT

The project has shown gains in student engagement, achievement of course learning outcomes, and teaching evaluation scores. First, students respond positively to the project and readily engage in all associated class discussions. As one student stated, “[t]he BBiT project was very educational. Although frustrating. I learned a lot from that.” Another noted that “stepping back and working on something like breakfast was a nice break from using data mostly used in our majors.” In addition, in their course evaluations, many students characterized the course as “fun” or “enjoyable” and called on the department to offer more courses in research methods. In fact, not a single student throughout the seven years of using this project has ever called the course boring or uninteresting.

Although student engagement is a key benefit of using this project, we also must consider the impact it has on learning the course content. The learning outcomes for the course are as follows:

1. 1. Identify the fundamental components, approaches, methods, and common practices of scientific social research.

2. 2. Evaluate the quality of scientific and nonscientific research claims and distinguish between the two, with special attention to claims in the “real world.”

3. 3. Articulate the process of crafting an excellent research design and paper.

4. 4. Analyze published work in political science, identifying the components (including thesis, methodology, assumptions, and data) and evaluating its claims.

5. 5. Justify theses and conclusions through the use and analysis of evidence.

6. 6. Craft a research design or paper to explore a social problem, implement the design, and evaluate the resulting data.

7. 7. Reproduce standards of professional behavior in a college classroom.

The project across its many dimensions aims to meet learning outcomes 1, 2, 3, 5, and 6. The final paper in the course is a good tool for evaluating whether students achieve these outcomes, particularly 3, 5, and 6. I compared scores on the paper in two sections of my methods course from 2014: one section worked on BBiT, the other completed individual projects on a political science topic of their choice. In both classes, the final paper comprised 25% of the course grade and was graded with a similar rubric. Students in the BBiT section performed remarkably better on the final paper, with an average of 86.7% compared to 69.9% for the traditional section. In the methodology section of the rubric, in which students were required to discuss and defend their methodological choices, the results were even starker: BBiT students earned an average of 82.4% on that section of the rubric, whereas non-BBiT students earned only 46.6%. Because the only significant difference between the two courses was the use of the project and all of its components, this provides evidence in favor of the BBiT project for improving student learning in the course.

Finally, because tenure and promotion committees continue to regularly consult teaching evaluations in their deliberations—and research methods courses tend to get lower scores (Fletcher and Painter-Main Reference Fletcher and Painter-Main2014)—it is worthwhile to investigate the impact of the project on student evaluations of instruction. I found small but notable improvements. Comparing the same two sections discussed previously, the BBiT-project students scored the instructor and the course higher on 12 of 15 items (i.e., on average, 0.25 points better on a 5-point scale). Overall, the averages on BBiT-using methods courses and other political science courses I teach are similar, with only a 0.1-point difference between them. My research methods class, therefore, does not experience any decrease in evaluations.

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit https://doi.org/10.1017/S1049096517001895.