In the Design, Implementation and Measurement Section of the Assessment/Learning Outcomes II Track of APSA's 2006 Teaching and Learning Conference, three papers relevant to measuring student learning outcomes in United States government courses were presented. Edleman and Kahn (Reference Edleman and Kahn2006) report that students at both the community college and university level scored similarly on pre- and post-knowledge tests. Males and those with higher GPAs show the most improvement. Quackenbush (Reference Quackenbush2006) uses test and control classes, as well as pretests and posttests, at the community college level and finds that both knowledge and civic awareness (such as likelihood of voting) are enhanced by active-learning techniques. His analysis includes the use of a U.S. government attitudinal survey, designed to measure the perceived importance of political knowledge, and a U.S. government self-confidence survey, designed to measure how confident students are in their political knowledge. Both of these instruments are employed in this study. Champney and Harris (Reference Champney and Harris2006) find, at the university level, that first-year political science majors, when encouraged in their courses to follow current events, will generally do so. “Following current events” is measured by actual news quizzes, rather than self-reports. Again, males with higher GPAs tend to follow current events more closely than others.
“Student learning outcomes” research is likely to become more common, given the emphasis on assessment of the regional accrediting groups, such as Middle States and the Higher Learning Commission, and given increasing national calls for accountability in higher education. Indeed, Arenson (Reference Arenson2006) points out that a national commission on higher education recently concluded there is a “lack of accountability” and called for “more proof of results.”
The participants in the Outcomes II Track drew several conclusions (Edleman et al. Reference Edleman, Evans, Kahn, Schattschneider and Williams2006). Our current empirical knowledge regarding the impact of variables such as gender and GPA on student learning outcomes, and on civic engagement, is limited. We need to begin to gather and share national data on the knowledge, skills, and participation levels of our students. In so doing, we need to improve our instruments, collaboratively collect data on student outcomes across varied institutions, and use these data in comparative analyses. Many of these same conclusions were reemphasized by members of the Assessment II Track at the 2007 APSA Teaching and Learning Conference (Beck et al. Reference Beck, Bennion and Persuad2007). The present research is designed to contribute to these efforts.
BACKGROUND
Of course, assessment work is done in the discipline of political science, and the pretest/posttest approach is not uncommon. Kelly and Klunk (Reference Kelly and Klunk2003) report that about 10% of 213 departmental surveys returned to them use the pretest and posttest method of assessment. Presumably, this is for political science courses, although the article does not state that explicitly.
The previous efforts demonstrate that learning facts and learning principles do indeed take place in political science classes in a variety of contexts. Anderson and Harsell (Reference Anderson and Harsell2005) administer pretests and posttests over 15 semesters to students in the Quantitative Methods in the Social Sciences course at the Maxwell School at Syracuse University. Students are asked to rate knowledge and comfort levels with computers, spreadsheets, equations, variables, and so on. The authors also randomly sample alumni and ask them whether they have had an opportunity to use information that was learned in the course. The analyses yield positive results on both counts.
Harkness (Reference Harkness2005) administers pretests and posttests to students in four sequential political science courses—America Government, Legislative Process, Research Skills, and Research Methods—over four semesters. Students in the courses show not only evidence of increased knowledge, but also evidence of retention of key concepts as they progress.
Alperin (Reference Alperin2007) describes how members at the University of Wisconsin–River Falls use pretests and posttests containing 65 multiple-choice questions to measure the government department's assessment goals for the introductory American government course. A portion of those questions are then inserted into traditional exams over the course of the semester, and students leave the class with “substantially more knowledge” than when they entered (Alperin Reference Alperin2007, 4). Similarly, the present analysis, as indicated below, employs 50 multiple-choice knowledge questions.
Omelicheva and Avdeyeva (Reference Omelicheva and Avdeyeva2008) measure student learning after exposure to six class periods of debate, on the one hand, and six class periods of lecture, on the other. They find that students score somewhat higher on factual knowledge after the lectures, but higher on comprehension after the debates. Comprehension is measured by asking students to explain what a concept means. Although exposure to debates results in students seeing how their knowledge can be used in situations not related to class is somewhat greater than in exposure to lectures, neither lectures nor debates excel in promoting application. The same is true of critical evaluation. Application is measured with hypothetical scenarios and critical evaluation is measured with ability to make a reasoned argument.
In connection with knowledge of current events, Huerta and Jozwiak (Reference Huerta and Jozwiak2008) investigate whether using the New York Times in general-education political science classes at Texas A&M University increases student engagement inside and outside the classroom. Students in four courses are given a pretest and a posttest containing questions on civic engagement. In a United States government course and a state and local government course, the New York Times is used. In a biology course and a United States government course, it is not. They find that exposure to the New York Times does not have a positive impact on the statement “I have a positive attitude toward politics,” which tends to be consistent with some of the other findings on political efficacy noted below. However, it does have a significant positive effect on community involvement measured as an index containing attitudes and actual activities.
The previous efforts yield mixed results with respect to stimulating political efficacy among students. In their study of traditional vs. Web-based American government classes, Botsch and Botsch (Reference Botsch and Botsch2001) administer a pretest and posttest on knowledge gained by students and administer a demographic survey. Consistent with the pattern of other studies, the value added for student knowledge of facts and principles is virtually identical in both versions of the course. The authors note, however, that GPA is a strong predictor of the knowledge gained in the traditional classes. Other factors, such as gender, have no impact. And students in both versions show increased political interest and political efficacy. Political interest is defined by asking (Botsch and Botsch Reference Botsch and Botsch2001, 137): “Would you say that you follow what is going on in government and public affairs (4) most of the time, (3) some of the time, (2) only know and then, or (1) hardly at all?” Political efficacy is defined by asking (137): “Sometimes politics and government seem so complicated that a person like me can't really understand what's going on. Do you (1) strongly agree, (2) agree, (3) have mixed feelings, (4) disagree, or (5) strongly disagree?”
Crawford (Reference Crawford2007) reports the results of pretests and posttests for seven introductory political science sections at a private midwestern university. Employing a 2–10 scale on internal efficacy, she compares sections emphasizing a rational-choice perspective to those that do not. The scale is based on the students' perceptions of how well qualified they are to participate in public affairs and whether they feel they are better informed than most. There are similar, statistically significant improvements for internal efficacy in both varieties of sections.
Strachan (Reference Strachan2006), however, does not find that political efficacy increases after employing a pretest and posttest for a one-credit course at the University of Albany designed to stimulate political efficacy, convince students that political participation can influence political outcomes, and make them feel more comfortable in engaging in such activities. Again, Likert scales are used to tap these dimensions. Efficacy is measured with a 5-point scale asking students to indicate their agreement that “my vote doesn't matter” and “it is difficult for someone like me to make a difference” (913).
The present analysis, as indicated below, includes 10 scaled questions regarding the extent to which students believe that knowledge of the United States government is important. Although somewhat different from the above indicators, this scale may also be used to tap political efficacy. The battery of questions is in appendix A.
DATA AND METHOD
We used a Solomon Four-Group Design to assess student learning outcomes in United States government classes at Sauk Valley Community College in Illinois and The University of Scranton in Pennsylvania (a private comprehensive institution). At each, a pretest and posttest was administered to students enrolled in United States government classes and students who had never enrolled in a college-level political science or government class, the latter selected to be representative of a variety of majors outside political science. In addition, the posttest was administered to a group of students who completed the United States government course, but were not given the pretest, and a group of students who have never enrolled in a college-level political science or government class and were not given the pretest. Hence, the four-group design: pretest/class/posttest (experimental group), pretest/posttest (control group), class/posttest, and posttest only.
The instrument is comprised of four parts. One contains demographic questions, such as gender, year, and major. Another contains 50 multiple-choice questions designed to measure knowledge of the United States government, previously employed at Sauk Valley Community College. The third contains 20 multiple-choice questions designed to measure knowledge of current events over the two-week period preceding administration. The fourth contains 10 scaled questions from the U.S. government attitudinal survey and the 10 scaled questions from the U.S. government self-confidence survey, both previously employed at Valencia Community College.
UNITED STATES GOVERNMENT LEARNING OUTCOMES
In the experimental group, background knowledge went from an average of 61% correct responses to an average of 72% correct responses, while current events went from 50% to 58%. Both are statistically significant at the .05 level, using the t statistic for difference of means. The control group went from 52% to 54% and 43% to 47%, respectively, and neither is statistically significant.
The United States government posttest only for background knowledge is 66% (statistically significant compared to the initial 61% in the experimental group). For current events, it is 54% (not statistically significant compared to the initial 50%). This suggests a modest possibility that the pretest may have impacted current-event performance. The control posttest only for background knowledge is 54%, very similar to the initial 52%. Somewhat surprisingly, the control posttest only for current events is 52% (statistically significant compared to the initial 43%). We have no explanation for this anomaly.
For both the attitudinal and self-confidence batteries, there are 10 questions arrayed on a scale of 1 to 5. Therefore, for each respondent, the lowest possible score across each set of 10 items is 10, and the highest 50. Low scores indicate more self-confidence and better attitudes. In the experimental group, for self-confidence, the pretest averages 25 and the posttest averages 19. This is statistically significant. On attitudes, the pretest and posttest are both right at 19. Apparently, self-confidence improves among students in the United States government classes, but their conviction that knowledge is important does not increase (although starting at a fairly healthy level). This is strikingly similar to Strachan's (Reference Strachan2006) finding that students' political efficacy did not improve significantly as a result of the one-credit course used in her study. She suggests that political efficacy is more likely based on long-term socializing agents. Our control group went from 21 to 23 on attitudes, while self-confidence was stable at 28. Neither is statistically significant.
The United States government posttest only for attitudes is 20, very similar to the experimental group. For self-confidence it is 21, similar to the posttest in the experimental group. The posttest only for the control group was remarkably similar to the other numbers the for control group.
A CLOSER LOOK
The previous section warrants the general conclusion that students in the United States government courses, compared to the control group, improve in their knowledge of the United States government and of current events. They also perceive their increased knowledge, although do not attach any more importance to it at the end of the semester than at the beginning.
However, it may be that the United States government courses are populated with students who are more likely to be receptive to increasing their knowledge. In particular, this might be true of political science majors, and those with higher GPAs. The literature, and our previous research, suggest it may also be true of males. A gender gap has been noted with respect to both interest in, and knowledge of, public affairs and current events (Ford Reference Ford2002).
Table 1 reports our investigation of these factors. As was the case with the initial analysis, for background knowledge and current events, the mean is the mean percentage of correct responses to the questions. Background knowledge improves from 61% to 72%. The degree of improvement is the same for males and females, although males start with a higher level of background knowledge. The pattern of improvement is also similar for high/low GPA and for political science majors vs. others. As with gender, political science majors begin with higher levels of knowledge.Footnote 1 Although not listed in the table, it is worth noting that female political science majors improve very substantially, from 57% to 73% (male political science majors go from 75% to 81%). Apparently, exposure to a United States government course has a very positive impact on female majors who start with less knowledge.
Table 1. Before and After Means in the Experimental Group
* Significant at the .05 level (t statistic for difference of means)
Knowledge of current events improves from 50% to 58%. The degree of improvement is the same for males and females, although, again, males start with higher levels of knowledge. Those with higher GPAs improve more than those with lower GPAs. In addition, political science majors improve more than other majors.
Self-confidence improves from 25 to 19. The improvement is remarkably similar across genders and GPAs. Political science majors start with more self-confidence and do not improve quite as much. (21 to 17). There is virtually no improvement in attitudes (20 to 19) and the pattern is consistent across genders, GPAs, and majors.
The improvement in the United States government classes in background knowledge, knowledge of current events, and self-confidence is not limited to those students who might come into the courses predisposed toward improvement. It is also true, however, that males and those with higher GPAs come into these courses with somewhat higher levels of background knowledge. Clearly, all students gain self-confidence in their knowledge of government, while political science majors bring more initial self-confidence. However, the students' perception of the importance of having such knowledge does not appear to improve at all. This may be a source of some concern.
We also look at the pattern in the United States government courses within categories of father's education, within categories of mother's education (at least some college and no college for each), and reported household income (divided at the median). For background knowledge and self-confidence, there are statistically significant improvements in all groups. However, for those whose fathers had at least some college, and for those whose mothers had at least some college, there is not a statistically significant difference for current events. In both instances, knowledge of current events starts relatively high and remains there. Therefore, for current events, parents' educational backgrounds may be as important as exposure to a United States government course.
CONCLUSIONS
The aggregated results for our two different varieties of institutions of higher education indicate that students' knowledge of government/politics, their knowledge of current events, and their self-confidence in their knowledge are all positively impacted by completion of a United States government course. And, for knowledge of government/politics, the impact is not limited to those students who might come into the courses predisposed toward improvement. Earlier research indicates that males and those with higher GPAs improve most with respect to background knowledge and knowledge of current events. The present analysis does not confirm that. For both background knowledge and knowledge of current events, males and females improve at a similar rate, although males start with greater knowledge. Those with higher GPAs show no greater improvement on background knowledge. The very substantial improvement of female political science majors in their knowledge of government/politics is notable.
The situation is a bit different regarding knowledge of current events. Those with higher GPAs and political science majors improve the most. In addition, parents' education appears to have a significant impact on knowledge of current events to begin with.
Perhaps most noteworthy, exposure to a United States government course does not have any positive impact on students' beliefs that such knowledge of politics and government is important. If this perception may be viewed as an indicator of political efficacy, it suggests such efficacy is not subject to short-term influences.
APPENDIX A: United States Government Attitudinal Survey
- 1 =
Totally Agree
- 2 =
Somewhat Agree
- 3 =
No Opinion
- 4 =
Somewhat Disagree
- 5 =
Totally Disagree
1. The three branches of U.S. government play an important role in my life. 1 2 3 4 5
2. It is important to vote in congressional elections. 1 2 3 4 5
3. It is important to vote in presidential elections. 1 2 3 4 5
4. It is important to read/watch the news related to the three branches of U.S. government. 1 2 3 4 5
5. It is important to do research about candidates who run for office. 1 2 3 4 5
6. Supreme Court decisions and opinions play an important role in my life. 1 2 3 4 5
7. It is important for me to know who my elected representatives are in the United States House of Representatives and the United States Senate. 1 2 3 4 5
8. It is important for me to know who my elected representatives at the state level of government. 1 2 3 4 5
9. It is important for me to know who my elected representatives are at the local level (city, township, county, school board, etc.). 1 2 3 4 5
10. It is important for me to know who the U.S. Supreme Court justices are. 1 2 3 4 5
