Hostname: page-component-5f56664f6-6wzj7 Total loading time: 0 Render date: 2025-05-08T07:05:41.986Z Has data issue: false hasContentIssue false
  • English
  • Français

Are Survey Experiments Externally Valid?

Published online by Cambridge University Press:  07 May 2010

JASON BARABAS  and
JENNIFER JERIT
JASON BARABAS*
Affiliation:
Florida State University
JENNIFER JERIT*
Affiliation:
Florida State University
*
Jason Barabas is Associate Professor, Department of Political Science, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 ([email protected]).
Jennifer Jerit is Associate Professor, Department of Political Science, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 ([email protected]).
Get access Rights & Permissions [Opens in a new window]

Abstract

Researchers use survey experiments to establish causal effects in descriptively representative samples, but concerns remain regarding the strength of the stimuli and the lack of realism in experimental settings. We explore these issues by comparing three national survey experiments on Medicare and immigration with contemporaneous natural experiments on the same topics. The survey experiments reveal that providing information increases political knowledge and alters attitudes. In contrast, two real-world government announcements had no discernable effects, except among people who were exposed to the same facts publicized in the mass media. Even among this exposed subsample, treatment effects were smaller and sometimes pointed in the opposite direction. Methodologically, our results suggest the need for caution when extrapolating from survey experiments. Substantively, we find that many citizens are able to recall factual information appearing in the news but may not adjust their beliefs and opinions in response to this information.

Type
Research Article
Information
American Political Science Review , Volume 104 , Issue 2 , May 2010 , pp. 226 - 242
Copyright
Copyright © American Political Science Association 2010

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

REFERENCES

Abrajano, Marisa A. 2005. “Who Evaluates a Presidential Candidate by Using Non-policy Campaign Messages?” Political Research Quarterly 58 (March): 55–67.CrossRefGoogle Scholar
Achen, Chris. 1982. Interpreting and Using Regression. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Arceneaux, Kevin, and Johnson, Martin. 2007. “Channel Surfing: Does Choice Reduce Videolmalaise.” Presented at the 2007 Annual Meeting of the Midwest Political Science Association, Chicago.Google Scholar
Arceneaux, Kevin, and Kolodny, Robin. 2009. “Educating the Least Informed: Group Endorsements in a Grassroots Campaign.” American Journal of Political Science 53 (October): 755–70.CrossRefGoogle Scholar
Barabas, Jason. 2004. “How Deliberation Affects Policy Opinions.” American Political Science Review 98 (November): 687–701.CrossRefGoogle Scholar
Barabas, Jason, and Jerit, Jennifer. 2009. “Estimating the Causal Effects of Media Coverage on Policy-specific Knowledge.” American Journal of Political Science 53 (January): 73–89.CrossRefGoogle Scholar
Bennett, W. Lance. 2006. News: The Politics of Illusion. 7th ed. New York: Longman.Google Scholar
Benz, Matthias, and Meier, Stephan. 2008. “Do People Behave in Experiments as in the Field? Evidence from Donations.” Experimental Economics 11: 268–81.CrossRefGoogle Scholar
Berinsky, Adam J. 2007. “Assuming the Costs of War: Events, Elites, and American Public Support for Military Conflict.” Journal of Politics 69 (November): 975–97.CrossRefGoogle Scholar
Blalock, H.M. Jr. 1979. Social Statistics. 2nd ed. New York: McGraw-Hill.Google Scholar
Brader, Ted. 2005. “Striking a Responsive Chord: How Political Ads Motivate and Persuade Voters by Appealing to Emotions.” American Journal of Political Science 49 (April): 388–405.CrossRefGoogle Scholar
Brehm, John. 1993. The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor: University of Michigan Press.Google Scholar
Cameron, A. Colin, and Trivedi, Pravin K.. 2005. Microeconometrics: Methods and Applications. New York: Cambridge University Press.CrossRefGoogle Scholar
Campbell, Donald T., and Stanley, Julian C.. 1963. Experimental and Quasi-experimental Designs for Research. Chicago: Rand McNally.Google Scholar
Chong, Dennis, and Druckman, James N.. 2007. “Framing Public Opinion in Competitive Democracies.” American Political Science Review 101 (November): 637–56.CrossRefGoogle Scholar
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Converse, Philip E. 1964. “The Nature of Belief Systems in Mass Publics.” In Ideology and Discontent, ed. Apter, David E.. New York: Free Press.Google Scholar
Cook, Fay Lomax, Jacobs, Lawrence R., and Kim, Dukhong. 2010. “Trusting What You Know: Information, Knowledge, and Confidence in Social Security.” Journal of Politics 72 (April): 397–412.CrossRefGoogle Scholar
Cook, Thomas D., and Campbell, Donald T.. 1979. Quasi-experimentation: Design & Analysis Issues for Field Settings. New York: Houghton Mifflin.Google Scholar
Cook, Thomas D., Shadish, William R., and Wong, Vivian C.. 2008. “Three Conditions under Which Experimental and Observational Studies Produce Comparable Causal Estimates: New Findings from Within-study Comparisons.” Journal of Policy Analysis and Management 27 (Autumn): 724–50.CrossRefGoogle Scholar
Delli Carpini, Michael X. 2009. “The Psychology of Citizen Learning.” In The Political Psychology of Democratic Citizenship, eds. Borgida, Eugene, Federico, Christopher M., and Sullivan, John L.. New York: Oxford University Press, 23–51.CrossRefGoogle Scholar
Druckman, James N. 2004. “Political Preference Formation: Competition, Deliberation, and the (Ir)relevance of Framing Effects.” American Political Science Review 98 (November): 671–86.CrossRefGoogle Scholar
Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur. 2006. “The Growth and Development of Experimental Research in Political Science.” American Political Science Review 100 (November): 627–35.CrossRefGoogle Scholar
Druckman, James N., and Nelson, Kjersten R.. 2003. “Framing and Deliberation: How Citizens’ Conversations Limit Elite Influence.” American Journal of Political Science 47 (October): 729–45.CrossRefGoogle Scholar
Dunning, Thad. 2008. “Improving Causal Inference: Strengths and Limitations of Natural Experiments.” Political Research Quarterly 61 (2): 282–93.CrossRefGoogle Scholar
Eagly, Alice, and Chaiken, Shelly. 1993. The Psychology of Attitudes. New York: Harcourt Brace.Google Scholar
Gaines, Brian J., and Kuklinski, James H.. 2009. “Including Self-selection in Random Assignment Experiments.” University of Illinois at Urbana–Champaign. Unpublished manuscript.Google Scholar
Gaines, Brian J., Kuklinski, James H., and Quirk, Paul J.. 2007. “The Logic of the Survey Experiment Reexamined.” Political Analysis 15 (Winter): 1–20.CrossRefGoogle Scholar
Gelman, Andrew. 2007. “Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22 (2): 153–64.Google Scholar
Gerber, Alan, and Green, Donald P.. 2000. “The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment.” American Political Science Review 94 (September): 653–63.CrossRefGoogle Scholar
Gibson, James. 1998. “A Sober Second Thought: An Experiment in Persuading Russians to Tolerate.” American Journal of Political Science 42 (July): 819–50.CrossRefGoogle Scholar
Gilens, Martin. 2001. “Political Ignorance and Collective Policy Preferences.” American Political Science Review 95 (June): 379–96.CrossRefGoogle Scholar
Gilens, Martin. 2002. “An Anatomy of Survey-Based Experiments.” In Navigating Public Opinion. eds. Manza, J., Cook, F. L., and Page, B. I.. New York: Oxford, 232–50.CrossRefGoogle Scholar
Gramlich, Edward M. 1998. Is It Time to Reform Social Security? Ann Arbor: University of Michigan Press.CrossRefGoogle Scholar
Grosser, Jens, and Schram, Arthur. 2006. “Neighborhood Information Exchange and Voter Participation: An Experimental Study.” American Political Science Review 100 (May): 235–48.CrossRefGoogle Scholar
Hovland, Carl. 1959. “Reconciling Conflicting Results Derived from Experimental and Survey Studies of Attitude Change.” American Psychologist 14: 8–17.CrossRefGoogle Scholar
Huber, Gregory A., and Arceneaux, Kevin. 2007. “Identifying the Persuasive Effects of Presidential Advertising.” American Journal of Political Science 51 (October): 957–77.CrossRefGoogle Scholar
Iyengar, Shanto, and Kinder, Donald. 1987. News That Matters. Chicago: University of Chicago Press.Google Scholar
Jacobs, Lawrence R., Cook, Fay Lomax, and Delli Carpini, Michael X.. 2009. Talking Together: Public Deliberation and Political Participation in America. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Jacobs, Lawrence R., and Shapiro, Robert Y.. 1998. “Myths and Misunderstandings about Public Opinion toward Social Security.” In Framing the Social Security Debate: Values, Politics, and Economics, eds. Arnold, R. Douglas, Graetz, Michael J., and Munnell, Alicia H.. Washington, DC: Brookings Institution Press, 355–88.Google Scholar
Jerit, Jennifer. 2009. “How Predictive Appeals Shape Policy Opinions.” American Journal of Political Science 53 (April): 411–26.CrossRefGoogle Scholar
Jerit, Jennifer, and Barabas, Jason. 2006. “Bankrupt Rhetoric: How Misleading Information Affects Knowledge about Social Security.” Public Opinion Quarterly 70 (Fall): 278–303.CrossRefGoogle Scholar
Jerit, Jennifer, Kuklinski, James H., and Quirk, Paul J.. 2009. “Strategic Rhetoric, Emotional Citizens, and the Rhetoric of Prediction.” In The Political Psychology of Democratic Citizenship, eds. Borgida, Eugene, Sullivan, John L., and Federico, Christopher M.. New York: Oxford University Press, 100–24.CrossRefGoogle Scholar
Kam, Cindy D., Wilking, Jennifer R., and Zechmeister, Elizabeth J.. 2007. “Beyond the ‘Narrow Database’: Another Convenience Sample for Experimental Research.” Political Behavior 29 (December): 415–40.CrossRefGoogle Scholar
Kellstedt, Paul M., and Whitten, Guy D.. 2009. The Fundamentals of Political Science Research. New York: Cambridge University Press.Google Scholar
Kennedy, Peter. 2003. A Guide to Econometrics. 5th ed. Cambridge, MA: MIT Press.Google Scholar
Keppel, Geoffrey. 1982. Design and Analysis: A Researcher's Handbook. 2nd ed. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
Kinder, Donald. 2007. “Curmudgeonly Advice.” Journal of Communication 57 (March): 155–62.CrossRefGoogle Scholar
Kinder, Donald R., and Palfrey, Thomas R., eds. 1993. Experimental Foundations of Political Science. Ann Arbor: University of Michigan Press.CrossRefGoogle Scholar
Kinder, Donald R., and Sanders, Lynn M.. 1996. Divided by Color: Racial Politics and Democratic Ideals. Chicago: University of Chicago Press.Google Scholar
King, Gary, Keohane, Robert O., and Verba, Sidney. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.CrossRefGoogle Scholar
Krosnick, Jon A., and Schuman, Howard. 1988. “Attitude Intensity, Importance, and Certainty and Susceptibility to Response Effects.” Journal of Personality and Social Psychology 54 (6): 940–52.CrossRefGoogle Scholar
Kuklinski, James H., Quirk, Paul J., Jerit, Jennifer, and Rich, Robert F.. 2001. “The Political Environment and Citizen Competence.” American Journal of Political Science 45 (April): 410–24.CrossRefGoogle Scholar
Kuklinski, James H., Quirk, Paul J., Jerit, Jennifer, Schwieder, David, and Rich, Robert F.. 2000. “Misinformation and the Currency of Citizenship.” Journal of Politics 62 (August): 790–816.CrossRefGoogle Scholar
Lassen, David Dreyer. 2005. “The Effect of Information on Voter Turnout: Evidence from a Natural Experiment.” American Journal of Political Science 49 (January): 103–88.CrossRefGoogle Scholar
Lavine, Howard. 2002. “On-line versus Memory Based Process Models of Political Evaluation.” In Political Psychology, ed. Monroe, Kristen. Mahwah, NJ: Lawrence Erlbaum Associates, 225–47.Google Scholar
Levitt, Steven D., and List, John A.. 2007. “What Do Laboratory Experiments Measuring Social Preferences Reveal about the Real World?” Journal of Economic Perspectives 21 (Spring): 153–71.CrossRefGoogle Scholar
Loughin, Thomas M. 2004. “A Systematic Comparison of Methods for Combining p-Values from Independent Tests.” Computational Statistics and Data Analysis 47: 467–85.CrossRefGoogle Scholar
Lupia, Arthur, Krupnikov, Yanna, Levine, Adam Seth, Grafstrom, Casandra, McMillian, William, and McGovern, Erin. n.d. “How ‘Point Blindness’ Dilutes the Value of Stock Market Reports.” Political Communication. Forthcoming.Google Scholar
Luskin, Robert C., Fishkin, James S., and Jowell, Roger. 2002. “Considered Opinions: Deliberative Polling in Britain.” British Journal of Political Science 32: 455–87.CrossRefGoogle Scholar
Marmor, Theodore R. 2000. The Politics of Medicare. 2nd ed. NY: Aldine de Gruyter.Google ScholarPubMed
McClelland, Gary H., and Judd, Charles M.. 1993. “Statistical Difficulties of Detecting Interactions and Moderator Effects.” Psychological Bulletin 114 (2): 376–90.CrossRefGoogle ScholarPubMed
McDermott, Rose. 2002. “Experimental Methods in Political Science.”Annual Review of Political Science 5: 31–61.CrossRefGoogle Scholar
Mondak, Jeffery J. 1995. “Newspapers and Political Awareness.” American Journal of Political Science 39 (May): 513–27.CrossRefGoogle Scholar
Mutz, Diana, and Reeves, Byron. 2005. “The New Videomalaise: Effects of Televised Civility on Political Trust.” American Political Science Review 99 (February): 1–15.CrossRefGoogle Scholar
Nadeau, Richard, and Niemi, Richard G.. 1995. “Educated Guesses: The Process of Answering Factual Knowledge Questions in Surveys.” Public Opinion Quarterly 59 (Autumn): 323–46.CrossRefGoogle Scholar
Norris, Pippa, and Sanders, David. 2003. “Message or Medium? Learning during the 2001 British General Election.” Political Communication 20: 233–62.CrossRefGoogle Scholar
Oberlander, Jonathan. 2003. The Political Life of Medicare. Chicago: University of Chicago Press.Google Scholar
O'Keefe, Daniel J. 2007. “Post Hoc Power, Observed Power, A Priori Power, Retrospective Power, Prospective Power, Achieved Power: Sorting Out Appropriate Uses of Statistical Power Analyses.” Communication Methods and Measures 1 (4): 291–99.CrossRefGoogle Scholar
Page, Benjamin I. 2000. “How Public Opinion Affects Reform: Is Social Security Reform Ready for the American Public?” In Social Security and Medicare: Individual versus Collective Risk and Responsibility, eds. Burke, Shelia, Kingson, Eric, and Reinhardt, Uwe. Washington, DC: National Academy of Social Insurance, 183–218.Google Scholar
Page, Benjamin I., and Shapiro, Robert Y.. 1992. The Rational Public: Fifty Years of Trends in Americans’ Policy Preferences. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Prior, Marcus, and Lupia, Arthur. 2008. “Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills.” American Journal of Political Science 52 (January): 169–83.CrossRefGoogle Scholar
Robinson, Gregory, McNulty, John E., and Krasno, Jonathan S.. 2009. “Observing the Counterfactual? The Search for Political Experiments in Nature.” Political Analysis 17: 341–57.CrossRefGoogle Scholar
Roth, Alvin. 1995. “Introduction to Experimental Economics.” In The Handbook of Experimental Economics, eds. Kagel, John H. and Roth, Alvin E.. Princeton, NJ: Princeton University Press, 3–109.Google Scholar
Shadish, William R., Cook, Thomas D., and Campbell, Donald T.. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.Google Scholar
Shaw, Greg M., and Mysiewicz, Sarah E.. 2004. “Trends: Social Security and Medicare.” Public Opinion Quarterly 68 (Fall): 394–423.CrossRefGoogle Scholar
Sniderman, Paul M., and Piazza, Thomas. 1993. The Scar of Race. Cambridge, MA: Harvard University Press.CrossRefGoogle Scholar
Sniderman, Paul M., and Theriault, Sean M.. 2004. “The Structure of Political Argument and the Logic of Issue Framing.” In Studies in Public Opinion: Attitudes, Nonattitudes, Measurement Error, and Change, eds. Saris, Willem E. and Sniderman, Paul M.. Princeton, NJ: Princeton University Press, 133–64.CrossRefGoogle Scholar
Turgeon, Mathieu. 2009. “‘Just Thinkin’: Attitude Development, Public Opinion, and Political Representation.” Political Behavior 31 (September): 353–78.CrossRefGoogle Scholar
Weisberg, Herbert F. 2005. The Total Survey Error Approach: A Guide to the New Science of Survey Research. Chicago: University of Chicago Press.CrossRefGoogle Scholar
Winship, Christopher, and Radbill, Larry. 1994. “Sampling Weights and Regression Analysis.” Sociological Methods and Research 23 (2): 230–57.CrossRefGoogle Scholar
Wooldridge, Jeffrey M. 2009. Introductory Economics: A Modern Approach. 4th ed. Mason, OH: South-Western Cengage Learning.Google Scholar
Zaller, John R. 1992. The Nature and Origins of Mass Opinion. New York: Cambridge.CrossRefGoogle Scholar
Zaller, John R. 2002. “The Statistical Power of Election Studies to Detect Media Exposure Effects in Political Campaigns.” Electoral Studies 21: 297–329.CrossRefGoogle Scholar
Zaller, John R., and Feldman, Stanley. 1992. “A Simple Theory of the Survey Response: Answering Questions versus Revealing Preferences.” American Journal of Political Science 36 (August): 579–616.CrossRefGoogle Scholar
Submit a response

Comments

No Comments have been published for this article.