Hostname: page-component-6bf8c574d5-8gtf8 Total loading time: 0 Render date: 2025-02-16T17:15:44.613Z Has data issue: false hasContentIssue false
  • English
  • Français

Cross-Sample Comparisons and External Validity

Published online by Cambridge University Press:  20 August 2014

Yanna Krupnikov  and
Adam Seth Levine
Yanna Krupnikov
Affiliation:
Department of Political Science, Stony Brook University, Stony Brook, NY 11794, USA; email: [email protected]
Adam Seth Levine
Affiliation:
Department of Government, Cornell University, Ithaca, NY 14850, USA; email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Experimentation is an increasingly popular method among political scientists. While experiments are highly advantageous for creating internally valid conclusions, they are often criticized for being low on external validity. Critical to questions of external validity are the types of subjects who participate in a given experiment, with scholars typically arguing that samples of adults are more externally valid then student samples. Despite the vociferousness of such arguments, these claims have received little empirical treatment. In this paper we empirically test for key differences between student and adult samples by conducting four parallel experiments on each of the three samples commonly used by political scientists. We find that our student and diverse, national adult sample behave consistently and in line with theoretical predictions once relevant moderators are taken into account. The same is not true for our adult convenience sample.

Keywords

Experimentsexternal validitysubjectssample comparisons
Type
Research Article
Information
Journal of Experimental Political Science , Volume 1 , Issue 1 , Spring 2014 , pp. 59 - 80
Copyright
Copyright © The Experimental Research Section of the American Political Science Association 2014 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Arceneaux, K. 2012. Cognitive Biases and the Strength of Political Arguments. American Journal of Political Science 56: 271–85.CrossRefGoogle Scholar
Banks, A. and Valentino, N. A. 2012. Emotional Substrates of White Racial Attitudes. American Journal of Political Science 56: 289–97.Google Scholar
Barabas, J. and Jerit, J. 2010. Are Survey Experiments Externally Valid? American Political Science Review 104: 226–42.CrossRefGoogle Scholar
Berinsky, A. J., Huber, G. A. and Lenz, G. S. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk. Political Analysis 20: 351–68.CrossRefGoogle Scholar
Brady, H. 2000. Contributions of Survey Research to Political Science. PS: Political Science and Politics 33: 47–57.Google Scholar
Brody, J. L., Gluck, J. P. and Aragon, A. S. 2000. Participants’ Understanding of the Process of Psychological Research: Debriefing. Ethics & Behavior 10: 13–25.Google Scholar
Brooks, D. J. and Geer, J. G. 2007. Beyond Negativity: The Effects of Incivility on the Electorate. American Journal of Political Science 51: 1–16.Google Scholar
Chong, D. and Druckman, J. N. 2007. Framing Public Opinion in Competitive Democracies. American Political Science Review 101: 637–55.Google Scholar
Chong, D. and Druckman, J. N. 2010. Framing Theory. Annual Review of Political Science 10: 103–26.Google Scholar
Cook, T. D., Bean, J. R., Calder, B. J., Frey, R., Krovetz, M. L., and Reisman, S. R. 1970. Demand Characteristics and Three Conceptions of the Frequently Deceived Subject. Journal of Personality and Social Psychology 14: 185–94.Google Scholar
Dalen, L. H., Stanton, N. A. and Roberts, A. D. 2001. Faking Personality Questionnaires in Personnel Selection. Journal of Management Development 20: 729–42.Google Scholar
Druckman, J. N., Fein, J. and Leeper, T. J. 2012. A Source of Bias in Public Opinion Stability. American Political Science Review 106: 430–54.CrossRefGoogle Scholar
Druckman, J. N. and Kam, C. D. 2011. Students as Experimental Participants: A Defense of the ‘Narrow Data Base’. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Druckman, J. N. and Leeper, T. 2012. Learning More from Political Communication Experiments: Pretreatment and Its Effects. American Journal of Political Science 56: 875–96.Google Scholar
Gaines, B. J., Kuklinski, J. H. and Quirk, P. J. 2007. The Logic of the Survey Experiment Reexamined. Political Analysis 15: 1–20.Google Scholar
Gartner, S. S. 2008. The Multiple Effects of Casualties on Public Support for War: An Experimental Approach. American Political Science Review 102: 95–106.Google Scholar
Gerber, A. S. and Green, D. P. 2008. Field Experiments and Natural Experiments. In The Oxford Handbook of Political Methodology. eds. Box-Steffensmeier, Brady, and Collier, Oxford: Oxford University Press.Google Scholar
Gerber, A. S., Huber, G. A., Doherty, D. and Dowling, C. M. 2011. Citizens’ Policy Confidence and Electoral Punishment: A Neglected Dimension of Electoral Accountability. Journal of Politics 73: 1206–24.Google Scholar
Haider-Markel, D. P. and Joslyn, M. R. 2001. Gun Policy, Opinion, Tragedy, and Blame Attribution: The Conditional Influence of Issue Frames. Journal of Politics 63: 520–43.Google Scholar
Henry, P. J. 2008. College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice. Psychological Inquiry: An International Journal for the Advancement of Psychological Theory 19: 49–71.Google Scholar
Hopkins, D. and King, G. 2010. Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability. Public Opinion Quarterly 74: 201–22.Google Scholar
Huber, G. A. and Paris, C. 2013. Assessing the Programmatic Equivalence Assumption in Question Wording Experiments: Understanding why Americans Like Assistance to the Poor More Than Welfare. Public Opinion Quarterly 77: 385–97.Google Scholar
Huddy, L. and Khatib, N. 2007. American Patriotism, National Identity, and Political Involvement. American Journal of Political Science 51: 63–77.Google Scholar
Iyengar, S. 2011. Laboratory Experiments in Political Science. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Jerit, J. 2009. How Predictive Appeals Affect Policy Opinions. American Journal of Political Science 53: 411–26.CrossRefGoogle Scholar
Jones, E. E. and Sigall, H. 1971. The Bogus Pipeline: A New Paradigm for Measuring Affect and Attitude. Psychological Bulletin 76: 349–64.CrossRefGoogle Scholar
Kam, C. D. 2007. Implicit Attitudes, Explicit Choices: When Subliminal Priming Predicts Candidate Preference. Political Behavior 29: 343–67.Google Scholar
Kam, C. D. and Simas, E. 2010. Risk Orientation and Policy Frames. Journal of Politics 72 (2): 381–96.Google Scholar
Kam, C. D., Wilking, J. R. and Zechmeister, E. J. 2007. Beyond the ‘Narrow Data Base’: Another Convenience Sample for Experimental Research. Political Behavior 29: 415–40.Google Scholar
Kinder, D. 2003. Communication and Politics in the Age of Information. In Oxford Handbook of Political Psychology. eds. Sears, Huddy, and Jerivs, Oxford University Press.Google Scholar
Kinder, D. and Palfrey, T. 1993. On Behalf of Experimental Political Science. In Experimental Foundations of Political Science. eds. Kinder and Palfrey, Ann Arbor, Michigan: University of Michigan.Google Scholar
Levine, M. and Ensom, M. H. H. 2001. Post hoc Power Analysis: An Idea Whose Time Has Passed? Psychometry 21: 405–09.Google Scholar
McDermott, R. 2002. Experimental Methodology in Political Science. Political Analysis 10: 325–42.Google Scholar
McDermott, R. 2011. Internal and External Validity. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
McGraw, K. 2011. Candidate Impressions and Evaluation. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Miller, J. M. and Krosnick, J. A. 2000. News Media Impact on the Ingredients of Presidential Evaluations: Politically Knowledgeable Citizens are Guided by a Trusted Source, American Journal of Political Science 44: 301–15.Google Scholar
Morton, R. B. and Williams, K. C. 2010. Experimental Political Science and the Study of Causality. New York: Cambridge University Press.CrossRefGoogle Scholar
Mutz, D. 1992. Impersonal Influence: Effects of Representations of Public Opinion on Political Attitudes, Political Behavior 14: 89–122.Google Scholar
Mutz, D. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.Google Scholar
Nelson, T. E., Bryner, S. M. and Carnahan, D. 2011. Media and Politics. In Handbook of Experimental Political Science. eds. Druckman, Green, Kuklinski, and Lupia, New York: Cambridge University Press.Google Scholar
Nelson, T. E., Clawson, R. A. and Oxley, Z. M. 1997. Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance. American Political Science Review 91: 567–83.CrossRefGoogle Scholar
Ostrom, T. M. 1973. The Bogus Pipeline: A New Ignis Fatuus? Psychological Bulletin 79: 252–59.Google Scholar
Roese, N. J. and Jamieson, D. W. 1993. Twenty Years of Bogus Pipeline Research: A Critical Review and Meta-Analysis. Psychological Bulletin 114: 363–75.Google Scholar
Schwarz, N., Hippler, H.-J., Deutsch, B., and Strack, F. 1985. Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments. Public Opinion Quarterly 49: 388–95.Google Scholar
Sears, D. 1986. College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature. Journal of Personality and Social Psychology 51: 515–30.Google Scholar
Shadish, W. R., Cook, T. D. and Campbell, D. T. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.Google Scholar
Sigall, H. and Page, R. 1972. Reducing Attenuation in the Expression of Interpersonal Affect via the Bogus Pipeline. Sociometry 35: 629–42.Google Scholar
Steffens, M. C. 2004. Is the Implicit Association Test Immune to Faking? Experimental Psychology 51: 165–79.Google Scholar
Taber, C. and Lodge, M. 2006. Motivated Skepticism in the Evaluation of Political Beliefs. American Journal of Political Science 50: 755–69.CrossRefGoogle Scholar
Transue, J. E., Lee, D. J. and Aldrich, J. H. 2009. Treatment Spillover Effects Across Survey Experiments. Political Analysis 17: 143–61.CrossRefGoogle Scholar
Vavreck, L. and Rivers, D. 2008. The 2006 Cooperative Congressional Election Study. Journal of Elections, Public Opinion, and Parties 18: 355–66.Google Scholar
Supplementary material: PDF

Krupnikov Supplementary Material

Appendix

Download Krupnikov Supplementary Material(PDF)
PDF 85.7 KB