Hostname: page-component-78c5997874-dh8gc Total loading time: 0 Render date: 2024-11-06T12:07:04.987Z Has data issue: false hasContentIssue false
  • English
  • Français

Thou shalt not cheat: how to reduce internet use in web surveys on political knowledge

Published online by Cambridge University Press:  22 December 2016

Cristiano Vezzoni  and
Riccardo Ladini Open the ORCID record for Riccardo Ladini [Opens in a new window]
Cristiano Vezzoni*
Affiliation:
Department of Sociology and Social Research, University of Trento, Trento, Italy
Riccardo Ladini
Affiliation:
Doctoral School of Social Sciences, University of Trento, Trento, Italy
*
*E-mail: [email protected]
Get access

Abstract

By means of a split-ballot survey experiment, we study whether a normative instruction not to use the internet when answering political knowledge questions reduces cheating in web surveys. The knowledge questions refer to basic facts about the European Union and the data come from the Italian National Election Study web panel carried out in Italy before the 2014 European Election. Our analysis shows that a simple normative instruction significantly reduces cheating. We also show that reducing cheating is important to achieve a correct assessment of reliability of knowledge scales, while a decrease of cheating leaves unaltered the knowledge gap between lower and higher educated respondents. These results invite caution when including political knowledge questions in an online survey. Our advice is to include a normative instruction not to search the internet to reduce cheating and obtain more genuine answers. More generally, we conclude by stressing the need to consider the implications of online data collection when building questionnaires for public opinion research.

Keywords

cheating in web surveyssurvey experimentspolitical knowledgeEuropean Unionreliability and validity
Type
Research Article
Information
Italian Political Science Review / Rivista Italiana di Scienza Politica , Volume 47 , Issue 3 , November 2017 , pp. 251 - 265
Copyright
© Società Italiana di Scienza Politica 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Atkin, C.K., Galloway, J. and Nayman, O.B. (1976), ‘News media exposure, political knowledge and campaign interest’, Journalism & Mass Communication Quarterly 53(2): 231–237.Google Scholar
Barabas, J., Jerit, J., Pollock, W. and Rainey, C. (2014), ‘The question (s) of political knowledge’, American Political Science Review 108(4): 840–855.Google Scholar
Berinsky, A.J., Margolis, M.F. and Sances, M.W. (2014), ‘Separating the shirkers from the workers? Making sure respondents pay attention on self‐administered surveys’, American Journal of Political Science 58(3): 739–753.CrossRefGoogle Scholar
Blair, G. and Imai, K. (2012), ‘Statistical analysis of list experiments’, Political Analysis 20(1): 47–77.Google Scholar
Boudreau, C. and Lupia, A. (2011), ‘Political knowledge’, in J.N. Druckman, D.P. Green, J.H. Kuklinski and A. Lupia (eds), Cambridge Handbook of Experimental Political Science, Cambridge: Cambridge University Press, pp. 171–183.Google Scholar
Callegaro, M., Lozar Manfreda, K. and Vehovar, V. (2015), Web Survey Methodology, London: Sage.Google Scholar
Carmines, E.G. and Zeller, R.A. (1979), Reliability and Validity Assessment, Beverly Hills, CA: Sage publications.Google Scholar
Cialdini, R.C. (1984), Influence. The Psychology of Persuasion, New York: Quill.Google Scholar
Clifford, S. and Jerit, J. (2014), ‘Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies’, Journal of Experimental Political Science 1(2): 120–131.CrossRefGoogle Scholar
Clifford, S. and Jerit, J. (2016), ‘Cheating on political knowledge questions in online surveys: An assessment of the problem and solutions’, Public Opinion Quarterly 80(4): 858–887.Google Scholar
Delli Carpini, M.X. and Keeter, S. (1993), ‘Measuring political knowledge: putting first things first’, American Journal of Political Science 37(4): 1179–1206, doi: 10.2307/2111549 Google Scholar
Delli Carpini, M.X. and Keeter, S. (1996), What Americans Know About Politics and Why it Matters, New Haven, CT: Yale University Press.Google Scholar
De Vreese, C.H. and Boomgaarden, H.G. (2006), ‘News, political knowledge and participation: the differential effects of news media exposure on political knowledge and participation’, Acta Politica 41(4): 317–341.Google Scholar
Eveland, W.P. Jr and Hively, M.H. (2009), ‘Political discussion frequency, network size, and “heterogeneity” of discussion as predictors of political knowledge and participation’, Journal of Communication 59(2): 205–224.Google Scholar
Feldt, L.S. (1969), ‘A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests’, Psychometrika 34(3): 363–373.Google Scholar
Iyengar, S., Curran, J., Lund, A.B., Salovaara‐Moring, I., Hahn, K.S. and Coen, S. (2010), ‘Cross‐national versus individual‐level differences in political information: a media systems perspective’, Journal of Elections, Public Opinion and Parties 20(3): 291–309.Google Scholar
Janus, A.L. (2010), ‘The influence of social desirability pressures on expressed immigration attitudes’, Social Science Quarterly 91(4): 928–946.Google Scholar
Jennings, M.K. (1996), ‘Political knowledge over time and across generations’, Public Opinion Quarterly 60(2): 228–252.Google Scholar
Jensen, C. and Thomsen, J.P.F. (2014), ‘Self-reported cheating in web surveys on political knowledge’, Quality & Quantity 48(6): 3343–3354.Google Scholar
Liu, Y.I. and Eveland, W.P. Jr (2005), ‘Education, need for cognition, and campaign interest as moderators of news effects on political knowledge: an analysis of the knowledge gap’, Journalism & Mass Communication Quarterly 82(4): 910–929.Google Scholar
Luskin, R.C. (1987), ‘Measuring political sophistication’, American Journal of Political Science 31(4): 856–899.Google Scholar
Mondak, J.J. and Davis, B.C. (2001), ‘Asked and answered: knowledge levels when we will not take “don’t know” for an answer’, Political Behavior 23(3): 199–224.CrossRefGoogle Scholar
Munzert, S. and Selb, P. (2015), ‘Measuring political knowledge in web-based surveys: an experimental validation of visual versus verbal instruments’, Social Science Computer Review. First published online on November 17, 2015: 1–17.Google Scholar
Prior, M. and Lupia, A. (2008), ‘Money, time, and political knowledge: distinguishing quick recall and political learning skills’, American Journal of Political Science 52(1): 169–183.Google Scholar
Schultz, P.W., Nolan, J.M., Cialdini, R.B., Goldstein, N.J. and Griskevicius, V. (2007), ‘The constructive, destructive, and reconstructive power of social norms’, Psychological Science 18(5): 429–434.Google Scholar
Shulman, H.C. and Boster, F.J. (2014), ‘Effect of test-taking venue and response format on political knowledge tests’, Communication Methods and Measures 8(3): 177–189.Google Scholar
Sirken, M.G., Herrmann, D.J., Schechter, S., Schwarz, N., Tanur, J.M. and Tourangeau, R. (eds) (1999), Cognition and Survey Research, New York: Wiley.Google Scholar
Strabac, Z. and Aalberg, T. (2011), ‘Measuring political knowledge in telephone and web surveys: a cross-national comparison’, Social Science Computer Review 29(2): 175–192.Google Scholar
Vavreck, L. (2012), ‘The myth of cheating on self-completed surveys’. Retrieved 1 December 2016 from http://today.yougov.com/news/2012/04/17/myth-cheating-self-completed-surveys/ Google Scholar
Supplementary material: Link

Vezzono et al. Dataset

Dataset

http://dx.doi.org/10.7910/DVN/4DNIIJ
Link