The Design of the Study: Mode, Sample and Size

The most striking aspect of the 2019 CES is that it was conducted in two modes—a random-digit-dial (RDD) telephone survey (n = 4,021) and a large-scale online collection (n = 37,822). Both datasets and relevant documentation are available at Project Dataverse (Stephenson et al., Reference Stephenson, Harell, Rubenson and Loewen2020a, Reference Stephenson, Harell, Rubenson and Loewen2020b) and through the Canadian Opinion Research Archive (https://www.queensu.ca/cora/home). This is not the first time the CES was conducted in different modes. Since 1988 the studies have involved multiple waves of questionnaires with the same respondents. Most commonly, this meant a campaign-period phone survey, a post-election phone survey and a follow-up mailback pen-and-paper questionnaire. In 2011 this model was expanded to include a fourth, web-based wave. In 2015, there was a parallel online survey conducted alongside a traditional three-wave study (phone and mailback). In 2019 each of the surveys had two waves, one during the campaign and one after the election.

The telephone survey was conducted by Advanis Inc. and largely followed the format of past studies in terms of content and order, though it was significantly shorter than previous studies (averaging approximately 20 minutes for each wave). For the campaign-period survey (CPS), the sample quotas were similarly distributed across the country (five regions: Atlantic, Quebec, Ontario, Prairies, British Columbia; 20 per cent sample each). Because of the rise of cell phone use, the study was intentionally dual-frame, utilizing both wireless and landline numbers. The CPS sample included 66 per cent wireless and 34 per cent landline numbers. Respondents were asked about the types of phone lines they have, and that information is incorporated into the weights provided in the dataset. The response rate for the CPS was 5.6 per cent and the post-election survey (PES) re-interview rate was 72 per cent.

The 2019 online data collection was designed to build upon the model used by the Local Parliament Project in 2015 (https://www.localparliament.ca) designed by Loewen and Rubenson, with Royce Koop. Online panel data has the advantage of being far less expensive than RDD telephone surveys, allowing a much larger scale than traditional telephone studies and increasing the likelihood of collecting significant numbers of respondents from all ridings across the country and from smaller demographic groups. This facilitates, for example, the analysis of local riding effects (Allen Stevens et al., Reference Allen Stevens, Islam, de Geus, Goldberg, McAndrews, Mierke-Zatwarnicki, Loewen and Rubenson2019). In the 2019 Canadian Election Study—Online Survey, we continued with this model. Of the 338 ridings in Canada, we have 50 or more responses in 95 per cent of ridings (see Figure 1). The online survey also used a cooperative model for data collection where other researchers were invited to contribute questions that were assigned to small subsamples of the study after they received the core instrument. This allowed us to further increase sample while promoting a range of separate studies outside the research team.

Figure 1 Number of Responses per Riding, 2019 CES - Online Survey, Campaign-Period Survey

Note: n= 37,262. The riding of 560 respondents could not be determined from the postal code they entered.

The online study survey was programmed by the CES team and associates and hosted by the University of Western Ontario through its licensed Qualtrics platform. Qualtrics also recruited respondents, aiming for three-day samples that were stratified by region (Ontario, Quebec, West, East and North)Footnote ¹ and balanced on gender and age within each region. Regions were sampled according to their approximate demographic weight. We also aimed for a language distribution of 80 per cent French and 20 per cent English within Quebec, 10 per cent French within the Atlantic region and 10 per cent French nationally. Respondents needed to be aged 18 or over and Canadian citizens or permanent residents in order to participate. The weights provided in the dataset are based upon age, gender, education and province census distributions. Traditional response rates cannot be calculated for online samples, but the re-interview rate for the PES was 27.3 per cent. The datasets were pre-processed by cleaning out any respondents who provided incomplete responses to initial demographics or the core survey, took the survey twice, completed either wave of the survey in less than 8.5 minutes, “straight-lined” multiple grid questions (“straightliners”) or provided a postal code not matching their province.

The 2019 CES continued the tradition, begun in the 1988 survey, of rolling cross-section samples (Johnston and Brady, Reference Johnston and Brady2002; Johnston et al., Reference Johnston, Blais, Brady and Crête1989). In both modes, responses were designed to be representative of the general population in three-day windows. This meant averaging about 100 surveys per day on the phone and 1,000 per day online. Given the importance of the last few days of the campaign for affecting the outcome, we gathered on average about 1,650 responses per day online in the last five days of the campaign.

The 2019 CES continued Canada's involvement with the Comparative Study of Electoral Systems (CSES) by administering Module 5 in both modes’ post-election waves (https://cses.org/data-download/cses-module-5-2016-2021/). Along with providing comparative items across a host of electoral democracies, this module includes interesting questions on nationalism and populism. Due to length restrictions (in terms of both time and not having a mailback component), the telephone surveys contain far fewer questions than the online surveys. Designing a shorter survey made us pay close attention to privileging questions that would be important for understanding the dynamics of the campaign and election outcome, as well as those that were crucial for important time series about the Canadian population. The core online CES instrument included two types of questions: those fielded to the full sample and those fielded to a series of split samples in order to ask a broader range of questions in the core instrument. For example, questions about positions on issues such as assisted dying were randomly asked of one-third of the sample.Footnote ² This format allowed us to make sure that we struck a balance between innovative research questions and ones that help us to understand the evolution of Canadian public opinion. Finally, in collaboration with electoral management bodies across Canada, we included several questions in the online study about electoral administration, such as sources of information about voting. Because turnout is a perennially interesting and important research question for political scientists and policy makers, understanding more about the actual cost and process of voting is relevant.

Key Contribution: Facilitating Subgroup Analysis

Traditionally, studies of vote choice are limited in their ability to analyze within-group variation because of smaller sample sizes. Those interested in studying the political behaviour of immigrants, for example, often have to pool several CES surveys in order to get a large enough sample (for example, Bilodeau et al., Reference Bilodeau, White and Nevitte2010; Harell, Reference Harell, Bittner and Koop2012), complicating the analysis with election-specific dynamics. The size of the 2019 CES online sample alleviates this problem and opens up new and promising avenues of research, as well as the possibility of examining important cleavages within groups. The 2019 online study includes a substantial number of foreign-born citizens (4,925) and, for the first time, includes permanent residents in Canada (1,342). There are also substantial samples of ethnic, linguistic and religious minority communities—which allow for finer-grained analyses than catch-all “visible minority,” allophone or non-Christian religion categories—as well as a large group of first-time eligible voters and LGBTQ respondents (see Table 1).

Table 1 Sample Size of Selected Subgroups in the 2019 CES—Online Survey

Note: Unweighted frequencies.

Subgroup analysis is particularly important because conventional wisdom often presumes that some groups of people lean one way or another politically—for example, that immigrants support the Liberal party (Blais, Reference Blais2005)—and this can have implications for party campaign activity and attention, not to mention for our understanding of electoral dynamics. Figure 2 provides examples of the power of our sample to detect subgroup variation. The Liberal advantage among immigrant-origin electors was reproduced in 2019 (see Figure 2), but there is substantial variation across immigration waves, with Conservatives doing particularly well among older cohorts of immigrant electors. Potential citizens (permanent residents) were also particularly favourable toward the Liberals, suggesting the Liberal advantage among immigrants may continue in future cohorts of citizens.

Figure 2 Vote Intention by Immigrant Status and Year of Entry into Canada

Note: 2019 CES - Online Survey. Graphs show unweighted means for vote intention, with 95 per cent confidence intervals. Permanent residents were asked which party they would vote for if they were eligible. LPC: Liberal Party of Canada; CPC: Conservative Party of Canada; NDP: New Democratic Party; BQ: Bloc Québécois.

The Consortium on Electoral Democracy (https://c-dem.ca) has administered the CES since 2019 and will coordinate election surveys at the federal and provincial levels until 2026. The network of more than 50 researchers and partners across Canada addresses urgent questions relating to political commitment, under-representation, levels of government, the evolution of public opinion between and through elections, as well as data collection practices, with a cooperative, evidence-based approach.