Sampling Practices in Applied Linguistics

There is no reason to assume that the field of Applied Linguistics is doing any better in terms of population representation than its neighboring disciplines. There are many indications (but little hard evidence so far) that certain groups are underrepresented. For example, in a plea for attention to illiteracy in second-language (L2) learning published in TESOL Quarterly, Bigelow and Tarone (Reference Bigelow and Tarone2004) pointed out that there had been no publications on illiterate L2 learners in TESOL Quarterly in the previous ten years, even though millions of people in the US and hundreds of millions of people worldwide are illiterate (currently estimated at 750 million by the UNESCO Institute for Statistics [2019]). Sixteen years later, this situation remains largely unchanged (but see Bigelow, Delmas, Hansen, & Tarone, Reference Bigelow, Delmas, Hansen and Tarone2006). Conversely, there is also evidence that some groups are overrepresented: Plonsky (Reference Plonsky2016) estimated that 67% of research samples in L2 research are university student samples. To date, however, no published research has systematically investigated sampling biases in Applied Linguistics research.

One way to gauge the sampling practices in the field is to examine published meta-analyses. The field has witnessed a plethora of meta-analyses, which collectively cover a large number of published studies on a wide array of topics. In some ways, meta-analyses are the hallmark of researchers’ wish to generalize. Meta-analysts typically synthesize a collection of studies that, by the nature of their designs, all aim for generalization, and in doing so, meta-analysts aim to present the state of the art in a particular area of inquiry. Meta-analysis also affords a possibility to evaluate the sampling practices in a given domain, given that many meta-analysts will summarize and present the demographic characteristics of the participant samples included in the primary studies. Table 1 lists sample information provided by meta-analyses published in the last five years that coded participants’ background in sufficient detail to allow insight into the field's sampling practices. As can be seen, this yielded a selection of 17 meta-analyses collectively covering 863 studies and a fair (but obviously far from exhaustive) range of topics that applied linguists are concerned with. The table shows that 65% of all samples were university student samples. Samples other than university students tended to consist of elementary or high school pupils. These samples of young learners accounted for 25% of the research database. There is thus a sizeable amount of research with young learners, but it still constitutes a heavy underrepresentation in light of the fact that L2 learning takes place in schools much more frequently than at universities (Kormos & Sáfár, Reference Kormos and Sáfár2008).

Table 1. Characteristics of participant samples (k) in recent meta-analyses (2014-2019)¹

¹ Notes: Elem. school = elementary school; Lang. ins. = language institute; Miss. = missing

² To be included in the table, the meta-analyses needed to be published between 2014 and 2019, were required to code for educational level, and were not allowed to exclude studies on the basis of educational level. To avoid overlap, the most recent meta-analysis was included in case studies covered the same topic. Journal selection was based on Al-Hoorie & Vitta (2018); journals were SSCI indexed and international in scope. They were: Applied Linguistics, Applied Psycholinguistics, Bilingualism: Language & Cognition, Computer Assisted Language Learning, ELT Journal, English for Specific Purposes, Foreign Language Annals, International Review of Applied Linguistics in Language Teaching, Language Assessment Quarterly, Language Learning, Language Learning & Technology, Language Teaching Research, Language Testing, The Modern Language Journal, ReCALL, Second Language Research, Studies in Second Language Acquisition, System, and TESOL Quarterly.

³ Number of articles and the total number of samples don't correspond exactly. Articles included in a meta-analysis often provide more than one sample, but some articles in the meta-analysis don't provide information about educational level.

⁴ This number represents elementary and high school samples together.

If we zoom in on the adult groups, the picture is particularly bleak. A staggering 88% of all adult samples are university student samples. The remaining 12% were samples coded as “language institute” or “other,” and, while it is not exactly clear what kind of learners fall in these categories, we might conclude that we know comparatively little about adult language learning that takes place outside the walls of academia.

It is important to acknowledge that an investigation of participant samples on the basis of meta-analyses has limitations. For one thing, descriptive and qualitative work remains under the radar, but such work often foregrounds the contextual and situated nature of language learning and use over its generalizability. Also, particular topics foreground particular populations: reading research is often conducted with younger learners and study abroad with university students. Furthermore, in meta-analyses, a large number of studies are typically dropped because of unfit study designs and insufficient detail in reporting. We cannot rule out (but do not assume either) that studies with nonuniversity learners present greater challenges in study design and are, therefore, excluded more often from meta-analysis. If that were the case, research with nonuniversity learners would be underrepresented in the meta-analytic literature and the proportion of research samples with non-university learners might actually be higher.

Zhang's (Reference Zhang2019) recently published bibliometric analysis of second language acquisition (SLA) research provides yet another view of the field's sampling practices. This analysis shows clearly that the field is dominated heavily by Western scholarship. Using the 16 most influential Applied Linguistics journals as determined by Al-Hoorie and Vitta (Reference Al-Hoorie and Vitta2018), Zhang (Reference Zhang2019) analyzed the origins of the publications in these journals. The analysis showed that the field has been heavily dominated by journals published in the U.S., publications produced at prestigious North American research institutes, and scholars working in Anglo-Saxon environments (although Japanese scholarship has been strong, and Chinese scholarship is on the rise). It is possible that the research produced at Anglo-Saxon universities was conducted elsewhere, such that there may not always be a one-to-one relationship between the university location and the study location, but overall, the picture emerging is that insight from Applied Linguistics research is strongly shaped by information gathered from university students, and these tend to be enrolled at prestigious research institutes in North America. The inescapable conclusion is that our participants are truly WEIRD.

It is interesting to consider briefly why sampling biases exist. Notions of convenience and accessibility are easily invoked as an explanation for sampling biases, but probably too simplistic (Ortega, Reference Ortega2005). Of course, many researchers are likely to resort to student samples, because they are easily available. Many programs, for example, maintain a database of potential research participants and sometimes have a credit system in place for student participation in research. But the reasons underlying sampling bias probably run deeper. One important reason could be a lack of engagement with language-learning contexts outside the walls of the university (Ortega, Reference Ortega2005). Many applied linguists teach languages at their own research institutes, and it is natural that their classroom observations generate hypotheses that inspire their research. Moreover, the tendency to look for participants within the walls of academia may be exacerbated by the “publish or perish” culture that governs much of the production of knowledge in academia (Chambers, Reference Chambers2019). Making it in academia tends to depend on the number of publications one produces and the perceived quality of the journals in which they appear, and this makes it attractive to go for the more easily accessible student populations.

What are the consequences for generalizability?

The ways in which existing sampling biases may have affected the state of our knowledge must remain a matter of speculation until we systematically gather knowledge from a wider array of language learning contexts. As Henrich et al. (Reference Henrich, Heine and Norenzayan2010) have demonstrated for several areas within psychology, there is every reason to be concerned about this. One consequence of sampling bias could be that behavior attested in particular groups is simply absent in other groups. This is illustrated by Tarone and Bigelow (Reference Tarone and Bigelow2005), who argued that the influential noticing hypothesis (Schmidt, Reference Schmidt1990, Reference Schmidt1994) may need reconsideration in light of evidence showing that illiterate learners typically do no exhibit noticing where literate learners do. Another consequence is that we may underestimate the range of variation in language learners and language learning outcomes. The Applied Linguistics field is replete with studies that try to establish links between language learning or use and particular individual characteristics. Such endeavors are likely to be compromised due to range restriction if observations are sampled from just a narrow band of the population distribution. How conclusive are findings on the role of working memory or aptitude in foreign language learning, if they are based on university students exclusively? Correlations are hard to detect in the absence of variation and may well be deflated or gone if one zooms in on a restricted range of variation. For this reason, at least one psychology lab at a northeastern U.S. elite institution no longer recruits study participants among its university students, although such a measure tends to be the exception rather than the rule.

Very few studies have directly tested how the nature of the sample affects the behavior under scrutiny. One exception is Andringa (Reference Andringa2014), who investigated how the characteristics of native-speaker comparison groups affect decisions about the incidence of native-like performance by second language learners. Second language learners were compared against a nonrepresentative (academic-only) native speaker norm group and a representative norm group, which approximately followed the education demographics of the society from which the native speakers were drawn, including participants with an academic background. Andringa showed that L1 proficiency ranges were consistently larger in the representative norm group, suggesting that it is both unrealistic and unrepresentative to hold L2 speakers’ language abilities up against academically trained L1 speakers only.

In reading research, the heavy reliance on participant samples learning to read in English (with the English language being an extreme outlier in terms of the inconsistency of the sound-spelling link) illustrates how sampling biases can negatively affect the state of our knowledge. Share (Reference Share2008) offered a fascinating and somewhat disturbing discussion of the consequences, which include, among others, an undue focus on accuracy in learning to read and in defining and diagnosing dyslexia, and the adoption of pedagogies that are misplaced when learning to read in languages with more regular orthographies. In Share's words, the overwhelming presence of English in reading research has led to “distorted theorizing with regard to many issues—including phonological awareness, early reading instruction, the architecture of stage models of reading development, the definition and remediation of reading disability, and the role of lexical-semantic and supralexical information in word recognition” (p. 584).

As Ortega (Reference Ortega2005) has poignantly argued, our sampling choices not only create problems for generalizability, but also pose ethical dilemmas. Our science may not provide answers to questions for a vast majority of language learners, even within WEIRD societies. Ultimately, the challenges we face as a discipline may lie elsewhere. They may lie in how to sustain linguistic diversity, in how to sustain multilingual development, in dealing with the language-learning needs of refugees and migrants, who may have had limited or interrupted formal education, in the relationship between (second) language skills and healthy aging, and in the dropping numbers of students learning languages other than English (e.g., Cox, Reference Cox2019; King, Schilling, Fogle, Lou, & Soukup, Reference King, Schilling, Fogle, Lou and Soukup2008; Ortega, Reference Ortega2019; Pot et al., Reference Pot, Keijzer and de Bot2018; Reynolds, Reference Reynolds2019). Does our research and the models that we generate represent the groups that face these challenges, and does it speak to their experiences? Based on the evidence presented above, we cannot be confident they do. Ultimately, for Applied Linguistics research to be considered relevant by stakeholders, researchers need to demonstrate and make a case for the impact of their work beyond the walls of the academy, in a society that faces many real linguistic needs and questions.

Making Applied Linguistics More Open and Less WEIRD: Some Suggestions

Changing current sampling practices will not be easy. It will require applied linguists to broaden the scope of their research. For this to happen, they may need to step outside their comfort zone, to take note of language learning contexts outside the university, to collaborate with scholars working in different settings, and to actively engage in more dialogue with stakeholders, such as teachers, policymakers, and funding agencies. Current publication practices and the emphasis on publication metrics in researcher quality evaluation may also need to change.

Some of these changes are already underway. The OASIS project, which stands for Open Accessible Summaries In Language Studies, is one cross-journal initiative that may help foster interaction between researchers and stakeholders (“OASIS,” n.d.), and has been embraced by a number of journals in our field already (e.g., Marsden, Trofimovich, & Ellis, Reference Marsden, Trofimovich and Ellis2019). Changes are also visible in the fact that many funding agencies now consider the broader impacts or societal impacts of research grant applications as a required and integral part of the grant evaluation process (e.g., NSF, n.d.; NWO, n.d.). And in Europe, the Plan S initiative, which mandates that publicly funded research from 12 European countries be published in open access journals by 2021, aims not only to make science freely available to everyone, but also tries to change the way scholarly achievement is evaluated by putting in place a system that acknowledges that an academic career is also shaped by a scholar's ability to engage with society and to teach (Science Europe, Reference Europe2018). All these initiatives are meant to break down walls between the academy and the wider society, so researchers may expand their research, relevance, and reach.

And of course, we should not take the current state of our knowledge for granted, but work directly to diversify participant samples. One powerful way to do this is to replicate existing research with different samples and evaluate to what extent the same conclusions still hold. This is the essence of the SLA for All? project (Andringa & Godfroid, Reference Andringa and Godfroid2018), in which we invited scholars to replicate influential studies in different, nonacademic samples. The goals of the project are to (1) raise awareness of the existing sampling biases in Applied Linguistics and the problems associated with them; (2) to get a sense of the extent to which the current state of knowledge in Applied Linguistics is indeed distorted by these sampling practices; (3) to gain experience doing research in different contexts and with learners who currently often remain invisible in research; and (4) to learn to what extent the tools applied linguists have developed are adequate for doing so, as they may well not be (Ortega, Reference Ortega2019). We hope that such an initiative will promote further efforts to make Applied Linguistics less WEIRD and, with that, a bit more fair and inclusive.