1. Introduction
Based on the rigorous systematicity assumed in systematic review methodology, it is no surprise that a prominent review such as Macaro et al.'s (Reference Macaro, Curle, Pun, An and Dearden2018) on English medium instruction (EMI) has been used as a basis for subsequent EMI research. However, in this article, we explore the ways in which the focus of systematic reviews can be necessarily narrowed and how this poses a risk to research when readers perceive them as offering definitive conclusions on all aspects of a subject. This article addresses two significant trends in applied linguistics. First, systematic review – that is, the use of formalised systems when reviewing literature – has become far more prominent and therefore more impactful than traditional reviews as a methodology (Chong & Plonsky, Reference Chong and Plonsky2023). Second, there has been an explosive growth in interest in EMI research (Curle et al., Reference Curle, Rose and Yuksel2024). There are further parallels between the two trends, given that both systematic review and EMI are umbrella terms that cover a wide range of research types. As we will see, there is perhaps more disagreement over how to conduct a systematic review than lay readers would suspect. Similarly, EMI is a broader field of research than appears in its most prominent systematic review article. Studies into EMI have explored policy, language learning, the effect on subject knowledge, attitudes towards EMI, ownership of English, and so on. Thus, while EMI is a growingly recognised field of study, it is not always clear what it means to ‘study EMI’.
As a result, systematic reviews of EMI are both highly desirable – to corral the high number of EMI papers – and fraught – because of the range of approaches to both EMI research and systematic review. In this position paper, we seek to draw attention to this difficulty by advancing two arguments:
1. That systematic review is generally a positivist methodology and that this significantly impacts how these reviews are conducted.
2. That positivism has had an impact on the results of Macaro et al.'s (Reference Macaro, Curle, Pun, An and Dearden2018) systematic review: a landmark paper that has had a significant role in shaping and potentially diverting the field of EMI.
This structure is intended to be quite flexible: this article can be read as a critique of how systematic reviews are often conducted, backed up with an example; conversely, it can be read as a case study about the importance of understanding the limits of the methodology when reading and interpreting one frequently read and cited systematic review (e.g., Macaro et al. Reference Macaro, Curle, Pun, An and Dearden2018). However, the present article should not be read as a critique of the paper, of systematic review in general, or even positivism. The findings of this, and other systematic reviews both of EMI (e.g., Dang et al., Reference Dang, Bonar and Yao2023; Feng et al., Reference Feng, He and Yiganmu2023; Lu et al., Reference Lu, Gu and Lee2023) and in applied linguistics more generally (Huang & Chalmers, Reference Huang and Chalmers2023; Kalyaniwala & Ciekanski, Reference Kalyaniwala and Ciekanski2021; Li, Reference Li2023) are useful but, we argue, not to be taken at face value or cited uncritically. We conclude by making suggestions for future reviews.
2. Systematic review and positivism
2.1 What is a systematic review?
A systematic review is a secondary research methodology that aims to be far more comprehensive than a typical literature review. Prima facie, a systematic review can be defined as a ‘review which uses systems’. This would contrast with a narrative or critical review, which would have much more leeway regarding how papers are selected and analysed. However, in applied linguistics, definitions of ‘systematic review’ have generally been more specific. For Norris and Ortega (Reference Norris and Ortega2006, Reference Norris and Ortega2007), a systematic review has three characteristics:
1. It has a system for collecting data.
2. It bases its analysis on the reported data used in original studies rather than the researcher's interpretations from them.
3. It has a system for collating data.
This typically involves trying to, within certain bounds, read all research answering a certain question and then trying to condense that research into an answer to that question. One commonly used definition is particularly strict: Petticrew and Roberts (Reference Petticrew and Roberts2006) describe systematic review as a method designed to reduce bias, achieved by pre-planning and reviewing every study in a field. This type of narrowing can be seen in studies in applied linguistics (e.g., Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2018; Rose et al., Reference Rose, Briggs, Boggs, Sergio and Ivanova-Slavianskaia2018, Reference Rose, McKinley and Galloway2021). As we will argue, however, this attempt to reduce bias can, ultimately, introduce its own kind of bias.
2.2 Positivism and systematic review
While these more restrictive definitions of systematic review frame themselves as scientific in general, we argue that they come from a certain set of beliefs about research, namely positivism. For Noblit and Hare (Reference Noblit and Hare1988), positivism is generally optimistic about the creation of generalisable theories and causes, which it seeks to generate through hypothesis testing and quantifying social events. Furthermore, they argue, it sees bias as controllable, and researchers as detached from the context of research.
For a holder of this type of positivist viewpoint, systematic review methodology is a good fit, especially in the more restrictive forms described above. The use of multiple researchers, a predefined transparent methodology, and a focus on exhaustiveness are all aligned with positivism. Similarly, the goal of eliminating bias to create a generalised answer to a question is highly positivist. Ultimately, if a researcher believes an object of study is broadly similar across contexts, it makes sense to attempt to aggregate research about that object using a systematic review. Therefore, as Ortega (Reference Ortega, Paltridge and Phakiti2015) notes, it is unsurprising that ‘research synthesis typically has a positivist and quantitative orientation’ (p. 230).
3. Problematising positivism
Of course, positivist beliefs are not held by all researchers, particularly when it comes to those doing qualitative research, which has gained increased significance in language teaching research (see Mirhosseini & Pearson, Reference Mirhosseini and Pearson2024). Perhaps, as a result, there has been a bifurcation of ‘qualitative systematic review’ (i.e., a systematic review of qualitative research) and ‘quantitative systematic review’ (i.e., a systematic review of quantitative research) – and, perhaps tellingly, that there are ‘two types of secondary research, systematic literature review and qualitative research synthesis’ (Chong & Plonsky, Reference Chong and Plonsky2023, p. 1587). However, there is not an isomorphic, or one-to-one relationship between positivism and quantitative research, or indeed positivism and quantitative systematic reivew. For example, Jinpei Zhang's (Reference Zhang2023) description of how qualitative systematic review should be done takes a positive stance on what research synthesis entails. As such, we argue that qualitative systematic review, if it remains positivist, can run into the same issues as quantitative systematic review.
Modern texts can be magnanimous in their description of qualitative and quantitative systematic review as complementary methods. However, for a more full-throated criticism of positivist forms of systematic review, we return to an older text that we find still relevant today. Noblit and Hare (Reference Noblit and Hare1988) argue that from a constructivist viewpoint, the object and findings of a study are socially constructed. Thus, they are dependent on society for their definition and can, therefore, differ depending on context. This means that analyses from multiple contexts cannot simply be combined without first determining their compatibility. To illustrate, if we are interested in how students' social class impacts their progress in EMI courses, it makes little sense to aggregate studies across contexts where social class acts differently. Thus, we would not, for example, expect the Polish and Indian middle class to engage with EMI in the same way. Conversely, when describing a single phenomenon in a single context, two authors might use different terminology that can make it appear they are referring to two separate constructs. This, then, also requires a significant amount of interpretation.
Furthermore, while we might have an object of study that is well-defined, this does not mean it exists similarly across contexts. For example, while EMI has a reasonably rigid definition, EMI policies and practices manifest differently in different contexts (see Rose & McKinley, Reference Rose, McKinley, Wei, Hua and Simpson2023). Thus, an attempt to ‘identify, appraise and synthesize all relevant studies (of whatever design) in order to answer a particular question (or set of questions)’ (Petticrew & Roberts, Reference Petticrew and Roberts2006, pp. 9–10), in the case of EMI, and in many other cases, will lead to problems of incommensurability (Rose et al., Reference Rose, Macaro, Sahan, Aizawa, Zhou and Wei2023). This is true even if a given systematic review does include both qualitative and quantitative studies.
A further criticism is that positivism often downplays the difficulty – or indeed impossibility – of controlling for bias. Crucially, rather than eliminating bias, the use of multiple researchers to conduct a systematic review may only ensure that shared biases prevail. For example, a team of researchers from a university in the Global North may unwittingly impose certain biases on their research. Similarly, reviewing a range of articles may not ‘average out’ bias if that bias is systematic. Finally, as we have seen, positivist systematic reviews are interested in evaluating the reliability of studies. As such, there is a dilemma when positivist systematic reviewers need to evaluate studies that take a non-positivistic view of bias – and indeed of how to evaluate the quality of research more broadly. Either reviewers need to evaluate studies based on positivist criteria, or they need to take a relativistic approach.
One particularly balanced response to these ideas comes from Gough and Thomas (Reference Gough and Thomas2016) in the field of education more broadly. They reject the labels qualitative and quantitative with regard to systematic review (see also Hampson & McKinley, Reference Hampson and Mckinley2023). Rather, they argue that a systematic review can be described on several continua. For example, a review might be highly interested in generating a new theory or interested in testing one or more hypotheses. It might also lie somewhere between those two poles. Similarly, the extent to which someone is interested in eliminating bias is better understood as a continuum rather than a binary. This stands in contrast to the narrow views of systematic review above by positioning it as a broad range of methods that can be applied as the situation dictates. Gough and Thomas (Reference Gough and Thomas2016) respond to criticisms that label systematic reviews as overly positivist by highlighting that this misperception stems from their early association with randomised controlled trials. They argue that the principles of systematic review are applicable to all types of research questions and methods, and that focusing solely on the positivist aspects has hindered progress in research methodologies and epistemologies. They also contend that criticisms of systematic reviews often miss the mark, stemming either from outdated views about the methodology's scope and potential, or from broader concerns about the purpose and nature of educational research.
Ultimately, we see this view as neither rejecting positivist systematic review nor assuming that this type of review is more ‘scientific’ than other types of systematic review. Rather, it places the onus on systematic reviewers to be aware of the ways in which paradigmatic concerns will influence their work. This, perhaps, requires nuance when it comes to assessing the quality of the studies under review. Correspondingly, readers of systematic reviews need to be aware that despite the reputation of systematic review as highly objective and scientific, the paradigm that informs a systematic review can impact how it is conducted and how it views its component studies. We do not believe the bifurcation of qualitative and quantitative systematic review is a solution to this problem, as this fails to recognise the diversity of systematic reviews.
4. Evaluating usage of Macaro et al. (Reference Macaro, Curle, Pun, An and Dearden2018)
In the first half of this article, we have argued that a positivist paradigm can impact the findings of a systematic review. We now move to an exploration of how these have played out in the case of the usage of Macaro et al.'s (Reference Macaro, Curle, Pun, An and Dearden2018) systematic review, which has been cited nearly 1,400 times to date. We explore how positivism has affected this systematic review across three areas:
1. How it views research.
2. How it views the researcher.
3. How it views the important questions of EMI.
4.1 View of research
First, while it includes qualitative research, this systematic review takes a positivist view of research. For example, Macaro et al. (Reference Macaro, Curle, Pun, An and Dearden2018) write:
From our total of 83 HE studies that we have included in the systematic map, only seven have in some way measured the impact, via objective language tests, that EMI programmes have had on English language learning or English language proficiency. (p. 29)
There is a dearth of research, using objective tests rather than self-report, on the impact of EMI on improving students’ English proficiency. (p. 39)
The preponderance of research based on case studies of single institutions further exacerbates the problem of not being able to ascertain the impact of EMI on either English proficiency or on content learning. The almost total absence of any comparative studies among institutions and/or among countries (except for the issue of EMI growth) means that the rigour offered by comparative education methodology is largely absent. (p. 39)
Overall, these quotations evince:
1. A belief that objectivity is possible.
2. A preference for objectivity, measurement, and large-scale studies comparing between groups.
3. A dispreference for subjective measurements that are viewed as not ‘empirical’.
This seems broadly in line with the description of positivism above. This does not invalidate this systematic review, nor does it make it objective. However, when citing it, we – as critical readers –must remain cognisant that the way this systematic review was conducted is shaped by a set of beliefs about research. This is particularly true, as such beliefs determine how the authors of a systematic review evaluate what they are reviewing.
4.2 View of the researcher
Second, Macaro et al. (Reference Macaro, Curle, Pun, An and Dearden2018; cf. Dearden, Reference Dearden2014) take a positivist stance on the role of researchers. This can be seen in the questions selected for the systematic review:
1. How is EMI in higher education labelled and defined (if at all) in studies purporting to be researching it as a phenomenon?
2. What are the growth and development patterns of EMI across the world?
3. What are the beliefs of EMI teachers towards EMI?
4. What are the beliefs of EMI students towards EMI, including their motivation for enrolling on EMI programmes?
5. What is the evidence that English proficiency increases as a result of EMI?
6. What is the evidence for the effect on academic subjects of being taught through EMI?
7. What research on HE classroom discourse has been carried out that might shed additional light on the above questions? (p. 39)
These seven questions, we argue, place the researcher as an observer seeking to define and then describe EMI. The ultimate goal is to produce a cost-benefit analysis, which can then be passed along to policymakers (Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2018, p. 39, 68). This is strikingly close to Guba and Lincoln's (Reference Guba, Lincoln, Denzin and Lincoln1994) description of the positivist view of researchers as being ‘“disinterested scientist[s]” who are informers of decision makers, policy makers and change agents’ (p. 112). However, EMI research can and does break this mould. One of many examples is Yuan's (Reference Yuan2020) description of a program for EMI teachers that, among other things, attempts to promote a decolonised version of English (see also Cao & Yuan, Reference Cao and Yuan2020).
This is important not only because of how the questions selected influence the systematic review, but also because of how frequently EMI is presented as inevitable in the literature. For instance, EMI has been famously described as a ‘“galloping” phenomenon’ by Chapple (Reference Chapple2015, p. 1) and as an ‘unstoppable train’ that ‘has already left the station’ by Macaro et al. (Reference Macaro, Hultgren, Kirkpatrick and Lasagabaster2019, p. 232). However, such a portrayal is not universally accepted. In fact, we would argue that such portrayals play into neoliberal discourse that suggest neoliberalism is the only game in town (Block et al. Reference Block, Gray, Holborow, Block, Gray and Holborow2012; Holborow Reference Holborow, Block, Gray and Holborow2012). Researchers should avoid describing EMI as inevitable, so as to not become complicit in this neoliberal agenda. As English language teacher–researcher Troudi (Reference Troudi and Wachob2009) writes:
Our role as educators and language teachers is not only to serve a curriculum but also to evaluate it, challenge it, play an active role, and even redesign it. We need to be fully aware of the underlying messages of governmental language policies. EFL professionals in the Arab world, consciously and subconsciously, play a dangerous role when they glorify the use of English. (pp. 212–213)
In the end, while a positivist view of EMI researchers is widely held, it is far from the only view. Therefore, we argue that researchers should avoid reading this, or indeed any, systematic review and thinking, ‘this is a scientific and definitive view of how researchers do and/or should position themselves vis-à-vis this field’.
4.3 View of the questions of EMI
Finally, and perhaps most importantly, there is an issue around which questions are deemed to be important in EMI. For example, sociolinguistic issues play a minor role in Macaro et al. (Reference Macaro, Curle, Pun, An and Dearden2018). When these are presented, it is as a factor in motivation and, in the conclusion, dismissed:
One thing is clear: policy makers and particularly university managers are not going to be swayed by sociolinguistic and sociocultural objections to the implementation of EMI as proclaimed in books on the subject. In view of that, it is hard to see anything but further expansion of EMI in HE. However, such leaders may be influenced by hard research evidence of the kind we have wished for in the above discussion and building on the findings of the current state of research. (Macaro et al., Reference Macaro, Curle, Pun, An and Dearden2018, p. 69)
However, many of the sociolinguistic issues raised by EMI studies go beyond the theoretical and have real-world impacts. Some we find particularly pressing are:
• To what extent is EMI about an investment in social capital (Zhang et al., Reference Zhang, Mittelmeier, Lomer and Lim2022)?
• What impact do EMI programs have on the language ideology of students (Hu & Lei, Reference Hu and Lei2014; Sung, Reference Sung2020)?
• Is access to EMI programs unfairly distributed? What impact does that have (Han & Dong, Reference Han and Dong2023)?
• Are socioeconomic inequalities replicated in the results of EMI programs (Muttaqin et al., Reference Muttaqin, Chuang, Lin and Cheng2022)?
• What is the relationship between globalisation, linguistic imperialism and EMI (R'boul, Reference R'boul2022)?
• What impact is EMI having on other languages (Troudi, Reference Troudi and Wachob2009)?
• What knock on social effect is tertiary EMI having on secondary education (Sah & Li, Reference Sah and Li2018)?
• How accurate are Galloway et al. (Reference Galloway, Kriukow and Numajiri2017) when they write ‘the global spread of EMI is perpetuating the stereotype that having a Western-style education is superior and something that is necessary for a successful future’ (p. 8)?
• Do differences in the quality of EMI provision ‘impose a social order on students that legitimizes the social differences and inequalities by restricting access to resources for some while providing it to others’ (Haidar, Reference Haidar2019, p. 845)?
We acknowledge that some of the studies above, particularly those that have moved beyond implications for language teaching, postdate Macaro et al.'s (2018) systematic review. However, the studies above still highlight that sociological impacts can potentially have a significant and material impact on people's lives. To illustrate, it is possible to imagine a situation where we know that EMI does not harm subject knowledge acquisition and is good for students' English, but where EMI is considered a net negative because it perpetuates sociolinguistic harms. Given this, even if we are only interested in material impacts, sociolinguistics seems to be an apt subject for EMI scholars. While positivism can answer certain types of questions well, there is a risk that a preference for what is quantifiable combined with the difficulty of quantifying sociolinguistic issues could lead them to be ignored by future systematic reviews. Given the sheer number of papers being published in the field of EMI, it is likely that secondary literature will increasingly be relied upon both for learning about EMI and for agenda-setting for EMI studies.
5. Conclusion
In conclusion, this article has had two goals. In the first half of this article, we have challenged the perception of systematic review as a methodology that represents a particularly scientific way of doing research. We have emphasised the impact a paradigm, and especially a positivist paradigm, can have on how a systematic review is done. In the second half of this article, we have explored how positivism may have influenced one particularly well-cited systematic review and our reading and usage of it. One limitation of this is that this study has focused on one study. As such, we recommend further research into other systematic reviews in applied linguistics and more broadly.
All methods have limitations, and all research is conducted with a set of assumptions. However, as Norris and Ortega (Reference Norris and Ortega2007) write, there is a danger that systematic reviews can be viewed as providing the ‘final answers’ (p. 811) on a topic. Although Macaro et al. (Reference Macaro, Curle, Pun, An and Dearden2018), from a positivist standpoint, have conducted a high-quality and expansive systematic review, it is potentially troubling that they may be cited uncritically – or at least without explicit criticality. This is particularly concerning when the study is used as a source of research agenda or methodological advice (as in the case of, inter alia, Abouzeid, Reference Abouzeid2021; Chung & Lo, Reference Chung and Lo2023; Genc & Yuksel, Reference Genc and Yuksel2021; Zhang & Pladevall-Ballester, Reference Zhang and Pladevall-Ballester2021).
For future systematic reviews of EMI, we recommend that researchers take a holistic view of what EMI research is in terms of methodology, researcher identity, and which questions are important. Furthermore, as Petticrew and Roberts (Reference Petticrew and Roberts2006) note, ‘systematic literature reviews usually aim to answer a specific question or test a specific hypothesis, rather than simply summarizing “all there is to know” about a particular issue’ (p. 10). Given the diffuse nature of EMI research, it is important to continue to ask, ‘What does it mean to study EMI?’ However, it is likely to also be important to conduct more focused systematic reviews into particular questions around EMI.
In terms of systematic review more broadly, we have argued that the bifurcation of quantitative and qualitative systematic review is unhelpful. Rather, systematic reviewers need to be able to include studies of a range of backgrounds while finding holistic ways of evaluating those studies. As Norris and Ortega (Reference Norris and Ortega2007) write:
Arguably the most important future development for research synthesis in our field is the formidable challenge of forging a new generation of capable research synthesists who are willing to integrate epistemologically diverse bodies of research, including quantitative, qualitative, and mixed-methods studies of many kinds, all bearing on a common research question or problem of interest to applied linguistics research communities. (p. 812)
As such, we echo Gough and Thomas’ (Reference Gough and Thomas2016) call for further ‘serious discussion’ (p. 84) of how this might be achieved in applied linguistics as well as systematic reviews in EMI and language teaching, which attempt to forward the craft of systematic review.
Acknowledgements
The authors would like to thank Graeme Porte and the anonymous reviewers for their dilligent and insightful feedback on this article. We would also like to thank Ernesto Macaro for his comments on a presentation that have been very helpful in shaping this work.
Timothy Hampson is a postgraduate researcher at University College London. He completed an M.Sc. in Applied Linguistics for Language Teaching at Oxford University, where he was given a Dissertation of the Year award. He has worked as an English language teacher in Saudi Arabia, China, and South Korea, focusing on creating learner-centred classrooms. He is a reviewer for System, TESOL Quarterly, and AILA Review. Tim's academic interests are centred on grounded theory, mixed methods research, systematic review, and critical realism. He is particularly interested in using these methods to ensure research has a practical benefit and does not gloss over sociological concerns.
Jim McKinley, SFHEA, is Professor of Applied Linguistics at University College London, IOE Faculty of Education and Society, and Honorary Research Fellow at the University of Oxford, Department of Education. He has taught in higher education in the UK, Japan, Australia, and Uganda, as well as US schools. His research targets implications of globalisation for L2 writing, language education, and higher education studies, particularly the teaching-research nexus and English medium instruction. Jim is co-author and co-editor of several books on research methods in applied linguistics. He is an Editor-in-Chief of the journal System, and a co-Editor of the Cambridge Elements in Language Teaching series (Cambridge University Press).
Open access
