2.1 Search strategy

Following the guidelines of the PRISMA framework to carry out a qualitative systematic review study, a rigorous search strategy was adopted. The key terms “chatbot,” “conversational agent,” “conversational system,” “dialogue-based,” “conversational dialog agent,” “human-computer dialog,” “language learning,” “English learning,” “foreign language,” “second language,” “L2,” “EFL (English as a Foreign Language),” “ESL (English as a Second Language),” “language acquisition,” “CALL,” and “MALL” were used with a mixture of Boolean expressions “AND” and “OR.” While in some studies researchers created web-based AI chatbots (Ayedoun, Hayashi & Seta, Reference Ayedoun, Hayashi and Seta2020; El Shazly, Reference El Shazly2021), others developed app-integrated AI chatbots (Belda-Medina & Calvo-Ferrer, Reference Belda-Medina and Calvo-Ferrer2022). Therefore, since researchers have used a variety of concepts to describe the utilization of AI chatbots depending on their digital format, we decided to include both “CALL” and “MALL” as keywords in our article search of databases. All keywords were searched in journal names, abstracts, and full articles in databases ranging from Web of Science, ERIC, Scopus, Elsevier, EBSCOhost, and Springer to access articles, book chapters, and conference proceedings to be reviewed matching our criteria for selection. In addition to the articles retrieved from these databases, articles found manually in the reference sections of the articles were also included. During the writing and revision process of the paper presented here, some of the initially reviewed studies (Chen, Yang & Lai, Reference Chen, Yang and Lai2023; M.-H. Hsu, Chen & Yu, Reference Hsu, Chen and Yu2023; Jeon, Reference Jeon2024) that were published with a DOI number and with early access date were published in the journals with an issue number at a later date. The present study’s authors updated the volume, issue numbers, and publishing dates. The selected studies for review have been shown with an asterisk in the reference section. The whole list of selected studies for review is given in the supplementary materials.

2.2 Selection criteria for the studies

Conceptual, theoretical papers, articles using languages other than English, studies outside of the English language teaching field, and studies that do not report the experiences of chatbot users were eliminated. The works on voice-based AI chatbots and English language learning were the main focus of the search. Under AI chatbots, we also included studies using intelligent personal assistants (IPAs) such as Google Assistant and Alexa because IPAs are also branched under closed-domain retrieval-based AI chatbots (H. Kim, Yang, Shin & Lee, Reference Kim, Yang, Shin and Lee2022). Although this meta-synthesis study is based on the English language learning experiences of chatbot users, feasibility studies using state-of-art technologies in creating this much-needed technology have also been included, as one dimension in the study was to explore these tools’ technical qualities.

This study’s focus on the time frame from 2010 to 2024 derives from our intention to concentrate on studies utilizing voice recognition chatbots with the latest AI, automatic speech recognition (ASR), and natural language processing technologies. The literature review showed that although the development of voice-based AI chatbots dates back to the 1980s, the research on voice-based chatbots in students’ English language learning experience did not surge until after 2016, as we could trace back to only one study published each year during 2010 and 2012. Table 1 demonstrates the number of papers reviewed and analyzed by journal name and year of publication (see supplementary materials).

2.3 Data extraction and analysis of the studies selected

Using PRISMA guidelines, an article search was carried out, and 57 articles were selected based on our selection criteria. Figure 1 outlines the selection of studies following PRISMA guidelines (Moher et al., Reference Moher, Liberati, Tetzlaff and Altman2009). The constructivist grounded theory (CGT) by Thornberg and Charmaz (Reference Thornberg and Charmaz2014) was used to analyze the findings and research trends in the studies. The main results of the AI chatbot studies were initially extracted in the form of raw data. The data were first read line by line in the initial coding phase and coded through a constant comparison method to examine data in contrast to other data, analyze data in contrast to codes, and compare code with code to identify any arising discrepancies or congruities. Subsequently, the major findings in the reviewed studies were specified and synthesized through a qualitative coding process while interpreting the emergent codes in the focused coding stage.

Figure 1. PRISMA flow chart (Moher et al., Reference Moher, Liberati, Tetzlaff and Altman2009).

Considering every article as an analytic component, we comprehensively and iteratively analyzed all the articles. First, we read the literature on reviews of chatbots in general (e.g. Pérez, Daradoumis & Puig, Reference Pérez, Daradoumis and Puig2020) to create a scheme for a deductive coding scheme. We constructed an initial coding framework of the methodology, settings, and participants based on the previous reviews. Through data-driven inductive coding, four further categories were constructed: theoretical frameworks, strengths and challenges, technologies, and pedagogical implications. These dimensions were deemed particularly important to this study, as examining these factors can provide insights into the technical-pedagogical aspects of voice-based AI chatbots used in past studies. The findings can be employed to understand the implications of using such chatbots with particular student profiles in various educational settings.

This study followed the CGT in the data analysis, as this theory allowed the enactment of an interpretative research hypothesis with a cyclical data analysis process through constant data comparison. This process enabled us to co-construct meaning during the analysis phase instead of validating a priori hypothesis. The selected studies for the present study were systematically synthesized through a meta-synthesis. The rationale behind using a meta-synthesis approach was that we aimed to reach a novel and unifying understanding of findings that was more substantial and profound than a single study and to get a broader understanding of the use of voice-based AI chatbots in English language learning (Finfgeld, Reference Finfgeld2003; Saini & Shlonsky, Reference Saini and Shlonsky2012). These unified findings were illustrated as an analysis table in Table 2 (see supplementary materials). After reaching a consensus on the data extraction form by two researchers, one researcher extracted related information from the reviewed studies. The validity of this process was ensured via data extraction of the reviewed studies by one other researcher.

3.1 Theoretical backgrounds in the reviewed studies

The first category is theoretical backgrounds, which comprise the theories employed in chatbot-assisted language learning research (see Table 3 in supplementary materials). Namely, theoretical frameworks point to theoretical foundations of AI chatbot research targeting English language learners at K–12 and higher education levels, which comprise linguistic, cognitive, social, and technological-pedagogical theories. The linguistic category is related to language learning theories (i.e. the input hypothesis). The cognitive category comprises cognitive learning theories like the interest development model. The social category refers to social learning theories such as the constructivist theory. At the same time, other theories explaining general technological frameworks applied to studies, such as community of inquiry (CoI), were categorized as technological-pedagogical.

As displayed in Figure 2, linguistic frameworks were the most frequently adopted theoretical backgrounds (n = 21), followed by technological-pedagogical (n = 11), cognitive (n = 4), and social (n = 2) theories. More particularly, the interaction hypothesis (n = 8), willingness to communicate (WTC) (n = 4), autonomous second language learning (n = 3), foreign language anxiety (FLA) (n = 3), and skill acquisition theory (n = 3) were most frequently used for grounding theories in studies.

Figure 2. Theoretical backgrounds.

Note. TAM = technology acceptance model; CHISM = chatbot–human interaction satisfaction model; WTC = willingness to communicate.

Based on Long’s (Reference Long, Ritchie and Bhatia1996) interaction hypothesis, studies sought to discover if chatbots would increase interaction in L2 (i.e. referring to a second language) by offering favorable circumstances for the negotiation of meaning, interaction, input, and output (Dizon, Reference Dizon2017, Reference Dizon2020). The studies reviewed were also based on the claim that the use of technology might help increase the WTC levels of learners – that is “a readiness to enter into discourse at a particular time with a specific person or persons, using an L2” (MacIntyre, Clément, Dörnyei & Noels, Reference MacIntyre, Clément, Dörnyei and Noels1998: 547), and it has been found that learners demonstrated a higher expected WTC after talking to a chatbot (Ayedoun, Hayashi & Seta, Reference Ayedoun, Hayashi and Seta2019a; Ayedoun et al., Reference Ayedoun, Hayashi and Seta2020; Tai & Chen, Reference Tai and Chen2023). Other studies also based their arguments on Gardner and Miller’s (Reference Gardner and Miller1999) self-access language learning theoretical framework, claiming that learners’ autonomy and independence can be promoted through the use of chatbots as supplementary language learning materials (Dizon & Tang, Reference Dizon and Tang2020; Moussalli & Cardoso, Reference Moussalli and Cardoso2020).

Another salient theory used by researchers was the interest development model (Hidi & Renninger, Reference Hidi and Renninger2006). Following this theory, researchers investigated whether AI chatbots could trigger students’ interest in language learning (Fryer, Ainley, Thompson, Gibson & Sherlock, Reference Fryer, Ainley, Thompson, Gibson and Sherlock2017; Fryer, Nakao & Thompson, Reference Fryer, Nakao and Thompson2019; Fryer, Thompson, Nakao, Howarth & Gallacher, Reference Fryer, Thompson, Nakao, Howarth and Gallacher2020). Basing their studies on the self-determination theory, researchers also explored if chatbots would facilitate autonomy, competence, and relatedness feelings of language learners (Annamalai et al., Reference Annamalai, Eltahir, Zyoud, Soundrarajan, Zakarneh and Al Salhi2023; Jeon, Reference Jeon2022; Morton & Jack, Reference Morton and Jack2010; Ryan & Deci, Reference Ryan and Deci2017). Moreover, based on the theory of FLA, researchers have examined the impact of using chatbots on learning anxiety (Dizon, Reference Dizon2017; Horwitz, Horwitz & Cope, Reference Horwitz, Horwitz and Cope1986; Tai & Chen, Reference Tai and Chen2023) and the meditation and relaxation levels of language learners (L. Hsu, Reference Hsu2022).

The investigation of chatbot usage has also been carried out under the principles of the skill acquisition theory (Lyster & Sato, Reference Lyster, Sato, García Mayo, Gutierrez Mangado and Martínez Adrián2013), revealing that chatbots would be constructive tools for language learners thanks to their potential to provide structured oral practice (Sydorenko, Smits, Evanini & Ramanarayanan, Reference Sydorenko, Smits, Evanini and Ramanarayanan2019). Studies also sought to discover the potential of AI-powered chatbots for fostering oral interaction by raising awareness of specific L2 pragmatic structures based on Schmidt’s (Reference Schmidt and Schmidt1995) noticing hypothesis (Timpe-Laughlin & Dombi, Reference Timpe-Laughlin and Dombi2020). Moreover, Ericsson, Sofkova Hashemi and Lundin (Reference Ericsson, Sofkova Hashemi and Lundin2023) also grounded their studies on the sociocultural theory, arguing that interaction with AI chatbots might provide instances for social interaction through human–computer dialogue (Lantolf, Thorne & Poehner, Reference Lantolf, Thorne, Poehner, VanPatten and Williams2014). Based on the constructivist theory in technological learning settings, Park (Reference Park2022) explored and identified the role of a chatbot in promoting student-centered and scaffolded learning as a learning tool for geographically remote learners (Jonassen, Davidson, Collins, Campbell & Haag, Reference Jonassen, Davidson, Collins, Campbell and Haag1995).

Besides the linguistic and social theories, learners’ perspectives in using chatbots for language learning concerning factors such as user engagement and linguistic and technological components were explored through the technology acceptance model (TAM) and the chatbot–human interaction satisfaction model (CHISM) (Belda-Medina & Calvo-Ferrer, Reference Belda-Medina and Calvo-Ferrer2022; Davis, Reference Davis1989). A similar theory, the theory of planned behavior by Fishbein and Ajzen (Reference Fishbein and Ajzen1975), was adopted by Wu et al. (Reference Wu, Lam, Kong and Wong2023) to analyze the relationship between the factors of the behavioral intention of using chatbot applications and learner beliefs, such as the usefulness of voice-based AI chatbots for learning.

The computers as social actors (CASA) theory by Nass, Moon, Fogg, Reeves and Dryer (Reference Nass, Moon, Fogg, Reeves and Dryer1995) has also been adopted to explore how AI chatbots affect language learners’ learning motivation. Researchers aimed to observe whether AI chatbots would create a feeling of social presence among learners in cases where they are described as having human-like properties (Ebadi & Amini, Reference Ebadi and Amini2024). Epley, Waytz and Cacioppo’s (Reference Epley, Waytz and Cacioppo2007) three-factor theory of anthropomorphism, consisting of “elicited agent knowledge, effectance, and sociality” factors, referring to knowledge about agents, motivation to interact with them, and intention to communicate with agents, respectively, was also used to analyze if AI chatbots were described with human-like characteristics (S. Lee & Jeon, Reference Lee and Jeon2022: 3). Adopting the affordance theory by Gibson (Reference Gibson1986), researchers also analyzed the experiences of language learners in using voice-based AI chatbots in terms of the technological, pedagogical, and social contributions to language learning (Jeon, Reference Jeon2024). Furthermore, the CoI framework by Garrison and Arbaugh (Reference Garrison and Arbaugh2007) has also been used in studies to observe the effect of the social, cognitive, and teaching presence on social dynamics, learning outcomes, and learning performance of language learners after interacting with voice-based AI chatbots (Wang et al., Reference Wang, Liu, Pang, Tan, Lei, Wallace and Li2023; Wang, Pang, Wallace, Wang & Chen, Reference Wang, Pang, Wallace, Wang and Chen2022). However, 19 studies were not based on any theoretical groundings. Therefore, a clear-cut theoretical framework still needed to be included in many studies.

3.2 Methodological trends in the reviewed studies

The second category is the methodologies, consisting of four variables: design features, countries, education level, and sample size. Regarding the methodology, there were 4,062 participants in these 57 studies. In 23 of the studies using an experimental methodology, data were collected from 1,408 participants. Two studies used sequential explanatory mixed methods with 432 participants. Ten feasibility studies with 246 participants were also conducted, and the five case studies comprised 57 participants. However, the remaining studies did not specify their methodologies explicitly, and they had 1,674 participants. Most participants were undergraduate and graduate students (49%, n = 1,984), while high school students constituted 23% of the population (n = 947), and primary school students constituted 23% of the population (n = 951). The remaining studies did not specify the education level of the students.

The meta-synthesis study presented here included studies conducted in various countries. However, most studies, namely 13 out of 57, have been conducted in Japan. The sample consisted of a total of 986 participants from Korea, followed by Japan (n = 604), China (n = 528), and Taiwan (n = 447), respectively. Figure 3 below displays the distribution of the participants in the reviewed studies by countries.

Figure 3. Research regions.

Moreover, the publication dates of the reviewed articles were evenly distributed over six years, with an upward trend in the number of publications beginning in 2016. Figure 4 denotes the trend for voice-based AI chatbot research in English language teaching over the years.

Figure 4. The course of voice-based AI chatbot publications over the years.

3.3 The types of technologies and/or chatbots used in the studies analyzed

The next category of analysis is the types of voice-based AI technologies, which comprised three codes: general audience chatbots (i.e. Alexa, Siri, Cortana, etc.), specific-purpose chatbots created for language learners, and AI chatbots embedded in virtual learning environments. Twenty-seven types of chatbots were identified (see Figure 5).

Figure 5. The distribution of chatbots used by studies.

The results showed that the these chatbots had been adopted to support English language learning. The most common practice among the synthesized studies was employing already present open-access or commercial applications created for native speakers or L2 learners. Thus, two such chatbots, Alexa and Google Assistant, were used the most. Two studies among the reviewed studies also employed Cleverbot, an open-access chatbot (Fryer et al., Reference Fryer, Ainley, Thompson, Gibson and Sherlock2017, Reference Fryer, Nakao and Thompson2019), which is an updated version of Jabberwacky, a Loebner prize winner for producing the most similar output to human speech.

Another trend in the synthesized studies was the creation of unique AI chatbots by the authors and testing their effectiveness on language learners. However, these chatbots were created for academic purposes and were not publicly available. For instance, Ayedoun et al. (Reference Ayedoun, Hayashi and Seta2019a, Reference Ayedoun, Hayashi and Seta2019b, 2020) designed DiMaCA, a dialogue management model consisting of the dialogue flow management and the strategies management modules to create a chatbot, Peter, which used affective backchannels and communication strategies. Sydorenko et al. (Reference Sydorenko, Smits, Evanini and Ramanarayanan2019) also created a voice-based HALEF application targeting students of English as a second language.

The results also showed that chatbots aiming at lower language proficiency groups were rare in the studies reviewed. In one study, TextEvaluator was created targeting third- and fifth-grade English language learners in the USA (Forsyth et al., Reference Forsyth, Luce, Zapata-Rivera, Jackson, Evanini and So2019); another chatbot, SPELL, was created for junior high school students in China (Morton & Jack, Reference Morton and Jack2010), and one other chatbot was designed for elementary school learners in China (Wang et al., Reference Wang, Liu, Pang, Tan, Lei, Wallace and Li2023). Additionally, in a few studies, chatbots were embedded in virtual learning environments (Hassani, Nahvi & Ahmadi, Reference Hassani, Nahvi and Ahmadi2016; Morton, Gunson & Jack, Reference Morton, Gunson and Jack2012; Park, Reference Park2022), yet the integration of virtual reality technology into AI chatbots was relatively uncommon among the reviewed studies.

3.4 Strengths and challenges in the use of chatbots in the reviewed studies

Lastly, strengths and challenges were determined by analyzing the results and discussion parts of the reviewed articles. The strengths and challenges discovered in the reviewed studies are illustrated in Table 4 (see supplementary materials). The most commonly encountered strengths in studies were developing language skills (n = 27), enhancing affective factors (n = 27), facilitating communicative competence (n = 14), and increasing pragmatic awareness and competence (n = 10). It has been discovered in our synthesis that language learners can develop their linguistic skills to a great extent thanks to the interaction with voice-based AI chatbots during English language learning. Significant gains in speaking scores (Dizon, Reference Dizon2020; El Shazly, Reference El Shazly2021; D.-E. Han, Reference Han2020; M.-H. Hsu et al., Reference Hsu, Chen and Yu2023; Hwang et al., Reference Hwang, Guo, Hoang, Chang and Wu2022; Moussalli & Cardoso, Reference Moussalli, Cardoso, Zoghlami, Brudermann, Sarré, Grosbois, Bradley and Thouësny2021; Park, Reference Park2022; C. T.-Y. Yang, Lai & Chen, Reference Yang, Lai and Chen2022) and listening scores (N.-Y. Kim, Reference Kim2018; Tai & Chen, Reference Tai and Chen2022) were reported after engaging in negotiating meaning with chatbots.

The findings of the analyzed studies have also demonstrated positive aspects for affective factors in the use of chatbots by English language learners: higher attention and meditation levels, reduced anxiety, higher WTC, and motivation levels after interacting with voice-based AI chatbots (K.-A. Lee & Lim, Reference Lee and Lim2023; Tai & Chen, Reference Tai and Chen2023; C. T.-Y. Yang et al., Reference Yang, Lai and Chen2022). For instance, L. Hsu (Reference Hsu2022) carried out a neuroscientific experimental study using a chatbot they created and tested it on EFL learners to analyze their attention – the level of mental “focus” and meditation and calmness/relaxation states – by measuring their brainwave activities when they practice speaking in three different environments (face-to-face, virtual human to human, and human to an AI chatbot conditions) to explore the interlocutor effect. Meditation level was the highest in human chatbot conditions, which implied the beneficial effects of chatbots in reducing English language learning anxiety.

The synthesized studies also report the challenges of using chatbot systems. The low intelligibility of learner utterances due to problems with speech recognition (n = 15), the unnaturalness of AI chatbot–human interaction (n = 9), and a lack of explicit corrective feedback (n = 5) were the most commonly reported problems. These challenges were faced due to various issues. According to the studies analyzed, learners appeared less motivated to converse with chatbots than human partners due to novelty factors and unnaturalistic speaking conditions (Ericsson, Lundin & Sofkova Hashemi, Reference Ericsson, Lundin and Sofkova Hashemi2023; Fryer et al., Reference Fryer, Ainley, Thompson, Gibson and Sherlock2017). For instance, the results of a study by Belda-Medina and Calvo-Ferrer (Reference Belda-Medina and Calvo-Ferrer2022) showed only moderate interest in using AI chatbots for a more extended period. Furthermore, Çakmak (Reference Çakmak2022) and El Shazly (Reference El Shazly2021) identified increased levels of anxiety after interacting with the AI chatbots, which was attributed to factors such as the educational context that could ignite worries about failing the course as well as self-consciousness or language ego of students exposed to error correction, and lack of empathy and complex reasoning in chatbots.

3.5 Pedagogical implementations in the studies analyzed

The last category pertained to the pedagogical implementations, which are concerned with guiding and helping learners use AI chatbots to interact as conversational partners in learning languages through activities such as training and scaffolding and designing conversational scenarios that best fit this purpose. Chatbots have substantial potential for teaching languages, yet there is a need for more guidance on classroom implementations. Therefore, the researchers’ use of these tools in classroom settings is relevant to this study. Learners need guidance to utilize chatbots effectively for language learning purposes. An analysis of the reviewed studies on pedagogical practices in AI chatbot implementation has shown that two strategies guide AI chatbot users: (1) providing structured or semi-structured prompts, and (2) providing linguistic strategies.

First, regarding the prompts, researchers have discovered that several strategies were implemented to direct learners to utilize AI chatbots for English language learning. The analyzed studies displayed that teachers applied methods containing structured and semi-structured prompts for learners’ guidance during the conversations. Students were given topics, questions, and hints in question-and-reply stems (Ayedoun et al., Reference Ayedoun, Hayashi and Seta2019a, 2020; Fryer et al., Reference Fryer, Ainley, Thompson, Gibson and Sherlock2017, Reference Fryer, Nakao and Thompson2019, Reference Fryer, Thompson, Nakao, Howarth and Gallacher2020; L. Hsu, Reference Hsu2022; Johnson, Reference Johnson2019; Tai & Chen, Reference Tai and Chen2023). Furthermore, while interacting with chatbots, learners were given prompts of pre-established questions and language commands such as “Play me a song/joke/story” and instructed to ask chatbots like Google Home Hub and Alexa other general-knowledge-related facts such as the weather, temperature, and the capital of the countries. This way, students could learn to utter appropriate commands during conversations (Chen et al., Reference Chen, Yang and Lai2023; Dizon, Reference Dizon2017; Moussalli & Cardoso, Reference Moussalli and Cardoso2020).

The synthesized studies also reported that students were provided with linguistic strategies in the pre-task phase before using the chatbots to familiarize them with these tools. The strategies they were suggested to use to get into a dialogical conversation included reflecting on the misunderstood aspects of their speech, modifying their output by using more precise and slower speech, altering the pronunciation of words (Dizon, Reference Dizon2020), and asking for the definition, spelling, synonym, or the translation of the words to be able to notice the linguistic gaps in their knowledge in the face of a communication breakdown (Chen et al., Reference Chen, Yang and Lai2023; Dizon, Reference Dizon2017; Moussalli & Cardoso, Reference Moussalli and Cardoso2020).

Another theme regarding the pedagogy of AI chatbots in English language learning was the design methodology of pedagogical activities in the reviewed studies. Our review illustrated that researchers used the principles of Ellis’s (Reference Ellis2003) task-based language teaching using tasks in speaking scenarios for language practice in designing AI chatbots (Ayedoun et al., Reference Ayedoun, Hayashi and Seta2019a, 2020; Li, Chang & Wu, Reference Li, Chang and Wu2020; Sydorenko, Daurio & Thorne, Reference Sydorenko, Daurio and Thorne2018; Sydorenko et al., Reference Sydorenko, Smits, Evanini and Ramanarayanan2019). To illustrate, in a study using HALEF, the users practiced speaking English in a coffee shop simulation. The chatbot and the users interacted to complete an order (Sydorenko et al., Reference Sydorenko, Daurio and Thorne2018, Reference Sydorenko, Smits, Evanini and Ramanarayanan2019; Timpe-Laughlin & Dombi, Reference Timpe-Laughlin and Dombi2020). Similarly, students talked to an animated character in task-based simulation modules in Enskill English. The tasks included asking for directions and buying train tickets (Ericsson, Lundin & Sofkova Hashemi, Reference Ericsson, Lundin and Sofkova Hashemi2023; Ericsson, Sofkova Hashemi & Lundin, Reference Ericsson, Sofkova Hashemi and Lundin2023; Johnson, Reference Johnson2019). Thus, through partially scripted task-based or non-task-based dialogue activities in studies (Ayedoun et al., Reference Ayedoun, Hayashi and Seta2019a, Reference Ayedoun, Hayashi and Seta2020; Johnson, Reference Johnson2019; Sydorenko et al., Reference Sydorenko, Smits, Evanini and Ramanarayanan2019), students got engaged in two-way reciprocal communication with voice-based AI chatbots.