Hostname: page-component-f554764f5-c4bhq Total loading time: 0 Render date: 2025-04-21T06:18:25.073Z Has data issue: false hasContentIssue false
  • English
  • Français

Meta-analysis of second language research with complex research designs

Published online by Cambridge University Press:  19 July 2023

Reza Norouzian Open the ORCID record for Reza Norouzian [Opens in a new window]  and
Gavin Bui
Reza Norouzian*
Affiliation:
Oregon Department of Education, Salem, OR, USA
Gavin Bui
Affiliation:
Department of English, The Hang Seng University of Hong Kong, China
*
Corresponding author: Reza Norouzian; Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Meta-analyses play an instrumental role in informing second language (L2) theory and practice. However, current (i.e., classic) approaches to meta-analysis are limited in their ability to do so because they often fail to capture the complexity inherent in primary studies’ research designs. As we argue in this article, when complex L2 studies are represented by simplistic meta-analyses, the latter cannot reach its potential to contribute to the development of cumulative knowledge. To mitigate this issue, we first discuss the fundamental problems of the classic approaches to meta-analysis of complex L2 research. Second, we introduce an alternative meta-analytic framework that will address those problems. Third, we apply the meta-analytic framework discussed to a complex L2 domain. Fourth, we offer free software to facilitate the conduct of the alternative meta-analytic approach described. Finally, we discuss the implications of this alternative framework for making evidence-based recommendations to the relevant stakeholders.

Type
Methods Forum
Information
Studies in Second Language Acquisition , Volume 46 , Issue 1 , March 2024 , pp. 251 - 276
Check for updates
Open Practices
Open data Open materials
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Aloe, A. M., Thompson, C. G., & Reed, D. K. (2020). CUTOS: A framework for contextualizing evidence. In Albers, B., Shlonsky, A., & Mildon, R. (Eds.), Implementation Science 3.0 (pp. 39–52). Springer.CrossRefGoogle Scholar
Becker, B. J. (1996). The generalizability of empirical research results. In Benbow, C. P. & Lubinski, D. (Eds.), Intellectual talent: Psychometric and social issues (pp. 362–383). Johns Hopkins University Press.Google Scholar
Becker, B. J. (2017). Improving the design and use of meta-analyses of career interventions. In J. P. Sampson, E. Bullock-Yowell, Dozier, V. C., Osborn, D. S., & Lenz, J. G., Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 95–107). Florida State University.Google Scholar
Becker, B. J., & Aloe, A. M. (2008, March 24–28). Modeling heterogeneity in meta-analysis: Generalizing using Cronbach’s (M)UTOS framework and meta-analytic data [Paper presentation]. Annual meeting of the American Educational Research Association, New York, New York, USA.Google Scholar
Berlin, J. A., & Antman, E. M. (1992, May 10–13). Advantages and limitations of meta-analytic regressions of clinical trials data [Paper presentation]. Thirteenth annual meeting of the Society for Clinical Trials. Adam’s Mark, Philadelphia, PA, USA. https://doi.org/10.1016/0197-2456(92)90151-OCrossRefGoogle Scholar
Brown, D., Liu, Q., & Norouzian, R. (2023). Effectiveness of written corrective feedback in developing L2 accuracy: A Bayesian meta-analysis. Language Teaching Research [Online]. https://doi.org/10.1177/13621688221147374CrossRefGoogle Scholar
Byrnes, H. (2013). Notes from the editor. The Modern Language Journal, 97, 825–827. https://doi.org/10.1111/j.1540-4781.2013.12051.xCrossRefGoogle Scholar
Cheung, A. C., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45, 283–292.CrossRefGoogle Scholar
Chong, S. W., & Plonsky, L. (2023). A typology of secondary research in applied linguistics. Applied Linguistic Review. https://doi.org/10.1515/applirev-2022-0189CrossRefGoogle Scholar
Cook, T. D. (1991). Meta-analysis: Its potential for causal description and causal explanation within program evaluation. In Albrecht, G. & Otto, H. U. (Eds.), Social prevention and the social sciences (pp. 245–285). de Gruyter.Google Scholar
Cook, T. D. (1993). A quasi-sampling theory of the generalization of causal relationships. New Directions for Program Evaluation, 1993, 39–82. https://doi.org/10.1002/ev.1638CrossRefGoogle Scholar
Cook, T. D. (2013). Causal generalization: How Campbell and Cronbach influenced my theoretical thinking on this topic, including in Shadish, Cook, and Campbell. In Alkin, M. C. (Ed.), Evaluation roots: Tracing theorists’ views and influences (2nd ed., pp. 85–113). Sage.Google Scholar
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. Jossey-Bass.Google Scholar
Cunnings, I. (2012). An overview of mixed-effects statistical models for second language researchers. Second Language Research, 28, 369–382. https://doi.org/10.1177/0267658312443651CrossRefGoogle Scholar
Ellis, R. (2008). The study of second language acquisition (2nd ed.). Oxford University Press.Google Scholar
Fernández-Castilla, B., Maes, M., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2019). A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis. Behavior Research Methods, 51, 1286–1304. https://doi.org/10.3758/s13428-018-1063-2CrossRefGoogle ScholarPubMed
Fisher, Z., & Tipton, E. (2015). Robumeta: An R-package for robust variance estimation in meta-analysis. ArXiv. https://arxiv.org/abs/1503.02220Google Scholar
Fox, J., & Weisberg, S. (2018). Visualizing fit and lack of fit in complex regression models with predictor effect plots and partial residuals. Journal of Statistical Software, 87, 1–27.CrossRefGoogle Scholar
Gass, S., Loewen, S., & Plonsky, L. (2021). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching, 54, 245–258.CrossRefGoogle Scholar
Gries, S. T. (2021). (Generalized linear) Mixed-effects modeling: A learner corpus example. Language Learning, 71(3), 757–798.CrossRefGoogle Scholar
Hedges, L. V. (2019). Stochastically dependent effect sizes. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis (3rd ed., pp. 281–297). Russell Sage Foundation.CrossRefGoogle Scholar
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Sage Publications.Google ScholarPubMed
Mathur, M. B., & VanderWeele, T. J. (2019). New metrics for meta-analyses of heterogeneous effects. Statistics in Medicine, 38, 1336–1342.CrossRefGoogle ScholarPubMed
Matt, G. E., & Cook, T. D. (2019). Threats to the validity of generalized inferences from research syntheses. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis (3rd ed., pp. 489–516). Russell Sage Foundation.CrossRefGoogle Scholar
Norouzian, R. (2020). Sample size planning in quantitative L2 research: A pragmatic approach. Studies in Second Language Acquisition, 42, 849–870.CrossRefGoogle Scholar
Norouzian, R. (2021). Interrater reliability in second language meta-analyses: The case of categorical moderators. Studies in Second Language Acquisition, 43, 896–915.CrossRefGoogle Scholar
Norouzian, R., & Plonsky, L. (2018). Eta- and partial eta-squared in L2 research: A cautionary review and guide to more appropriate usage. Second Language Research, 34, 257–271.CrossRefGoogle Scholar
Norouzian, R., de Miranda, M., & Plonsky, L. (2018). The Bayesian revolution in second language research: An applied approach. Language Learning, 68, 1032–1075.CrossRefGoogle Scholar
Norouzian, R., De Miranda, M., & Plonsky, L. (2019). A Bayesian approach to measuring evidence in L2 research: An empirical investigation. Modern Language Journal, 103, 248–261.CrossRefGoogle Scholar
Olkin, I., & Gleser, L. (2009). Stochastically dependent effect sizes. In Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). Russell Sage Foundation.Google Scholar
Plonsky, L., & Ghanbar, H. (2018). Multiple regression in L2 research: A methodological synthesis and guide to interpreting R2 values. The Modern Language Journal, 102, 713–731. https://doi.org/10.1111/modl.12509CrossRefGoogle Scholar
Plonsky, L., Hu, Y., Sudina, E., & Oswald, F. L. (2023). Advancing meta-analytic methods in L2 research. In Mackey, A. & Gass, S. (Eds.), Current approaches in second language acquisition research (pp. 304–333). Wiley Blackwell.CrossRefGoogle Scholar
Plonsky, L., & Oswald, F. L. (2017). Multiple regression as a flexible alternative to ANOVA in L2 research. Studies in Second Language Acquisition, 39, 579–592. https://doi.org/10.1017/S0272263116000231CrossRefGoogle Scholar
Plonsky, L., Sudina, E., & Hu, Y. (2021). Applying meta-analysis to research on bilingualism: An introduction. Bilingualism: Language and Cognition, 24, 819–824. https://doi.org/10.1017/S1366728920000760CrossRefGoogle Scholar
Pustejovsky, J. E. (2022). ClubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections (R package version 0.5.5) [Computer software]. https://CRAN.R-project.org/package=clubSandwichGoogle Scholar
Pustejovsky, J. E., & Tipton, E. (2022). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science, 23(3), 425–438. https://doi.org/10.1007/s11121-021-01246-3CrossRefGoogle ScholarPubMed
R Core Team. (2022). R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/Google Scholar
Raeisi-Vanani, A., Plonsky, L., Wang, W., Lee, K., & Peng, P. (2022). Applying meta-analytic structural equation modeling to second language research: An introduction. Research Methods in Applied Linguistics, 1(3), 100018.CrossRefGoogle Scholar
Raudenbush, S. W., & Bryk, A. S. (1985). Empirical bayes meta-analysis. Journal of Educational Statistics, 10, 75–98. https://doi.org/10.3102/10769986010002075CrossRefGoogle Scholar
Rodgers, M. A., & Pustejovsky, J. E. (2021). Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26, 141–160. https://doi.org/10.1037/met0000300CrossRefGoogle Scholar
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company.Google Scholar
Shintani, N., & Ellis, R. (2013). The comparative effect of direct written corrective feedback and metalinguistic explanation on learners’ explicit and implicit knowledge of the English indefinite article. Journal of Second Language Writing, 22, 286–306.CrossRefGoogle Scholar
Stevens, J. R., & Taylor, A. M. (2009). Hierarchical dependence in meta-analysis. Journal of Educational and Behavioral Statistics, 34, 46–73. https://doi.org/10.3102/1076998607309080CrossRefGoogle Scholar
Tipton, E., Pustejovsky, J. E., & Ahmadi, H. (2019). A history of meta-regression: Technical, conceptual, and practical developments between 1974 and 2018. Research Synthesis Methods, 10, 161–179. https://doi.org/10.1002/jrsm.1338CrossRefGoogle ScholarPubMed
Trikalinos, T. A., & Olkin, I. (2012). Meta-analysis of effect sizes reported at multiple time points: A multivariate approach. Clinical Trials, 9, 610–620. https://doi.org/10.1177/1740774512453218CrossRefGoogle Scholar
Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning, 46, 327–369.CrossRefGoogle Scholar
Truscott, J. (2004). Evidence and conjecture on the effects of correction: A response to Chandler. Journal of Second Language Writing, 13, 337–343.CrossRefGoogle Scholar
Van Beuningen, C. G., De Jong, N. H., & Kuiken, F. (2012). Evidence on the effectiveness of comprehensive error correction in second language writing. Language Learning, 62, 1–41.CrossRefGoogle Scholar
Viechtbauer, W. (2007). Confidence intervals for the amount of heterogeneity in meta‐analysis. Statistics in Medicine, 26, 37–52. DOI:10.1002/sim.2514CrossRefGoogle ScholarPubMed
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48.CrossRefGoogle Scholar
Viechtbauer, W. (2021). Model checking in meta-analysis. In Schmid, C. H., Stijnen, T., & White, I. R. (Eds.), Handbook of meta-analysis (pp. 219–254). CRC Press.Google Scholar