Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-gxg78 Total loading time: 0 Render date: 2024-12-23T12:38:02.847Z Has data issue: false hasContentIssue false

29 - Meta-Analysis

Integration of Empirical Findings through Quantitative Modeling

from Part VII - General Analytic Considerations

Published online by Cambridge University Press:  23 March 2020

By
Kristian E. Markon
Edited by
Aidan G. C. Wright  and
Michael N. Hallquist
Aidan G. C. Wright
Affiliation:
University of Pittsburgh
Michael N. Hallquist
Affiliation:
Pennsylvania State University
Get access

Summary

Meta-analysis is a well-established approach to integrating research findings, with a long history in the sciences and in psychology in particular. Its use in summarizing research findings has special significance given increasing concerns about scientific replicability, but it has other important uses as well, such as integrating information across studies to examine models that might otherwise be too difficult to study in a single sample. This chapter discusses different forms and purposes of meta-analyses, typical elements of meta-analyses, and basic statistical and analytic issues that arise, such as choice of meta-analytic model and different sources of variability and bias in estimates. The chapter closes with discussion of emerging issues in meta-analysis and directions for future research.

Keywords

random effectsfixed effectsestimation errorheterogeneitypublication bias
Type
Chapter
Information
The Cambridge Handbook of Research Methods in Clinical Psychology , pp. 383 - 395
Publisher: Cambridge University Press
Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Berkey, C. S., Hoaglin, D. C., Mosteller, F., & Colditz, G. A. (1995). A Random-Effects Regression Model for Meta-Analysis. Statistics in Medicine, 14(4), 395–411.Google Scholar
Bickel, P. J., & Blackwell, D. (1967). A Note on Bayes Estimates. Annals of Mathematical Statistics, 38, 1907–1911.Google Scholar
Blackwell, D., & Girshick, M. A. (1954). Theory of Games and Statistical Decisions. New York: John Wiley.Google Scholar
Bodnar, O., Link, A., Arendacká, B., Possolo, A., & Elster, C. (2017). Bayesian Estimation in Random Effects Meta-Analysis Using a Non-informative Prior. Statistics in Medicine, 36(2), 378–399.CrossRefGoogle ScholarPubMed
Carlsson, R., Schimmack, U., Williams, D. R., & Bürkner, P.-C. (2017). Bayes Factors from Pooled Data Are no Substitute for Bayesian Meta-Analysis: Commentary on Scheibehenne, Jamil, and Wagenmakers (2016). Psychological Science, 28(11), 1694–1697.Google Scholar
Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015). A Series of Meta-Analytic Tests of the Depletion Effect: Self-Control Does Not Seem to Rely on a Limited Resource. Journal of Experimental Psychology: General, 144, 796–815.Google Scholar
Center for Open Science. (2018). Transparency and Openness Promotion Guidelines. Retrieved from https://cos.io/our-services/top-guidelines/Google Scholar
Cheung, M. (2015). Meta-Analysis: A Structural Equation Modeling Approach. Chichester: Wiley.Google Scholar
Clarke, B. S. (1999). Asymptotic Normality of the Posterior in Relative Entropy. IEEE Transactions on Information Theory, 45(1), 165–176.Google Scholar
Cochran, W. G. (1954). The Combination of Estimates from Different Experiments. Biometrics, 10(1), 101–129.Google Scholar
Copas, J. B. (1999). What Works? Selectivity Models and Meta-Analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society), 162, 95–109.Google Scholar
Copas, J. B. (2013). A Likelihood-Based Sensitivity Analysis for Publication Bias in Meta-Analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 62, 47–66.Google Scholar
Duval, S., & Tweedie, R. (2000). Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis. Biometrics, 56, 455–463.Google Scholar
Fleiss, J. L., Cohen, J., & Everitt, B. S. (1969). Large Sample Standard Errors of Kappa and Weighted Kappa. Psychological Bulletin, 72(5), 323–327.Google Scholar
Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect Size Estimates: Current Use, Calculations, and Interpretation. Journal of Experimental Psychology: General, 141(1), 2–18.Google Scholar
Furlow, C. F., & Beretvas, S. N. (2005). Meta-Analytic Methods of Pooling Correlation Matrices for Structural Equation Modeling under Different Patterns of Missing Data. Psychological Methods, 10(2), 227–254.CrossRefGoogle ScholarPubMed
Glass, G. V. (2015). Meta-Analysis at Middle Age: A Personal History. Research Synthesis Methods, 6(3), 221–231.Google Scholar
Greenland, S., & O’Rourke, K. (2001). On the Bias Produced by Quality Scores in Meta-Analysis, and a Hierarchical View of Proposed Solutions. Biostatistics, 2(4), 463–471.Google Scholar
Gustafson, P., & Clarke, B. (2004). Decomposing Posterior Variance. Journal of Statistical Planning and Inference, 119(2), 311–327.CrossRefGoogle Scholar
Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., … Zwienenberg, M. (2016). A Multilab Preregistered Replication of the ego-Depletion Effect. Perspectives on Psychological Science, 11(4), 546–573.Google Scholar
Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. D. (2010). Ego Depletion and the Strength Model of Self-Control: A Meta-Analysis. Psychological Bulletin, 136, 495–525.Google Scholar
Hardy, R. J., & Thompson, S. G. (1996). A Likelihood Approach to Meta-Analysis with Random Effects. Statistics in Medicine, 15(6), 619–629.Google Scholar
Hedges, L. V. (1984). Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences. Journal of Educational and Behavioral Statistics, 9, 61–85.Google Scholar
Higgins, J. P. T., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., … Cochrane Statistical Methods Group (2011). The Cochrane Collaboration’s Tool for Assessing Risk of Bias in Randomized Trials. British Medical Journal, 343, d5928.Google Scholar
Higgins, J. P. T., Thompson, S. G., & Spiegelhalter, D. J. (2009). A Re-evaluation of Random-Effects Meta-Analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1), 137–159.Google Scholar
Huang-Pollock, C. L., Karalunas, S. L., Tam, H., & Moore, A. N. (2012). Evaluating Vigilance Deficits in ADHD: A Meta-Analysis of CPT Performance. Journal of Abnormal Psychology, 121(2), 360–371.Google Scholar
Ioannidis, J. P. A. (2005). Differentiating Biases from Genuine Heterogeneity: Distinguishing Artifactual from Substantive Effects. In Rothstein, H., Sutton, A. J., & Borestein, M. (Eds.), Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments (pp. 287–302). Chichester: Wiley.CrossRefGoogle Scholar
Ioannidis, J. P. A. (2008). Interpretation of Tests of Heterogeneity and Bias in Meta-Analysis. Journal of Evaluation in Clinical Practice, 14(5), 951–957.Google Scholar
Iyengar, S., & Greenhouse, J. B. (1988). Selection Models and the File Drawer Problem. Statistical Science, 3(1), 109–117.Google Scholar
Jin, Z.-C., Zhou, X.-H., & He, J. (2015). Statistical Methods for Dealing with Publication Bias in Meta-Analysis. Statistics in Medicine, 34(2), 343–360.Google Scholar
Jüni, P., Witschi, A., Bloch, R., & Egger, M. (1999). The Hazards of Scoring the Quality of Clinical Trials for Meta-Analysis. Journal of the American Medical Association, 282(11), 1054–1060.Google Scholar
Kass, R. E., Eden, U. T., & Brown, E. N. (2014). Analysis of Neural Data. New York: Springer.Google Scholar
Kraemer, H. C. (2005). A Simple Effect Size Indicator for Two-Group Comparisons? A Comment on Requivalent. Psychological Methods, 10(4), 413–419.Google Scholar
Kraemer, H. C., & Andrews, G. (1982). A Nonparametric Technique for Meta-Analysis Effect Size Calculation. Psychological Bulletin, 91(2), 404–412.Google Scholar
Krueger, R. F., & Markon, K. E. (2006). Reinterpreting Comorbidity: A Model-Based Approach to Understanding and Classifying Psychopathology. Annual Review of Clinical Psychology, 2(1), 111–133.Google Scholar
Lee, C. H., Cook, S., Lee, J. S., & Han, B. (2016). Comparison of Two Meta-Analysis Methods: Inverse-Variance-Weighted Average and Weighted Sum of z-Scores. Genomics & Informatics, 14(4), 173–180.Google Scholar
Li, S., Jiang, H., Yang, H., Chen, W., Peng, J., Sun, M., … Zeng, J. (2015). The Dilemma of Heterogeneity Tests in Meta-Analysis: A Challenge from a Simulation Study. PLOS ONE, 10(5), e0127538.Google Scholar
Lin, D. Y., & Zeng, D. (2010). On the Relative Efficiency of Using Summary Statistics versus Individual-Level Data in Meta-Analysis. Biometrika, 97(2), 321–332.CrossRefGoogle ScholarPubMed
Lumley, T. (2002). Network Meta-Analysis for Indirect Treatment Comparisons. Statistics in Medicine, 21(16), 2313–2324.Google Scholar
McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016). Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes. Perspectives on Psychological Science, 11(5), 730–749.Google Scholar
Munafo, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., … Ioannidis, J. P. A. (2017). A Manifesto for Reproducible Science. Nature Human Behaviour, 1(1), 0021.Google Scholar
Noorbaloochi, S., & Meeden, G. (1983). Unbiasedness as the Dual of Being Bayes. Journal of the American Statistical Association, 78, 619–623.Google Scholar
Park, S., Serpedin, E., & Qaraqe, K. (2013). Gaussian Assumption: The Least Favorable but the Most Useful. IEEE Signal Processing Magazine, 30(3), 183–186.CrossRefGoogle Scholar
Revelle, W., & Zinbarg, R. E. (2009). Coefficients Alpha, Beta, Omega, and the GLB: Comments on Sijtsma. Psychometrika, 74(1), 145–154.Google Scholar
Rosenberg, M. S. (2010). A Generalized Formula for Converting Chi-Square Tests to Effect Sizes for Meta-Analysis. PLoS ONE, 5(4), e10059.Google Scholar
Rosenthal, R., & Rubin, D. B. (2003). Requivalent: A Simple Effect Size Indicator. Psychological Methods, 8(4), 492–496.Google Scholar
Ruscio, J. (2008). A Probability-Based Measure of Effect Size: Robustness to Base Rates and Other Factors. Psychological Methods, 13(1), 19–30.Google Scholar
Scheibehenne, B., Jamil, T., & Wagenmakers, E.-J. (2016). Bayesian Evidence Synthesis Can Reconcile Seemingly Inconsistent Results: The Case of Hotel Towel Reuse. Psychological Science, 27(7), 1043–1046.Google Scholar
Schmidt, F. L., & Hunter, J. E. (2015). Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Thousand Oaks, CA: Sage.Google Scholar
Shadish, W. R. (2015). Introduction to the Special Issue on the Origins of Modern Meta-Analysis: Introduction to Special Issue. Research Synthesis Methods, 6(3), 219–220.CrossRefGoogle Scholar
Sharma, L., Markon, K. E., & Clark, L. A. (2014). Toward a Theory of Distinct Types of “Impulsive” Behaviors: A Meta-Analysis of Self-Report and Behavioral Measures. Psychological Bulletin, 140(2), 374–408.Google Scholar
Smith, T. C., Spiegelhalter, D. J., & Thomas, A. (1995). Bayesian Approaches to Random-Effects Meta-Analysis: A Comparative Study. Statistics in Medicine, 14(24), 2685–2699.Google Scholar
Stein, M. S., & Nossek, J. A. (2017). A Pessimistic Approximation for the Fisher Information Measure. IEEE Transactions on Signal Processing, 65(2), 386–396.Google Scholar
Stram, D. O. (1996). Meta-Analysis of Published Data Using a Linear Mixed-Effects Model. Biometrics, 52, 536–544.CrossRefGoogle ScholarPubMed
Switzer, F. S., Paese, P. W., & Drasgow, F. (1992). Bootstrap Estimates of Standard Errors in Validity Generalization. Journal of Applied Psychology, 77(2), 123–129.Google Scholar
Van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2016). Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve. Perspectives on Psychological Science, 11(5), 713–729.Google Scholar
Van Houwelingen, H. C., Zwinderman, K. H., & Stijnen, T. (1993). A Bivariate Approach to Meta-Analysis. Statistics in Medicine, 12(24), 2273–2284.Google Scholar
Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., … Salanti, G. (2016). Methods to Estimate the Between-Study Variance and Its Uncertainty in Meta-Analysis. Research Synthesis Methods, 7(1), 55–79.CrossRefGoogle ScholarPubMed
Viswesvaran, C., & Ones, D. S. (1995). Theory Testing: Combining Psychometric Meta-Analysis and Structural Equations Modeling. Personnel Psychology, 48(4), 865–885.Google Scholar
Xu, A., & Raginsky, M. (2017). Information-Theoretic Lower Bounds on Bayes Risk in Decentralized Estimation. IEEE Transactions on Information Theory, 63(3), 1580–1600.Google Scholar
Zwaan, R., Etz, A., Lucas, R., & Donnellan, M. (2018). Making Replication Mainstream. Behavioral and Brain Sciences, 41, e120.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×