Objective: To explore the impact of methodologic issues on
the results of meta-analyses. The following issues were examined:
the type of literature search strategy used; inclusion or exclusion
of non–peer-reviewed studies; the inclusion or exclusion of
non-English language publications; the effect of trial quality; and the
inclusion or exclusion of non–placebo-controlled studies.
Methods: The International Study of Perioperative Transfusion (ISPOT)
meta-analyses were used to evaluate each of
the methodologic issues. The 10 meta-analyses consisted of
technologies to reduce the need for perioperative red blood cell
transfusion. The number of trials for each of the meta-analyses
varied from 2 to 45. Both EMBASE and MEDLINE searches were
conducted, including the use of systematic search strategies.
Results: MEDLINE identified the vast majority of trials. Alone,
MEDLINE would have missed 8 studies compared to 10 for EMBASE. Use of
the systematic search strategies greatly reduced the number of
articles to be reviewed compared to open searches. Type of
publication, country of study origin, inclusion of non-English
publications, and trial quality had very little impact on the
estimates of effect. The use of placebo versus open-label control
affected the magnitude of the odds ratio for two of the
meta-analyses. The results of the two meta-analyses were not statistically
significant if only placebo-controlled trials were included.
Conclusions: While methodologic issues had very little impact
on the ISPOT meta-analyses, further studies are needed in a variety
of other clinical settings. Because MEDLINE, coupled with a review
of the references in the identified trials, identified the vast
majority of trials, one needs to consider the costs and benefits of
searching EMBASE and the pursuance of unpublished and unindexed
trials.