Identification of studies

For a generic systematic review, empirical evidence shows that literature searches limited to electronic databases retrieve only approximately half of all relevant studies (Stroup et al. Reference Stroup, Berlin, Morton, Olkin, Williamson, Rennie, Moher, Becker, Sipe and Thacker2000). Therefore, it is recommended to search other sources, such as study registries, conference proceedings, grey (i.e., not controlled by commercial publishers) literature, and to contact experts in the field (Trespidi et al. Reference Trespidi, Barbui and Cipriani2011). In systematic reviews of RCTs, the search in sources other than electronic databases is made easier by the availability of specific repositories of randomised trials, such as clinicaltrials.gov, and by the requirement of ethical approval, which makes RCTs more easily traceable. Unfortunately, this is not always the case for pharmacoepidemiological studies. In this regard, the ENCePP suggests to considering additional sources, such as the Clinical Practice Research Database (CPRD) and the EU PAS Register, that are perhaps less well known to a number of researchers but that may turn out to be a precious source. Another challenge in the literature search process for systematic reviews of pharmacoepidemiological studies relates to the poor reliability of databases filters and indexing terms specifically tapping this type of studies. Indeed, whilst, for example, in Pubmed filters are available for RCTs, other study design labels are unlikely to capture all observational studies, in part owing to their heterogeneous nature. Therefore, the ENCePP urges caution in using such filters/indexing.

Data extraction

Whereas the process of data extraction would follow the same recommendations as for other systematic review (e.g., it should be conducted by two researchers independently), some of the extracted variables reflect the specificity of pharmacoepidemiological studies of safety outcomes. More specifically, the ENCePP highlights the relevance of reporting if the study was primarily aimed at assessing efficacy of intervention, safety endpoints, or if it was an aetiological study where medication use was only one of a range of factors considered. Additionally, the type of harm that the study could detect (related to the length of time participants were under observation), and the definition of safety outcomes are important variables to extract.

Study quality assessment

Many readers will be familiar with the GRADE system, including the Cochrane risk of bias tool, for RCTs, which is currently recommended as the standard for quality assessment of this type of studies (Barbui & Cipriani, Reference Barbui and Cipriani2011). There is less consensus on quality assessment tools for pharmacoepidemiological studies. Whilst the Cochrane collaboration has often used the Newcastle Ottawa Scale 7 (NOS) (Wells et al. Reference Wells, Shea, O'Connell, Peterson, Welch, Losos and Tugwell2010), the ENCePP points out that a drawback of this scale is the lack of an ‘Other Bias’ item and, of relevance in particular for pharmacoepidemiological studies, the inability to adapt the bias assessment to domains highly relevant for such studies, such as the so-called ‘confounding by indication’ (i.e., when a variable is a risk factor for a disease among non-exposed individuals and is associated with the exposure of interest in the population from which the cases derive). By contrast, more recent tools such as the RTI Item Bank, commissioned by the Agency for Healthcare Research and Quality (AHRQ) (Viswanathan & Berkman, Reference Viswanathan and Berkman2011), do take into account the ‘confounding by indication’. Arguably, the development and refinement of tools to appraise the quality of pharmacoepidemiological studies of safety outcomes will be one of the most relevant and expanding areas in the field.

Statistical plan

Two main issues arise when it comes to meta-analyse data from pharmacoepidemiological studies of safety outcomes. First, high inter-study variability is frequent due to, biases in treatment effect estimates arising from the lack of control over treatment allocation, absence of blinding, differences in length and completeness of follow-up. Whilst heterogeneity can be taken into account by means of random effect analytical approaches, the ENCePP points to the possibility of using Bayesian methods, especially when the number of meta-analysed studies is small. Furthermore, sensitivity analyses should always be planned to assess important sources of variability. Second, and relatedly, confounders need to be taken into account. Two approaches suggested by the ENCePP are: (1) to meta-analyse only the most credible adjusted results from the studies; and (2) to collect individual participant data with as many potential confounders as possible and to repeat the adjustments for the entire dataset.

The reader is invited to carefully examine the full text of the ENCePP guidance for further details on the recommended procedures for conducting systematic reviews and meta-analyses of pharmacoepidemiological studies of safety outcomes.

To conclude, although the ENCePP guidance should not be considered as a definitive document, since it will likely be refined in several iterations arising from researchers’ feedback, it is expected that it will be highly influential and useful for the field, with the aim to improve and standardise the conduction and reporting of systematic reviews and meta-analyses of pharmacoepidemiological studies of safety outcomes.