Previous studies have found an association between allegiance bias and methodological quality of psychological treatment comparisons.

To test the hypothesis whether poorer designed randomised controlled trials (RCTs) of psychotherapeutic interventions could be yield increased allegiance effects.

Using a systematic approach with meta-analysis we aimed to assess the relationships between allegiance effect and outcomes with respect to treatment integrity, quality of treatment and overall quality of RCTs.

We searched meta-analyses of RCTs of different types of psychotherapies in the Cochrane Database of Systematic Reviews. Eligible articles included were meta-analyses of RCTs with at least 1 study with allegiance of the experimenter. Effect sizes in allegiance trials were compared with non-allegiance by sensitivity analysis using random and fixed models and a summary relative odds ratio was calculated. Heterogeneity was quantified with the I2 metric.

A total of 30 meta-analyses including 240 RCTs were analyzed. The analysis addressing the quality of studies did not show any significant association between allegiance and non-allegiance studies, whereas significant differences were observed in cases where treatment integrity were not evaluated (sROR=1.54 [95% CI 1.01 to 2.35], P=0.047 and I2=74). Regarding the overall quality of administered psychotherapy the magnitude of the effect was stronger only in the case of low quality (sROR=1.61 [95% CI 1.00 to 2.59], P=0.049 and I2=76).

The results support the view that experimenter's allegiance acts as a systematic bias and can be explained through low quality and integrity of delivered psychotherapy.