Swanson et al (Reference Swanson, Swartz and Wagner2000) reanalysed the results of the North Carolina trial (Reference Swartz, Swanson and WagnerSwartz et al, 1999) and their findings are becoming increasingly influential in current debates about mental health legislation in the UK. Our recent systematic review (Reference Churchill, Owen and SinghChurchill et al, 2007), which included these articles, demonstrated that there was no robust evidence to indicate that community treatment orders are associated with either significant benefit or harm. The secondary analyses performed by Swanson et al are, we believe, misleading for two reasons.

First, based on everyone in the trial the intention-to-treat (ITT) effect of randomisation to an involuntary out-patient commitment (OPC) was of a modest and non-significant reduction in violence (risk difference of 4.5%). This overall ITT effect of OPCs is a weighted average of the ITT effects in the two subgroups of participants defined by their post-randomisation management (those who received short-term OPCs and those who eventually received long-term OPCs). These two subgroups would exist in the control arm had they been placed on OPCs. Assuming that there was no benefit in those who received the short-term OPCs (i.e. risk difference 0), the results of Swanson et al suggest that the reduction in violence in those with long-term OPCs would be 12.4%. However, even if considered clinically significant, this finding would still not be statistically significant because the overall ITT effect was not significant (assuming a zero ITT effect in those receiving short-term OPCs implies that a test of the hypothesis concerning those receiving long-term OPCs is equivalent to the test for the overall ITT effect). The only way in which there could have been a beneficial effect in those receiving long-term OPCs is if the effects in those receiving short-term OPCs were actually detrimental (i.e. increased the rate of violence). It is improbable that they would be, and in policy terms it would be unacceptable to impose OPCs in the knowledge that they would cause harm to those in whom they are only applied for a short period.

Second, a post hoc comparison of the outcomes in groups defined by management decisions or patient behaviour following randomisation is potentially subject to selection effects (hidden confounding). That this is in fact the case is illustrated by the results of other subgroup analyses by the same research group (Reference Swartz, Swanson and WagnerSwartz et al, 1999: Fig. 1). The group destined to be on long-term OPC have a better clinical outcome in the first 1–2 months. In other words there is evidence that the group destined to receive long-term OPCs have a favourable clinical profile before the OPC is renewed. We believe that it is likely that long-term OPCs will only be contemplated under certain circumstances, such as when the short-term OPC has apparently made a difference. Those who have intractable problems or in whom a short-term OPC has failed to make any change might not have their OPC renewed.

The investigators responsible for the North Carolina trial accomplished one of the most extraordinary trials ever performed and as such deserve enormous praise. However, the results described in these and similar secondary analyses are, we believe, flawed and misleading, and should not be taken as evidence for a beneficial effect of OPC. We made a similar point (Reference Szmukler and HotopfSzmukler & Hotopf, 2001) following the publication of the original trial. The trial data are best interpreted using the main ITT analyses, which show no evidence of benefit or harm.