Transparent performance of research

There are inherent differences in the priorities of academics and policy-makers. Academics are incentivised to generate research that is publishable, leads to funding and, to some extent, is translatable. In comparison, governments and other authorities are interested in the profitability and translatability of research (Ioannidis, Reference Ioannidis2014). These differences in priorities mean that the research conducted by teams such as BIT in collaboration with policy-makers is typically much more rapidly translated and applied to policy settings than research conducted in traditional academic settings. However, as Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) describe, this has meant that (until recently) their research efforts have been focused on the “low-hanging fruit”; projects that are more likely to be of interest to policy-makers, but perhaps of less scientific interest.

Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) also discuss how a misalignment of research priorities means that “there may be differences over the proposed timing, framing and conclusions of any potential publication.” While academics may engage in HARKing (Hypothesising After the Results are Known) in order to generate publishable results (Kerr, Reference Kerr1998), policy-makers or private investors may be motivated to ensure that research findings are in line with their interests, and may therefore be prone to CoRKing (our own phrase – ‘Concluding before the Results are Known’).

Pre-registration of study protocols on platforms such as the Open Science Framework (Foster & Deardorff, Reference Foster and Deardorff2017) or the ISRCTN registry (https://www.isrctn.com) creates a permanent record of the protocol prior to the start of testing. By including a data analysis plan and a comprehensive record of all outcome measures, pre-registration makes it harder for those with vested interests from consciously or unconsciously HARKing or CoRKing (Rifai et al., Reference Rifai, Bossuyt, Ioannidis, Bray, Mcshane, Golub and Hooft2014; Munafò et al., Reference Munafò, Nosek, Bishop, Button, Chambers, Du Sert, Simonsohn, Wagenmakers, Ware and Ioannidis2017) and increases the methods reproducibility of the research (see Box 1). BIT have already started to engage in pre-registration practices, although there is room for improvement: at the time of our searches (in October 2017), of the 11 studies published by BIT in academic journals, only one (Hallsworth et al., Reference Hallsworth, Chadborn, Sallis, Sanders, Berry, Greaves, Clements and Davies2016) has an accompanying study protocol on the BIT website (Chadborn & Sanders, Reference Chadborn and Sanders2014). Protocols can be pre-registered but embargoed for a period (the Open Science Framework has this functionality) where those commissioning the research are concerned about making research plans public (especially where the research involves changing the behaviour of the general public). Once the research is complete, the protocol can be made public and will retain its date stamp proving its pre-registration status.

Transparent verification of research

Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) describe how senior officials are particularly concerned about “any transfer of data … (even anonymised data).” This concern is not unique to policy-makers, as academics have also been shown to be reluctant to share their data (Wicherts et al., Reference Wicherts, Borsboom, Kats and Molenaar2006). However, the benefits of open data (and accompanying meta-data) are compelling. Open data means that results can be reproduced independently and verified, interventions can be better understood, alternative explanations can be explored and CoRKing can be identified. Together, this is a way of assessing the inferential reproducibility of research (see Box 1). Recent initiatives spearheaded by individual researchers (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin and Christensen2015; Morey et al., Reference Morey, Chambers, Etchells, Harris, Hoekstra, Lakens, Lewandowsky, Morey, Newman and Schönbrodt2016), publishers and research councils have encouraged the sharing of study data. Study data can be published on the Open Science Framework alongside pre-registered study protocols.

Transparent communication of research

There is evidence of publication bias in BIT. Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) distinguish between “initiatives that are not made public and those that were made public but did not go through the additional step of peer-reviewed publication.”

As studies are typically not pre-registered, the extent of any file drawer of unpublished studies (i.e., publication bias) in BIT is unknown. However, with over 300 RCTs conducted over the past six years, the 30 ‘academic publications’ on their website and 39 ‘policy publications’ at the time of our searches (in October 2017) are unlikely to tell the whole story of BIT activity. As Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) acknowledge, publication bias reduces the “transparency of government,” causing a “‘public file drawer’ problem” that distorts the literature, ultimately to the detriment of future research and policy practices.

The reasons for publication bias have been described elsewhere (Rosenthal, Reference Rosenthal1979; Joober et al., Reference Joober, Schmitz, Annable and Boksa2012; Franco et al., Reference Franco, Malhotra and Simonovits2014; Ioannidis et al., Reference Ioannidis, Munafo, Fusar-Poli, Nosek and David2014). However, they are worth describing here in the context of research conducted in policy settings. An underlying reason for publication bias in both academic and policy settings is the pressure to ‘find’ interesting results or perhaps, in the case of policy-makers, findings that fit with their policy objectives. It is also widely acknowledged that null findings are often not made public (Franco et al., Reference Franco, Malhotra and Simonovits2014). Withholding null findings can have serious implications for the transparency of research. Rather than being seen as interventions that failed to ‘work’, null findings resulting from well-designed and adequately powered studies should be made public in order to prevent the draining of valuable resources into research questions that have already been answered. Increasingly, there is recognition of the importance of reporting null findings, and it is encouraging that BIT have recently published null findings from a trial investigating methods of priming honesty among individuals completing tax returns (Kettle et al., Reference Kettle, Hernandez, Sanders, Hauser and Ruda2017).

Another underlying reason for publication bias is a lack of time to devote to writing up ‘uninteresting’ findings. As Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) say, “there are often few resources provided to support the publication process, which is likely to be seen as a luxury.” Time is not a luxury many policy-makers have. Importantly, for teams such as BIT, their findings can alter government action as soon as the results are in, long before publication in an academic journal. Where the impact comes first, and those commissioning the work have already moved to the next problem, publication in an academic journal becomes less of a priority. Where time and limited resources would otherwise prevent publication of manuscripts, BIT could consider working with academic partners, for whom publishing in academic journals is not a luxury, but a necessity. BIT has a history of employing PhD students to conduct research internships and, using this scheme, a mutually beneficial relationship could be established such that the student gains experience working in a policy setting while writing up a manuscript for publication that otherwise would not have been published. As a minimum, brief summaries of research findings could be published online (on the Open Science Framework, for example) alongside the pre-registered protocol and data. In this instance, this final step would be trivial, but would ensure that research findings are made open. Together, these components could be ‘published’ by assigning a DOI to the deposit, which would allow other researchers to cite it.

Transparent evaluation of research

Where BIT findings are written up, these are rarely published in peer-reviewed journals. Of the 30 academic publications on their website, 11 have been published in academic journals and thus peer reviewed. Where publication in academic journals is not possible, Sanders and colleagues (Reference Sanders, Snijders and Hallsworth2018) suggest that manuscripts should “provide enough detail to allow a reasonable judge of their quality.” This is already encouraged for RCTs – the CONSORT statement (Schulz et al., Reference Schulz, Altman and Moher2010) provides an evidence-based minimum set of requirements for reporting the results of RCTs. It also includes extensions for specific types of RCTs, such as cluster-randomised trials, which are often used by BIT (Campbell et al., Reference Campbell, Piaggio, Elbourne and Altman2012). Using internationally recognised standards would increase the reproducibility of their research.

In recent years, peer review has been heavily criticised (Bohannon, Reference Bohannon2013) and post-publication peer review has been provided as an alternative (Hunter, Reference Hunter2012; Kriegeskorte, Reference Kriegeskorte2012; Teixeira da Silva & Dobránszki, Reference Teixeira da Silva and Dobránszki2015). Using platforms such as PubPeer, academics can support independent scrutiny and comment on the research methodology, the reliability of the findings and the appropriateness of the conclusions. Arguably, this kind of evaluation is particularly important for the research conducted by BIT, given that many of the users of their research are not academics but policy-makers and members of similar teams worldwide who may have less experience in evaluating research designs and findings.

A nudge towards transparency

We suggest an incentive structure that rewards BIT's non-academic partners for engaging in practices that increase research transparency. Munafò and colleagues (Reference Munafò, Nosek, Bishop, Button, Chambers, Du Sert, Simonsohn, Wagenmakers, Ware and Ioannidis2017) describe how providing incentives can increase the uptake of practices promoting open science. For example, badges for open science practices that accompany published manuscripts may increase the uptake of these practices (Kidwell et al., Reference Kidwell, Lazarević, Baranski, Hardwicke, Piechowski, Falkenberg, Kennett, Slowik, Sonnleitner and Hess-Holden2016). No doubt BIT would consider these a simple, low-cost and effective ‘nudge’, increasing the transparency and reproducibility of the research reported.

We envisage a tiered approach (see Figure 1) where those commissioning research can choose from the bronze, silver and gold options. Research can then be badged on the BIT website under one of these headings. This has the additional benefit of bringing the focus onto those commissioning the research, rather than BIT themselves. This is important given that BIT have been taking important steps to increase the transparency of their research, with these steps arguably hampered by those they work with.

Figure 1. Tiered incentive structure to encourage transparent performance, verification, communication and evaluation of research. CoRKing = Concluding before the Results are Known.

The bronze level requires that a study protocol is registered online prior to the start of testing. The silver level requires data (and accompanying meta-data) to also be published, and the gold level requires a summary of the data to be published once the data have been analysed. Where time and resources permit, a pre-print or working paper that follows reporting guidelines (such as CONSORT; Schulz et al., Reference Schulz, Altman and Moher2010) could be published, or a manuscript could be submitted to a peer-reviewed journal. Protocols, data and results could all be published in a single location, such as the Open Science Framework, and assigned a persistent link via a DOI.