Introduction
The self-help genre represents a specific mode of reading that values books as tools to promote action, facilitate self-change and enhance well-being (Blum, Reference Blum2020). The concept of self-help targeting ‘self-actualisation’ and offering general advice on an area leading to self-improvement has now become mainstream (Harwood and L’Abate, Reference Harwood and L’Abate2010). Although evidence regarding the effectiveness of specific titles remains limited (Harwood and L’Abate, Reference Harwood and L’Abate2010), written or computerised self-help is increasingly recognised as a legitimate way to provide psychological therapy (Delgadillo, Reference Delgadillo2018). The most well-established use of self-help interventions to serve as a platform for the delivery of psychological therapy is associated with cognitive behavioural therapy (CBT; Farrand, Reference Farrand and Farrand2020).
Written CBT self-help became popularised through the publication of titles such as Living with Fear (Marks, Reference Marks1978), The Feeling Good Handbook (Burns, Reference Burns1999) and Mind Over Mood (Greenberger and Padesky, Reference Greenberger and Padesky1995). In general, these and subsequent written CBT self-help interventions follow a standard format providing psychoeducation about mental health problems and use a CBT model to help the reader appreciate ways they may experience the difficulty, followed by several specific CBT techniques to address the problem. Informed by a CBT approach (Beck, Reference Beck2020), the user engages with the written self-help intervention through interactive worksheets setting out specific CBT techniques. In effect the written self-help intervention guides the reader through a CBT treatment protocol (e.g. CG113; National Institute for Health and Care Excellence, 2020). In the case of Mind Over Mood (Greenberger and Padesky, Reference Greenberger and Padesky1995) a Clinician’s Guide (Padesky and Greenberger, Reference Padesky and Greenberger1995) has also been developed to facilitate use of the written intervention as adjuvant to therapist-delivered CBT. Requiring the reader to engage with written CBT self-help interventions, the approach extends beyond standard psychoeducation (Goldman, Reference Goldman2006) and different from self-help books that provide advice rather than encouraging active engagement through worksheets (Schindler Zimmerman et al., Reference Schindler Zimmerman, Holm and Haddock2004).
The effectiveness of CBT self-help for the treatment of common mental health difficulties has been demonstrated through an evidence base consisting of over 30 systematic reviews and 50 controlled trials (Delgadillo, Reference Delgadillo2018). Additionally, no significant differences in effectiveness or drop-out up to one-year post-assessment have been reported between CBT self-help and face-to-face psychotherapies (Cuijpers et al., Reference Cuijpers, Donker, van Straten and Andersson2010). Informed by this evidence base, guided CBT self-help is implemented within the Improving Access to Psychological Therapies (IAPT) programme (National Collaborating Centre for Mental Health, 2020). However, there is considerable heterogeneity regarding the characteristics of the CBT self-help interventions adopted within studies that form the basis of NICE recommendations (Farrand, Reference Farrand and Farrand2020).
CBT self-help interventions vary in the specific CBT techniques and content they adopt (Lewis et al., Reference Lewis, Pearce and Bisson2012). Such heterogeneity is problematic given that CBT self-help interventions represent Step 2 of the stepped care service delivery model adopted by the IAPT programme (Richards, Reference Richards, Bennett-Levy, Richards, Farrand, Christensen, Griffiths, Kavanagh, Klein, Lau, Proudfoot, Ritterband, White and Williams2010). These interventions have several characteristics that distinguish them from therapist-delivered high-intensity CBT (HICBT) (Farrand, Reference Farrand and Farrand2020). Whilst HICBT recognises the presenting difficulty in the ‘here and now’, emphasis is also placed upon factors that may have precipitated or maintain the difficulty, through a developmental formulation (Beck, Reference Beck2020). This helps inform the range of CBT techniques that HICBT therapists may use to address cognitive distortions in the form of negative automatic thoughts in the ‘here and now’, alongside unhelpful assumptions and negative schemas that exist at different levels of cognition (Beck, Reference Beck2020). In contrast, the focus of low-intensity CBT (LICBT) is solely on the ‘here and now’, with CBT self-help interventions comprising a ‘single-strand’, where during assessment a single CBT technique that will comprise the LICBT self-help intervention is identified (Turpin et al., Reference Turpin, Baguley, Farrand, Hope, Leibowitz and Lovell2010). This contrasts with HICBT, where evidence-based treatment protocols usually specify the delivery of several different CBT techniques.
As the characteristics of LICBT within the IAPT programme have become better established (Farrand, Reference Farrand and Farrand2020), it is surprising that guidance to evaluate CBT self-help interventions (e.g. UCL, n.d.-a and n.d.-b) have largely focused on factors such as their evidence base, quality (Charnock, Reference Charnock1998) and engagement (Martinez et al., Reference Martinez, Whitfield, Dafters and Williams2008). However, identifying fundamental characteristics that distinguish LICBT from HICBT is needed to ensure CBT self-help interventions demonstrate fidelity to the LICBT clinical method and wider fundamental assumptions of CBT (Beck, Reference Beck2020). With respect to written CBT workbook based self-help interventions commonly adopted by IAPT services, this paper reports on the development and application of a single criteria framework to evaluate both quality and consistency with the fundamentals of LICBT informing the IAPT programme.
Method
Developing criteria for rating written workbook based CBT self-help interventions
Recruiting the criteria application panel
To recruit qualified psychological wellbeing practitioners (PWPs) still in practice or trainers on LICBT programmes onto the criteria application panel, advertisements were sent to all LICBT accredited training providers with a request to forward to all services commissioning training. There were nine respondents, with the first four from different IAPT services (located in London, the Midlands and the South-East) selected to form the criteria application panel, along with P.F. An Expert by Experience who had engaged with CBT self-help as a treatment (M.D.) was also recruited by a PWP panel member.
Criteria framework development
Following consultation between P.F. and A.R., a single criteria framework to rate written workbook based CBT self-help interventions and make recommendations for IAPT services was developed. Consultation was based on a review of the literature surrounding written CBT self-help interventions within the IAPT programme, recommendations regarding written CBT self-help interventions and criteria associated with good quality written patient information (Table 1).
Table 1. Literature informing criteria framework development
Literature review identified 26 criteria incorporated to create a single criteria framework. These criteria were organised into five domains – Scope Consistent with Low-Intensity CBT, Accurate and Clear Information, Engagement, Usability and Behavioural Principles. The criteria for demonstrating fidelity was set as a minimum average score of 4 for individual items within each domain (highlighted in grey in Appendix 1 of the Supplementary material) except all Behavioural Principles criterion (University College London, 2015) given that these were a recent addition to the curriculum (UCL, n.d.-b).
Following development of the pilot criteria (Appendix 1 of the Supplementary material), discussion was undertaken with panel members to ensure it appropriately captured the use of written CBT self-help interventions by PWPs and addressed areas of importance from an ‘Experts by Experience’ perspective. To reach consensus and inform development of the pilot criteria, adaptations were made to the name given to one of the domains (Current adapted to Accurate and Clear Information). Additionally, minimum average scores that were required to be met to meet criteria (highlighted in grey in Appendix 1 of the Supplementary material) was reduced from 4 to 3 on two specific criteria (Complementary, Collaborative), on the basis that support could compensate for these criteria if not fully addressed in the intervention directly. Following adaptation, it was ensured that the pilot criteria framework maintained consistency with the characteristics of LICBT stated in the IAPT Manual (National Collaborating Centre for Mental Health, 2020). At the end of this meeting, M.D. identified domains or individual criterion she did not feel comfortable rating due to lack of knowledge. Furthermore, it was agreed that P.F. would not be involved in rating any intervention he was associated with to avoid a conflict of interest. Following application, further discussions were planned with members of the criteria panel to discuss the process generally and to particularly focus on specific criterion if poor agreement was identified.
Identifying written workbook based CBT self-help interventions adopted by IAPT services
A member of the national IAPT programme team sent a request to IAPT services in the 156 Clinical Commissioning Groups asking them to identify a member of staff who could list the written CBT self-help interventions adopted by their service. On return, these were screened by P.F. to ensure these interventions met basic criteria – written, free to use, did not represent a single or collection of individual worksheets addressing different techniques and were accessible for use via a weblink.
Pilot application of the criteria framework
Members of the panel were sent a list of weblinks to the written workbook based CBT self-help interventions that met criteria (and for pragmatic reasons, were adopted by more than four services). Panel members were asked to independently apply the criteria (Appendix 1 of the Supplementary material) to each written LICBT intervention, rating each individual criteria using the 5-point Likert scale (1, not at all; 5, completely).
Inter-rater reliability
Each member of the criteria application panel submitted ratings for each criterion included within the criteria framework and were independently coded and analysed by a researcher not otherwise associated with the study. For each domain addressed in the criteria framework (see Table 3), ratings for individual criteria were combined and an average domain rating separately calculated for each panel member. Fleiss’ Kappa (κ) was used to examine the level of agreement between members for each domain. Reporting reliability results was informed by Kottner et al. (Reference Kottner, Audig, Brorson, Donner, Gajewski, Hrobjartsson, Roberts, Shoukri and Streiner2011) with level of agreement interpreted according to guidelines (Landis and Koch, Reference Landis and Koch1977).
Results
Intervention rating
Responses were received from 51 IAPT services, with at least one service from each of the seven NHS commissioning regions. Twenty-three free to use and accessible written workbook based CBT self-help interventions were identified, of which 16 (70%) met criteria for a written workbook based LICBT self-help intervention (Table 2).
Table 2. Recommendations for written CBT self-help interventions for common mental health difficulties treated by the IAPT programme and adopted by four or more services
On all occasions, interventions targeting a common mental health difficulty but not meeting criteria failed to achieve the minimum rating for the Single Strand criterion. One intervention (Depression; Centre for Clinical Interventions) was also rejected because it employed a HICBT technique to challenge core beliefs, which is outside of the LICBT clinical method. Goal Setting (Farrand and Woodford, Reference Farrand and Woodford2013a, Reference Farrand and Woodford2013b) failed to meet criteria as it was not considered to represent an IAPT LICBT intervention.
Sixteen (31%) of the 51 services reported only using CBT technique-specific worksheets that do not represent a written CBT workbook self-help intervention. Worksheets developed by the Centre for Clinical Interventions to address anxiety and sleep were reported as being used by 10 (20%) and seen (14%) of services, respectively. Six (12%) services also reported using Reach Out (Richards and Whyte, Reference Richards and Whyte2011) training resources for Cognitive Restructuring and Exposure. Furthermore, 21 (41%) services reported using worksheets freely available on several websites. Six (29%) of these services also indicated employing worksheets that were directed at a mental health problem or behaviour not treated with LICBT within the IAPT programme (e.g. binge-eating disorder, bipolar disorder, body dysmorphia, self-harm). A range of intervention-specific worksheets unrepresentative of the LICBT clinical method were also adopted (e.g. coping strategies, interoceptive exposure sheet, positive log, responsibility pie chart).
Inter-rater reliability
Level of agreement between panel member ratings varied across each of the domains addressed in the criteria (Table 3).
Table 3. Agreement (Fleiss’ kappa, κ) between panel members on average score for each domain
Substantial agreement was identified for Behavioural Principles (κ = 0.77) and Accurate and Clear Information (κ = 0.64). With respect to Scope Consistent with IAPT LICBT (κ = 0.58) and Engagement (κ = 0.51), moderate levels of agreement arose, with only fair agreement identified for Usability (κ = 0.39). Given that only a fair level of agreement was identified for the Usability domain, inter-rater reliability was calculated for individual criteria within this domain. Moderate levels of agreement were identified for each individual criterion except for Support Risk Management (κ = 0.34) and Inviting Presentation (κ = 0.32).
Discussion
This study describes the development and application of a single criteria framework to evaluate written workbook CBT self-help interventions adopted by the IAPT programme. The framework was practical and easy to use, with at least moderate levels of inter-rater reliability across four of the domains addressed. A fair level of agreement was identified for Usability, however, with lower inter-rater reliability on the Support Risk Management and Inviting Presentation domains.
Pilot application of the criteria identified that only a small number of written CBT self-help interventions targeting specific common mental health difficulties were consistent with LICBT adopted within the IAPT programme. Not adopting a single LICBT technique but continuing to employ multi-strand CBT techniques associated with HICBT (Farrand, Reference Farrand and Farrand2020), was the most common reason for written workbook CBT self-help interventions failing to meet criteria. Perhaps unsurprisingly, the majority of recommended written workbook CBT self-help interventions were developed by training providers (e.g. CEDAR; University of Exeter, University of Manchester) or services (e.g. TalkPlus) with close connections to the IAPT programme.
Except for Usability, criteria demonstrated at least moderate levels of inter-rater reliability across domains. The minimum level of inter-rater reliability required to recommend criteria use is the subject of debate, although it is proposed that the situation to which criteria are applied should be considered when interpreting inter-rater reliability and inform adoption (O’Neill, Reference O’Neill2017). As the aim of the criteria were solely to provide a tool to evaluate written CBT self-help interventions for recommendation, moderate inter-rater can potentially be argued to justify adoption. Within the Usability domain, inter-rater reliability was reduced to fair, dependent on lower levels of agreement on the specific criterion Support Risk Management and Inviting Presentation.
Given lower levels of agreement on these selected criteria, a consultation process was undertaken between members of the review panel to identify reasons and generate solutions. Differences on the risk management criterion arose from varied perspectives regarding the need for written CBT self-help interventions to have a dedicated section addressing action to take were the patient to consider themselves at risk. Adopting recommendation regarding the need for specific and clear information regarding risk (e.g. Duman and Farrell, Reference Duman and Farrell2000), it was decided to revise the criterion to state the need to have a specific section of the intervention directing patients to take appropriate action and providing signposting information (Appendix 1 of the Supplementary material). Lower levels of agreement on Inviting Presentation potentially highlights differences in general aesthetic preferences between people and when applied to product design (Hoegg, Reference Hoegg2015). During consultation it was recognised that whilst differences will exist, reducing the number of guidance documents associated with good quality written information addressing aspects of presentation could serve to improve agreement. For future applications of the criteria framework therefore, for issues related to presentation it was decided to solely consult a review of the evidence regarding good patient information developed by the Patient Information Forum (Treadgold and Grant, Reference Treadgold and Grant2014). This review also highlighted benefits of personalising and tailoring presentation to enhance acceptability for different groups. Involving Experts by Experience from groups with diversity to adapt the criteria framework for specific populations could help ensure greater inclusivity.
Encouragingly, the majority of interventions selected for review were consistent with many criteria surrounding the core assumptions of CBT such as interactivity (e.g. Richards and Farrand, Reference Richardson, Richards and Barkham2010; University College London, n.d.-a) and guidance regarding good written patient information (e.g. EQIP: Moult et al., Reference Moult, Franck and Brady2004; NHS Patient Information Toolkit: Department of Health, 2003). However, there were some notable exceptions regarding specific criteria that may be helpful for authors to include when planning to develop LICBT self-help interventions. For example, providing a brief description of author background and experience (rather than relying on the name of an organisation) is recommended to establish competency (e.g. DISCERN; Charnock et al., Reference Charnock1998). Doing so has potential to enhance a patient’s sense of treatment credibility and improve effectiveness (Constantino et al., Reference Constantino, Vîslă, Coyne and Boswell2018). Additionally, as LICBT interventions are intended to be supported by the IAPT programme (National Collaborating Centre for Mental Health, 2020), setting out the way in which this support is offered within the LICBT self-help intervention is important and may help foster and maintain engagement. For example, enabling the patient to spend similar amounts of time engaging with the self-help intervention between support sessions as a patient spends engaging with HICBT within treatment sessions (van Straten et al., Reference van Straten, Hill, Richards and Cuijpers2015). Finally, interventions should include signposting so that users are directed to other sources of support if their mental health difficulty becomes worse between support sessions (e.g. Treadgold and Grant, Reference Treadgold and Grant2014).
Although results from this pilot study do not represent all IAPT services, applying the criteria framework highlights that a wide range of written LICBT interventions varying across services are adopted. This highlights the utility of developing a criteria framework to evaluate interventions already being employed and new ones being considered for adoption. This is important given that use of the criteria framework highlights several areas of concern regarding the adoption of LICBT self-help interventions. Given the CBT self-help genre generally employs techniques associated with HICBT (e.g. Burns, Reference Burns1999; Greenberger and Padesky, Reference Greenberger and Padesky1995), it is perhaps unsurprising that these CBT self-help interventions adopted multiple CBT techniques and therefore did not meet the single-strand criteria (Turpin et al., Reference Turpin, Baguley, Farrand, Hope, Leibowitz and Lovell2010). The addition of several techniques resulted in extending the CBT clinical method from a focus on the presenting difficulty in the ‘here and now’. This included techniques to address different levels of cognitive distortion at the level of core beliefs (Depression; Centre for Clinical Interventions). Whilst not being consistent with the characteristics of LICBT self-help interventions adopted in the IAPT programme (Farrand, Reference Farrand and Farrand2020), such CBT self-help books continue to have utility when adopted as a HICBT intervention.
Many services reported using individual CBT worksheets rather than incorporated within a self-help intervention to be guided by PWPs (National Collaborating Centre for Mental Health, 2020). Whilst the criteria have only been developed to evaluate written CBT self-help workbooks and not single worksheets, caution would need to be exercised if worksheets are employed outside of a workbook format. Using a range of individual worksheets may result in PWPs applying competencies more commonly associated with HICBT and result in ‘therapeutic drift’ (Waller, Reference Waller2009; Waller and Turner, Reference Waller and Turner2016). This is problematic because many HICBT competencies go beyond those identified by the national PWP national training curriculum (University College London, 2015). Furthermore, several services reported using worksheets targeting mental health problems beyond those treated with LICBT and not supported by the evidence base (National Collaborating Centre for Mental Health, 2020). Adopting interventions or resources that are inconsistent with the LICBT clinical method threatens IAPT recovery rate targets.
Development of a criteria framework to evaluate written CBT self-help demonstrated moderate levels of agreement between raters. Although higher levels of agreement would enhance confidence, restricting the use of the adapted criteria framework to solely serve as the basis of recommendations for discussion by services could justify adoption (O’Neill, Reference O’Neill2017). If adopted, the adapted criteria framework (Appendix 1 of the Supplementary material) could serve as the basis of a systematic process to review other interventions adopted by services that did not participate in this pilot or are under development. Given knowledge and competency guiding LICBT self-help interventions, the process should be led by practising PWPs, providing opportunities for national involvement in the IAPT programme. Implementing the process would address concerns regarding interventions currently adopted by IAPT services and direct services to written LICBT workbook-based self-help interventions appropriate for use within the IAPT programme (National Collaborating Centre for Mental Health, 2020).
Limitations
Given the number of services participating in this pilot study, findings may not be representative of IAPT services nationally. As such, caution should be exercised when considering concerns raised in this paper regarding the selection of interventions.
This paper is solely focused on written workbook based LICBT interventions that remain the format most adopted within IAPT services. However, other formats are available, and this may require criteria to be adapted.
Different definitions regarding written LICBT exist, therefore the application of criteria is directly relevant to the evaluation of written workbook based LICBT interventions based on a definition employed within the IAPT programme.
Although an Expert by Experience was fully involved in all stages developing and applying the criteria framework and recognised as an author, there was potential for greater Expert by Experience involvement.
Key practice points
-
(1) Many IAPT services are currently adopting written interventions for use at Step 2 of the stepped care model that do not share characteristics of LICBT associated with the IAPT programme.
-
(2) There is a need to implement an independent standardised criteria-driven process to make recommendations regarding written LICBT self-help interventions employed within the IAPT programme.
-
(3) Using the criteria framework to inform development of written LICBT interventions for use within the IAPT programme offers promise to significantly enhance suitability and fidelity with the LICBT clinical method.
-
(4) Given knowledge and competency supporting LICBT self-help interventions, practising PWPs should lead a process applying the criteria framework to evaluate LICBT interventions.
-
(5) Involving appropriate Experts by Experience to adapt the criteria framework for specific populations with diversity could help ensure the IAPT programme is more inclusive.
-
(6) There remains a need to apply and/or adapt a criteria framework to interventions targeting treatment of mental health difficulties associated with physical health problems.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1754470X22000241
Data availability statement
The data that support the findings of this study are available on request from the corresponding author, P.F. The data are not publicly available given it contains information that could compromise service policy regarding identification of interventions being provided.
Acknowledgements
We would like to thank IAPT services that responded to the initial request to identify written self-help interventions being adopted and members of the IAPT programme team supporting data collection.
Author contributions
Paul Farrand: Conceptualization (lead), Data curation (lead), Formal analysis (lead), Investigation (lead), Methodology (lead), Project administration (lead), Writing – original draft (lead), Writing – review & editing (equal); Adam Dawes: Investigation (supporting), Methodology (supporting), Writing – review & editing (equal); Michelle Doughty: Investigation (equal), Methodology (equal), Writing – review & editing (equal); Sundeep Phull: Investigation (equal), Methodology (equal), Writing – review & editing (equal); Sally Saines: Investigation (equal), Methodology (equal), Writing – review & editing (equal); Simon Winter: Investigation (equal), Methodology (equal), Writing – review & editing (equal); Anthony Roth: Formal analysis (supporting), Investigation (supporting), Methodology (supporting), Writing – review & editing (equal).
Financial support
This research received no grant or funding from any funding agency, commercial or not-for-profit sectors.
Conflicts of interest
P.F. was involved in the development of many of the interventions identified in this paper. However, to avoid the potential for a conflict of interest he was not involved in applying criteria to any intervention he was involved with. All interventions are available freely for clinical purposes with no potential for monetary gain. Paul Farrand is an Associate Editor of the Cognitive Behaviour Therapist. He was not involved in the review or editorial process for this paper, on which he is listed as an author.
Ethical standards
Relevant ethical standards have been complied with fully, with governance undertaken through the national IAPT programme team.
Open access
Comments
No Comments have been published for this article.