Origins of cognitive bias modification research
This journal recently published an ambitious and novel meta-analysis by Cristea et al Reference Cristea, Kok and Cuijpers1 on the topic of cognitive bias modification (CBM) research. Meta-analysis is a powerful analytical approach, with the capacity to answer a range of vital questions. However, for legitimate conclusions to be drawn, it is crucial to ensure clarity concerning the precise question that the design of any given meta-analysis permits it to resolve. As we will argue in the present commentary, the term ‘cognitive bias modification’ has come to be used in an ambiguous manner, which compromises clarity concerning the specific question that Cristea et al's meta-analysis was designed to resolve, thereby permitting potential misinterpretation of their findings. In this paper, we will distinguish two quite separate questions that were not clearly differentiated by Cristea et al. Our reanalyses of the studies considered by these colleagues demonstrates that, although procedures intended to modify cognitive bias may not reliably evoke the target process of cognitive change, when this CBM process is successfully elicited then a reliable impact on emotional vulnerability is observed.
The past 5 years have witnessed a rapid burgeoning of interest in CBM research. Experimental psychologists originally developed the CBM approach to shed light on the causal nature of associations between certain types of cognitive bias and variation in emotional vulnerability, and bias modification approaches continue to be used as an experimental tool to illuminate such theoretical issues. Reference MacLeod and Mathews2 Clinical interest in CBM has been driven by the recognition that the direct modification of cognitive bias may represent a method of reducing emotional vulnerability, with potential therapeutic benefits for anxiety and depressive disorders. The escalation of clinical interest in this field of research was fuelled by some encouraging early findings, reporting positive outcomes of CBM in the treatment of dysfunctional anxiety. Reference Amir, Beard, Burns and Bomyea3,Reference Beard and Amir4 As is the case with all new lines of discovery, however, conclusions ultimately must be based on the weight of empirical evidence that accrues across multiple studies.
The seminal CBM experiments, carried out in the early years of this century, were designed to determine whether the successful modification of cognitive bias would serve to significantly alter emotional vulnerability. Investigators focused on two quite different types of cognitive bias: (1) selective attentional responding to unambiguously negative information (attentional bias); and (2) selective interpretation of emotionally ambiguous information (interpretive bias). In the former case, early researchers developed rudimentary procedures intended to encourage attentional avoidance of negative information, Reference MacLeod, Rutherford, Campbell, Ebsworthy and Holker5 and in the latter case, they developed simple procedures intended to encourage the consistent imposition of a benign interpretation on ambiguous information. Reference Wilson, MacLeod, Mathews and Rutherford6 In these initial studies, it was shown that some (although not all) of these procedures were capable of successfully modifying attentional bias and/or interpretive bias, as intended. Moreover, when this process of bias change was successfully evoked, it served to significantly alter emotional vulnerability (as evidenced, for example, by reduced emotional reactivity to stressors). A great deal of CBM research has subsequently been carried out to extend this seminal work. Unfortunately, as the field has progressed, exactly what is meant by the term ‘cognitive bias modification’ has become muddied, with unhelpful consequences. We will now turn to consider the basis of this terminological confusion.
The importance of distinguishing process from procedures
The term cognitive bias modification is now sometimes used to refer to the process of actual cognitive bias change, as was the case in early CBM research. By this definition, when a procedure fails to successfully modify cognitive bias then ‘cognitive bias modification’ has not taken place. However, other times the term is instead used to refer to the procedures (typically, computerised methods) that are intended to bring about change in cognitive bias, irrespective of whether these procedures successfully elicit the intended process of cognitive bias change. By this definition, even when a procedure demonstrably fails to successfully modify cognitive bias ‘cognitive bias modification’ has nevertheless taken place, so long as it was the investigator's intention to evoke a change in cognitive bias through delivery of this procedure. This lack of terminological precision compromises clarity concerning the specific question that the recent meta-analysis by Cristea et al sought to answer. Specifically, these investigators set out to determine whether CBM, on average, alters emotional vulnerability. However, the authors did not distinguish between these two potential meanings of the term ‘cognitive bias modification’, thereby leaving unclear whether the purpose of their meta-analysis was to address the question of whether the process of cognitive bias change reliably alters emotional vulnerability, or to address the quite different question of whether procedures employed with the intention of inducing change in cognitive bias reliably alter emotional vulnerability, irrespective of whether or not they bring about this intended process of cognitive bias change.
Critically, confusion concerning the precise question that the Cristea et al meta-analysis was designed to address introduces the grave danger that readers could readily misconstrue the authors' findings as indicating that the process of cognitive bias modification is unreliable in delivering emotional benefits, when in fact these findings indicate only that procedures intended to evoke the process of CBM are unreliable in delivering emotional benefits. The observation that procedures which fail to reliably elicit the process of CBM also fail to reliably influence emotional vulnerability, permits no legitimate conclusion concerning whether the process of CBM reliably influences emotional vulnerability. The important general point here is that the therapeutic benefit of evoking a target process cannot be established by examining only the therapeutic impact of procedures merely intended to evoke this target process, without regard to whether the process was indeed evoked. This point can readily be illustrated using self-evident examples. For instance, consider a researcher who wants to determine whether the process of exercising can deliver health benefits. If no health benefits were found to result from a procedure intended to elicit the behavioural process of exercising (e.g. an advertising campaign), this would not justify the conclusion that the process of exercising delivers no health benefits, as this procedure may or may not have evoked this process. Indeed, the conclusion that exercise does or does not deliver health benefits can only be drawn when it is confirmed that the target process of increasing exercise has been evoked (e.g. by using exercise diaries). The present commentary is intended to reduce the prospect of confusion that is invited by Cristea et al's meta-analysis in two ways. First, we clarify the specific question Cristea et al's meta-analysis was capable of resolving, which we will point out concerns the emotional impact of procedures intended to elicit the process of cognitive bias change. Second, we report supplementary analyses that address the rather different but extremely important question that this previous meta-analysis did not resolve, concerning whether the process of cognitive bias change serves to alter emotional vulnerability.
Cristea et al adopted thoughtful and rigorous inclusion criteria when selecting experimental studies for their meta-analysis. However, one criterion not taken into account was whether the procedure adopted with the intention of modifying cognitive bias actually achieved this objective. The decision to include studies in which an intended CBM procedure failed to elicit the process of CBM reveals which question Cristea et al's meta-analysis was designed to address. Specifically, the meta-analysis can determine only whether, when one averages across studies in which researchers have delivered a diversity of procedures with the intention of changing cognitive bias, without regard to whether these procedures elicited the intended process of CBM, one observes a significant mean effect of these studies on emotional vulnerability. The findings reported by Cristea et al permit the conclusion that the range of procedures adopted across the reviewed studies do not reliably alter emotional vulnerability.
We would not wish to dispute the legitimate inference that such findings mitigate against future routine adoption of these procedures within clinical interventions intended to reduce emotional vulnerability. However, it is crucial to recognise that Cristea et al's meta-analysis permits no conclusion whatsoever concerning the equally important, but quite different, question of whether the process of successfully modifying cognitive bias reliably serves to alter emotional vulnerability. Determining this would require adopting the inclusion criterion, when selecting studies for consideration in a meta-analysis, that the target cognitive bias was significantly modified by the adopted procedure, confirming that the process of cognitive bias change was successfully evoked. In the present commentary, we reconsider the studies selected for review by Cristea et al to address this critically important question of whether the process of successfully modifying cognitive bias (rather than simply the delivery of procedures intended to evoke this cognitive change process, that sometimes fail to do so) reliably alters emotional vulnerability.
Before addressing this question, we must distinguish ‘emotional vulnerability’ from ‘mood state’ per se. The predominant purpose of most CBM research has been to determine whether the modification of cognitive bias serves to alter emotional vulnerability. Early experimental studies showed that the successful induction of change in attentional bias, and in interpretive bias, often did not influence resting mood state, but instead altered emotional vulnerability, for example by reducing the degree to which subsequent exposure to stressors elicited large elevations in negative mood. Reference Wilson, MacLeod, Mathews and Rutherford6,Reference Baert, De Raedt, Schacht and Koster7 The impact of CBM on emotional vulnerability also can be assessed by using trait measures of emotional disposition. However, the Cristea et al meta-analysis overlooked this distinction between the impact of CBM on emotional vulnerability, as opposed to its impact on resting mood state. Studies that examined the impact of intended CBM procedures only on mood state were mixed together with studies that examined their impact on emotional vulnerability. Regardless of whether the question under scrutiny concerns the impact on emotional vulnerability of procedures merely intended to induce change in cognitive bias, or instead concerns the impact on emotional vulnerability when this process of cognitive bias change is successfully induced, only those studies that assess emotional vulnerability (rather than resting mood state alone) can provide an answer. Determining whether CBM affects emotional vulnerability would require adopting the inclusion criterion, when selecting studies for consideration in a meta-analysis, that the outcome measures assess emotional vulnerability (rather than resting mood state alone), and do so in a manner that has the potential to reveal change in such vulnerability following CBM. In the next section, we reanalyse the data from the studies included in Cristea et al's meta-analysis to address the question of whether the process of successfully modifying cognitive bias reliably alters emotional vulnerability.
Reanalysis of Cristea et al
To determine whether the process of modifying cognitive bias reliably alters emotional vulnerability, we reanalysed the data from the studies selected by Cristea et al for inclusion in their meta-analysis. For this purpose, we included in this present reanalysis those studies selected by Cristea et al that met the following three criteria:
-
1 the intended CBM procedure specifically targeted either interpretive bias or attentional bias;
-
2 the impact of the intended CBM procedure on the target cognitive bias was assessed;
-
3 the effect size reported by Cristea et al was computed from an emotional assessment approach sensitive to detection of changes in emotional vulnerability.
A summary of the studies selected by Cristea et al, thus catagorised, is provided in Table 1.
Table 1 Overview of studies selected by Cristea et al that did and did not meet the inclusion criteria in our present reanalysis
AB, attentional bias; CBM, cognitive bias modification; IB, interpretive bias; N, no; Y, yes.
Within the resulting pool of studies, we distinguished those studies in which the intended CBM procedure successfully induced the process of bias modification, and those studies in which the procedure failed to induce the process of bias modification. A summary of the studies, thus categorised, is provided in Table 2. Our analyses reveal that, across these studies, when the intended CBM procedure has successfully elicited the process of CBM (n = 12), then this process has been reliably accompanied by a significant change in emotional vulnerability. We considered the average impact of successful CBM on emotional vulnerability, first when this bias modification process involved change in attentional bias (n = 3), and next when this process involved change in interpretive bias (n = 9). This revealed a significant positive impact of the bias modification process on emotional disposition in both cases, with g = 0.602 (95% CI 0.139–1.066), and g = 0.400 (95% CI 0.213–0.586) respectively. In each case heterogeneity was negligible (I 2 = 0%) and not significant (P > 0.05).
Table 2 Success of intended CBM procedures in eliciting change process in targeted cognitive bias, and impact on emotional vulnerability
| Study | Bias targeted by intended CBM procedure |
Process of bias modification change achieved |
Change in emotional vulnerability observed |
Hedges' g (95% CI) |
|---|---|---|---|---|
| Amir et al (2009) Reference Amir, Beard, Burns and Bomyea3 | AB | Y | Y | 0.662 (−0.113 to 1.438) |
| Hazen et al (2009) Reference Hazen, Vasey and Schmidt11 | AB | Y | Y | 0.607 (−0.201 to 1.415) |
| Li et al (2008) Reference Li, Tan, Qian and Liu14 | AB | Y | Y | 0.530 (−0.296 to 1.356) |
| Beard & Amir (2008) Reference Beard and Amir4 | IB | Y | Y | 0.656 (−0.098 to 1.411) |
| Bowler et al (2012) Reference Bowler, Mackintosh, Dunn, Mathews, Dalgleish and Hoppitt9 | IB | Y | Y | 0.553 (−0.053 to 1.159) |
| Hirsch et al (2007) Reference Hirsch, Mathews and Clark12 | IB | Y | Y | 0.535 (−0.253 to 1.322) |
| Hoppitt et al (2010) Reference Hoppitt, Mathews, Yiend and Mackintosh13 | IB | Y | Y | C1 0.236 (−0.329 to 0.801) C2 0.091 (−0.461 to 0.643) |
| Mackintosh et al (2006) Reference Mackintosh, Mathews, Yiend, Ridgeway and Cook15 | IB | Y | Y | 0.755 (0.093 to 1.418) |
| Mathews et al (2007) Reference Mathews, Ridgeway, Cook and Yiend16 | IB | Y | Y | 0.430 (−0.192 to 1.053) |
| Murphy et al (2007) Reference Murphy, Hirsch, Mathews, Smith and Clark17 | IB | Y | Y | C1 0.644 (0.049 to 1.240) C2 0.331 (−0.253 to 0.916) |
| Salemink et al (2009) Reference Salemink, van den Hout and Kindt20 | IB | Y | Y | 0.333 (−0.328 to 0.995) |
| Steinman & Teachman (2010) Reference Steinman and Teachman22 | IB | Y | Y | 0.164 (−0.385 to 0.713) |
| Baert et al (2010) Reference Baert, De Raedt, Schacht and Koster7 | AB | N | N | C1 −0.218 (−0.777 to 0.341) C2 0.246 (−0.411 to 0.904) |
| Boettcher et al (2013) Reference Boettcher, Leek, Matson, Holmes, Browning and Macleod8 | AB | N | N | 0.008 (−0.421 to 0.436) |
| Carlbring et al (2012) Reference Carlbring, Apelstrand, Sehlin, Amir, Rousseau and Hofmann10 | AB | N | N | −0.060 (−0.497 to 0.377) |
| Neubauer et al (2013) Reference Neubauer, von Auer, Murray, Petermann, Helbig-Lang and Gerlach18 | AB | N | N | −0.009 (−0.513 to 0.495) |
| Rapee et al (2013) Reference Rapee, MacLeod, Carpenter, Gaston, Frei and Peters19 | AB | N | N | −0.134 (−0.530 to 0.262) |
| Schoorl et al (2013) Reference Schoorl, Putman and Van Der Does21 | AB | N | N | 0.142 (−0.422 to 0.529) |
AB, attentional bias; C, comparison; CBM, cognitive bias modification; IB, interpretive bias; N, no; Y, yes.
In contrast, when the intended CBM procedure has not successfully modified the target cognitive bias, as has been the case in some studies in which the intention was to modify attentional bias (n = 6), then this failure to modify bias has been accompanied by a concurrent lack of impact on emotional vulnerability, g = −0.01 (95% CI −0.184 to 1.63). Again, heterogeneity was negligible (I 2 = 0%) and was not significant (P > 0.05).
As can be seen in Table 1, in every study where the intended CBM procedure has successfully elicited the process of cognitive bias change, this process has been accompanied by a significant change in emotional vulnerability. Conversely, in every study where the intended CBM procedure has not successfully elicited the process of cognitive bias change, there has been no significant change in emotional vulnerability.
Thus, the results of the present reanalysis of Cristea et al's data, here carried out to determine whether the process of successfully modifying cognitive bias reliably alters emotional vulnerability, indicate that eliciting this CBM process does reliably impact on emotional vulnerability, whether the process involves change in attentional bias or change in interpretive bias. By omitting this critical distinction between studies in which the intended CBM procedure did or did not successfully induce the target process of cognitive bias change, Cristea et al's meta-analysis obscures the evidence that the process of CBM reliably influences emotional vulnerability. Hence, the conclusion drawn by Cristea et al, that CBM does not reliably alter emotional vulnerability, is warranted only if one defines CBM as being any procedure intended to change cognitive bias, regardless of whether it elicits this intended process of bias change. However, when one instead defines CBM as being the process of cognitive bias change itself, then the very different conclusion must be drawn that CBM does reliably alter emotional vulnerability. Thus, although procedures adopted to date by experimenters with the intention of modifying cognitive bias have not consistently induced this target process of cognitive bias change across past studies, the process cognitive bias change has proven to reliably alter emotional vulnerability across past studies. In the following we briefly consider the implications of these findings.
Implications and future directions
The pattern of findings revealed by our present analysis invites a more parsimonious and cogent account of previously noted cross-study inconsistencies in the observed emotional impact of intended CBM procedures than the account proposed by Cristea et al. Cristea and colleagues simply attribute positive findings to unspecified ‘experimenter effects’. Instead, our analysis reveals that such positive findings are the norm when the intended CBM procedure successfully elicits the process of modifying cognitive bias. Hence, inconsistency in the observed emotional impact of putative CBM procedures can plausibly and parsimoniously be attributed to cross-study variability in whether intended CBM procedures succeed in modifying the target cognitive bias.
Next, consider Cristea et al's contention that changes across the past few years in the reported impact of intended CBM procedures on emotional vulnerability should be attributed to publication bias. Publication bias is an important issue across all sciences. However, there is a simpler potential explanation for Cristea et al's observation that early CBM studies, which virtually always delivered the intended bias modification procedure within well-controlled laboratory or clinical research settings, tended to obtain effects on emotional vulnerability more often than has been the case across more recent studies. Specifically, in these recent studies the intended bias modification procedures have more often been delivered online via the internet to participants, who complete them in their unconstrained, highly variable, home environments. Hence, reductions in the degree to which emotional vulnerability has been influenced by intended CBM procedures, across recent years, may in part reflect a reduction in the degree to which these procedures have successfully elicited the intended process of cognitive bias change, when delivered in real-world environments with complex additional variance, rather than in more controlled and consistent settings.
A recent review lends support to this account, by showing that the use of online methods to home-deliver procedures intended to modify attentional bias has increased nearly threefold since 2010. However, such online procedures have successfully elicited the intended process of attentional bias change in only 11% of studies, as opposed to 74% of studies in which these procedures are delivered in the controlled laboratory or clinical research setting. Reference Macleod and Clarke39 A similar pattern is mirrored in the subset of studies reviewed by Cristea et al. Specifically, the proportion of these studies in which the procedure intended to modify attentional bias has been remotely delivered into the home environment has increased by a factor of three since 2010. These remotely delivered procedures have failed to successfully elicit the intended process of attentional bias modification in every study, whereas this process has been successfully elicited in 60% of the studies in which these procedures have been delivered within the controlled laboratory or clinical research setting. Thus, although the process of attentional bias change has consistently served to alter emotional vulnerability, the prospect of successfully eliciting this attention bias modification process has declined in recent years, as investigators have progressively adopted more ambitious but less rigorous approaches to the delivery of the procedures intended to evoke this CBM process.
The present observation that successfully inducing the process of CBM reliably influences emotional vulnerability, but that procedures intended to elicit this process of cognitive bias change do not always successfully do so, underscores the central challenge that we believe should be the focus of future CBM research. Investigators must systematically delineate the procedural factors that influence capacity to successfully induce the intended process of CBM, and use this knowledge to develop more powerful procedures that can more reliably bring about this process of cognitive bias change. This compelling conclusion sits at odds with Cristea et al's contention that the development of new CBM techniques will hinder research progress within the CBM field. This discrepancy may appear puzzling, until one remembers that the question Cristea et al have sought to address is not whether the process of CBM influences emotional vulnerability, but rather is whether procedures motivated by the intention of evoking the CBM process reliably influence emotional vulnerability, regardless of whether they elicit this intended process of bias change. The present negative answer to Cristea et al's chosen question is most likely to remain valid if the range of CBM procedures remains narrow and unchanging. In contrast, the affirmative answer to the quite different question we have focused on in the present commentary, concerning whether the process of successfully modifying cognitive bias exerts a reliable impact on emotional vulnerability, suggests that the development and evaluation of new CBM procedures capable of more consistently eliciting this process should become one of the most important avenues for future CBM researchers to energetically pursue. Increasing access to the potential therapeutic benefits that can result from successfully inducing the CBM process will depend on refining intended CBM procedures to enable such procedures to more consistently and effectively induce this process of cognitive bias change. The procedures that ultimately prove most effective in this regard may be quite different from the presently limited set of available options, and indeed there is even evidence that some psychoactive medications may directly modify cognitive bias. Reference Harmer and Cowen40
In this commentary, we have distinguished two important questions concerning CBM research, which we do not believe are adequately differentiated in the recent meta-analysis published by Cristea et al in this journal. These questions are: (1) on average, do all the varying procedures that investigators have employed with the intention of modifying cognitive bias significantly influence emotional vulnerability?; and (2) does the process of successfully modifying cognitive bias significantly influence emotional vulnerability? We have emphasised the need to ensure that a negative answer to the former question is not misconstrued as representing a negative answer to the latter, and we are concerned that Cristea et al's meta-analysis could readily lead the unwary reader to make this error. The analyses reported in Cristea et al's paper invite a negative answer to the first question, but we caution that some of the claims and contentions made based on these analyses could encourage the unwarranted inference that the meta-analysis instead indicates that the process of modifying cognitive bias does not influence emotional vulnerability. This is not the case. By dividing the studies originally selected for inclusion by Cristea et al into those in which the process of CBM was, or was not successfully elicited, we have been able to demonstrate that when the process of CBM took place then this process was reliably accompanied by a change in emotional vulnerability.
To avoid confusion in the future, we suggest that the term ‘cognitive bias modification’ should be used to refer only to the process of cognitive bias change, rather than being employed as a label for any procedure intended to elicit this process regardless of whether it successfully does so. Such procedures should themselves each be referred to by a distinctive label (e.g. identifying published protocol), as this will assist reviewers in distinguishing the quite different procedures that investigators may have employed (or may yet develop) with the objective of inducing the process of CBM. By minimising the prospect of confusion in this way, future reviewers of this expanding literature will be able to contrast the relative capacity of alternative procedures to successfully elicit the intended process of cognitive bias change, while keeping this procedural issue distinct from questions concerning the emotional impact of successfully evoking this process of CBM. This will assist in the development of increasingly powerful procedures to more effectively elicit the intended bias change process to deliver the greatest therapeutic benefit. The resulting refinement of such procedures, to maximise their capacity to successfully drive the intended CBM processes, will best serve to establish the value of the contribution that CBM ultimately is able to make to the future treatment of anxiety and depressive disorders.
Funding
B.G. is supported by Australian Research Council Grant . C.M. is supported by Australian Research Council Grants , , and , and by a grant from the Romanian National Authority for Scientifc Research, CNCS–UEFISCDI, project number . E.A.H. is supported by the Medical Research Council (United Kingdom) intramural programme []. E.F. is supported by a European Research Council Advanced Investigator award (CogBIAS project REF: ).
eLetters
No eLetters have been published for this article.