For some people with psychosis, identifying and reducing violence risk can form part of a prognosis-focused approach, given that violence perpetration can fragment care and support networks and is associated with poorer functional outcomes Reference Cotton, Lambert, Schimmelmann, Filia, Rayner and Hides1 and stigma. Reference Pescosolido, Halpern-Manners, Luo and Perry2 It is also an adverse outcome with societal implications and economic costs. Reference Senior, Fazel and Tsiachristas3 First presentation to clinical services is a higher risk phase of illness for violence. Reference Youn, Guadagno, Byrne, Watson, Murrihy and Cotton4 Risk is typically assessed in an unstructured manner, however, and clinicians can lack confidence and vary in their subjective weighting of clinical information. Reference Whiting, Glogowska, Fazel and Lennox5 Resource-intensive assessment tools used by specialist forensic services are not feasible for non-forensic services, and so there is a lack of consistency and structured, practical support. Evidence-based scalable tools could improve assessment by augmenting and complementing clinical judgement. The Oxford Mental Illness and Violence (OxMIV) tool was developed with a focus on routinely available clinical information. Reference Fazel, Wolf, Larsson, Lichtenstein, Mallett and Fanshawe6 It has performed well in external validation using pragmatic clinical predictor definitions and routine data from UK early intervention in psychosis (EIP) services, across a range of performance metrics (area under the curve (AUC) 0.75, sensitivity 71%, specificity 66%, with adequate calibration after updating). Notably, the OxMIV tool compared favourably on measures of net benefit with unstructured clinical judgement. Reference Whiting, Mallett, Lennox and Fazel7 To be useful, however, such new tools need to be implemented in practice. This study aimed to begin to bridge this translational gap to clinical use for the OxMIV tool by using mixed methods to examine (a) the acceptability of the approach to clinicians, patients and carers, (b) the practical feasibility and uptake of the tool and (c) its perceived utility, potential impact and optimal role within clinical pathways.
Method
In two EIP services, we examined use of the newly implemented OxMIV risk assessment tool to support routine risk assessments over 12 months (July 2020–July 2021). Qualitative and quantitative data were collected concurrently and analysed separately in a convergent parallel design, Reference Creswell and Clark8 with integration at the level of interpretation and reporting to reflect on combined meaning Reference Fetters, Curry and Creswell9,Reference Fetters and Molina-Azorin10 (Fig. 1). Joint display of quantitative and related qualitative results in a table to form an integrated results matrix was done to support the generation of meta-inferences (derived from integrating findings from both modalities). Reference Guetterman, Fetters and Creswell11,Reference McCrudden, Marchand and Schutz12
Fig. 1 Outline of mixed methods study with concurrent qualitative and quantitative data collection. OxMIV, Oxford Mental Illness and Violence.
OxMIV tool
The OxMIV model, openly published, combines 15 categorical predictors (e.g. previous drug misuse) and one continuous predictor (age) Reference Fazel, Wolf, Larsson, Lichtenstein, Mallett and Fanshawe6 (Table 1) to calculate individual risk of violence perpetration within 12 months, presented as percentage risk and a categorical rating (low/increased). The OxMIV tool has been externally validated and updated (re-calibrated) using clinical data from UK EIP services to assess risk of violence perpetration resulting in police contact or involving a weapon or physical injury. Reference Whiting, Mallett, Lennox and Fazel7
Table 1 Predictors making up the Oxford Mental Illness and Violence tool and aligned definitions
For the current study, the OxMIV tool was built into the risk assessment section of the electronic health record (EHR) for services in two English counties, Oxfordshire and Buckinghamshire (with a total catchment area of 1.2 million people). Pragmatic definitions for the predictors were specified (Table 1), complemented by a brief user guide. The OxMIV tool was assistive, not directive, that is, there were no directions with the risk score. Six of the predictors can be rated ‘unknown’ (leading to a risk range rather than a point estimate).
Quantitative methods
We investigated the experience and utility of the tool. Data on use over 12 months following introduction were retrieved anonymously. This examined uptake (number completed), point used in the clinical pathway (interval between referral and OxMIV assessment completion) and availability of information (completeness of assessments and interval between when each OxMIV assessment was commenced/completed). For each assessment, clinicians categorically indicated (a) whether they had ‘identified any needs related to violence risk’ before completing the OxMIV assessment, and whether they deemed these needs ‘higher than for a typical patient’ in their service, (b) whether the OxMIV tool had ‘assisted in the overall assessment of needs in this case’ and (c) whether the OxMIV tool had ‘prompted any changes to careplans’. Fisher’s exact test Reference Fisher13 was used to test for associations between categorical ratings. Whether the mean number of unknown predictors per assessment varied by pre-identified needs around violence was examined using Welch’s two-sample t-test. Reference Welch14 Quantitative analyses were undertaken with R version 4.1 for Windows (R Core Team, Vienna, Austria; www.R-project.org/), 15 and visualisations with package ggplot2. Reference Wickham16
Qualitative methods
Data were collected using semi-structured individual interviews to examine the overall experience of the tool. Clinicians were recruited from the same EIP services using purposive sampling. Reference Ritchie, Lewis, Nicholls and Ormston17,Reference Palinkas, Horwitz, Green, Wisdom, Duan and Hoagwood18 Eligible clinicians were clinically qualified, with some role in violence risk assessment/management, who worked within the service during the relevant period. Target quotas were set according to professional background and duration of clinical experience. Recruitment continued until sufficient information power was achieved to address the study aims. Reference Malterud, Siersma and Guassora19 Participants provided written informed consent.
Patients and carers were recruited from the same two services. Eligible patients and carers were male or female, aged 14–65 years, who were able to give written informed consent for participation (or written parental consent if aged 14–15 years), take part in an interview in English, were deemed suitable to participate by their usual clinician and had been involved in any reviews, assessments or discussions of risk subsequent to July 2020. Patient and carer participants received a £10 online shopping voucher.
For all participants, semi-structured interviews using topic guides were conducted by D.W., with length capped at 60 min. Transcripts were imported into NVivo 12 for Windows (QSR International; www.lumivero.com). 20 Data were analysed thematically. Idea-by-idea open coding was built into wider categories that were refined as data were added and developed into themes discussed with the research team. Reference Ritchie, Lewis, Nicholls and Ormston17 Analysis was informed by the constant comparative method Reference Boeije21 whereby collected data iteratively informed ongoing data collection.
Public and patient engagement
A public and patient advisory group of five individuals with personal or carer experience informed study design, including topics for interview schedules, and reviewed participant-facing documentation
Ethical considerations
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2013. All procedures involving human participants/patients in the OxMIV tool qualitative acceptability study of clinicians, patients and carers were approved prospectively by the West Midlands – Solihull Research Ethics Committee (reference 20/WM/0011). This was nested within a local project to improve violence risk assessment and documentation by making the OxMIV tool available within the EHR, approved by Oxford Health National Health Service (NHS) Foundation Trust’s quality and audit team. Quantitative data related to patterns of use of the OxMIV tool were obtained anonymously using the UK Clinical Record Interactive Search (UK-CRIS) system, approved by the local independent CRIS oversight group. Qualitative methods embedded core principles outlined by the Economic and Social Research Council.
Results
In total, 141 OxMIV assessments were completed in 12 months (Supplement 1 available at https://doi.org/10.1192/bjp.2024.293). In n = 51 (36%), the clinician identified ‘any needs’ around violence risk before OxMIV assessment, and in 27 (19%) these were deemed ‘higher than average’. In parallel, 20 clinicians were interviewed from across professional backgrounds (Supplement 2). Six hours of interview dialogue was transcribed and analysed (average interview length 19 min). Clinician data were analysed across ten themes (Supplement 3). Twelve patients and carers were also interviewed (nine patients and three carers, Supplement 4), from which 4 h of dialogue was analysed (mean interview length 18 min). There were difficulties in recruiting patients with direct experience of the OxMIV tool, so interviews were amended to involve a more general discussion of the OxMIV tool.
Integrated qualitative and quantitative findings
Table 2 presents integrated results where quantitative and qualitative findings examined the same clinical issue. To align with the study aims, these are considered together across the domains of (a) uptake, reach and role, (b) acceptability and availability of information and (c) utility and impact. Additional quotes are presented by theme in Supplement 3.
Table 2 Integrated results matrix combining qualitative and quantitative results examining the same clinical issue
OxMIV, Oxford Mental Illness and Violence; EIP, early intervention in psychosis.
Other qualitative findings
Qualitative data also provided insights that were not directly examined by quantitative data.
Interpretation of output
Clinicians typically described using the categorical and probabilistic outputs of the tool in combination. The percentage score was a novel approach for most but was deemed easy to interpret. Several clinicians described this as more richly informative, by showing the magnitude of risk differences between individuals:
‘…low or moderate…is just one jump, whereas to look at a number…compared to somebody else you might’ve done the day before, it does show the big difference in the risk I think when…you’re not just jumping from one category to another. You’re looking at the difference in score completely…’ (P19, occupational therapist)
Stigma and labelling
Inadvertent stigmatisation or labelling was not found to be a concern. Many clinicians felt that whether risk was assessed formally with a tool or in the standard unstructured manner made no difference to such issues. The intended purpose of the tool for positive input and support was highlighted. One clinician was concerned that a tool focused on violence brought this to the forefront when not always relevant, and that integration within the general risk assessment processes could help this:
‘…if you’re doing a risk assessment then you’re already starting to…classify levels of risk and whether you…do it in the OxMIV way…with some numbers or you don’t…I don’t, personally…think it makes that much difference and… what we’re trying to do is…to be…as aware as possible of levels of risk so that we can help manage them…that’s potentially very much in the patient’s benefit if we can help them…manage their risks and not end up in trouble…then that’s gonna help them as well as others.’ (P15, psychiatrist)
Patient and carer perspectives
Patients and carers expressed positivity and thought the approach could improve openness. None would have objected to being involved in completing it during their assessment:
‘If you are feeling hesitant, a structure is always a useful way forward, isn’t it? And it also kind of opens up…other situations where there might be aggressive behaviour like…drugs or alcohol…So I think it’s good. It’s good to get it out in the open and it will cover more specific detail than somebody might disclose at the time…’ (P9, carer)
‘[OxMIV] looked pretty good…it’s easy to use, it was straightforward, the questions weren’t too invasive…they were put nicely.’ (P8, patient)
As highlighted in the integrated results, it was important to several participants that the assessment led to a helpful change in management. For patients, this was the aspect that they most wanted to be involved in, and carers highlighted that they also needed to be included. One carer described how worrying about potential violence was a stressful unknown around illness onset, and that the OxMIV tool could have a role in communicating the often-low magnitude of risk to families. This was in the context of all carers having some experience of feeling sub-optimally informed in their family member’s care, including risk, because of boundaries around confidentiality. One carer recounted a positive experience of interacting openly with services, where a frank discussion of violence risk was helpful in supporting the family during a crisis.
Discussion
This mixed methods study examined the first use of a novel assessment approach (OxMIV tool) to support clinical violence risk management in 141 first-episode psychosis patients. Quantitative data on utility and patterns of use was complemented and expanded by qualitative interviews of 20 clinicians. Acceptability was further examined through qualitative interviews with 12 patients and carers. The integration of these methods yielded a detailed understanding of OxMIV tool adoption, its feasibility and acceptability and how the tool meets an identified clinical need to support risk assessments. As well as addressing the issue of why and how clinicians may choose to use the OxMIV tool to assist their decision-making, this work has implications for the clinical integration of other novel scalable digital tools, and how to study their implementation at scale in psychiatric settings.
Acceptability and feasibility of the OxMIV tool
The OxMIV tool achieved good reach, being used in around 65% of new assessments. Clinicians found it straightforward, and it did not add significantly to administrative burden. Required information matched well with what was clinically available, with only family history items missing in many cases, reflecting clinical practice in not asking about this in standard assessments. Categorical predictors were well received. One reported cognitive effort for clinicians was considering diagnostic thresholds for previous drug or alcohol use disorders. This could be clarified in guidelines for the tool.
The OxMIV tool was also broadly acceptable from a service perspective. The balance between resource and benefits was thought to be favourable. There was also statistical concordance between the tool and clinical judgement, which was important for acceptability. Patients and carers stated that using a structured approach to improve assessments was acceptable, and could help with the openness of risk assessment.
Uptake and barriers to use
Several findings were relevant to the resources and process required to integrate even a simple digital tool. There was a lag of 3–4 months between the tool’s availability and it being adopted to a notable degree. Clinicians highlighted how it was a novel addition for a non-forensic service. Another issue was varying awareness of whether it should be part of routine risk assessment for all, or only for individuals with significant forensic history. Promoting understanding required repeated communication, with staff turnover an additional challenge. Also, whilst the tool’s interface within the EHRs was acceptable, many thought that it needed fuller integration within general risk assessments. Among clinicians who did not use the OxMIV tool frequently, however, the main reason was that violence risk was a low clinical priority for their individual patient group.
One theme previously cited as a barrier to clinicians raising the topic of violence risk is stigma. Reference Whiting, Glogowska, Fazel and Lennox5 However, in the current study, stigma was not felt to be a barrier to use of the OxMIV tool. There was consensus that this was no more a concern with the OxMIV tool than with other methods of assessing risk. Clinicians felt that uptake was correlated with the emphasis placed on violence risk within the team, which while important, is not considered their primary function.
Utility and impact
Feedback from clinicians on a case-by-case basis was that the OxMIV tool assisted assessment in most cases, more so when there were existing concerns around violence risk. Some clinicians also found the reassurance of confirming low risk helpful. A clear finding was that helpfulness did not depend on the OxMIV tool altering the clinical view of risk. Clinicians spoke more around how it clarified and focused assessment, and that the percentage score provided richer information than categorical ratings by highlighting differences in the magnitude of risk. There were also broad benefits to confidence and knowledge. There was more limited impact in terms of the OxMIV tool prompting additional clinical support, although this was more common in those at highest risk, of whom around one in five had some extra clinical input (such as regarding substance misuse) incorporated following assessment.
Clinical role and moving towards linked interventions
The OxMIV tool was preferred as part of the initial assessment, at first assessment and/or during the early period of contact with a care coordinator. By integrating it here, the OxMIV tool aligned with the information gathering and documentation already completed. This also aligns clinically in that people at first assessment are unknown to the service and are more likely to be actively unwell, making risk considerations timely. A secondary role was for refining risk assessments for those already known for whom there were concerns about violence risk. Clinicians saw less value in its use for those already known to them where risks were low, aside from providing a summary when documents had become unclear.
A strong theme from interviews with carers was feeling ‘in the dark’ around treatment because of confidentiality barriers, and carer participants were supportive of a tool as a way of framing their inclusion in conversations around risk. Rather than focusing solely on elevated risk, there was an identified role to reassure families where risks are low.
In line with their general approach to risk assessments, clinicians regarded the role of the OxMIV tool to assist formulation following assessment, rather than a tool to actively complete with patients. Some thought that in theory it could be used more directly, and patients and carers thought this would be acceptable. However, a more typical view was that collaboration could more helpfully be focused on developing a support plan in response to the assessment, rather than the risk assessment itself. This finding is in the wider context that collaboration in risk assessment is also limited in other settings, including forensic settings where it is a research priority. Reference Ryland, Davies, Kenney-Herbert, Kingham and Deshpande22
The limited impact of the OxMIV tool on what support clinicians offered is in keeping with previous studies of prediction models that suggest an assessment tool has less impact on its own than when presented with linked therapeutic recommendations. Reference Kappen, Vergouwe, van Wolfswinkel, Kalkman, Moons and van Klei23 An example of this is a trial to reduce suicidal behaviour in people presenting to emergency departments with suicidal ideation or attempts that compared treatment as usual, universal risk screening and risk screening plus an intervention. Reference Miller, Camargo, Arias, Sullivan, Allen and Goldstein24 Outcomes were only significantly better than treatment as usual in the intervention group. Similarly, in a cluster randomised controlled trial in forensic psychiatric out-patient settings, risk assessment alone did not reduce reoffending. Reference Troquete, van den Brink, Beintema, Mulder, van Os and Schoevers25 Therefore, to directly influence interventions to reduce risk, the OxMIV tool will require linked treatment pathways. The acceptability of this was discussed, with clinicians responding positively, with the caveats that such guidance should remain suggestive rather than directive, to avoid care becoming protocolised. Further, patients and carers placed high value on the assessment leading to something helpful, and highlighted this as the aspect they most wanted to be directly involved in.
Limitations
It proved difficult to recruit patients to interviews who had a significant violence history. Such individuals may have a different perspective on the issues explored. This difficulty seemed to mirror the wider perception by clinicians of violence being a sensitive topic, reported elsewhere, and they were perhaps more hesitant to invite those for whom it may be personally sensitive.
Work took place during the COVID-19 pandemic, which meant interviews were undertaken remotely, although conversely this may have facilitated discussion of more challenging topics. The service was also operating in a different fashion to usual. Finally, the researcher undertaking interviews became known to some of those in included services by working clinically during the project. To mitigate any risk of bias in interview responses, it was clarified at interviews that whether views of the tool were positive or negative had no implications for that researcher.
Implications for future research
There are two main implications for future research. First, the study has shown the relevance of considering a prediction tool as a complex intervention in how it interacts with clinical systems, Reference Greenhalgh and Papoutsi26,Reference Petticrew27 given the flexibility with which clinicians may integrate it, the range of behaviours that can be affected and the importance of systems and senior leadership for embedding sustainable use within a service. For example, in this study clinicians used the OxMIV tool in different ways, from a tool to summarise documentation and communicate with colleagues, to a way to discuss substance misuse with patients. They also described a range of impacts including on their wider practice, confidence and knowledge. Drawing on aspects of the framework for developing and evaluating complex interventions Reference Skivington, Matthews, Simpson, Craig, Baird and Blazeby28,Reference Craig, Dieppe, Macintyre, Michie, Nazareth and Petticrew29 will therefore be important for future research on the clinical translation of prediction tools like the OxMIV tool, particularly in moving towards linking therapeutic interventions.
Second, this is a demonstration of how mixed methods can be used to examine clinical use of a prediction tool. In turn, this can also inform clinical translation work in psychiatry. This is a key area of need in the field, where a large gap remains between models developed and those translated into clinically impactful tools. Reference Cavanagh, Deligianni, Fenton, Gkoutos, Lee and Leighton30 A novel aspect developed here was harnessing the potential of EHRs Reference Oliver, Arribas, Perry, Whiting, Blackman and Krakowski31 to collect use-by-use structured feedback on utility and data on the patterns of use, and combining this with qualitative interviews. Integrating these sources of data provided clear added value and facilitated a more detailed understanding than any of these approaches would have provided in isolation. The contemporaneous use-by-use feedback also provides a perspective that is free from the recall bias that distal surveys may be prone to.
Future for the OxMIV tool
For a prediction tool to be of potential clinical use, three broad areas need consideration: (a) addressing an identified clinical need; (b) the tool is sufficiently accurate in the target clinical setting; and (c) the tool is acceptable and feasible in that setting, with a defined clinical role. For the OxMIV tool, there is now evidence for predictive accuracy, acceptability and feasibility in clinical services for early psychosis. The current study substantially increases understanding of how the tool could be integrated clinically to address the challenges identified by previous work, and prevent violence outcomes, Reference Whiting, Glogowska, Fazel and Lennox5 beyond simply increasing accuracy compared to unstructured judgement alone Reference Whiting, Mallett, Lennox and Fazel7 (Table 3). Future research should therefore focus on implementation studies as part of stepwise progress towards a more definitive trial or observational study of impact. For planning such work, the current study identified three specific considerations. First, a run-in period of several months is required for adoption, which needs inclusion in study designs. Second, a clearly specified role, such as for all new assessments, would need to be established by a sustainable process such as embedding within standard mandatory documentation. Once it is established within a team, use with 70% of all new assessments would be a realistic target. Finally, measuring impact of a risk assessment tool on a clinical service will likely need to consider the importance of outcomes that are more proximal than, for example, reductions in violent offending. This study has identified that examining the provision of specific additional support to reduce risk would be a candidate outcome. The importance of aspects such as team systems, administration and clinical leadership support were also well evidenced in the current study, and resources such as the Consolidated Framework for Implementation Research could provide a structure for future clinical implementation research. Reference Damschroder, Reardon, Widerquist and Lowery32
Table 3 Challenges of clinical violence risk assessment and possible solutions
OxMIV, Oxford Mental Illness and Violence.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1192/bjp.2024.293
Data availability
Qualitative data that support the findings of this study (in the form of exemplar quotes) are available in the main text and supplementary material of this article. Additional qualitative data that support the findings of this study are available from the corresponding author, D.W., upon reasonable request. Quantitative data for this work is owned by Oxford Health National Health Service (NHS) Foundation Trust using anonymised patient records via CRIS powered by Akrivia Health. The data cannot be made publicly available but can be accessed with permissions from Oxford Health NHS Foundation Trust for UK NHS staff and UK academics within a secure firewall, in the same manner as the authors.
Acknowledgements
The authors are grateful to the clinicians, patients and carers of Oxford Health NHS Foundation Trust’s EIP services for their engagement with the work. We would also like to acknowledge the work and support of the Oxford Research Informatics Team, Tanya Smith, Head of Research Informatics, Adam Pill and Suzanne Fisher, Research Informatics Systems Analysts, and Lulu Kane, Research Informatics Administrator.
Author contributions
D.W., M.G., S.M., B.L. and S.F. conceived and designed the work; D.W. and D.M. acquired the data; D.W. analysed the data; M.G., S.M., B.L. and S.F. provided supervision; all authors contributed to interpretation of results; D.W. drafted the manuscript; all authors critically revised and approved the manuscript.
Funding
D.W. was funded by the National Institute for Health Research (NIHR) (Doctoral Research Fellowship DRF-2018-11-ST2-069). D.W., B.L. and S.F. were supported by the NIHR Oxford Health Biomedical Research Centre (BRC-1215-20005). This study was supported by CRIS powered by Akrivia Health, using data, systems and support from the NIHR Oxford Health Biomedical Research Centre Research Informatics Team. M.G. receives funding from the NIHR Community Healthcare MedTech and In Vitro Diagnostics Co-operative at Oxford Health NHS Foundation Trust. S.M. receives funding from the NIHR and from the UCL/UCLH Biomedical Research Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.
Declaration of interest
None.
Open access
eLetters
No eLetters have been published for this article.