Mental health service users and caregivers

In a systematic review of the literature, we identified several initiatives to increase the involvement of service users and caregivers in activities to strengthen the mental health system in LMICs, for example, in the areas of advocacy, quality control, training of health workers, policy development, service planning and research evaluation (Semrau et al. Reference Semrau, Lempp, Keynejad, Evans-Lacko, Mugisha, Raja, Lamichhane, Alem, Thornicroft and Hanlon2016). However, only four of the identified studies included an explicit evaluation of service user/caregiver involvement, and the methodological quality was considered to be low in most cases. Most evaluations focused on the impact of service user and/or caregiver involvement in the development of national level policies and plans, using qualitative methods or mixed qualitative-quantitative studies with service users, caregivers and service user representatives involved as subjects of research. An ecological study design examined the association between service user and/or caregiver involvement in development of mental health legislation and access to psychotropic medications, but such an approach is vulnerable to ecological bias (McBain et al. Reference McBain, Norton, Morris, Yasami and Betancourt2012). The other main group of evaluations focused on measurement of satisfaction with services, again with service users as subjects of research. Overall, there was a weak evidence base on the most effective models of building capacity of service users and caregivers to support involvement in mental health system strengthening in LMICs (Semrau et al. Reference Semrau, Lempp, Keynejad, Evans-Lacko, Mugisha, Raja, Lamichhane, Alem, Thornicroft and Hanlon2016). This is a missed opportunity because of the current global push to scale up person-centred and co-produced mental health services (World Health Organization, 2016a ). With better evidence, powerful arguments can be made to ensure that building the capacity of service users and caregivers to enable greater involvement in system issues (e.g., service development, monitoring and advocacy) become part and parcel of scale-up efforts.

Evidence from high-income country settings supports the use of participatory research with service users and caregivers for studies relating to mental health interventions, services and systems (Rose, Reference Rose2014), with limited examples from LMICs (Hann et al. Reference Hann, Pearson, Campbell, Sesay and Eaton2015). Potential benefits of involvement of service users and caregivers in the process of evaluation of capacity-building include ensuring that the goals of capacity-building are relevant and aligned with priorities, and that service users and caregivers are empowered through the process of co-production of care and knowledge. This leads to a greater chance of longer-term impact and sustainability of capacity-building efforts (Thornicroft & Tansella, Reference Thornicroft and Tansella2005; Rose, Reference Rose2014).

Service planners and policy-makers

A further systematic review was conducted to synthesise knowledge about evaluation of efforts to build the capacity of policy-makers and planners to strengthen the mental health system (Keynejad et al. Reference Keynejad, Semrau, Toynbee, Evans-Lacko, Lund, Gureje, Ndyanabangi, Courtin, Abdulmalik, Alem, Fekadu, Thornicroft and Hanlon2016). Rigorous evaluation was conducted in only a few of the identified studies. Evaluation approaches usually employed mixed methods. Lower quality evaluations were descriptive and not guided by any conceptual framework, whereas higher quality studies sought to combine various data sources, with one study making use of both an evaluator who was external to the programme, as well as an ‘insider’ evaluation (Patton, Reference Patton1997). Many of the capacity-building interventions for this target group involved long-term engagement and mentoring, with emphasis placed upon the need to develop sustainable, good quality relationships. Evaluation strategies need to incorporate these important indicators of a successful intervention, as well as capturing impacts on the health system.

Mental health researchers

Best practice guidelines for indicators of successful capacity-building for researchers in LMICs have been published (TDR/World Health Organization, 2011, 2016) and are better developed than the frameworks for evaluating capacity-building for other target groups. The ESSENCE framework recommends considering the impact of research capacity-building at the individual, organisational and sub-national or national levels, with emphasis given to understanding the country-specific relationships between these levels (TDR/World Health Organization, 2011). The selected indicators for measuring capacity-building success include a focus on publications and grants. The importance of publication as a hard outcome of capacity-building success has been echoed by others (Zachariah et al. Reference Zachariah, Reid, Van den Bergh, Dahmane, Kosgei, Hinderaker, Tayler-Smith, Manzi, Kizito, Khogali, Kumar, Baruani, Bishinga, Kilale, Nqobili, Patten, Sobry, Cheti, Nakanwagi, Enarson, Edginton, Upshur and Harries2013; Kohrt et al. Reference Kohrt, Upadhaya, Luitel, Maharjan, Kaiser, MacFarlane and Khan2014). The range of data sources that may be relevant and acceptable for evaluation of research capacity-building include: annual reports, mid-term and final interviews, publications, citation index, grant agreements, certificates of training and documentation of personal interactions (TDR/World Health Organization, 2011). A need has been identified for existing evaluative frameworks for capacity-building to be applied, tested and adapted for LMIC settings (Thornicroft et al. Reference Thornicroft, Cooper, Van Bortel, Kakuma and Lund2012).

Emerald evaluation of capacity-building for service users and caregivers

Low levels of literacy in many of the Emerald country sites meant that it was not possible to use self-completed questionnaires to assess the indicators of success. As well as the cost implications of interviewer-administered questionnaires, there was also concern that an interview format would lead to social desirability bias, for example, when measuring satisfaction with the capacity-building workshop. Another important adaptation for this target group was to evaluate change in understanding, e.g., ‘I understand about types of mental health problems’ (strongly disagree/disagree/don't know/agree/strongly agree), rather than change in knowledge.

Qualitative in-depth interviews were conducted at baseline to identify priority goals for service user and caregiver interventions and capacity-building needs (Samudre et al. Reference Samudre, Shidhaye, Ahuja, Nanda, Khan, Evans-Lacko and Hanlon2016; Abayneh et al. Reference Abayneh, Lempp, Alem, Alemayehu, Esehtu, Lund, Semrau, Thornicroft and Hanlon2017; Gurung et al. Reference Gurung, Upadhyaya, Magar, Giri, Hanlon and Jordans2017). Follow-up qualitative interviews are planned in order to explore perceptions of the impact of Emerald capacity-building upon service user and caregiver involvement in mental health system strengthening, the level of empowerment and mobilisation, the experience of participation in capacity-building, perceived limitations to the capacity-building approach and recommendations on how to improve capacity-building efforts.

Two of the Emerald-linked Ph.D. students (in Ethiopia and India) focused their research on service user involvement in mental health system strengthening. Although not possible at the cross-country level, the Ph.D. students are using participatory research methods to develop, pilot and evaluate models of service user involvement, with the evaluation of capacity-building as a nested component. See Table 3.

Table 3. Service user involvement in Ethiopia: participatory research for mental health systems strengthening

Emerald evaluation of capacity-building for service planners and policy-makers

The hierarchical nature of relationships with policy-makers or service planners and the research team were apparent across the Emerald partner countries, which affected the nature of the evaluation that was appropriate and possible. As with service users and caregivers, tests of knowledge were considered to be unacceptable for service planners and policy-makers who participated in the short courses and were replaced by pre-post questionnaires examining understanding of mental health systems. Attitudinal change was felt to be of paramount importance in this target group, but also a sensitive area and so attitudinal measures were not used in several of the Emerald countries.

Questions exploring the extent to which organisational capacity-building needs had been met were nested within in-depth interviews being conducted with key informant planners and policy-makers for other objectives of the Emerald project (Petersen et al. Reference Petersen, Marais, Abdulmalik, Ahuja, Alem, Chisholm, Egbe, Gureje, Hanlon, Lund, Shidhaye, Jordans, Kigozi, Mugisha, Upadhaya and Thornicroft2017).

Emerald evaluation of capacity-building for mental health researchers

The Emerald cross-country indicators for research capacity-building were adapted from the ESSENCE framework (TDR/World Health Organization, 2011). An important indicator of equity and sustainability for research capacity-building was the percentage of course participants who were working in public sector institutions. From the inception of Emerald, each of the country teams emphasised the importance of becoming self-sufficient in delivery of short courses and so an indicator was included to capture the number of courses delivered without external assistance. In the revised ESSENCE framework (TDR/World Health Organization, 2016), corresponding authorship by LMIC partners was emphasised, but for Emerald-linked Ph.D. students first authorship was considered to be essential, and in some of the Emerald LMIC institutions corresponding authorship had no value in terms of professional recognition and promotion opportunities. The indicators for obtaining grants were expanded to measure involvement in system-related projects, which may not have required external funding but provided an indicator of local uptake of the training.

The measurement of quantitative indicators was supplemented by qualitative in-depth interviews with key informants from the Emerald LMIC research partner institutions as well as with participants in the short courses. The interview topic guides explored the extent to which Emerald had contributed to successful capacity-building in mental health systems research, and what could have made the capacity-building efforts more successful. Perspectives on the experience of being part of a multi-country research consortium were obtained from Ph.D. fellows and mid-level researchers through an anonymous online survey. This included feedback on the experience of annual meetings, the extent and usefulness of opportunities to present their work and receive feedback from other consortium members, and the opportunity to be part of a network of Ph.D. researchers working in the area of mental health systems.