Introduction
Antimicrobial resistance (AMR) is a One Health issue requiring the development of well-performing surveillance systems in the human, environmental and animal sectors, as stated in the World Health Organisation (WHO) Global Action Plan on AMR [1] and the European Union (EU) One Health Action Plan against AMR [2]. Currently, in the animal sector of the EU and European Economic Area, the European Food Safety Authority coordinates active AMR monitoring covering zoonotic pathogens and indicator bacteria isolated from meat samples at retail and healthy animals at slaughterhouses, with the perspective to protect consumers from AMR transmission through the food chain. This scheme does not cover AMR monitoring in bacterial pathogens of animals isolated in veterinary clinics and farms, i.e. where antimicrobials are used. Therefore, it does not take into account the need to protect animal health and well-being, as well as the need for efficient, resilient and sustainable food production systems. Similar to human medicine where most – if not all – surveillance systems are implemented in the clinical setting, it is of utmost importance to develop AMR surveillance in diseased animals as part of national action plans (NAPs) in Europe and beyond.
In this context, the EU Joint Action on AMR and Healthcare Associated Infections (EU-JAMRAI) aimed to evaluate existing surveillance systems for AMR in diseased animals in the EU, as part of a larger goal to study the feasibility of coordinated surveillance of AMR in animal health in Europe and to support the establishment of such systems in European member states without them. Indeed, such evaluations are essential to allow more transparent interpretation of outputs, more objective decision-making and resource allocation, as well as improvements in system design and enhanced acceptance of system outputs by stakeholders [Reference Peyre3]. In addition, evaluations can prove useful to other systems operating in different regions, as well as for the setup of new surveillance systems, through the identification of success and failure factors. Therefore, it seems essential to share evaluation outcomes. Here, we report the results of the evaluation of the French surveillance network for AMR in bacteria from diseased animals (RESAPATH), which is the oldest surveillance system of its kind in Europe at least in livestock (set up in 1982) [Reference Schrijver4].
Materials and methods
The RESAPATH network
RESAPATH performs passive (or event-based) phenotypical AMR surveillance in cattle, sheep, goats, swine, chickens, turkeys, rabbits, fish, horses, dogs, cats and exotic animals. It has a steering committee composed of representatives of public and private diagnostic laboratories, the ministry in charge of agriculture, veterinary professional organisations, as well as microbiologists and epidemiologists from the French Agency for Food, Environmental and Occupational Health & Safety (ANSES). RESAPATH is coordinated by two laboratories of ANSES, located in Ploufragan-Plouzané-Niort and Lyon, France. In 2017, it was composed of 71 volunteer public or private veterinary diagnostic laboratories collecting resistance data for 56 286 isolates from diverse specimens. From 2008 to 2017, it experienced an important development, with an increase of 39% in the participating laboratories and 211% in the data collected [5]. RESAPATH became an important component of the French NAP to tackle AMR in the animal sector, so-called ECOANTIBIO 2 (2017–2021) [6]. The objectives of RESAPATH, as defined in a mutual agreement between participating laboratories and ANSES, are to:
• follow AMR trends in pathogenic bacteria of animals;
• collect and store a panel of isolates that can be needed for in-depth molecular investigations;
• provide solid technical and scientific support to field laboratories;
• enable comparisons of animal and human AMR data through the French national observatory for epidemiology of bacterial resistance to antimicrobials (ONERBA), to which RESAPATH is federated.
The mutual agreement also defines the roles and duties of the coordination team (ANSES) and participating laboratories.
All member laboratories are required to perform antimicrobial susceptibility testing (AST) by disk diffusion according to the French norm NF U47-107 [7] and to interpret AST results according to the veterinary guidelines of the Antibiogram Committee of the French Society of Microbiology [8]. They are also required to send to ANSES all AST data (antibiotics tested and inhibition diameters) together with appropriate anonymised epidemiological data (geographical origin, animal species, age category, specimen, disease and bacterial species) every three months using a dedicated Excel® template. Upon request, laboratories also provide bacterial isolates with specific phenotypical resistance profiles for further molecular investigations (ANSES bearing shipping costs).
ANSES is committed to organise and financially support annual proficiency testing (PT) and continuous scientific and technical support to laboratories on the AST technique and results interpretation. ANSES is also responsible for the analysis of surveillance data, performed jointly by a team of epidemiologists and microbiologists, and is in charge of communication activities such as (i) editing a publicly available annual surveillance report, (ii) managing the RESAPATH website (https://resapath.anses.fr/), (iii) issuing a regular newsletter to the network and (iv) organising an annual one-day RESAPATH meeting where all participating laboratories and other partners are invited.
Evaluation method
RESAPATH was evaluated with the Outil d'Analyse des Systèmes d'Information en Santé (OASIS) [9, Reference Hendrikx10]. This qualitative method was selected as it enables standardised, detailed and comprehensive evaluations of the organisation and operations of surveillance systems. Due to its participatory approach with the professionals involved in the surveillance system under evaluation, it supports the acceptability of results and the uptake of recommendations. This method is also frequently used as the reference method of the French platform for epidemiological surveillance in animal health [Reference Marcé11]. OASIS is based on an evaluation grid composed of 78 criteria which are marked from 0 (lowest possible score) to 3 (highest possible score), according to a precise scoring guide (grid and scoring guide available on https://www.plateforme-esa.fr/article/l-outil-d-evaluation-oasis). Due to the generic nature of the OASIS method, only criteria relevant to RESAPATH were evaluated. Evaluators also provided commentaries including justifications for attributed scores and recommendations for improvement. It is commonly acknowledged that comments are of great importance and complementary to scores that assess the performance of a surveillance system. OASIS outputs are displayed in the form of three complementary figures on (i) 10 functional sections (defined according to the structure and activities of a surveillance system), (ii) seven critical control points (CCPs) which are the operations where improvement measures can be implemented and (iii) 10 attributes, i.e. measurable characteristics such as representativeness or timeliness which indicate the system's quality. In these figures, results are indicated as proportions of the maximum possible score.
Once the RESAPATH steering committee approved the principle of an evaluation of RESAPATH using the OASIS tool, a joint team was built, composed of two external assessors (not involved in RESAPATH activities) and two internal assessors (members of the RESAPATH coordination team) (Table 1). One of the two external assessors was experienced with the OASIS method, for having used it several times, and ensured its proper implementation. This team conducted a series of semi-open interviews from 11 June 2018 to 10 July 2018, either face to face or by phone (lasting between 1 and 1.5 h), with a panel of 23 representative partners of RESAPATH (Table 2). This panel was selected to represent all categories of actors or beneficiaries of the surveillance performed by RESAPATH. The representatives of field laboratories were selected to represent different geographic areas, animal species and volumes of data provided to RESAPATH. All professionals contacted by the evaluation team agreed to be interviewed. Before each meeting, the evaluation team prepared a series of open questions enabling them to score the evaluation criteria which are relevant to the position of the interviewed person. Assessors took notes manually during the interviews. Following these interviews, the two external assessors filled the OASIS grid (including commentaries), which was then discussed with the two internal assessors. When necessary, information was cross-checked between the notes of the four evaluators and some interviewees were recontacted to provide additional information or clarification. Then, the grid was sent for review to a panel composed of seven representative partners of RESAPATH (Table 3). At this point, the review panel and the evaluation team discussed the preliminary results of the evaluation during a full-day meeting on 17 July 2018. Some scores were adapted and additional comments were collected. When no consensus was reached at any stage of the evaluation process (a situation that rarely occurred), the point of view of the external assessors took precedence.
Table 1. External and internal assessors of the OASIS evaluation of RESAPATH in 2018
Table 2. Professional organisations and positions of the 23 RESAPATH partners interviewed during the OASIS evaluation of RESAPATH in 2018
Table 3. Professional organisations and positions of the seven members of the evaluation grid review group (in addition to the four assessors) during the OASIS evaluation of RESAPATH in 2018
Results
The three OASIS outputs, presented in Figures 1–3, show high scores (all but one above 67%). The lowest scores were obtained for the field institutional organisation, surveillance tools and communication among functional sections (Fig. 1), sampling and information distribution for CCPs (Fig. 2) and representativeness and timeliness regarding attributes (Fig. 3). The filled evaluation grid can be found in Supplementary Material 1, available on the Cambridge Core website.
Fig. 1. Scores of RESAPATH obtained for the 10 OASIS functional sections (indicated in proportions of the maximum possible score) in 2018
Fig. 2. Scores of RESAPATH obtained for the seven OASIS CCPs (indicated in proportions of the maximum possible score) in 2018
Fig. 3. Scores of RESAPATH obtained for the 10 OASIS attributes (indicated in proportions of the maximum possible score) in 2018.
The surveillance objectives were considered relevant (Figs 1 and 2) although some were not strictly speaking surveillance objectives and should be reformulated (Table 4). The surveillance scope of RESAPATH is very large, covering multiple combinations of animal species, production types (when relevant), specimens, bacterial species and antibiotics. However, limitations in human resources at ANSES combined with a decreasing budget over the years have prevented the integration of additional laboratories, significantly impacted communication and laboratory training activities, and reduced the proportion of samples subjected to molecular analyses. To enable these developments, an increase in financial and human resources in the coordination team was recommended (Table 4).
Table 4. Main recommendations for improvement following the OASIS evaluation of RESAPATH in 2018
The field institutional organisation is composed of veterinary clinicians and laboratories. Due to the presence of member laboratories in all French administrative regions, as well as a good geographical overlapping of distributions of animal populations and AST data [Reference Boireau12], the geographical coverage and representativeness of animal populations (Fig. 3) of RESAPATH was considered satisfactory, although not perfect. The sub-optimal score for the field institutional organisation (Fig. 1) was also due to financial and practical limitations in ordering an AST (such as delays in the processing of samples and obtaining results or less frequent sample pick up from veterinary clinics) and the simultaneous development of rapid AMR diagnostic tests directly performed by veterinarians and therefore not collected by the network. It was recommended to follow these developments carefully (Table 4).
The central institutional organisation defined simple and pragmatic surveillance procedures (Fig. 1) contributing to the good scores obtained for simplicity and acceptability (Fig. 3). However, the passive data collection entails possible sampling biases (Fig. 2) leading to sub-optimal representativeness of pathogenic bacterial populations of animals (Fig. 3). Indeed, animals may be more prone to being sampled in specific circumstances, such as previous treatment failures or chronic infections. Nonetheless, this hypothesis proved invalid in poultry production where Bourély et al. showed that AST was performed in nearly all cases of suspected infection [Reference Bourély13]. Therefore, we recommended quantifying such possible biases outside of the poultry sector and suggested a solution to address sampling bias issues (Table 4).
The score for surveillance tools was considered satisfactory (Figs 1 and 2) but negatively impacted by the lack of inclusion of roles and duties for veterinarians in the mutual agreement. Thus, RESAPATH has no control over the collection, storage and transfer of clinical samples and related epidemiological data (animal species, sample type, disease, location, etc.) from veterinarians to laboratories, bringing possible limitations (although never assessed) in terms of data quality, completeness and harmonisation. As such, laboratories could disseminate sampling and storing procedures to veterinarians, make a harmonised sample form readily available to veterinarians and raise awareness on the need to submit more complete epidemiological data (Table 4).
Participating laboratories are the basis of the system. They have strong methodological skills and the disk diffusion method is fully harmonised within the network and checked during the annual PT. All laboratories have the opportunity to undertake a free technical training to improve their skills in the AST method upon demand. These characteristics contributed to high scores in the training and laboratory functional sections (Fig. 1). However, interviews revealed that laboratories can be hesitant at asking for such training, so we recommended organising formal training sessions on fixed dates in parallel to convenient training on-site and uploading a video tutorial on the disk diffusion technique on the RESAPATH website (Table 4).
RESAPATH received a good score on its data management functional section (Fig. 1), but at the price of significant time-consuming processes for both the laboratories and the coordination team. Indeed, many laboratories do not have optimal information technology (IT) material. Some have to manually report their AMR data in the RESAPATH Excel® template, while others can make data extraction from their laboratory information management system, but not in the required format and codification. In the end, the coordination team receives a great diversity of file formats and had to develop a series of semi-automated solutions to convert laboratory files into the appropriate format to reduce the workload of data cleaning and harmonisation. In addition, most laboratories do not send their data every three months, leading to heavier workloads during some periods. This situation led to a sub-optimal score for data collection (Fig. 2) and timeliness (Fig. 3). However, all laboratories send their data at least once a year, enabling the analysis of the complete RESAPATH database each year. Another weakness is the existence of two RESAPATH databases in two different ANSES laboratories which are not integrated: one in Ploufragan-Plouzané-Niort with data from swine, chickens, turkeys, rabbits and fish and another one in Lyon with data from other animal species, mainly ruminants, equids and companion animals. This leads to a lack of efficiency in processes such as data cleaning, integration or analyses, which are carried out separately. Moreover, the database in Ploufragan-Plouzané-Niort is not as secure as the one in Lyon. Thus, data management was assessed as the main area for improvement of RESAPATH and the main cause for not accepting additional laboratories in the network. At the time of evaluation, ANSES was planning to set up a more efficient electronic data interchange where laboratories would upload their data on a dedicated website that includes automated data quality controls. This project may solve some of these issues and has become a recommendation from this evaluation too (Table 4). Specific attention should nevertheless be paid not to exclude small laboratories with limited IT capacities from the network. In the meantime, more frequent data submissions from laboratories should be encouraged (Table 4). Finally, it was recommended to merge both databases and keep the database management system of Lyon (Table 4).
Despite these weaknesses, RESAPATH manages to perform well in terms of data analyses (Fig. 2), with good scores for sensitivity, specificity and reliability (Fig. 3). Its capacity to conduct multi-disciplinary analyses due to skills in bacteriology, epidemiology, biostatistics and IT within the coordination team was considered a strong asset.
A particularity of RESAPATH, which drives its whole performance, is its capacity to develop a volunteer network of laboratories. Indeed, RESAPATH is not regulated by law, not supported by dedicated funds and relies on the participation of all members. According to the interviews, the main motivation for integrating RESAPATH was the so-called ‘free’ and high-quality technical support that laboratories receive. This support includes an annual PT which was recognised as a key benefit for laboratories. This PT has the double advantage of enabling laboratories to check and continuously improve their own performance and for ANSES to check the quality of the AMR data collected. Some laboratories also use their participation in this PT as a quality label for marketing purposes. In turn, laboratories agree to send AMR data and isolates of particular interest, allowing long-term AMR surveillance at the national level and providing material to researchers in microbiology and epidemiology at ANSES. This win–win approach, where ANSES and participating laboratories always need to meet each other's expectations, is fundamental to the success of RESAPATH and the basis of its strong acceptability, flexibility and stability scores (Fig. 3). Beyond acceptability, the RESAPATH community seems to have even developed a sense of belonging according to our interviews. Moreover, RESAPATH enables its member laboratories to gain a better recognition as indisputable actors in the fight against AMR in the animal sector.
Scores for communication and information distribution were lower than others (Figs 1 and 2) due to the absence of any newsletter distributed to the network. Issuing a newsletter on a regular basis is mentioned in the mutual agreement and is expected by laboratories, so we recommended direct efforts in this area (Table 4). However, the assessors also acknowledged the numerous and successful efforts of the coordination team to develop internal and external communication. It organises the annual RESAPATH meetings, frequently exchanges emails with laboratories, participates in the ONERBA and in numerous veterinary, epidemiology and microbiology events. It publishes scientific articles and the annual RESAPATH reports (available in French and English on its website) and develops a web application (using the Shiny package [Reference Chang14] of R software [15]) to make RESAPATH results more readily accessible to all, especially veterinary practitioners. In the end, we considered that RESAPATH succeeded in maintaining the momentum and motivation of all partners through its numerous communication and information distribution activities.
Regarding evaluation, RESAPATH received a high score (Fig. 1) due to the annual calculation and analysis of relevant performance indicators (listed in Supplementary Material 2, available on the Cambridge Core website) and the undertaking of two OASIS evaluations in 2010 and 2018, despite the fact that limited human resources prevented the implementation of some recommendations of the first evaluation. It was also suggested to add two performance indicators, one on compliance by laboratories on quarterly data submission and another one on the completeness of epidemiological data (Table 4).
Finally, RESAPATH proved to be a useful system (Fig. 3), which is the key to its sustainability. An illustration of this is its recent recognition as a key partner of the French NAP to tackle AMR. Indeed, RESAPATH enables the ministry in charge of agriculture to monitor the efficiency of the NAP and to address some urgent issues, such as in 2015 when a rapid assessment of the spread among animals of the newly discovered plasmid-mediated mcr-1 colistin resistance gene was required [Reference Haenni16]. Veterinarians also benefit from the regular production of epidemiological results, including numerous tables with resistance proportions for a large panel of combinations of animal species, bacterial species and antibiotics that can guide their treatment decisions. However, the frequency at which field veterinarians consult the RESAPATH surveillance reports and to what extent their content enables them to rationalise their prescription of antimicrobials was not assessed. The reports are also particularly useful to the regional reference veterinary practitioners for antimicrobials, whose role is to respond to technical questions from other veterinary practitioners on antimicrobial therapy, in the framework of the French NAP. In addition, some antimicrobial therapy guidelines produced by veterinary professional organisations refer to the RESAPATH reports [17, 18]. Finally, the regular improvement of the performance of diagnostic laboratories leads to the delivery of higher quality AST results to veterinary practitioners with likely clinical and economic impacts.
Discussion
To our best knowledge, it is the first time that an in-depth evaluation of an AMR surveillance network in animal health has been published. Performance of Veterinary Services evaluations and Joint External Evaluations, respectively coordinated by the World Organisation for Animal Health and the WHO, also provide online information on country capacities for AMR surveillance in animal health, but do not provide the same amount of details as in an OASIS evaluation [19, 20].
From this evaluation, key success factors for RESAPATH were identified and included:
• A strong central institutional organisation with the inclusion of many relevant actors in the animal sector defining clear surveillance objectives, scope and procedures that meet their expectations.
• Strong skills in epidemiology and microbiology at the central level, including the capacity to complement phenotypical surveillance with molecular surveillance.
• A collaborative win–win approach between member laboratories and the coordination team, leading to the voluntary participation of numerous field laboratories throughout the whole country.
• The provision of free technical support to laboratories including an annual PT enabling the production of high-quality and harmonised AST data.
• Strong internal and external communication with all possible end-users of surveillance data, including in the human health sector.
• Continuous efforts to assess its performance due to laboratory PT, performance indicators and OASIS evaluations, followed by the implementation of training and improvement measures.
However, this evaluation pointed out several areas for improvement. The most important one referred to data management and a lesson to learn would be to always consider scalability, a parameter that is often overlooked at the setup of a system. Taking into account the limited IT capacities of laboratories is key to maintaining a strong volunteer network, but this requires a lot of flexibility from the coordination team and can be very time-consuming. Another weakness of RESAPATH lies in its possible sampling biases, as a passive laboratory-based surveillance network that does not integrate the sampling stage in its procedures. However, these possible biases were not considered as having a major impact on representativeness in this evaluation. On the other hand, the current organisation of RESAPATH brings a lot of simplicity, which is key to its sustainability.
This detailed investigation of the strengths and limitations of RESAPATH provides valuable information to countries aiming to set up a national AMR surveillance system in diseased animals using a passive surveillance approach or to those wishing to improve their current system. It shows how participative win–win surveillance networks are of strong value for national authorities and can succeed in triggering highly positive collateral impacts, such as capacity building in veterinary diagnostics through greater skills acquired by all members of the network. Although this was not explored here, such systems are also expected to be able to operate at more limited costs compared to non-participative systems, which is a critical parameter. Veterinary medicine is structured similarly in other European countries, so we believe that passive and voluntary AMR surveillance systems in diseased animals could be relevant in other countries too. However, similar systems may only be set up in countries where sufficient AMR data are produced routinely by diagnostic laboratories, with samples sent by veterinary practitioners or farmers. Also, it may be less adapted to cover some food-producing animal species when most AMR data are produced within laboratories of large food production corporations, which may be less likely to share them. In any case, duplication of existing systems may not always be appropriate. On the contrary, a preliminary situational analysis and wide inclusion of all relevant actors (including local actors) are essential to design a system. Setting up a participative system remains a long-term process, where trust is key and needs to be built over time. The current performance of RESAPATH is the fruit of several decades of development. Of note, the major increase in the number of RESAPATH members in the last 10 years was directly correlated with an increase in time and efforts dedicated by ANSES to coordinate the network. Therefore, we may foresee similar successes for any participative surveillance system provided that key factors are there, i.e. trust and mutual commitment.
The OASIS tool proved successful with fruitful exchanges and a strong acceptability of its results and recommendations. Despite being a qualitative method, its detailed scoring guide limited the opportunity for subjective answers. At the time of writing, the implementation of some recommendations had already started with the organisation of fixed-date training sessions, the ongoing merging of the two databases and the development of the electronic data interchange between laboratories and ANSES.
However, OASIS remains a generic tool and some debates have occurred on the relevance and interpretation of some of its evaluation criteria in the specific case of a passive laboratory-based AMR surveillance system. For example, should the data collection be evaluated from the laboratory stage and/or from the veterinary stage? The review group succeeded in taking consensual decisions, but this can slightly hinder the comparability of subsequent evaluations if such decisions are not consistent in time (this was namely the case for several criteria between this evaluation and the one performed in 2010). We recommend having at least one external assessor being experienced with the OASIS method to advise such decisions, as it was the case during our study, and to record decisions for future evaluations. As a qualitative method, OASIS does not enable the quantitative measurement of performance attributes, such as sensitivity or timeliness of a system nor does it look at its economic efficiency. Moreover, it does not investigate multisectoral collaboration, something of particular value for One Health issues such as AMR. To find the most appropriate method, depending on the evaluation question and surveillance attributes to evaluate, assessors may follow the steps suggested in the RISKSUR EVA tool, a framework providing guidance in the planning, implementation and reporting of evaluations [Reference Peyre3]. Regarding multisectoral collaboration, a specific tool called ECoSur (https://survtools.org/wiki/surveillance-evaluation/doku.php?id=quality_of_the_collaboration) was recently developed to allow for an in-depth analysis of the organisation and functioning of collaboration taking place in a multisectoral surveillance system [Reference Bordier21].
Of note, a method specifically dedicated to the assessment of national AMR surveillance systems has been developed since 2015 by the Food and Agriculture Organization of the United Nations (FAO). It was based on the FAO Surveillance Evaluation Tool, itself inspired by OASIS [22], and on the FAO Laboratory Mapping Tool [23], adapted to assess the specific issues linked to AMR. This method uses the FAO Assessment Tool for Laboratories and AMR Surveillance Systems (FAO-ATLASS) [24], which consists of two complementary modules (laboratory and surveillance) covering the key components of a national AMR surveillance system in the food and agriculture sectors. It also includes a Progressive Improvement Pathway scoring system, designed to assist policymakers in prioritising actions for building reliable national AMR surveillance systems for the sectors assessed. As such, conducting an FAO-ATLASS assessment could complement our results by providing a more global picture of the performance of France in terms of AMR surveillance in both animal and environmental sectors.
Conclusion
Overall, RESAPATH exhibited good scores, proving that a well-performing participative surveillance system for AMR in diseased animals is a realistic option to be included in the frame of a NAP. The thorough description and analysis of its organisation and operations led to the identification of key success factors including (i) a strong and inclusive central institutional organisation defining clear and well-accepted surveillance objectives, scope and procedures, (ii) strong skills in epidemiology and microbiology and (iii) a win–win approach enabling the voluntary participation of 71 field laboratories and where a free annual PT plays a pivotal role. The OASIS evaluation also allowed the identification of areas for improvement and provided a series of recommendations. Some of them have already been implemented by RESAPATH, illustrating the usefulness of such evaluations. In the context where AMR surveillance systems should be set up or improved in veterinary medicine, these results are most likely very helpful to other countries designing or improving their own system. Because of their multiple benefits, such evaluations should be encouraged and their results shared at the European and/or global levels.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0950268821000856.
Acknowledgements
We are very grateful to all the professionals interviewed for this evaluation and/or participating in the full-day review meeting on 17 July 2018. We are also thankful to Pascal Hendrikx for his technical support on the OASIS tool and to Nicolas Keck, Michaël Treilles, Francesca Latronico and Béatrice Mouillé for their inputs regarding the FAO-ATLASS tool.
Financial support
The project EU-JAMRAI has received funding from the Health Program of the EU (2014–2020) under grant agreement No. 761296.
Conflict of interest
The authors have no conflict of interest to disclose.
Data availability statement
All data supporting the findings of this study are in Supplementary Material 1, openly available on the Cambridge Core website.
Open access
