Introduction
From viral variants of concern, staffing and supply shortages, and noncompliance with public health measures to concurrent outbreaks and disasters, COVID-19 response efforts have encountered constant hurdles. Response typically functions in triage mode during initial surges of an emerging health threat. As soon as time allows, taking a moment to evaluate response efficacy to-date and prepare for what may lie ahead can prevent or reduce the impact of subsequent surges. Reference Mayigane, de Vázquez and Vente1
Intra-Action reviews (IARs) provide retrospective analysis of strengths and gaps during responses to strengthen readiness and resiliency. Reference Chamberlin, Okunogbe, Moore and Abir2 In contrast, tabletop exercises (TTXs) allow responders a chance to prospectively assess current levels of readiness and explore resiliency to potential future scenarios. 3,Reference Honhold, Douglas, Geering, Shimshoni and Lubroth4 During Action Review and Tabletop (DART), the methodology described here builds off strengths from both of these approaches into a single methodology incorporating retrospective and prospective participant-led analysis to improve both readiness and resilience to health emergencies. For the purposes of this paper, readiness is the ability to respond to an emergency in a ‘timely, efficient, and effective manner.’ Resiliency is the ability to effectively manage and recover from adverse events as they progress. Reference Chamberlin, Okunogbe, Moore and Abir2
Retrospective response evaluation
The World Health Organization (WHO) developed the IAR toolkit to empower localities to evaluate response during protracted health emergencies; in other words, ‘learning while doing.’ Reference Chamberlin, Okunogbe, Moore and Abir2 IARs include a facilitated discussion of trigger questions among participants representing the whole of a community and may include review of plans and root cause analysis. 5 14 trigger questions focus on different aspects of response including laboratory capacity, infection control, and case management. Once complete, a final report with recommendations is developed using a provided template with the option of sharing a success story. 5
The COVID-19 WHO IAR toolkit has been used in 33 countries as of early 2021. Reference Greiner, Nguyen, Shamout, Krishnan and Stowell6 A recent commentary noted the WHO IAR’s retrospective approach ‘does not sufficiently address this ongoing and protracted response planning,’ and recommended questions on prospective response planning. Reference Greiner, Nguyen, Shamout, Krishnan and Stowell6
Prospective planning
While IARs are designed to review ‘what has happened? or ‘what is happening?,’ TTXs are tools employed for preparing for the ‘what ifs?.’ TTXs use simulations of future scenarios to test response plans and enable participants to realize potential impacts of competing priorities or progressively worsening situations. This approach allows participants to understand how current plans ensure an effective, timely response and to identify areas for improving readiness in advance of a crisis. Reference Gary, Clauss, Bonbon and Myers7 For example, COVID-19 TTXs have been conducted in many countries to inform vaccine deployment, Reference Copper, de Vázquez, Bell, Mayigane, Vedrasco and Chungong8 assess the impact of lockdown measures, Reference Lo, Lin and Cheng9 support continuity of operations of schools and businesses, and address cybersecurity. Reference Wendelboe, Miller and Drevets10–13 Often, the After Action Review (AAR), which summarizes the TTX findings, is conducted soon after the TTX by an external evaluator. Participant feedback is included in the AAR report, but participants typically do not lead the AAR analysis. 14
Value of an integrated retrospective and prospective approach
DART is based on the hypothesis that a single methodology integrating both retrospective and prospective approaches using participant-led analysis will allow countries to improve readiness and resiliency of current and future challenges with COVID-19 and other health emergencies. DART’s multistep design intentionally enables participants to review data and identify key recommendations along the entire process.
Methods
To effectively assess response and planning both retrospectively and prospectively, DART incorporates5 unique steps (See Figure 1). DART is designed to be flexible and modular to allow for use across different types of health emergencies, with different groups of participants across varying geographies. In-country leadership co-design DARTs to ensure appropriate participation and realistic scenarios. Each step involves input from facilitators, in-country coordinators, and participants. Facilitators may be people within or external, to participating organizations with expertise in health readiness, and skills in facilitated discussion and TTX design. In-country coordinators are individuals who, through their position and expertise, can identify the optimal participants, provide expert input into scenario design, and assist with coordinating facilitated discussions. Participants are experts involved in the outbreak response at a level that allows them to be able to assess readiness and resilience from the perspective of their roles. In-country coordinators, as well as participants, may be responsible for sharing findings with appropriate leadership and policy makers. DART is designed to engage multiple sectors to capture gaps and strengths using a 1 Health lens, accounting for human, animal, and environmental health. DARTs have been conducted in 3 countries in 2021, focusing on COVID-19.
Figure 1. During Action Review and Tabletop (DART) methodology overview and key features.
Startx (Ending Pandemics, San Francisco, California, USA) (Ending Pandemics, San Francisco, California, USA),15 is an online tool for tabletop exercises.
Step 1: During action review questionnaire and discussion
The ‘During Action Review’ expands beyond the type of retrospective analysis IAR provides by using a participant-led approach to identify challenges that may complicate response in the future. Facilitators develop an initial questionnaire to gather information from participants in critical response roles. A modular questionnaire template includes specific questions for multiple, unique roles including animal health/ environmental health response, case management and countermeasures, epidemiology, and surveillance, as well as laboratory testing, coordination and emergency response, communications, and travel/ points of entry. In-country coordinators tailor the questions to explore capabilities and response needs related to specific roles during the COVID-19 response of 2020 - 2021; some questions reflected those of WHO’s IAR and others were developed by health readiness experts. Countries may include all or a subset of roles to focus on specific aspects of response that are of greatest concern.
An in-country coordinator reviews and revises the questionnaire to ensure it is culturally, politically, and operationally appropriate. Limiting the length of questionnaire to 1 hour is recommended to acknowledge that those involved have limited time. The questions are emailed in advance of the discussion session to allow participants to reflect and document responses in advance. 1 or more participants per role are selected by the in-country coordinator to complete the questionnaire based on their direct or advisory role to the response. In instances with multiple participants responding for a given role, all answers are collected even if contradictory.
Facilitators identify basic trends and organize findings for a video conference discussion of the review (During Action Review), generally limited to 2 hours. Using a participant-led analysis format, facilitators guide the discussion in which participants review, update, and prioritize findings of key strengths and gaps, and highlight future challenges which might further complicate responses. The future challenges are collected to inform DART’s next step, TTX Design.
Step 2: Tabletop exercise (TTX) design
Based on the initial findings from Step 1, a Concept and Objectives meeting is held among facilitators and in-country coordinators to: (1) create objectives for the TTX based on gaps identified, (2) determine which identified future scenarios of concern to incorporate in the TTX, and (3) establish a timeline for the future scenarios.
Analogous to Step 1, the TTX is designed from role-based perspectives to enable an accurate assessment of capabilities and to track cross-sector coordination and communication. As such, each participant receives information unique to their role in a response. For example, a laboratoryparticipant might be addressing biosafety challenges due to inadequate PPE and disinfectant supplies while a case management participant might tackle case overload due to multiple, concurrent outbreaks.
The TTX incorporates multiple scenarios in ‘phases’ looking at different advance time frames. For COVID-19 DARTs, these phases were short-term (1 - 3 months), medium-term (6 months) and long-term (2 - 3 years). Between each phase, participants conduct their own immediate analysis through questions designed to share lessons learned from the scenario just completed. Questions include: ‘What did you feel most prepared for?,’ ‘What did you feel least prepared for?,’ ‘What actions could be taken today to strengthen your ability to respond to this scenario?,’ and ‘Did the scenario highlight or bring to mind any other potential scenarios to plan for in the future?’
Step 3: Conduct TTX in STARTx
The multiplayer, multi-scenario TTXs are designed and conducted in the Scenario-based Tool for Assessing Readiness through Tabletop Exercises, Startx (Ending Pandemics, San Francisco, California, USA)), 15 a web-based platform that includes features to track communication among the different roles. Participants are provided with Startx (Ending Pandemics, San Francisco, California, USA) online tutorials and remote training sessions. Startx (Ending Pandemics, San Francisco, California, USA) communication tools allow for emails to be sent among participants to share their unique information or to ask questions. TTXs are designed to be completed in less than 90 minutes and may be administered asynchronously over days or weeks to allow for those actively engaged in a response to complete when they have available time. Participants are asked to answer questions and communicate with each other as they would during the actual response, based on current plans and protocols. In Startx (Ending Pandemics, San Francisco, California, USA), facilitators monitor progress by tracking communications and, with in-country coordinator support, send email reminders to ensure all roles complete the TTX. Multilingual capabilities in Startx (Ending Pandemics, San Francisco, California, USA) allow countries the option of conducting the TTXs in English or another language of their choice.
Step 4: TTX after Action review (TTX AAR)
Following completion of the TTX, participants are provided an immediate draft TTX After Action Review (AAR) report downloaded from Startx (Ending Pandemics, San Francisco, California, USA) that incorporates each role-based scenario, all decisions made by each participant, how/ when information was communicated among participants, and hotwash responses. Scenario overviews are shared prior to the TTX AAR discussion to ensure all participants have a common operating picture since they were each playing pieces of the scenario unique to their roles. During the discussion, facilitators guide participants to review overarching needs and recommendations for necessary levels of cross-sector communication, and hotwash responses. The Startx (Ending Pandemics, San Francisco, California, USA) tool captures communication between roles during the various phases, allowing for the assessment of inter- and intra-sectoral communication and collaboration. The TTX AAR provides a means for honing in on consistent needs, identifying critical gaps, setting priorities, and arriving at creative solutions necessary to address the recognized opportunities for strengthening readiness and resiliency.
Step 5: Comprehensive assessment report
Facilitators compile a comprehensive assessment report incorporating systematically collected data based on the participants’ findings from DART. In-country coordinators review and revise the report to ensure participant input is captured accurately. The report includes identified priorities and specific next steps critical for strengthening resiliency and readiness for the ongoing COVID-19 response and future health emergencies. Appendices to the report are included to share the specific questions and scenarios used within a given DART.
Participants are provided an online survey-based evaluation following DART to garner feedback on the approach and its impact on readiness activities using a Likert scale and open-ended text questions. Evaluation findings are used by facilitators to review opportunities to improve DART. Participants receive certificates upon completion of these activities.
Results
3 COVID-19 focused DARTs proved the hypothesis that integrating retrospective during action reviews with prospective tabletop exercises allow countries to improve 1 Health readiness and resilience for current and future challenges from COVID-19 and other health emergencies. Throughout each step, all participants joined and actively provided input into the qualitative findings.
During DART, participants identified a variety of concerning future scenarios. In-country coordinators and facilitators were able to seamlessly integrate these likely scenarios into the TTXs. DARTs resulted in both sector-specific and response system/ multisector recommendations. As a result of DARTs conducted, in-country coordinators shared the comprehensive assessment report to delineate needs for sustainable readiness with policymakers and external funding agencies.
DART evaluations have acknowledged that the methodology met the objectives to identify sustainable, effective ways to strengthen readiness and resiliency from a 1-Health approach. Almost all participants agreed or strongly agreed that DART allowed them to recognize critical, actionable needs and priorities for strengthening resilience to COVID-19 and to identify strengths and gaps in multisectoral coordination. They also agreed or strongly agreed DART helped them to identify current and best new practices to sustain for long-term readiness and resilience, and to identify ways to improve 1- Health response. Participants agreed and strongly agreed the TTX provided a way to work through future potential complications that may arise during the pandemic through realistic scenarios.
In open text questions, participants noted the most effective aspects of DART: addressing multisector coordination, the 1- Health approach, identifying challenges, discussion sessions, providing a baseline review, increasing community level of response, and providing realistic scenarios. Some participants identified the least effective aspects of DART to include challenges in identifying an effective approach to address a specific gap, non-participation at the same level, the fact that DART was virtual and not in-person, and the 1- Health approach.
A participant recommended improving DART’s multisectoral approach by increasing the number of sectors involved, such as wildlife, law enforcement, and environmental health. Another recommended holding separate breakout sessions for each sector. Participants recommended that exercises should be conducted on a regular basis; when possible, in-person with additional demonstrations and meetings should take place in advance of the exercise. Others suggested holding a session with experts to discuss specific mitigation measures, providing online guidance for addressing future pandemics in developing countries, providing case studies from other countries, and increasing the length of time for the DART process.
Time did not seem to be an issue, based on the feedback from participants; in fact, a few recommended additional exercises or more time to be spent on DART. Flexibility for online use allowed for effective implementation even during times of limited travel. Development of a user-friendly tool is critical for deployment in different cultures and systems Reference Haeberer, Tsolova and Riley16 ; participants did not note technical challenges using Startx (Ending Pandemics, San Francisco, California, USA) during DARTs.
Limitations
3 DARTs were conducted within 3 different countries, which limits the inferences that can be drawn from the methodology compared to if it were conducted in a greater number of countries or repeated for validation within a given country. Additionally, the long-term impact of DART can only be fully assessed several months or years after completion. DARTs in this publication were conducted virtually during COVID-19. If the DARTs were conducted in person or during a different outbreak, they may have a different level of participation with subsequently different results. In person, DARTs may also help build stronger relationships among participants. The depth of discussions may have been reduced due to time limitations as participants were also responders to the pandemic. Up to 3 scenarios were conducted for each DART; numerous additional scenarios could have been added if time availability allowed. As all DARTs were conducted in a multi-sector format, information bias may be present if sectors were not willing to completely share with other sectors. The authors encourage participants in-country to publish the results of their respective DART as appropriate for their country; as such, their results are not included within this paper that focuses on the methodology.
Discussion
Resilient health systems have been described as not only ready for the crisis at hand but able to maintain core functions, to be informed by the lessons learned during the crisis, and to adjust planning as needed. Reference Kruk, Myers, Varpilah and Dahn17 The combined retrospective and prospective aspects of DART address these resilience requirements. While individual TTXs and IARs inform planning, DART brings additional value by integrating a multistep participant-led approach to inform readiness and resiliency planning. DART’s scenario-based approach provides prospective analysis, which was recommended in recent commentary. Reference Greiner, Nguyen, Shamout, Krishnan and Stowell6 This allows for a unique, experiential opportunity to ‘play out’ the plans and assess cross-sector communication when faced with future unknowns.
From Steps 1-5, DART uses participant-led analysis to ensure it reflects on-the-ground realities and that solutions come from those who are responsible for or impacted by the results. Participatory analysis recognizes that those closest to the issues are often the best at identifying what has not worked and recommending solutions that are more likely to be adopted. Minimizing external evaluation and involvement may allow for increased candor among participants in some countries, due to potential security vulnerabilities in real-life and simulated incidents; so, it may be more appropriate for facilitators to be local. Reference Copper, Mayigane and Pei21 In addition to including participant-led analysis, DART follows a participatory-based research model approach in which in-country participating partners also play key roles in design and execution. In-country coordinators ensure the right questions are asked of the right people. Identification of appropriate participants leads to rich, in-depth dialogue regarding frontline response strengths, gaps, and recommendations. Full participation throughout all the steps has been shown to be possible, despite initial concerns that participants may not have enough time during an active response.
In light of the 1- Health nature of most emerging infectious disease outbreaks, participation among human, animal, and environmental health sectors during DART is essential for informing coordinated planning and joint operations. Reference Jones, Patel and Levy22 Multi-scenario, multi-role DARTs may help identify solutions for addressing challenges in timely communication and coordination given that outbreak response requires coordination across multiple disciplines. COVID-19 demonstrated, for example, how animal and environmental health entities can play roles in providing consistent community outreach and messaging, participating in laboratory and epidemiologic surge capacity, and engaging in surveillance. Reference Aguiar-Oliveira, Campos and Matos23,Reference Häsler, Bazeyo and Byrne24 DARTs provide opportunities for analysis of the integration of the multi-sector capabilities and performance in response, a key need for health emergency preparedness. Reference Copper, Mayigane and Pei21 Multi-sector DARTs also uncover compounding factors to be addressed such as concurrent animal health outbreaks that may limit surge support from the animal health sector.
As is typical in TTX design, COVID-19 TTXs generally incorporate multiple sectors for coordinated problem solving. Many TTXs, however, do not always provide different scenarios for each role to see how and if information from each scenario is shared among the unique sectors. Reference Copper, de Vázquez, Bell, Mayigane, Vedrasco and Chungong8–Reference Wendelboe, Miller and Drevets10,20 Role-based exercises provide not only a way to review performance by response sector and functional role (i.e., epidemiology and risk communications) but also to review communication and coordination among roles. Reference McNeil, Sanzero and Noel25 Both are critical capabilities in response.
Participants may opt to repeat DART during an extended response to assess progress and inform future planning. Sharing DART findings may be instrumental in helping countries learn new strategies for tackling similar challenges. DART does not currently link findings with the WHO’S IAR ‘success stories template;’ this could be a potential way to allow greater sharing of findings. Reference Mayigane, de Vázquez and Vente1 The possibility for a remote format is also conducive to multi-country or regional TTXs, to areas where travel is limited, and during outbreaks that impact the safety of in-person activities.
Conclusions
While looking back may help emergency managers see where response may have gone off track, looking ahead makes sure the tires are pointed in the right direction and bags are packed for whatever they may encounter down the road.
DART’s single methodology integrating both retrospective and prospective analysis allows countries to improve readiness and resilience, not only for COVID-19, but for any other health emergency. While DART has been implemented to inform ministry or provincial-level planning, its flexible format allows for effective use at any level, from a sub-national health facility, or laboratory to a multinational response. Multilingual options and remote functionality increase DART’s usability globally. The modular format is easily adapted to incorporate policymakers, law enforcement, transportation, or other sectors that play vital roles during response.
Over a year into the COVID-19 pandemic, countries continue to face ongoing challenges and will need to continually evaluate plans. WHO Global Preparedness Monitoring Board is encouraging countries to review plans and conduct exercises to strengthen readiness. 26 DART addresses these recommendations within a single methodology to ensure countries have systematically identified what has worked, what needs improvement, and what to start planning for on the road ahead.
Acknowledgements
We thank Ann Hammer and Matthew Sumner, Sandia National Laboratories for their expertise in development of the role-specific DART questionnaires; Jessica Rowland, Helene Walker, Alexis Joiner, and Vibeke Halkjaer-Knudsen, Sandia National Laboratories Global Chemical and Biological Security Group, for their contributions to DART; Theresa Wilson, Sandia National Laboratories, for coordinating Sandia’s role in implementing DARTs; Dr M Salim Uzzaman, Bangladesh Institute of Epidemiology, Disease Control And Research; Dr Syed Mursalin, CEO Pak 1- Health Alliance; Dr Dani Cohen and Dr Khitam Muhsen, Tel Aviv University, for leading their countries in implementation of the DART.
Authors contribution
Carrie McNeil conducted background research, created the initial concept, developed the methodology with Nomita Divi, and coordinated implementation of the DARTs. She also drafted the original manuscript. Nomita Divi, in addition to methodology development, also contributed to implementation of the DARTs and contributed substantially to writing of the manuscript. Mark Smolinski substantially contributed to the writing of the manuscript and provided review of the project throughout.
Funding
Ending Pandemics, a project of TIDES Center, a not-for-profit organization (501c3) in the United States, funded this effort.
Abbreviations
AAR, After Action Review; COVID-19, Coronavirus disease 2019; DART, During Action Review and Tabletop; IAR, Intra-action Review; Startx, Scenario-based Tool for Assessing Readiness through Tabletop Exercises; TTX, Tabletop Exercise; WHO, World Health Organization.
Open access
