Emergency services organizations routinely undertake internal quality improvement activities following training and operational activities. One form of a quality improvement activity is the After Action Review (AAR). An AAR is a facilitator-led post-event review where data is collected from individuals and groups of individuals using either surveys, focus group-like meetings, or individual interviews. Data is collected using a semi-structured data collection tool which is usually customized to the specific tasks being reviewed and the priorities and goals of the organization. The quality of AARs has been noted to vary significantly across organizations with reporting biases and differences between exercise or real event AARs and biases introduced by utilization of internal rather than external evaluators, and limited justification for data selection and collection rationales. The Australian Capital Territory Emergency Services Agency (ACT ESA) has identified that interoperability and integration between internal ACT ESA sub-agencies and externally with other agencies is a key problem. The aim of this study was to identify specific areas of consistent interoperability concern through the examination of historical AARs.

In this qualitative study a Framework Analysis of historical AAR text documents was conducted, following the standard five step process. This was followed by detailed thematic analysis in aggregate and examined longitudinally, with emergent themes undergoing further analysis.

AARs relating to major emergency response and training activities 2016-2022 were examined. There was a substantial variation in quality, level of detail, completeness and structure across the different AAR documents. Consistent themes emerging from the data highlighted concerns regarding inter-agency culture, uncertainty when conducting joint activities and its impact on decision making, logistics concerns, resourcing and breakdown of shared understanding and communication between teams.

Despite a centralized command structure, historical AAR analysis shows ACT ESA has had persistent operationally impacting interoperability and integration problems within the organization and with external agencies.