1 Introduction
The COVID-19 pandemic has posed major public health, policy and economic challenges to governments around the world. In navigating these challenges, governments have drawn on a range of scientific advice including from public health, infectious disease epidemiology, genomics, immunology, social science and mathematical sciences.
Australia and New Zealand have taken similar approaches to managing the pandemic, with both countries rapidly enacting strict border controls to minimize imported cases, and pursuing aggressive suppression through strict public health and social measures, until high vaccination coverage was achieved. Despite numerous border-related incursions of varying sizes [Reference Douglas, Geoghegan, Hadfield, Bouckaert, Storey, Ren, de Ligt, French and Welch15, Reference Grout, Katar, Ait Ouakrim, Summers, Kvalsvig, Baker, Blakely and Wilson27], community transmission of SARS-CoV-2 was eliminated entirely from New Zealand and from the majority of states and territories of Australia for long periods of 2020 and 2021 [Reference Baker, Wilson and Anglemyer5, Reference Golding, Price, Ryan, McVernon, McCaw and Shearer22, Reference Price70]. Both countries achieved high vaccination rates in late 2021, enabling them to adopt a revised COVID-19 strategy, progressively relaxing border restrictions and community interventions while keeping anticipated health impacts manageable. As a result, levels of pandemic mortality in both countries, whether measured through confirmed COVID-19 deaths or excess mortality, were among the lowest in the world.
Mathematical modelling has played a significant role in both Australia and New Zealand in informing the policy and operational response to the pandemic. Because of the extended period of time under elimination or strict suppression strategies, this has required the development of a different set of modelling tools to those used in countries where there were significant levels of community transmission continuously since early 2020. For example, there has been a focus on modelling the effect of border controls [Reference Shearer, Walker, Tellioglu, McCaw, McVernon, Black and Geard75, Reference Steyn, Plank, James, Binny, Hendy and Lustig81] and an intensive test-trace-isolate-quarantine (TTIQ) system [Reference James, Plank, Hendy, Binny, Lustig, Steyn, Nesdale and Verrall35, Reference Shearer, McCaw, Ryan, Hao, Tierney, Lydeamore, Ellis, Ward, Wood, McVernon and Golding72] as key interventions within an elimination or strict suppression strategy. There has also been a need for tools that provide situational awareness when prevalence is very low or zero. In these settings, traditional epidemiological data, for example, on reported cases, are insufficient to provide a comprehensive view of the epidemic situation and public health priorities [Reference Golding, Price, Ryan, McVernon, McCaw and Shearer22].
In this article, we will review some of the modelling tools used in Australia and New Zealand for policy advice and describe the impact these have had on the pandemic response. We will compare and contrast the approaches taken in the two countries, identifying strengths and areas where increased collaboration could potentially be beneficial. We will explore how the characteristics and requirements of those tools changed with the establishment of widespread community transmission following the emergence of the Omicron variants of SARS-CoV-2 in late 2021 [Reference Viana88]. We will also discuss how modelling groups have interacted with government officials and policymakers and identify some challenges and examples of good practice.
Our review and evaluations should not be considered systematic or exhaustive, but rather reflective of our own experiences as applied mathematicians working in leadership roles supporting our governments’ responses to COVID-19. The work described is the outcome of the dedication and efforts of a large number of mathematicians and statisticians collaborating with epidemiologists and public health agencies. Much of the work has been carried out by early career researchers (ECRs), often employed on short-term contracts. Although not the main focus of this article, consideration of the perspectives of ECRs, and the suitability of academic reward systems and funding structures for supporting the delivery of urgent policy advice have been acknowledged [Reference Kucharski, Funk and Eggo42, Reference Sherratt76] and are important areas for further reflection and action.
2 Modelling tools used in Australia and New Zealand
Australia has used short-term statistical and mechanistic forecasting models to provide regular weekly situational assessment advice to government [Reference Golding, Price, Ryan, McVernon, McCaw and Shearer22–Reference Golding26, Reference McCaw51, Reference Moss, Price, Golding, Dawson, McVernon, Hyndman, Shearer and McCaw55, Reference Price, Shearer, Meehan, McBryde, Golding, McVernon and McCaw69, Reference Price70] based on real-time data on cases, behaviour and vaccination coverage. Outputs were agreed upon through formal processes of government, documented in Australia’s National Surveillance Plan for COVID-19 [3], and a national consortium provided rapid internal peer-review of methods and outputs each week. These activities have been complemented by the periodic use of strategic scenario modelling for longer-term planning at key points in the pandemic. These include the onset of the pandemic in January 2020 [Reference Moss, Wood, Brown, Shearer, Black, Glass, Cheng, McCaw and McVernon56], and when Australia began its transition to “living with COVID” in mid-to-late 2021 based on achieving high vaccination coverage [Reference Conway9, 65, Reference Ryan, Shearer, McCaw, McVernon and Golding71, Reference Shearer, McCaw, Ryan, Hao, Tierney, Lydeamore, Ellis, Ward, Wood, McVernon and Golding72] (see Figure 1(a)).
Figure 1 Schematic diagrams showing key data requirements, workflows and stakeholder relationships for modelling groups in Australia and New Zealand. These are not official organisational charts, but rather a subjective representation of how some of the various modelling activities described in the text (central box) have used a range of data streams (bottom box) and interacted with government advisory groups (top box) at the data-modelling–policy interface. Abbreviations: DPMC, Department of the Prime Minister and Cabinet; MOH, Ministry of Health;
$R_{\mathrm {eff}}$
, effective reproduction number; ODE, ordinary differential equation; TP, transmission potential; TTIQ, test-trace-isolate-quarantine; VE, vaccine effectiveness. Panel (a) adapted from original produced by Dr Freya Shearer.
New Zealand has primarily used mechanistic transmission models, fed with real-time data on vaccination rates, to provide advice to government, particularly at key points when policy changes were considered or implemented. Where important parameters were uncertain or context-dependent, an approximate Bayesian computation (ABC) approach was used to calibrate models to epidemiological data. Models have been periodically updated as needed, for example, with the arrival of new variants, new data on measures such as vaccine effectiveness or when epidemic dynamics began to clearly deviate from model projections. Reports describing model assumptions and results were initially informally reviewed by a multi-disciplinary rapid peer-review panel and publicly released as pre-prints concurrently with the policy decisions they were designed to support (see www.covid19modelling.ac.nz/reports ).
The rapid peer-review processes used in Australia and New Zealand provided some level of quality assurance for results being used for policy advice in real time. This was complemented with traditional scientific peer review and publication over longer timescales. This allowed a robustly peer-reviewed methodological model base to be built up over time, which could be rapidly applied to different contexts and policy questions as the need arose.
Both countries drew on, and in places extended, analyses and/or models published by researchers in the UK’s Scientific Pandemic Influenza Group on Modelling (SPI-M-O) [Reference Barnard, Davies, Jit and Edmunds6, Reference Davies, Jarvis, Edmunds, Jewell, Diaz-Ordaz and Keogh12–Reference Davies14, Reference Keeling, Brooks-Pollock, Challen, Danon, Dyson, Gog, Guzman Rincon, Hill, Pellis, Read and Tildesley38, Reference Keeling, Dyson, Tildesley, Hill and Moore39, Reference Moore, Hill, Tildesley, Dyson and Keeling53] and international data and analyses published by the UK Health Security Agency and others [Reference Andrews2, Reference Golding26, Reference Herrera-Esposito and de Los Campos32, Reference Nyberg64, Reference Verity87]. In this section, we give a description of some of the key model developments and examples of how they were used in each country.
2.1 Australia
2.1.1 Early scenario modelling to support initial planning and response
In mid January 2020, Professor Jodie McVernon (Director of Epidemiology, Peter Doherty Institute) and Professor James McCaw (Head, Infectious Disease Dynamic Unit (IDDU), University of Melbourne) were engaged by the Australian Government Department of Health and Ageing to advise on the emerging threat from the virus now known as SARS-CoV-2. By the first week of February 2020, with few recorded cases globally and only preliminary information on the rate of transmission from early studies in Wuhan [Reference Kucharski43, Reference Li44], they had drawn on earlier work on pandemic influenza and SARS (that is, SARS-CoV-1) to develop a simple SEIR model to explore the potential impacts of COVID-19 in Australia. They used this to evaluate how measures such as case isolation, quarantine of contacts and modest levels of social distancing may suppress transmission.
Figure 2 Schematic diagram of the deterministic transmission model used in Australia in early 2020. Compartments are: susceptible (S), exposed (
$E_1$
,
$E_2$
), infectious (
$I_1$
,
$I_2$
), “managed” (M) and recovered (R). A proportion
$p_M$
of presenting cases (themselves a proportion
$\alpha $
of all infections) is ascertained and isolated (compartment M). Quarantined persons are shown as compartments with superscript q (dashed borders). Managed and quarantined compartments exert a lesser force of infection than nonmanaged, nonquarantined compartments. Full mathematical details, including the expression for the force-of-infection
$\lambda $
and process by which susceptible persons (S) are routed to the quarantine pathway (dashed borders), are provided in [Reference Moss, Wood, Brown, Shearer, Black, Glass, Cheng, McCaw and McVernon56, Appendix].
A key element of the early modelling work in Australia was a preliminary epidemiological investigation conducted in China [Reference Du, Xu, Wu, Wang, Cowling and Meyers16] which indicated that the serial interval distribution included probability mass at negative values. The serial interval is the time between symptom onsets in an infector–infectee pair and density at negative intervals is highly suggestive of pre-symptomatic transmission, now well understood to be a key feature of SARS-CoV-2 transmission.
The model included two key elements beyond the foundational SEIR equations (Figure 2). First, it included compartments for those who were contagious but not yet displaying symptoms, allowing the interplay between case-based isolation and pre-symptomatic transmission to be explored. And second, it modelled contacts, allowing for quarantine strategies to be evaluated. This rapid model-based risk assessment was completed by extending on earlier work [Reference McCaw and McVernon50] that introduced a method for modelling contacts in a simple ordinary differential equation (ODE) framework, which had laid the foundations for around 15 years of pandemic preparedness modelling in Australia.
The value of this early work [Reference Moss, Wood, Brown, Shearer, Black, Glass, Cheng, McCaw and McVernon56] was not in its quantitative findings. Rather, it demonstrated that case isolation and quarantine, particularly when combined with moderate social distancing, could achieve a valuable reduction in the rate of transmission and effectively, perhaps dramatically, “flatten the curve”. The work provided senior health officials with a clear reason to advise government to act. Measures that were considered feasible and appropriate—a modest reduction in contacts of approximately 25%–33% and enactment of quarantine and isolation policies—were expected to have a substantive, potentially game-changing, impact on the epidemic.
As history records, action was taken and by late March 2020—supported by strict border closures (also informed by model-based risk assessment [Reference Shearer, Walker, Tellioglu, McCaw, McVernon, Black and Geard75]) that slowed the rate of importation and strong (as opposed to the modest as modelled) social distancing measures enacted—transmission in Australia was in decline, leading to predicted elimination of local transmission [Reference Price70].
2.1.2 Real-time analysis, transmission potential and forecasts: COVID-19 situational assessment
Elimination of SARS-CoV-2 from circulation presented a major methodological challenge for infectious disease dynamics. Best practice for monitoring transmission is to estimate the effective reproduction number
$R_{\mathrm {eff}}(t)$
. At the heart of any method to compute
$R_{\mathrm {eff}}(t)$
is the renewal equation:
$$ \begin{align} \text{Inc}(t) = R_{\text{eff}}(t) \int_0^{\infty} \text{Inc}(t-\tau) f(\tau) \; d\tau, \end{align} $$
where
$\text {Inc}(t)$
is the incidence of infection at time t (that is, rate of new infectives) and
$f(\tau )$
is the generation interval density function. As equation (2.1) implies, there is no need for all infections to be recorded as cases: recording of a constant fraction of infections is sufficient to estimate
$R_{\text {eff}}(t)$
.
Evaluation of
$R_{\text {eff}}(t)$
—a key metric for risk-assessment—clearly requires established transmission and the recording of incident cases in health databases. With local (but of course, temporary) elimination achieved, there was no way to make such a risk assessment. However, the ability for a respiratory pathogen, such as SARS-CoV-2, to spread through the population depends on a number of behavioural factors—who we mix with and how we act during encounters.
In April 2020, with temporary elimination achieved, Professor Nick Golding from Curtin University worked in collaboration with members of the IDDU at the University of Melbourne to develop a novel metric, the “transmission potential” (TP), to provide risk-assessment in the absence of a computable estimate of
$R_{\text {eff}}(t)$
. The semi-mechanistic approach [Reference Golding, Price, Ryan, McVernon, McCaw and Shearer22] draws on behavioural data streams (and case data where available) to provide an assessment of risk during periods of zero, low and high transmission. Conceptually, this method can be considered a generalization of Bayesian methods for estimating
$R_{\text {eff}}(t)$
, where behavioural time-series data are used to compute TP, which may be interpreted as a time-dependent prior for
$R_{\text {eff}}(t)$
. In the absence of case data (because it is not available or there are no infections occurring in the community), we interpret that prior (TP) as an estimate of the ability of the virus, if it were present, to spread in the population. During periods of virus circulation, the methodology provides an estimate of
$R_{\text {eff}}(t)$
and the discrepancy between the prior (TP) and posterior (
$R_{\text {eff}}$
) quantifies the difference between the expected and realized transmission. This discrepancy is directly interpretable through an epidemiological and public health lens [Reference Golding, Price, Ryan, McVernon, McCaw and Shearer22].
The other major element of situational assessment is forecasting. In Australia, Dr Rob Moss (IDDU) and Professor James McCaw had worked in collaboration with Defence Science Technology Group (DSTG) and public health authorities for many years to establish a seasonal influenza forecasting capability. Time-series data from case-based surveillance systems were analysed with SIR-type transmission models using a Bayesian particle filter [Reference Moss, Zarebski, Dawson and McCaw57, Reference Moss, Zarebski, Dawson and McCaw58], with extensions to include evaluation of seasonality [Reference Zarebski, Dawson, McCaw and Moss91], healthcare-seeking behaviour [Reference Moss, Zarebski, Carlson and McCaw59] and the role of forecasting in public health decision-making processes [Reference Moss, Fielding, Franklin, Stephens, McVernon, Dawson and McCaw54, Reference Moss, Zarebski, Dawson, Franklin, Birrell and McCaw60]. Combined with US Defence-funded research in pandemic decision support [Reference Shearer, Moss, McVernon, Ross and McCaw73, Reference Shearer, Moss, Price, Zarebski, Ballard, McVernon, Ross and McCaw74], led by Dr Freya Shearer (IDDU), this provided the foundation from which Australia’s national COVID-19 forecasting capability was rapidly developed. By mid 2020, the national situational assessment consortium had established a suite of models for producing weekly month-ahead ensemble forecasts of COVID-19 case incidence. A team at Monash University contributed a time-series analysis statistical model and a team at the University of Adelaide produced a stochastic branching process model. In Melbourne, the EpiFX platform used for seasonal influenza forecasting was adapted for COVID-19 [Reference Moss, Price, Golding, Dawson, McVernon, Hyndman, Shearer and McCaw55]. From 2022, DSTG contributed an additional compartmental model to the ensemble forecast. Over the course of the pandemic, numerous enhancements to the case forecasting models have been made including the role and impact of: vaccination and waning immunity on short- and medium-term transmission dynamics; variants of concern, that is, new strains of SARS-CoV-2 with enhanced intrinsic transmissibility and/or immune-escape properties; anticipated or modelled changes in mixing patterns and behaviour; and changes in case ascertainment due to, for example, changes in testing practices.
Forecasting clinical loads, and in particular hospital ward and ICU occupancy levels, was also a major focus. In early 2020, Dr David Price and Dr Freya Shearer used estimates of admission rates and length-of-stay statistics to map from case-incidence forecasts to hospital ward and ICU occupancy [Reference Price70]. A more sophisticated approach, accounting for age- and jurisdictional-specific factors, as well as variants, was developed over 2021 and 2022 in the IDDU, informed by a detailed epidemiological study on length-of-stay statistics in New South Wales [Reference Tobin, Wood, Jayasundara, Sara, Walker, Martin, McCaw, Shearer and Price83]. The full details of those clinical forecasting methods are being prepared for scientific peer-review.
2.1.3 Australia’s national re-opening plan 2021
While not detailed here, major undertakings in scenario analyses were conducted in mid- to late-2021 to support Australia’s “national re-opening plan”, a major shift in Australia’s strategic response to COVID-19. That high-impact work, co-led by McVernon and McCaw and delivered by a large national consortium [65], is not reviewed here. Details are available in a number of technical reports (see aforementioned website) and publications [Reference Conway9, Reference Ryan, Shearer, McCaw, McVernon and Golding71, Reference Shearer, McCaw, Ryan, Hao, Tierney, Lydeamore, Ellis, Ward, Wood, McVernon and Golding72].
2.2 New Zealand
2.2.1 Early scenario modelling to support initial response and elimination strategy
In March 2020, a group of researchers at Te Pūnaha Matatini (New Zealand’s Centre of Research Excellence in Complex Systems), led by Professor Shaun Hendy, used simple SEIR models to estimate the potential scale of the health impact of COVID-19 in New Zealand under different interventions [Reference James, Hendy, Plank and Steyn34]. Interventions were modelled in a very simplistic way and their effectiveness was highly uncertain at the time. Nevertheless, the models were sufficient to show that a rapid and highly effective response would be needed to avoid quickly overwhelming the healthcare system and thousands of deaths. These results were communicated via the Prime Minister’s Chief Science Advisor and Ministry of Health Chief Science Advisor and helped to inform decisions to close the border and, on 23 March 2020, to instigate a strict nationwide “Alert Level 4” lockdown [Reference Hendy30].
Two main modelling tools were subsequently developed by groups at COVID-19 Modelling Aotearoa (formerly Te Pūnaha Matatini): a network contagion model, led by Dr Dion O’Neale and Dr Emily Harvey, and a nonlinear branching process model (and later a compartment-based ODE model), led by Professor Michael Plank and Professor Alex James. Both models were stochastic: New Zealand’s very low case numbers in 2020–21 meant stochastic effects were important and deterministic models were less useful.
The network contagion model is an agent-based model that simulates transmission on a synthetic contact network representing New Zealand’s population. The network was constructed from individual-level employment and education micro-data in New Zealand’s Integrated Data Infrastructure [77], with separate network layers for interactions occurring in different settings (home, work, school and community) [Reference Harvey, Maclaren, O’Neale, Ortiz-Cervantes, Patten-Elliott, Turnbull, Vasques Filho and Wu28].
Most of this section focuses primarily on the branching process model (the workstream led by MP). The model is similar in structure to a stochastic SEIR model, but allows for non-Markovian transitions and individual variability among infectious individuals (Figure 3). The most basic form of the model can be expressed as the mean number of individuals infected by individual i at time t:
$$ \begin{align} \Lambda_i(t) = R_0 Y_i F_i(t) C(t) \hat{f}\left(t-t_{\mathrm{inf},i} \right) s(t), \end{align} $$
where
$R_0$
is the basic reproduction number,
$Y_i$
is a random variable representing individual heterogeneity in transmission [Reference Lloyd-Smith, Schreiber, Kopp and Getz46],
$F_i(t)$
represents the effect of isolation or quarantine on individual i at time t,
$C(t)$
represents the effect of social distancing measures at time t,
$\hat {f}$
is the probability mass function for the generation interval and
$s(t)$
is the proportion of the population that is susceptible at time t.
Figure 3 Schematic diagram of the stochastic nonlinear branching process model used in New Zealand. (a) A branching process model generates an explicit transmission tree and allows for a right-skewed offspring distribution, where a high proportion of individuals infect few others and transmission is dominated by a minority of superspreaders. This was incorporated into the model by assigning each infected individual a random transmission rate multiplier
$Y_i$
drawn from a gamma distribution. (b) Individual transmission rate over time is governed by a generation time distribution
$w(t)$
(blue); if an individual is quarantined or isolated, their transmission rate is reduced (green). (c) Simplified test-trace-isolate-quarantine model: symptomatic individuals have a prescribed probability of testing, with some time delay from symptom onset to test; confirmed cases are isolated, and their contacts are traced and quarantined with a prescribed probability, with some time delay from confirmation of the index case to quarantine of the contact. (d) Diagram of main model compartments: the susceptible population
$S_i$
in age group i is split according to vaccination status and prior infection (susceptible compartments are further divided to account for number of vaccine doses and waning immunity—not shown here). Each susceptible compartment is associated with different levels of immunity against infection, hospitalization and death. (Colour available online.)
Having a combination of modelling approaches proved useful. The branching process model was less computationally intensive, required fewer parameter assumptions and could be readily adapted to a variety of policy questions, for example, how testing policies for international arrivals and hotel quarantine workers affected the risk of border-related outbreaks [Reference Plank, Binny, Hendy, Lustig and Ridings66, Reference Steyn, Plank, James, Binny, Hendy and Lustig81]. However, it only offered a high-level picture. The network model provided a more detailed view, for example, the ability to provide spatial outputs, to stratify by ethnicity, employment type or deprivation index, and to explicitly model household and nonhousehold transmission [Reference Harvey, Maclaren, O’Neale, Ortiz-Cervantes, Patten-Elliott, Turnbull, Vasques Filho and Wu28]. Some policy questions were more suitable for one model than the other and often a summary of complementary outputs from both models was communicated to policymakers. In some instances, the network model was used to simulate the relative effect of a specific intervention [Reference Harvey, Maclaren, O’Neale, Patten-Elliott, Turnbull and Wu29] and the results were used to estimate parameters for scenario analysis in the branching process model [Reference Vattiatio, Lustig, Maclaren, Binny, Hendy, Harvey, O’Neale and Plank84]—see Figure 1(b).
By April 2020, New Zealand had formally adopted an elimination strategy [Reference Baker, Kvalsvig, Verrall, Telfar-Barnard and Wilson4], aiming to stamp out community transmission and use border controls to prevent re-introduction. Between April and June 2020, simulations of the branching process model were run with seed infections representing reported travel-related cases. These results were used to investigate the potential consequences of relaxing restrictions, informed by international estimates of the effects of various nonpharmaceutical interventions [Reference Flaxman19, Reference Hsiang33]. The model was also used to estimate the probability of elimination, conditional on fitting observed data on daily reported cases [Reference Hendy, Steyn, James, Plank, Hannah, Binny and Lustig31].
2.2.2 Modelling border controls and resurgent outbreaks
On 8 June 2020, the New Zealand government declared that community transmission of SARS-CoV-2 had been eliminated and removed all social distancing requirements [63]. On several occasions between June 2020 and August 2021, a case of COVID-19 was detected with no known link to the border [Reference Douglas, Geoghegan, Hadfield, Bouckaert, Storey, Ren, de Ligt, French and Welch15]. In these situations, the branching process model was used to estimate the potential number of undetected community cases. New Zealand lacked a computable estimate comparable to Australia’s TP, a clear gap in situational awareness capability. Instead, simulations were run across a range of assumptions for
$R_{\mathrm {eff}}(t)$
, as well as the probability of symptomatic community testing and the number of transmission chains from the index to the detected case [Reference Steyn, Plank, James, Binny, Hendy and Lustig81].
When a new case was detected in Auckland on 11 August 2020, based on the fact that there was no known link to the border, the model estimated there were 10–50 people already infected. This prompted an immediate decision to move to “Alert Level 3” in Auckland, which, coupled with an intensive TTIQ operation, ultimately eliminated the outbreak after 179 confirmed cases [Reference Hendy30]. In November 2020, another unlinked case was detected. However, whole genome sequencing subsequently suggested a link to another case known to have been exposed at a hotel quarantine facility. In this case, the model estimated there were likely to be 4–20 other infections. This result, combined with new estimates of the effect of TTIQ on transmission [Reference James, Plank, Hendy, Binny, Lustig, Steyn, Nesdale and Verrall35], helped avoid another costly lockdown [Reference Hendy30, 62]. The cluster was subsequently eliminated via TTIQ after 6 confirmed cases [Reference Douglas, Geoghegan, Hadfield, Bouckaert, Storey, Ren, de Ligt, French and Welch15].
2.2.3 Modelling the vaccine rollout and shift away from elimination
Over time, the branching process model was developed to include additional variables and processes such as age structure, vaccination status, waning immunity and the effect of different SARS-CoV-2 variants (Figure 3(d)). This work drew on models developed by the Infectious Disease Epidemiology group at the University of Warwick [Reference Dyson, Hill, Moore, Curran-Sebastian, Tildesley, Lythgoe, House, Pellis and Keeling17, Reference Keeling, Thomas, Hill, Thompson, Dyson, Tildesley and Moore40, Reference Moore, Hill, Tildesley, Dyson and Keeling53] and made similar assumptions to the subsequently published Doherty National Plan Modelling in Australia [Reference Conway9]. Age structure was modelled via a contact matrix, derived from pre-pandemic social survey data from European countries [Reference Mossong61, Reference Prem, Cook and Jit68] and projected onto New Zealand’s population.
The age-structured model was used to estimate the reduction in
$R_{\mathrm {eff}}(t)$
and potential health impacts at different stages of the vaccine rollout [Reference Steyn, Plank, Binny, Hendy, Lustig and Ridings80]. By early 2021, a cross-agency modelling steering group had been established to liaise between modellers and policy makers, although the Chief Science Advisors were still playing an important role in communicating results to decision makers (see Figure 1(b)). The results of the model showed that, with the B.1.617.2 (Delta) variant becoming dominant globally, it would not be possible to reach herd immunity by vaccination alone due to Delta’s higher transmissibility. This implied that additional public health measures would likely be needed to control the epidemic and manage the transition from elimination to living with the virus. These results also informed decisions around the progressive relaxation of border restrictions [Reference Steyn, Lustig, Hendy, Binny and Plank79].
Previous studies showed that Māori and Pacific People were likely to be at higher risk of severe illness and death as a result of COVID-19 after controlling for age [Reference Steyn, Binny, Hannah, Hendy, James, Ridings, Plank and Sporle78, Reference Steyn82]. Māori have a younger age demographic and, combined with the higher risk, this suggested that the age-based vaccine rollout strategy should be adjusted so that Māori would become eligible to receive the vaccine at a younger age. However, the Crown chose not to implement this recommendation, a decision that was ultimately found to be in breach of Te Tiriti o Waitangi [Reference Tribunal89].
In August 2021, New Zealand experienced an outbreak of the Delta variant, most likely linked to a traveller from New South Wales [Reference Jelley36]. At the time, approximately 32% of the population had received at least one dose of the Pfizer/BioNTech vaccine and 19% had received two doses. Our model estimated that vaccination was providing approximately a 15% reduction in
$R_{\mathrm {eff}}(t)$
and that, without a stringent response, healthcare systems would likely be overwhelmed within weeks [Reference Plank, Hendy, Binny, Vattiato, Lustig and Maclaren67]. Outbreak size and risk of inter-regional spread were also estimated with the network model [Reference Gilmour, Harvey, Looker, Mackenzie, Maclaren, O’Neale, Patten-Elliott, Trent, Turnbull and Wu20, Reference Gilmour, Harvey, Looker, Mackenzie, O’Neale and Turnbull21]. For the first time since March 2020, the government imposed a nationwide Alert Level 4 lockdown. Although this ultimately failed to eliminate the outbreak, it kept it geographically localized to Auckland and controlled case numbers sufficiently until vaccine coverage was much higher.
From October to December 2021, reduction in
$R_{\mathrm {eff}}(t)$
as a result of increasing vaccine coverage was partially counteracted by the phased easing of restrictions and increasing contact rates. An ABC method was developed for inferring the value of the time-varying “control function”
$C(t)$
in equation (2.2) from data on new daily cases [Reference Plank, Hendy, Binny, Vattiato, Lustig and Maclaren67]. Real-time estimation of
$R_{\mathrm {eff}}(t)$
was also provided using the EpiNow2 package, which uses a semi-mechanistic model based on the renewal equation, equation (2.1) [Reference Abbott1, Reference Binny, Lustig, Hendy, Maclaren, Ridings, Vattiato and Plank8]. This was suitable for short-term forecasts. However, the more mechanistic branching process model [Reference Plank, Hendy, Binny, Vattiato, Lustig and Maclaren67] enabled projections of epidemic dynamics over a longer period of time, taking account of future vaccinations based on comprehensive data on booked vaccination appointments.
2.2.4 Modelling the impact of Omicron
When the B.1.1.529 (Omicron) variant began spreading globally in December 2021 [Reference Viana88], emerging estimates of Omicron’s severity [Reference Nyberg64, Reference Ward90] and time-dependent vaccine effectiveness [Reference Andrews2] were combined with projected booster uptake to produce scenarios for an Omicron outbreak starting in New Zealand on different dates [Reference Vattiato, Maclaren, Lustig, Binny, Hendy and Plank86].
Up to January 2022, New Zealand had recorded a cumulative total of only 3 cases per 1,000 people. Once Omicron began to spread in February 2022, the model was calibrated to real-time data to produce scenarios for the height and timing of the peak. These results were incorporated into dashboards for government policymakers and District Health Board planners. Scenarios were updated periodically as the model was refined to account for relaxation of social distancing behaviours, changing age-structured contact rates and reinfections [Reference Vattiatio, Lustig, Maclaren and Plank85].
The model was extended to investigate the effect of a new variant and subsequently used to model New Zealand’s BA.5 wave, which occurred in July 2022 [Reference Lustig, Vattiato, Maclaren, Watson, Datta and Plank47]. Immunity parameters were calibrated using the model of [Reference Cromer, Steain, Reynaldi, Schlub, Wheatley, Juno, Kent, Triccas, Khoury and Davenport10, Reference Khoury, Cromer, Reynaldi, Schlub, Wheatley, Juno, Subbarao, Kent, Triccas and Davenport41] for the relationship between neutralising antibody titre and protection against different clinical endpoints, coupled with whole genome sequencing data on community cases [18]. Around June 2022, the stochastic branching process model was replaced with a deterministic ODE model with similar underlying structure. This less computationally expensive model allowed an improved method for calibrating model parameters using an ABC approach. At the time of writing in April 2023, the ODE model is still being updated and fitted to data, and used to evaluate intervention effectiveness and inform policy decisions [Reference Datta11].
3 What has modelling delivered?
Mathematical modelling has delivered quantitative results and qualitative interpretations of raw epidemiological and clinical data streams to support government decision-making in response to the pandemic. These include the following.
-
• Estimates of
$R_{\mathrm {eff}}$
(and TP) and short-term forecasts, which aim to predict new cases and hospital admissions in the next 1–4 weeks, supporting week-to-week monitoring and response to the highly dynamic situation. -
• Medium-term projections, produced by calibrating a model to recent data and then running it forward in time under the assumption that key factors affecting transmission (for example, contact patterns) do not change, or change only in certain instructive ways over the relevant time frame.
-
• Long-term scenario modelling, which is more hypothetical and may be used to compare alternative policy choices or the outcomes of, for example, different levels of vaccine uptake, over a period of months or years. The qualitative outcomes of scenario modelling have supported decisions on overall strategic response as described in Section 2.
For all three use-cases of mathematical modelling listed above, the ability to incorporate mechanistic assumptions, for example, about the size of the susceptible population, vaccination coverage and roll-out, and alternative future behavioural mixing patterns, is of great importance. However, short-term forecasting typically makes use of simpler models with fewer mechanistic assumptions, for both practical (for example, computational run-time) and theoretical (for example, statistical overfitting) reasons. Scenario modelling, with a focus on exploratory, more qualitative analyses, typically benefits from development of more detailed models capturing more of the intricacies of the transmission process.
Among many challenges faced by applied mathematicians and data analysts supporting the COVID-19 response, communication to both the public and decision makers on the distinction between these different uses of mathematical modelling has been prominent. For example, the outputs of scenario models, including both the simulation results and the reports describing them, were prone to misinterpretation. Interpretation must be nuanced due to the potential sensitivity to model assumptions and mechanisms that are either not included in the model or are subject to high uncertainty, such as behavioural change, antigenic evolution of the pathogen, or development and use of new pharmaceuticals. Scenario models were typically developed for, and of greatest value in, making relative comparisons among different scenarios (for example, alternative policy settings). However, there was a tendency for the results to be considered as absolute predictions, particularly in the media but also at times in decision-making contexts. These challenges are, of course, not unique to Australia and New Zealand, nor to epidemiological modelling, and have been well documented across a range of model-based scientific fields, for example, climate modelling [Reference Maslin and Austin48].
The types of results described above will be familiar to many readers and are typically quite visible: they are described in written reports and scientific publications. In short, they are the direct outputs of our mathematical and statistical work. However, there is a second, less tangible way in which the mathematical sciences have contributed to the COVID-19 response, by providing public health leaders and decision makers (that is, politicians) with insights derived from a distinct mathematical way of thinking. For example, an intuitive grasp of nonlinear dynamics and stochasticity can provide insights or perspectives that might not otherwise be made without this type of training.
As described earlier, this was evident in the first weeks of the pandemic, when “simple” modelling in Australia and New Zealand was able to demonstrate to decision makers that not only were our countries at grave risk, but that an early, definitive response could dramatically and effectively mitigate that risk. The findings were a natural, and to the applied mathematician entirely unsurprising, consequence of the nature of transmission dynamics, yet surprisingly and deeply influential to those charged with determining our nations’ responses.
Later in the pandemic, an ability to reason on the probability of importation, and the consequences of re-establishment of transmission in different epidemiological contexts helped decision makers to quantify and respond to risk in an uncertain setting. As an example, when the trans-Tasman quarantine-free travel zone was first being considered in 2021, New Zealand modellers were asked to estimate how this would affect the risk of community outbreaks. At the time, prevalence in Australia was so low that the risk of a trans-Tasman traveller causing an outbreak in New Zealand was minimal. It would have been tempting not to bother with a formal model at all. However, the process of setting up a model led to the realization that if the hotel quarantine slots previously taken by arrivals from Australia were re-allocated to other countries where infection rates were much higher, the number of infected arrivals would likely increase substantially. That would inevitably translate into a higher risk of a case leaking out into the community. This realization, obvious in retrospect, was not something that had been considered at the time as the focus was on the risk posed by trans-Tasman travellers.
More generally, the approach to problem solving inherent in the mathematical and physical sciences, and the contribution it can make to decision making, is harder to articulate and evaluate compared to technical outputs such as those described in Section 2. While the capability is almost taken for granted by members of the mathematical and physical sciences communities, it is of fundamental importance in a range of application areas, and has proven invaluable during the COVID-19 pandemic. We suggest that as a community, we need to challenge ourselves to describe this skill set more clearly and more explicitly, to notice when we are using it, and to value it. We are of course not suggesting that mathematical models have the answers to all questions. Often, answers from a mathematical model disintegrate on contact with reality and should be quickly abandoned, although even then there are lessons. Which assumption(s) invalidated the work and why? What could be changed or included to improve the model and its role in decision support? Furthermore, domain expertise (here in infectious disease epidemiology and public health), gained either directly through professional roles or through deep, multi-disciplinary collaboration, is absolutely crucial. Elegant, and perhaps even accurate, mathematical models developed or presented without an understanding of the public health domain and context are unlikely to have impact. However, when offered from an informed position, simple mathematical perspectives on complex problems are often illuminating and at times critical to informed decision making.
4 Communication between modellers and stakeholders
Communicating the purposes of modelling and how model outputs should and should not be interpreted has been a key challenge of working with policymakers and advisors in a fast-moving environment. Some stakeholders who were initially unfamiliar with modelling as a discipline had a tendency to view models as crystal balls, whereas others tended to dismiss any model as being too assumption-dependent or lacking sufficient complexity to be of value. The reality and appropriate use of model-based analyses lies in between these extremes, and we believe this has become much better understood by policymakers over time. At their core, models provide a way of representing a complicated system in a simplified way, thinking systematically about the consequences of a range of assumptions, and testing which of those assumptions are consistent with observed data.
It is the task of the modellers to find a balance between models that are too simplistic and miss essential features, and models that are overly complicated or include irrelevant variables or mechanisms. This balance will depend on the policy question the model is designed to address. In our experience, navigating this complex landscape benefits from strong and sustained stakeholder engagement and a rich bidirectional dialogue. Engagement and clear communication of the scope and purpose of modelling in general helps give stakeholders the confidence to interpret particular models, critically evaluate findings, engage in the process of model improvement and thereby incorporate findings from modelling into the decision-making process in a nuanced, robust way. If results can be transparently explained with reference to specific key assumptions, particularly if the results are unexpected or counter-intuitive, then decision makers can draw on their complementary expertise and knowledge from outside of the modelling domain. We argue that this enhances the value of modelling to the decision-making process. A corollary is that black-box-type models or models developed and delivered in the absence of stakeholder input can be problematic, and are (in both our considered opinion and from experience) unlikely to be adopted by decision-makers and have impact.
Communicating limitations and uncertainty in model outputs is an ongoing challenge but an essential one: to avoid over-confidence in point estimates; to help understand the differences between the types of model output mentioned above; to help interpret outputs such as sensitivity analyses and confidence intervals; and to enable evaluation of model performance against subsequent observed data [Reference McCabe, Kont, Schmit, Whittaker, Løchen, Walker, Ghani, Ferguson, White, Donnelly and Watson49]. Both frequentist and Bayesian statistical tools, such as maximum likelihood or approximate Bayesian computation, provide practical ways to fit models to observed data and produce associated uncertainty intervals on model outputs. Choice of appropriate summary statistics for the outputs of dynamic models is important to accurately convey the nature of the uncertainty. For example, these could include curve-based statistics as opposed to commonly used but often misleading fixed-time statistics [Reference Juul, Græsbøll, Christiansen and Lehmann37], or probabilistic estimates of certain outcomes such as healthcare demand exceeding a specified threshold [Reference McCabe, Kont, Schmit, Whittaker, Løchen, Walker, Ghani, Ferguson, White, Donnelly and Watson49]. Capturing uncertainty due to model mis-specification or unmodelled processes such as future new variants or unanticipated behavioural change is a bigger challenge.
Distinguishing between situations with relatively high and low levels of uncertainty is important. For example, during the exponential growth or decline phase of a wave, the epidemic trajectory was relatively predictable and forecast uncertainty was typically low. In contrast, around the peak of a wave (or trough between waves), future epidemic activity is typically far more uncertain. While there may be a temptation to think that model outputs with very wide uncertainty intervals are not useful, the finding that future epidemic states cannot be predicted with confidence in some situations is itself valuable to decision-makers.
Communication is not just about mathematical modellers helping stakeholders to interpret model results. Two-way communication between policymakers and modellers about the policy questions that the model is being designed to address and the appropriate model design is equally important. Ideally, this is an interactive and iterative process so that models can be refined and updated as needed, for example, according to data availability or updated policy objectives. Building awareness among policymakers of what models can and cannot deliver helps ensure modelling questions are designed appropriately and efficiently. Likewise, rich two-way communication can help influence surveillance practices, leading to enhanced availability of high-quality data for more accurate modelling. Conversely, it is essential that modellers are aware of the biases and limitations of the data that are used as model inputs, which requires engagement with public health agencies gathering the data [Reference Lipsitch, Finelli, Heffernan, Leung and Redd45].
Although modelling has mostly been very positively received in the policy arena in both Australia and New Zealand, it has at times been heavily criticized in other areas. Much of that criticism is valid: challenging particular modelling assumptions, mechanisms and trade-offs that were not included in models, or inadequate communication of model interpretation. There are many aspects where modelling infectious disease dynamics for policy advice can be improved and this remains an active area of research [Reference Bedson7, Reference McCabe, Kont, Schmit, Whittaker, Løchen, Walker, Ghani, Ferguson, White, Donnelly and Watson49]. At other times, criticism stemmed from a misunderstanding of the purposes and limitations of modelling, or simply a dislike of its conclusions or of the decisions which it informed. Many people working in this area, including both authors, have invested significant time and effort into communicating the purposes, interpretations and limitations of modelling to the general public via mainstream media [Reference Medley52]. We believe this is an essential part of mathematical modelling for public policy advice, promoting transparency, trust and understanding of the modelling process, including what it can and cannot achieve. Communicating the value of mathematical and statistical training more broadly to policymakers and the general public is an ongoing challenge for our community.
5 Future challenges and opportunities
As Australia and New Zealand transition out of the acute phase of the COVID-19 pandemic, there are a number of opportunities, but also risks, for the mathematical sciences community and those involved in infectious disease modelling and response. First and foremost, the impact of mathematical modelling on the pandemic response has once again demonstrated the fundamental importance of our broad discipline to societal prosperity, health and well-being. While our observations on the value and practice of applied mathematical modelling have necessarily come from the epidemiological context, many of them apply equally to applications of mathematical modelling in other content and policy areas.
Mathematical epidemiology provided insight into the pandemic, demonstrated how we could respond to it, and had a significant impact on government decision making and pandemic response strategy in both Australia and New Zealand. Of course, a mathematical model cannot tell society what to do: policy decisions rightly rest with elected politicians, often involve value judgements and are typically made using a variety of sources of evidence. Nevertheless, mathematical modelling provided insight that was arguably not available through any other discipline base alone, including traditional public health and epidemiological training. Modelling and model-based analyses of data proved particularly valuable in fast-moving situations where decisions needed to be made before more complete information became available. Ultimately, the impact of modelling lay not in its ability to tell policymakers what to do (despite what some in society, the media and even political circles claimed at certain times), but in the fact that governments made decisions with a clearer understanding of the likely consequences of their choices than they would otherwise have had.
The success of applied mathematics and modelling during COVID-19 affords a huge opportunity for governments and the public health sector, in both academia and the public service, to set up new collaborations and training pathways in and with the mathematical sciences. Training in infectious disease epidemiology and public health could be fundamentally transformed with a renewed commitment to rigorous training in the mathematical sciences. We suggest this would need to begin in high school.
However, with such an opportunity, there is also risk that must be managed. When is the right time to “specialize”? Neither of the authors of this work ever trained formally in epidemiology or public health, acquiring those skills in their professional post-PhD careers. If and when the skills of infectious disease modelling are taught routinely to those undertaking training outside of the mathematical sciences—in say clinical, biomedical, health and life sciences—do we risk losing the deeper expertise in applied mathematics, statistics and data science that were foundational to the teams and consortia delivering high-impact support to government during the pandemic?
Our view is that this risk is manageable and the benefits that flow from strengthening and growing interdisciplinary collaborations between the mathematical sciences, epidemiology and public health, and other research disciplines and application areas are significant. There are enormous benefits to be had from training a future epidemiology and public health workforce in the core competencies of mathematical epidemiology. These include enhancing modelling capabilities outside academia, uncovering new research questions and developing application-oriented methodologies. This will benefit both the mathematical sciences and epidemiology and public health communities, building capacity that can be drawn on in a future emergency.
Acknowledgements
The mathematical modelling and data analytics work described in this paper were developed and performed by a large number of people from across Australia and New Zealand, only a few of whom are mentioned by name in the manuscript. Rather than list them here and risk omissions, the authors direct the reader to the full co-authorship lists of referenced papers and the multiple consortium webpages referenced in the main text. M. Plank acknowledges funding in support of the work described in this manuscript from Te Pūnaha Matatini, the New Zealand Ministry of Business, Innovation and Employment, Department of the Prime Minister and Cabinet, and Ministry of Health. M. Plank also acknowledges the role of the Ministry of Health, StatsNZ, and the Institute of Environmental Science and Research (ESR) in supplying data in support of this work. M. Plank is grateful to members of the COVID-19 Modelling Government Steering Committee for helping to design the modelling questions that the work described in this manuscript addressed and for their role in communicating model outputs to decision makers. J. McCaw acknowledges funding support for the work described in this manuscript from the Australian Research Council, National Health and Medical Research Council, Defence Science Technology Group, United States Defence, and Australian Government Department of Health and Ageing. J. McCaw thanks fellow members of numerous national working groups and committees which contributed to the design, implementation and reporting of modelling and data-analytics studies described in this manuscript. The authors are grateful to two anonymous reviewers for comments on an earlier version of this manuscript.
Open access
