1.0 Introduction

In air combat training, a great deal of instructor pilots’ (IPs’) work concerns evaluating the quality of upgrade pilots’ (UPs’) decision-making as they progress through fast jet training. Success in air combat is predominantly about making sound tactical decisions. Fast-paced air combat is highly challenging – even for pilots with extensive knowledge of its dynamics. Windows of opportunity open and close rapidly, and a single poor decision can quickly lead to a situation where no good decision alternatives are left.

Fighter pilots make decisions in a cycle, often described as the OODA (Orient, Observe, Decide, Act) loop [Reference Bryant1]. An objective of air combat training is to increase the pace around this loop to achieve a dominant tactical tempo [2]. To maintain this tempo, pilots must make decisions swiftly, often with uncertain and incomplete information. The decision-making challenge is aggravated by limited ability to extract information from the tactical environment. Furthermore, as those engaged in air combat strive to hide or deceive their actual intentions, pilots’ resulting awareness of the environment may deviate from its real state. As a result, tactical decisions may seem rational with respect to the subjective state of air combat, but less rational from the perspective of its objective state.

Air combat training is a form of Naturalistic Decision Making (NDM) training. NDM originated from the requirement to understand decision-making in a command-and-control context. It examines how subject matter experts use their experience to make decisions in an operational setting [Reference Zsambok and Klein3]. NDM is characterised by ill-structured decision-making problems [Reference Orasanu, Connolly, Klein, Orasanu, Calderwood and Zsambok4], situated in uncertain, rapidly changing dynamic environments, where information is incomplete, imperfect, unreliable and/or ambiguous. Simon [Reference Simon5] argued that the information processing requirements underlying rational decision-making in such environments may exceed the limits of the information processing system of decision-makers. As a result, the decision-makers usually do not have complete knowledge of the consequences of all the possible alternatives. Moreover, when under time pressure, the decision-makers are likely to consider only a limited number of options and tend to adopt decision-making strategies using heuristics [Reference Payne, Bettman and Johnson6].

Early NDM studies suggested that people first try to understand the situation and establish if, from their experience, they had encountered something similar before. They use their experience to frame the situation and determine a potential course of action. They rarely consider all potential options. If a similar situation is not immediately recognised, they work sequentially until they generate a potentially satisfactory outcome. Klein et al. [Reference Klein, Calderwood and Clinton-Cirocco7] observed that experts often pattern-matched the current situation with previous experience and determined a course of action without making a conscious decision. Klein labelled this as Recognition-Primed Decision making (RPD). RPD training approaches tend to be undertaken using simulations involving demonstration and practice with little emphasis on classroom-based instruction [Reference Cohen, Freeman, Thompson, Cannon-Bowers and Salas8]. Zakay and Tsal [Reference Zakay and Tsal9] found that practice without stress and time pressure did not enhance decision-making when real-life decisions were made under similar conditions.

Air combat training involves all the main characteristics of NDM and RPD. The pilots’ pre-existing knowledge, or mental models, about air combat is situated in their long-term memory (LTM). The content of LTM is relatively stable and is therefore referred to as static knowledge. When fighter pilots engage in air combat training, they sample the air combat environment and update their knowledge with new information. This process of information acquisition and updating, together with the short-term storage of the updated information, takes place in working memory. The resulting situation or dynamic knowledge is the pilots’ subjective understanding of their tactical environment, or situation awareness (SA). Endsley [Reference Endsley10, Reference Endsley, Zsambok and Klein11] argued that RPD requires the decision-maker to have a mental model of the decision-making environment based on past experience, but which is dynamically updated by current events.

Higher levels of SA enable decision-makers to act in a more timely and effective way. In Endsley’s three-level concept, SA is “the perception of the elements in the environment within a volume of time and space (SA-Level 1), the comprehension of their meaning (SA-Level 2), and the projection of their status in the near future (SA-Level 3)” [Reference Endsley10], p 36]. Unfortunately, SA and the real tactical environment are seldom perfectly aligned. Based on their SA, the pilots attempt to identify decision points (DPs). A DP is a state of the air combat environment from where the situation has potential to evolve in various directions and from which point the pilot’s decision can affect how the situation will unfold. The pilots seldom have all the necessary information or time to evaluate the feasibility of all possible alternatives. Instead, they are often forced to make a rapid decision with whatever SA they have at the DP. When making a rapid decision, the pilot seeks to identify cues from the tactical environment associated with some learned decision alternative. These alternatives typically deal with the pilot’s tactical and procedural options. If the identified cues are associated with just a single option, the pilot selects and executes it. If the cues are associated with several alternatives, the pilot is likely to select the first satisfactory option as it allows the fastest reaction to the tactical situation. If, however, they cannot associate the cues with any existing decision alternative, they must either adjust an existing alternative or create an ad-hoc solution. Both options are less desirable as they take time and tax the pilot’s limited working memory capacity. Once the pilots have selected a decision alternative, they run a mental simulation to evaluate the expected outcome. If the expected outcome is not satisfactory and time permits, they will try to seek more information and repeat the cue-decision pairing process (see (Reference Orasanu, Fischer, Zsambok and Klein12)). In all other cases, the pilot executes the selected decision. The executed decision changes the state of the air combat, which in turn creates a need to update SA. Such a fast-paced cyclical decision-making and information processing loop (see Fig. 1) is constantly repeated throughout the air combat mission.

Figure 1. Pilot’s decision-making and information processing in air combat.

In air combat, the outcome is what ultimately matters. Motivated by such a premise, an IP may put too much emphasis on the outcome of a UP’s decision at the cost of omitting the analysis of the decision-making process [Reference Mansikka, Virtanen and Harris13]. A decision leading to a good outcome may be prematurely assessed as ‘good’ regardless of the rationale behind it. Similarly, an UP’s decision resulting in an undesired outcome may be assessed as ‘bad’ irrespective of subjective reality at the time. Air combat decisions are made under uncertainty. Pilots can apply an appropriate process, but this may result in a bad outcome because of factors such as unanticipated actions of enemy aircraft. The quality of a decision cannot be solely assessed by its outcome [Reference Brown, Kahr and Peterson14]. Instead, the effectiveness of decision-making training should be gauged on underpinning processes [Reference Li and Harris15–Reference Mansikka, Virtanen, Harris and Jalava17].

When UPs’ information processing is seen as a type of NDM, air combat training should emphasise recognition of patterns characteristic of domain-specific situations, i.e. RPD. To do this, IPs can use a Critical Decision Method (CDM) procedure to assist UPs to verbalise the static and dynamic knowledge underlying their decisions. CDM is a widely used semi-structured interview procedure for eliciting knowledge underlying RPD-type decisions [Reference Crandall, Klein and Hoffman18, Reference Klein, Calderwood and MacGregor19]. Klein [Reference Klein20] outlined four requirements for RPD training: engage in deliberate practice; develop a wide range of experience; obtain accurate, diagnostic and timely feedback; and review prior experiences to derive new insights and learn from mistakes. The role of IPs is critical, especially in the last two requirements. To improve UPs’ decision-making by providing enhanced process-related performance feedback, the assessment of their decision quality should be more than simply ‘good’ or ‘bad’. Unfortunately, this has traditionally not been the standard protocol.

The quality of UP’s decisions should be evaluated against their static and dynamic knowledge relevant to the DP in question. The role of IPs in this evaluation is essential. If a pilot’s decision leading to a desired outcome is based on accurate and correct knowledge, in terms of decision quality the pilot should be considered as ‘good’. However, if the pilot’s underlying knowledge appears to be inaccurate or incorrect but s/he still achieves a desired outcome, the pilot should be regarded as ‘lucky’. If the pilot’s underlying knowledge is not evaluated at all, his/her decision-making is seen as ‘good’ in both cases. However, if the pilot’s decision leads to an undesired outcome but the decision is based on accurate and correct knowledge, the decision is simply ‘unlucky’. But if the same decision is founded on an inaccurate and incorrect knowledge, it would be reasonable to consider the pilot’s decision-making as ‘bad’. If the pilot’s knowledge is not assessed, his/her decision would be regarded as ‘bad’ in both cases. A failure to associate correctly the outcome resulting from a decision with the knowledge leading to it can have critical, and sometimes fatal, consequences in pilot training. An ‘unlucky’ pilot incorrectly assessed as ‘bad’ is likely to question what is actually correct knowledge, resulting in ineffective or even negative training. In contrast, a ‘lucky’ pilot assessed as ‘good’ is actually encouraged to strengthen his or her false representation of air combat, resulting in negative training and potentially developing inappropriate or incorrect mental models. Simply asking the UPs whether they had correct and accurate static knowledge or SA when they made a decision is not a fruitful approach. A person with a stable – but inaccurate and incorrect – SA is an unreliable assessor of his/her own SA [Reference Endsley21]. Table 1 summarises the alternative judgements provided in the performance feedback based on either traditional or CDM-type feedback protocols.

Table 1. Performance feedback judgements with respect to desired and undesired outcomes of decisions when using traditional and CDM-type performance feedback protocols

Timely and accurate feedback has long been recognised as an essential factor for improving performance (e.g. [Reference Gagné22]). Instructing modern air combat is primarily about building UPs’ accurate and detailed mental models of different states of air combat to support effective tactical decision-making. During mission briefs, it is the role of IPs to prepare UPs for upcoming air combat training missions by providing them with the most accurate mental models possible. During the mission, the UPs’ mental models and decision-making are tested. A great deal of learning, in terms of development of the mental models, takes place during mission debriefs, where the IPs evaluate the UPs’ decisions and give feedback on the mental models underlying those decisions.

This paper aims at improving the IPs’ ability to assess UPs’ decision quality by identifying the difference between the UPs’ subjectively perceived state and the objective state of air combat. The objective of the training programme described is to improve the IPs’ ability to identify and correctly classify the knowledge upon which the UPs base their tactical decisions during air combat training, enabling the IPs to provide enhanced feedback. The CDM-based training intervention received by the IPs is evaluated using Kirkpatrick’s four-level evaluation model [Reference Kirkpatrick and Kirkpatrick23]. At Level 1 (Reactions), the training content, materials, instructors and methods of delivery are evaluated. These evaluations help to develop the training and promote uptake. Positive reactions do not, however, suggest that the training has been effective [Reference Kirkpatrick and Kirkpatrick23]. Level 2 (Learning) typically comprises an end-of-course test to determine what knowledge and skills have been acquired. Learning does not guarantee that practices from the training will migrate to the taskwork itself, which is a fundamental objective. This is addressed at Level 3 (Behaviour), which evaluates such transfer of Learning. Kirkpatrick & Kirkpatrick [Reference Kirkpatrick and Kirkpatrick23] suggested that evaluation at this level is important but often omitted in the assessment of training. Level 4 (Results) is concerned with organisational changes resulting from the training. It has been described as the most important and most challenging level to evaluate [Reference Reio, Rocco, Smith and Chang24].

In this study, UPs were pilots transitioning from the BAe Hawk, an advanced jet trainer aircraft, onto fast jets such as the Lockheed Martin F-35 or Boeing F/A-18. A training intervention was developed to enhance the quality of UPs’ decision-making. It was assumed that after the training the IPs would be better positioned to provide detailed and accurate feedback, ultimately resulting in UPs’ better tactical decisions in air combat. A long-term field test was implemented for this purpose.