1.0 Introduction
Advancements in commercial flight deck technology have influenced the ability for flight crew members to utilise multiple sources of information to monitor the safety of their flight. Redundancy amongst aviation systems and sources of information is commonplace in any aircraft flight deck. For example, recent technological advancements have added redundancy in the form of the Electronic Flight Bag (EFB). An EFB is capable of displaying dynamic information that can be updated in real time and information that is generated from multiple sources (e.g. an overlay of GPS location on top of the enroute chart). These overlays of data pulled from multiple sources can provide similar information to existing systems, such as navigation displays (ND) that are affixed to an aircraft instrument panel (FAA [1]).
Although new technology can serve as an excellent aid to situational awareness, redundancy can also increase the potential for an information conflict. For example, the location of an aircraft displayed on the ND can differ compared to the location displayed on the EFB. Recent research has found that pilots commonly experience information conflicts between the EFB and the information systems in their panel, including information conflicts related to weather, traffic and navigation information (Carroll et al. [Reference Carroll, Sanchez and Wilt2]). If an information conflict is detected by the flight crew, the pilots must determine which source of information to ultimately act on. There is not currently a theoretical model that describes the cognitive processes that take place on the flight deck when a pilot makes a decision regarding two sources of conflicting information. However, there is a need to understand how pilots make decisions under these circumstances, including the factors that influence this decision. The purpose of this study was to examine this decision-making process, in airline pilots, making decisions with conflicting information, in realistic simulation-based scenarios. Both a top-down theoretically driven approach, in conjunction with a bottom-up data driven approach, were utilised to develop a theoretical model of pilot decision-making with conflicting information.
This paper serves as a continuation of previous work attempting to understand pilot decision-making with conflicting information and fills a gap in the related literature that previously lacked a model of pilot decision making with conflicting information. This model could aid in the further advancement of cockpit sources of information and a deeper understanding of how a pilot will respond to a situation that involves a conflict between one or more of these sources. This deeper understanding can help inform the design of systems, training, and SOPs to aid pilots’ decision-making process, resulting in safer skies.
1.1 Foundational theory
There are a number of theoretical models of decision making that can serve as a basis for a model of pilot decision making with conflicting information. Lipshitz and Strauss [Reference Lipshitz and Strauss3] developed a framework that modeled how individuals make decisions when presented with uncertainty. One hundred and two cases of self-reported decision-making experiences with an uncertain situation were analysed. The self-report data combined with findings from a review of related literature led to Lipshitz and Strauss conceptualising five strategies for coping with uncertainty. The strategies were reducing uncertainty, assumption-based reasoning, weighing the pros and cons of alternatives, suppressing uncertainty and forestalling. In the Lipshitz and Strauss model the decision maker facing uncertainty starts by determining if the situations make sense. If it does, the decision maker either forestalls or determines if there is a single, good-enough option. If there is a single good option, that option is implemented; however, if there are multiple options, the pros and cons of each option are weighed and the best one is selected. If there is no best course of action, the decision maker either suppresses, forestalls or generates a new alternative. If the situation does not make sense and there is no additional information available, then the decision maker either uses assumption-based reasoning or forestalling.
One of the main limitations of the Lipshitz and Strauss [Reference Lipshitz and Strauss3] model, is that it does not consider some of the key factors that influence decision making in the cockpit, such as time pressure, risk severity and workload. Carroll and Sanchez [Reference Carroll and Sanchez4] identified several key system factors, individual factors, and environmental factors that impact decision making with conflicting information. Specifically, individuals will typically trust and act on more reliable systems (e.g. Skitka [Reference Skitka5]), that are more transparent (Fleming & Pritchett [Reference Fleming and Pritchett6]) and require less workload (Pritchett [Reference Pritchett7]). Further, training can lead to increased trust and utilisation (e.g. Bahner et al. [Reference Bahner, Huper and Manzey8]), individuals tend to act on the source of information that minimised risk (e.g. Pritchett et al. [Reference Pritchett, Fleming, Cleveland, Zoetrum, Popescu and Thakkar9]), and time pressure can lead to significantly less effective decision making when an information conflict is present (e.g. Mosier et al. [Reference Mosier, Sethi, Mccauley, Khoo and Oransanu10]). Also, experience has been shown to have a range of effects on decision making with conflicting information. St. John et al. [Reference St John, Callan and Holste11] found that decision makers with large amounts of experience were more decisive in their decision making.
In the case of decision making with conflicting information, pilots with experience may intuitively know which source of information to trust based on their prior experience negating some of the negative effects of information conflicts. Lee and See [Reference Lee and See12] explain that positive experience can turn into trust, and trust in an information source can reduce uncertainty by serving as a social heuristic. If the source that pilots utilise frequently, and trust is conflicting with a source that they use infrequently then they may be more likely to use the trusted source.
Orasanu [Reference Orasanu13] developed a model that described the process of how pilots make decisions in threatening environments. This model, unlike Lipshitz and Strauss [Reference Lipshitz and Strauss3], includes some of the critical factors that affect the decision process, such as time pressure and risk severity. In the Orasanu model, a pilot first assesses a situation. Based on this assessment, the pilot identifies what the problem is, how much time they have to resolve it and how severe the potential consequences are (risk severity), and as a result takes one of three decision making paths. If there is limited time available, and the situation involves hazards that are deemed to be high risk, pilots rely on rules and Standard Operating Procedures (SOPs) that they appropriately apply to the situation. If there is ample time and the pilots are familiar with, or understand, the problem, they will evaluate multiple options and choose the best option for the given situation or follow the rule or SOP. If no rule or SOP is available, then the pilot has to create a novel solution. If ample time is available, however, the problem is not understood, the pilots will gather additional information and reassess the situation. A limitation of the Orasanu pilot decision-making model is that the framework does not consider judgement regarding which information sources to utilise in the decision-making process. When pilots are required to make a decision regarding which source of information to trust, they need to be able to analyse all of the available information and decide which information source they will give credence to. The weighing of alternatives, which was present in Lipshitz and Strauss [Reference Lipshitz and Strauss3] but absent from Orasanu [Reference Orasanu13], should be included in any future models that describe the decision-making process.
A key limitation with both the Lipshitz and Strauss [Reference Lipshitz and Strauss3] and the Orasanu [Reference Orasanu13] models is that they both describe a situation in which there is uncertainty, in general. There are different types of uncertainty related to information discrepancies. Chen and Li [Reference Chen and Li14] present three different types of information discrepancies, including missing information, ambiguous information and conflicting information. Chen and Li found that decision makers respond to these discrepancies in different ways. As such, it is important to create a model specific to decision making with conflicting information.
1.2 Empirical support
There is also empirical evidence to inform the development of a model of pilot decision making with conflicting information. Carroll et al. [Reference Carroll, Rebensky, Wilt, Pittorie, Hunt, Chaparro and Sanchez15] surveyed 108 pilots across different sectors of aviation, commercial airlines, corporate and general aviation, to determine the prevalence and the type of information conflicts that pilots have experienced first-hand. It was found that a majority of participants experienced some sort of information conflict on the flight deck related to weather, traffic or navigation information. Most participants reportedly trusted and ultimately acted on information from certified sources of cockpit information (e.g. navigation display, onboard weather radar), compared to uncertified EFB sources. Carroll et al.’s results also revealed several factors that impacted which source of information pilots trusted, including source accuracy, source reliability, recency of the source information and knowledge of the strengths and weaknesses of each source. Results revealed that the pilots ultimately acted on a source due to their trust in the source, being trained to use a source, being required to use the source and the source indicating a more hazardous or risky situation. These factors are relevant to the proposed model because ultimately the source that pilots trust (in lieu of a mandatory source) will most likely be the source that they act on. Therefore, understanding how pilots come to trust the source is important.
Carstens et al. [Reference Carstens, Pittorie, Carroll and Sanchez16] examined four aviation safety databases in order to identify the trends in information discrepancies and how pilots are responding to them. They searched through the Aviation Safety Reporting System (ASRS) database, the National Transportation Safety Board (NTSB) accident database, the FAA Accident and Incident Data System (AIDS) database and the Aviation Safety Action Program (ASAP) database from a major United States-based airline. From these databases, they identified 56 information discrepancies. Fifty percent of the discrepancies involved conflicting information, 39% of discrepancies involved incomplete/ambiguous information and 11% of discrepancies involved missing information. The Carstens et al. study found that pilots responded differently based on the type of discrepancy experienced. Pilots who experienced an information conflict were more likely to investigate the discrepancy and alter their course of action. They also found that pilots who had incomplete, ambiguous, or missing information were less likely to detect the conflict. Because pilots who experience information conflicts were more likely to alter their course of action, it is important to understand how pilots make decisions during information conflicts.
The purpose of this paper is to present the methods used to develop, and the resulting model of the decision-making process that pilots undergo when confronted with conflicting information on the flight deck. The model integrates factors and processes from the theoretical models discussed in the previous section, with those identified through a qualitative analysis of airline pilot response to information conflicts in a simulation scenario. The resulting model aims to elucidate how pilots make decisions when confronted with information conflicts. The following sections detail the methods, results and implications of this research.
2.0 Methods
2.1 Experimental design
The data utilised in this study was previously collected in a study that examined the effects of conflicting information on B737 pilot decision making. Carroll et al. [Reference Carroll, Rebensky, Wilt, Pittorie, Hunt, Chaparro and Sanchez15] conducted a within-subjects repeated measures experiment in which 35 participants type-rated in the Boeing 737, who had experience using an EFB on a Boeing 737 flight deck, flew a series of short vignette scenarios. During the study, participants flew six separate scenarios in which an event required each participant to react to an unexpected situation. The presence of an information conflict was manipulated within subjects. Half of the scenarios included an information conflict in which information presented on the uncertified EFB conflicted with information presented on a primary source of information (i.e. ND or ATC). The type of information on which the scenarios focused was manipulated between subjects, with participants receiving scenarios related to airspace, traffic or navigation events. Only data associated with participants who flew scenarios related to airspace and traffic were included in the present study due to missing data and the fact that the navigation conflict elicited a different response from the participants due to lower levels of risk in the scenarios. The methods utilised in this experiment are summarised below; for full details, see Carroll et al. [Reference Carroll, Rebensky, Wilt, Pittorie, Hunt, Chaparro and Sanchez15].
2.1.1 Participants
The study participants consisted of 35 pilots (2 female, 33 male) who were type-rated to fly the B737 for airlines operating in the United States. The average age of the participants was 50 years old with 16 participants being between 30-49 and 19 being between 50 and 65. On average, the participants had 13,897 flight hours, with on average 5,295 in the Boeing 737. However, for the creation of the model, only 25 participants (2 female, 23 male) were included. This represented a sub-sample of the Carroll, Rebensky et al. study participants who experienced conflict scenarios in the airspace and traffic groups. The sample included data associated with 62 information conflicts experienced by these 25 pilots.
2.1.2 Experimental task
Each participant completed a series of short vignette scenarios in the simulator. In all scenarios, the participants had two sources of information: an uncertified source in the form of EFB information and an approved source in the form of either ATC or information on a certified ND. For the ATC source, one of the researchers served as the controller and utilised scripted instructions and responses that were developed by an ATC Subject Matter Expert (SME). Information conflicts were created for each scenario by creating an event in which there was a mismatch between the two information sources. For Airspace scenarios, the conflicting information was the location or status of Prohibited Areas, Class C, and D Airspace or Temporary Flight Restrictions (TFRs). For traffic scenarios, the conflicting information was the difference in the position of, or movement of, traffic on TCAS compared to traffic on the EFB application. An example of an airspace information conflict scenario would be a TFR that is displayed on the EFB, but ATC would be unable to verify the location or status of the TFR. An example of a traffic scenario was a traffic alert from the Traffic Collision Avoidance System (TCAS) but the same traffic displayed in a different location on the EFB. During each scenario, the participants had to decide which source of information to trust and how to respond and execute the appropriate course of action.
2.1.3 Measures
The current study analysed three different types of measures, including participant perceptions collected via a questionnaire, participant behavioural responses captured via an event-based observer checklist and participant post-study interviews.
Perception measures. After flying the scenarios, participants were administered a perceptions questionnaire that measured their perceptions of the scenario they just flew. The associate questions are summarised in Table 1.
Table 1. Participant perception survey
Behavioural response measures. Two behavioural response measures were captured by researchers using a behavioural checklist. The first measure was a process-level measure that captured whether the participants detected and investigated the conflict. The second measure was an outcome-level measure that captured whether the participant continued or modified their flight profile. For this study, modifying the flight profile meant conducting any action that was not in the initial description, such as climbing or descending in altitude. The behavioural checklist resulted in a numerical value that was used to understand how the participants responded to the information conflict.
Participant interviews. After the participants flew all scenarios and filled out post-scenario perception questionnaires, each participant was interviewed by a researcher to gain insight into their decision-making process during the information conflict scenarios. Prior to the interview, the participants were shown a picture of the discrepancy. Then they were asked the post-scenario interview questions. The first question in the post-study interview asked, “Did you detect any discrepancies between the information presented on your EFB and other sources of information available such as your ND, ATC or other sources?” This was scored Yes/No. If the participant answered “Yes”, they were then asked the questions in Table 2. If the participant answered “No”, the discrepancy was pointed out and they were asked the questions in Table 3. All post-scenario interviews were recorded on video and later transcribed.
Table 2. Post-scenario interview questions and response format: Discrepancy detected
Note. *Question 6 was contingent on the scenario being a traffic scenario. It was skipped in airspace scenarios.
Table 3. Post-scenario interview questions and response format: discrepancy not detected
Table 4. Influencing factors and frequency counts
2.2 Data analysis and model formation
2.2.1 Qualitative data analysis
The qualitative analysis had two goals. The first goal was to extract the factors that influenced the decision process. The second goal was to determine the process that pilots went through when making the decision. The process of extracting the factors that influenced decision-making was a six-step process.
First, the video recordings of the participants’ interviews were transcribed. The transcription process involved two researchers reviewing and transcribing the video recordings into an electronic document. Second, the interview notes and questionnaire responses were input into a database and filtered so only conflict scenarios from the airspace and traffic groups remained. Third, using this database and the interview transcripts as supplemental information, a thematic analysis was conducted in which the data was analysed, themes extracted, and pilot responses categorised into themes. The categorisation was conducted by two researchers working independently to extract categories of influencing factors and categorise the participants’ questionnaire and interview responses into categories. There was a preliminary set of categories that were identified from the literature and previous research, and the remaining categories emerged during the analysis process based upon the participant responses. Participant responses were either assigned to an existing category, or if it did not fit into an existing category, a new category was created. The final list of categories for influencing factors and common responses is outlined in Table 4. After the researchers completed their categorisations, all responses were viewed side by side and discussed by the entire research team of five researchers. During the discussion, comments were reassigned to existing categories, and new categories were formed. This discussion aimed to reach a consensus on a final set of categories and the categorisation of each participant’s response. The research team repeated the process until all the scenarios were completed.
Fourth, after the initial categorisation, the researchers re-analysed the categories and associated participants to determine if some of the categories could be consolidated, especially those in which there was only one participant response. Fifth, the participant responses were then assigned to categories that used standardised wordings. Sixth, a frequency count for responses in each category was generated to allow the categories to be scrutinised quantitatively. This process allowed us to come up with a list of factors that influence pilot decision making.
While extracting the factors that influenced the decision-making process, we also began to extract the steps of the decision-making process. This was not a formal process; however, as we discussed the participant responses to compile the factors that influence pilot decision making, we conceptualised a flow for the decision-making process. We did this by talking through the steps that pilots took to make a decision and trying to determine in what order they were conducted. This process flow was incorporated into the model formulated in the next section.
2.2.2 Model formation
Utilising the factors and process flow that were extracted by the qualitative analysis described in the previous section, we began to formulate the final model. The model formation strategy consisted of both a top-down and bottom-up strategy. From a top-down perspective, the conceptual framework was rooted in two previous theoretical works. First, the model was grounded in a decision-making model created by Lipshitz and Strauss [Reference Lipshitz and Strauss3] that illustrates how individuals make decisions when there is uncertainty in the situation, such as when there is inadequate understanding, incomplete information, or undifferentiated alternatives. Second, the conceptual framework was also based on a framework of factors identified by Carroll and Sanchez [Reference Carroll and Sanchez4] to influence decision making with conflicting information. These factors aided in the model formation process by providing themes to look for in the data. Between the Lipshitz and Strauss [Reference Lipshitz and Strauss3] framework’s components and the Carroll and Sanchez [Reference Carroll and Sanchez4] factors, the researchers were able to conceptualise a structure and key factors to include in the preliminary model. To generate a preliminary conceptual model, each researcher drafted a conceptual model, and over a series of two meetings, the research team discussed, came to a consensus, and consolidated the various frameworks into a preliminary conceptual model.
In parallel, a bottom-up, data-driven approach was conducted by examining data resulting from the qualitative analysis presented in the previous section. The data illustrating the decision-making process performed by the participants and factor categorisations were reviewed to find evidence to provide support for, and expand upon, the initial model components. For example, if multiple participants responded that their company policy forbade them from making decisions based on EFB information and, as such, did not even consider the EFB display, then the data would support that company policy is a factor that influences the early detection stages in the decision making of pilots. In this example, the data also supported the inclusion of training/SOP into the final model. If no evidence supported the initial model component, that proposed component was removed from the model and replaced with a component that was supported by the data. This process occurred by having three researchers review the final data categorisations and determine which components of the initial model they fit into. If an initial component was not supported by the data, new components were created that had data-driven support. The three researchers involved in the qualitative categorisation and integration into the model draft were all certified pilots and could use flight experience and knowledge to ensure the new evidence-based model was logically consistent with the process of decision making in a real-world aircraft. Once the researchers agreed on a final model draft, the entire research team discussed the exact order of decision-making steps and where the influence factors came into play to ensure it was consistent and included a logical flow.
3.0 Results
3.1 Overview of model
The model resulting from the analysis aims to describe the pilot decision making process for a distinct situation in which an information conflict between two sources exists: a primary source such as an ND and a secondary source such as an EFB. The final model of decision making with conflicting information describes both the process and the factors that influence which source of information the decision maker will ultimately act on and includes three distinct stages: detection, sense making, and decide and act. Each stage of the model contains influencing factors that explain the cognitive process of the pilot as they progress through these stages. The model is illustrated in Fig. 1 and described in detail in the sections that follow.
Figure 1. Model of pilot decision-making with conflicting information.
Stage 1, detection, is concerned with whether the pilot is attending to both sources of information and detects the information conflict. If the pilot is only utilising one source, then the pilot advances to acting on the primary source in the decide and act stage. If the pilot is utilising both sources, it must then be determined if the pilot detects the conflict between the two sources. If the conflict is not detected, the pilot also advances to acting on the primary source. If the pilot does detect the information conflict, the pilot progresses into the second stage, sense making. Influencing factors that affect the pilot’s ability to detect the conflict include how experienced they are with both sources, the training they received on the sources, SOPs that govern how the sources are utilised, and the clarity of the information presented on the two sources.
Stage 2, sense making, is governed by whether or not a pilot has the necessary resources, specifically, the information, time and cognitive capacity, to perform the sense making process. If any of these resources are limited, the pilot will typically refrain from sense making and move straight to the decide and act stage. If resources available are sufficient to allow sense making, the pilot will progress through the sense making process, which involves collecting additional information and making sense of the situation until an adequate understanding is reached or resources are exhausted, at which point the pilot will move to the decide and act stage. During the process of sense making, the pilot attempts to understand why the information conflict is occurring and which source is more accurately representing ground truth. Influencing factors in Stage 2 include the levels of stress and workload that the pilot is experiencing, information clarity, how knowledgeable the pilot is regarding the functionality of each source, (e.g. sources strengths and weaknesses), and if the pilot has ever experienced a similar situation relative to the information conflict under consideration.
Stage 3, decide and act, focuses on consideration of the outcomes associated with a pilot’s decision to act on source 1 versus source 2. A pilot will typically decide to act on the source that shows the most life-threatening situation in order to err on the side of caution. A pilot will also tend to act on the source based on whether or not it is aligned with company SOPs and regulation, aiming to avoid potential repercussions, such as a regulatory or company violations that could result from acting on a particular source. If there is not a clear distinction between the risk associated with acting on each source, and if one source is not more in in line with company SOPs than the other source, pilots will ultimately act on the source that they trust is more accurately representing the situation. All of these considerations are influenced by the pilot’s level of stress, training and SOPs, and level of risk to safety or policy violation.
3.2 Influencing factors
There were nine factors that emerged from the qualitative data analysis as influencing the pilot decision making with conflicting information process: (1) experience with sources, (2) training/SOPs, (3) information clarity, (4) stress, (5) workload, (6) source functional knowledge, (7) experience with situation, (8) safety risk, and (9) violation risk. These factors are summarised in Table 3, including the common responses from participants that fell within these categories and the frequency of these responses. The following sections detail how these factors influence this decision-making process.
Training and SOPs were a factor that influenced all three stages of the model. Nineteen comments from participants indicated that training and SOPs influenced detection, sense-making and decide and act stages of the model. With respect to detection, a lack of training on the EFB or SOPs that discourage its use in normal operations could cause a pilot to focus on only the primary source. This would inhibit the pilot from being able to notice any information conflict between the two sources. For example, pilots operating under Part 121 are trained to use their ND to determine the location of other aircraft in proximity to their own location. Although EFBs are capable of displaying the same type of traffic information, pilots are not trained on this functionality, and it would violate their company’s SOPs if they displayed traffic on their EFB.
Eighteen comments from participants were categorised as information clarity, and this factor influenced Stage 1: detection and Stage 2: sense making. If a pilot is able to clearly understand the visual information that is presented to them, they will be more likely to detect an information conflict between two sources. Nine responses referenced the visual depiction of the hazard on the display as being easily detectible. In the detection stage, a pilot is more likely to notice a hazard displayed on the EFB if it is salient compared to its surroundings. In the sense making stage, a pilot will be able to further investigate the threat and the impact it could have on the safety of the flight if they are able to notice the threat in the first place. Five responses indicated that level of familiarity with the source symbology allowed the pilot to deduce what the available information was indicating. This level of familiarity, such as recognising that red airspace on the EFB display represented a TFR, helped the pilot make a decision in a shorter amount of time. Two comments referenced the EFB that used in the study was mounted to left-hand side of the main instrument panel, while EFBs in Part 121 operations are closer to a pilot’s field of view. If a source is outside of the field of view for the pilot, it may not have been used at all in any stage of the model. Two responses indicated that Source 2 was utilised during the sense making stage as it presented visual information that was novel to the situation. The presence of novel information into a hazardous situation influenced the pilot into including Source 2 in the process of making sense of the information conflict.
Fifteen comments from participants were categorised as functional knowledge of a particular source, and this factor influenced Stage 2: Sense Making. Data revealed that if a pilot is familiar with how a source works and receives information, they will be more likely to try to make sense of a conflict between two similar sources. A total of seven responses were categorised as a participant being unable to verify the source of ATC information or its accuracy. Although pilots are trained to follow ATC instructions, they may be unaware of the hardware and software with which ATC is equipped. Five responses indicated Source 1 was trusted and utilised because it was certified, while Source 2 was uncertified. If certain functionalities of Source 2 are uncertified, their use will not be permitted on the flight deck and a pilot would not receive training on these functionalities. A pilot who has never received training on an EFB or certain uncertified functionalities will be unfamiliar with the source of the data used to display airspace or traffic information. Important details such as the accuracy of this information or the delay between sending the data from the source until it’s received the EFB would be unknown to the pilot. The final three responses indicated that some pilots who may have had a lack of functional knowledge of an EFB still utilised the source for decision-making due to the novel information presented on its display. Other sources of cockpit information are unable to display a TFR, so the presence of this information that was deemed important by the pilot was included in their decision of which source to act upon.
Nine comments from participants indicated that any risk to the safety of the flight influenced their decision of acting on one source of cockpit information over another. This factor influenced Stage 3: decide and act. Two responses referenced an event that posed little threat to the physical safety of the aircraft. Therefore, participants acted on Source 1. In contrast, four responses indicated that pilots acted on the information from Source 2 if there was a depiction of an immediate danger to their aircraft. For example, a visual alert of an impending mid-air collision, poses an immediate physical threat to the safety of the pilot’s aircraft. Three responses were categorised as a participant acting on Source 2 due to it being the conservative choice regarding the hazard. Even if Source 2’s information is deemed less accurate than information from Source 1, a pilot will not take the risk of putting their aircraft in harm’s way and will avoid any chance of degrading the safety of the flight.
Experience with the source is a factor that influenced Stage 1: detection. Eight comments from participants indicated that prior experience with a source of cockpit information influenced their use of a source to detect an information conflict. Of these eight comments, five responses indicated that participants who had received training on the appearance of data and functionality of Source 1 detected the information conflict and did not need to utilise Source 2. If a pilot is more familiar or comfortable with a particular source, they will be more likely to utilise this source while investigating any situation, such as flying in close proximity to a TFR. Three participants responded that they had previously experienced errors associated with the source, which impacted the detection stage. If a pilot has previously penetrated a TFR due to an EFB incorrectly displaying its location, the pilot may exclude the EFB as a source of airspace information on future flights.
Seven participants reported that previous experience with a specific situation or location influenced Stage 2: sense making. A total of three responses were categorised based on prior experience with hazardous situations or returning to a particular airport. Two responses were categorised as experience with a particular aircraft and its systems, and two additional responses were categorised as experience with that particular instrument procedure. For example, if a pilot has prior experience with Source 1 from training and usage of the source on the flight deck, they will be more accustomed to its reliability. Source 1 has the ability to display traffic information. If this information was accurate in the past and helped to avoid hazardous situations, the pilot will be more likely to base decision-making off of this source.
Six comments from participants were categorised as being related to stress or workload, and this factor influenced Stage 2: Sense Making and Stage 3: Decide and Act. The cockpit of a commercial aircraft during a potentially hazardous situation can result in high levels of workload and stress for the flight crew. All of the responses identified a lack of time as being an influencing factor on a pilot’s stress and workload. Any situation involving conflicting information increases a pilot’s workload as attention will need to be split between two or more sources in order to identify which source could be in error. A time constraint will further increase the amount of stress the pilot is under and the workload of the situation, which will have an effect on the process the pilot uses to select which source to base their decision off of.
Three comments from participants were categorised as the threat of violating a regulation or SOP. This factor influenced Stage 3: Decide and Act. Although the safety of a flight is the number one priority for any flight crew, avoiding the violation of a regulation or rule is also a major concern in commercial aviation operations. A total of three responses cited fear of a violation risk as being an influencing factor to making a decision between two conflicting sources of information. In specific hazardous situations involving the proximity of an aircraft to a TFR, some participants decided to act on Source 1 because the information matched instructions from ATC information. If avoiding a TFR displayed on Source 2 required the pilot to deviate from a clear ATC instruction, the pilot would be influenced to rely on the information presented on Source 1 in fear committing a violation.
4.0 Discussion
4.1 Implications
4.1.1 Theoretical
The theoretical model presented in this paper was developed using both top-down and bottom-up approaches that leveraged decision making frameworks in the extant literature and data from an empirical study. The model describes the process of pilot decision making with conflicting information and incorporates many of the components discussed in literature while addressing several shortcomings that prevented current models from adequately describing the process of reaching a decision under uncertainty from an information conflict. The model proposed herein follows a chronological flow that first begins when the pilot detecting the information conflict, followed by sense making and ultimately making a decision and acting. This flow is similar to that used in other decision-making frameworks (Lipshitz & Strauss; Orasanu; Prelec & Lowenstein [Reference Lipshitz and Strauss3,Reference Orasanu13,Reference Loewenstein and Prelec17]). However, the model proposed herein extends beyond Lipshitz and Strauss [Reference Lipshitz and Strauss3] and Orasanu [Reference Orasanu13] models because it takes into account these discrete stages of the decision-making process, when reconciling multiple sources of conflicting information, and the factors that influence each stage.
The model proposes that pilots must first detect the information conflict, and this is influenced by experience with this source, training or SOPs, and information clarity of the source. If there are there is time, cognitive resources and enough information, the pilot must then make sense of the situation to determine why the information conflict exists and decipher ground truth. This stage too is influenced by training, SOPs and information clarity, but also by the amount of stress and workload, the decision maker’s knowledge of how the system works, and their experience with the situation. If there is not time or resources for sense-making, or once the pilot has made sense of the situation or exhausted their resources, they will proceed to the decision stage. In this final stage, the pilot will evaluate whether there is a life-threatening situation, whether any of the courses of action will result in administrative repercussions, and the degree to which they trust each source, and make a decision regarding which source to ultimately act on. This phase is influenced by the level of stress being experienced, training SOPs, the level of risks to safety or an administrative repercussion.
This model is not only in line with empirical data, but also extant research in the literature. For example, St. John et al. found that when information uncertainty was present, greater levels of experience led to more decisiveness. Pilots receive training on emergency situations, sources that should be utilised, and proper responses (Jensen [Reference Jensen18]). This training, in conjunction with SOPs, govern how a pilot will act in specific situations. A pilot who is experienced with using a particular source during a hazardous situation is going to be better able to discern which source is more accurately representing the situation than a pilot who is inexperienced with a particular situation or source. This is in line with extant research that has found that experienced pilots were not only quicker decision makers, but also better able to make decisions involving multiple cues (Schriver et al. [Reference Schriver, Morrow, Wickens and Talleur19]). Research has also shown that if information is not clear, or difficult for pilots to understand, they are less likely to be utilise it (Pritchett [Reference Pritchett7]). There is a great deal of research illustrating the deleterious effects of decision making on the decision-making process in general (See Carroll et al. [Reference Carroll, Hale, Stanney, Woodman, Devore, Scquire and Sciarni20]). The model also proposes that functional knowledge of a source also impacts the sense-making process. If the pilot is familiar with how a source receives data and associated limitations, the pilot will be better able to make sense of the situation. This is in line with literature stating that performers must have enough functional-level system knowledge to allow accurate mental model development that can facilitate an understanding of why systems respond in particular ways in certain situations (Gilson et al. [Reference Gilson, Deaton and Mouloua21]). A pilot making a decision between two sources is influenced by the likelihood of a hazard causing physical harm to their aircraft and the consequences of committing a violation of regulations or SOPs. This is in-line with the findings of Orasanu [Reference Orasanu13] wherein a pilot must make decisions in an environment that contains threats to their safety. If necessary, a pilot may make the conservative choice in regard to the safety of their aircraft, even if the information presented on this source turns out to be inaccurate. The pilot may be less likely to utilise the same source that displays a conflict if the consequence is less severe and would result in a violation of a SOP or regulation as opposed to a threat to the physical safety of the flight. This is in line with research that has shown that individuals making decisions with conflicting information will act to minimise risk (Pritchett et al. [Reference Pritchett, Fleming, Cleveland, Zoetrum, Popescu and Thakkar9]).
4.1.2 Practical
This model provides a practical tool to assist in understand how pilots will respond to information conflicts, which can help inform effective training and SOPs in commercial aviation operations. Current guidance from the FAA provides pilots and flight instructors with insight regarding how to respond to the failure of certain cockpit instruments and systems (FAA [22]). However, with portable devices becoming more common on the flight deck and more expansive in their functionality, there is greater chance of information conflicting with existing cockpit-based displays of information. Several participants commented that the visual information on an EFB aided them in detecting a conflict between other sources of information, indicating that the introduction of new technology could result in increasing prevalence of information conflicts. Additional training and guidance should be developed to aid pilots and operators with how to respond to a situation in which two or more sources of information conflict with one another. Further, if a pilot has received training on the function of a particular source or how the information from this source is obtained, the pilot will be more able to effectively discern which source of information is more accurately representing ground truth. This model can help inform the FAA when developing regulations and recommended practices, and by airlines when considering changes to their SOPs that are related to the usage of EFBs on the commercial flight deck.
4.2 Limitations and future research
The results of this research should be considered with caution as there are several limitations. First, data from only 25 Part 121 airline pilots, type-rated on the Boeing 737 aircraft were included. A larger sample size or participants type-rated on a different aircraft could yield different results. Further, the data was associated with only the types of information conflicts within the various scenarios, namely conflicts involving TFRs or possible mid-air collisions. Pilots may respond differently to different types of information conflicts. Lastly, much of the data utilised was captured after all of the scenarios had been performed and could have been subject to forgetting, resulting in inaccuracies in the data.
Each of the limitations identified above are recommended areas of focus for potential future research. A replication study to Carroll et al. [Reference Carroll, Rebensky, Wilt, Pittorie, Hunt, Chaparro and Sanchez15] should be conducted with a larger sample size and with additional types of information conflicts. The purpose of this study would be to validate the model proposed in this paper. Further, a replication study utilising responses from pilots who are type-rated in a different aircraft or who operate outside of the commercial aviation industry should be performed to validate the generalisability of the model to performance in other aircraft and other operations.
5.0 Conclusions
The purpose of this paper was to develop a model to describe the cognitive process of a pilot making a decision when presented with conflicting information from multiple sources on the flight deck. This study was a follow-up analysis to Carroll et al. [Reference Carroll, Rebensky, Wilt, Pittorie, Hunt, Chaparro and Sanchez15], a study that collected data from a sample of Boeing 737 airline pilots who performed a series of simulated scenarios that included realistic information conflicts and events. Quantitative data from Carroll, Rebensky et al. was complemented by opened-ended data related to the pilot’s thought process used while making these decisions during a time of information uncertainty. The opened-ended questions taken from Carroll et al. were analysed using qualitative methods, which along with the related literature identified in this paper, informed the final theoretical model of pilot decision-making with conflicting information. The theoretical model includes three stages: detection, sense-making, and decide and act, each with a range of influencing factors, including prior experience with sources and situations, training and SOPs on source utilisation, information clarity, the stress and workload of a particular situation, the knowledge of the source and accuracy of information, the severity of a hazard to the safety of a flight, and the potential for a violation of regulations or procedures.
Acknowledgements
The original data collection for this effort was funded by the Federal Aviation Administration NextGen Human Factors Division (ANG-C1) under Contract # DTFAWA-16-D-00003. Such funding must not be construed as the FAA endorsing or sponsoring the content of this paper.
