Hostname: page-component-6bf8c574d5-9nwgx Total loading time: 0 Render date: 2025-02-22T14:07:47.278Z Has data issue: false hasContentIssue false
  • English
  • Français

Using EEG and eye-tracking as indicators to investigate situation awareness variation during flight monitoring in air traffic control system

Published online by Cambridge University Press:  20 February 2025

Qinbiao Li Open the ORCID record for Qinbiao Li [Opens in a new window] ,
Kam K.H. Ng Open the ORCID record for Kam K.H. Ng [Opens in a new window] ,
Simon C.M. Yu ,
Cho Yin Yiu Open the ORCID record for Cho Yin Yiu [Opens in a new window] ,
Fan Li Open the ORCID record for Fan Li [Opens in a new window]  and
Felix T.S. Chan
Qinbiao Li
Affiliation:
Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, China
Kam K.H. Ng*
Affiliation:
Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, China
Simon C.M. Yu
Affiliation:
Department of Aerospace Engineering, Khalifa University of Science and Technology, UAE
Cho Yin Yiu
Affiliation:
Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, China
Fan Li
Affiliation:
Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, China
Felix T.S. Chan
Affiliation:
Department of Decision Sciences, Macau University of Science and Technology, China
*
*Corresponding author: Kam K.H. Ng. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Identifying the absence of situation awareness (SA) in air traffic controllers is critical since it directly affects their hazard perception. This study aims to introduce and validate a multimodal methodology employing electroencephalogram (EEG) and eye-tracking to investigate SA variation within specific air traffic control contexts. Data from 28 participants executing the experiment involving three different SA-probe tests illustrated the conceptual relationship between EEG and eye-tracking indicators and SA variations, using behavioural data as a proxy. The results indicated that both EEG and eye-tracking metrics correlated positively with the SA levels required, that is, the frequency spectrum in the β (13–30 Hz) and γ (30–50 Hz) bands, alongside the fixation/saccade-based indicators and pupil dilation increased in response to higher SA levels. This research has substantial implications for investigating SA using a human-centric approach via psychophysiological indicators, revealing the intrinsic interactions between the human capability envelope and SA, contributing to the development of a real-time monitoring system of SA variations for air transportation safety research.

Keywords

ergonomicsaviationEEGeye-trackingsituation awareness
Type
Research Article
Information
The Journal of Navigation , First View , pp. 1 - 22
Check for updates
Copyright
Copyright © The Author(s), 2025. Published by Cambridge University Press on behalf of The Royal Institute of Navigation

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aricò, P., Borghini, G., Flumeri, G. D., Bonelli, S., Golfetti, A., Graziani, I., Pozzi, S., Imbert, J. P., Granger, G., Benhacene, R., Schaefer, D. and Babiloni, F. (2017). Human factors and neurophysiological metrics in air traffic control: A critical review. IEEE Reviews in Biomedical Engineering, 10, 250–263.CrossRefGoogle ScholarPubMed
Behrend, J. and Dehais, F. (2020). How role assignment impacts decision-making in high-risk environments: Evidence from eye-tracking in aviation. Safety Science, 127, 1–7.CrossRefGoogle Scholar
Borghini, G., Aricò, P., Di Flumeri, G., Cartocci, G., Colosimo, A., Bonelli, S., Golfetti, A., Imbert, J. P., Granger, G. and Benhacene, R. (2017). EEG-based cognitive control behaviour assessment: An ecological study with professional air traffic controllers. Scientific Reports, 7, 1–16.CrossRefGoogle ScholarPubMed
Cak, S., Say, B. and Misirlisoy, M. (2019). Effects of working memory, attention, and expertise on pilots’ situation awareness. Cognition, Technology & Work, 22, 85–94.CrossRefGoogle Scholar
Charles, R. L. and Nixon, J. (2019). Measuring mental workload using physiological measures: A systematic review. Applied Ergonomics, 74, 221–232.CrossRefGoogle ScholarPubMed
Claramunt, C. and Fujino, I. (2023). Navigation pattern extraction from AIS trajectory big data via topic model. Journal of Navigation, 76(4-5), 506–524.Google Scholar
Dasari, D., Shou, G. and Ding, L. (2017). ICA-Derived EEG correlates to mental fatigue, effort, and workload in a realistically simulated Air traffic control task. Frontiers in Neuroscience, 11, 297.CrossRefGoogle Scholar
Dehais, F., Duprès, A., Blum, S., Drougard, N., Scannella, S., Roy, R. N. and Lotte, F. (2019). Monitoring pilot's mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors, 19, 1324.CrossRefGoogle ScholarPubMed
Dussault, C., Jouanin, J.-C., Philippe, M. and Guezennec, C.-Y. (2005). EEG and ECG changes during simulator operation reflect mental workload and vigilance. Aviation, Space, and Environmental Medicine, 76, 344–351.Google ScholarPubMed
Eklund, R. and Osvalder, A.-L. (2021). Optimising aircraft taxi speed: Design and evaluation of new means to present information on a head-up display. Journal of Navigation, 74, 1305–1335.CrossRefGoogle Scholar
Endsley, M. R. (1995). Measurement of situation awareness in dynamic systems. Human Factors, 37, 65–84.CrossRefGoogle Scholar
Endsley, M. R. (1999). Situation Awareness in Aviation Systems. Handbook of Aviation Human Factors. Mahwah, NJ, USA: Lawrence Erlbaum Associates Publishers.Google Scholar
Fabbri, T. and Vicen-Bueno, R. (2021). Decision-making methodology in environmentally-conditioned ship operations based on ETD–ETA windows of opportunity. Journal of Navigation, 74, 1219–1237.CrossRefGoogle Scholar
Fernandez Rojas, R., Debie, E., Fidock, J., Barlow, M., Kasmarik, K., Anavatti, S., Garratt, M. and Abbass, H. (2019). Encephalographic assessment of situation awareness in teleoperation of human-swarm teaming. In: Gedeon, T., Wong, K. W. and Lee, M. (eds.). Neural Information Processing, 2019, Cham: Springer International Publishing, 530–539.CrossRefGoogle Scholar
Hu, X. and Lodewijks, G. (2020). Detecting fatigue in car drivers and aircraft pilots by using non-invasive measures: The value of differentiation of sleepiness and mental fatigue. Journal of Safety Research, 72, 173–187.CrossRefGoogle ScholarPubMed
Jung, T.-P., Makeig, S., Stensmo, M. and Sejnowski, T. J. (1997). Estimating alertness from the EEG power spectrum. IEEE Transactions on Biomedical Engineering, 44, 60–69.CrossRefGoogle ScholarPubMed
Kästle, J. L., Anvari, B., Krol, J. and Wurdemann, H. A. (2021). Correlation between situational awareness and EEG signals. Neurocomputing, 432, 70–79.CrossRefGoogle Scholar
Li, Q., Ng, K. K. H., Fan, Z., Yuan, X., Liu, H. and Bu, L. (2021a). A human-centred approach based on functional near-infrared spectroscopy for adaptive decision-making in the air traffic control environment: A case study. Advanced Engineering Informatics, 49, 101325.CrossRefGoogle Scholar
Li, Q., Yiu, C. Y., Yu, S. C. M. and Ng, K. K. H. (2021b). Situational Awareness and Flight Approach Phase Event Recognition Based on Psychophysiological Measurements. 2021 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM).CrossRefGoogle Scholar
Li, Q., Ng, K. K. H., Chu, S. T., Lau, T. Y. and Leung, C. H. (2023a). Revealing the Effects of Increased Workload and Distraction on the Pilot's Situation Awareness Neurobehavioral Activities. AIAA AVIATION 2023 Forum. American Institute of Aeronautics and Astronautics.CrossRefGoogle Scholar
Li, Q., Ng, K. K. H., Yu, S. C. M., Yiu, C. Y. and Lyu, M. (2023b). Recognising situation awareness associated with different workloads using EEG and eye-tracking features in air traffic control tasks. Knowledge-Based Systems, 260, 110179.CrossRefGoogle Scholar
Li, Q., Chen, C.-H., Ng, K. K. H., Yuan, X. and Yin Yiu, C. (2024). Single-pilot operations in commercial flight: Effects on neural activity and visual behaviour under abnormalities and emergencies. Chinese Journal of Aeronautics, 37, 277–292.CrossRefGoogle Scholar
Liang, N., Yang, J., Yu, D., Prakah-Asante, K. O., Curry, R., Blommer, M., Swaminathan, R. and Pitts, B. J. (2021). Using eye-tracking to investigate the effects of pre-takeover visual engagement on situation awareness during automated driving. Accident Analysis & Prevention, 157, 106143.CrossRefGoogle ScholarPubMed
Lu, Z., Happee, R. and de Winter, J. C. (2020). Take over! a video-clip study measuring attention, situation awareness, and decision-making in the face of an impending hazard. Transportation Research Part F: Traffic Psychology and Behaviour, 72, 211–225.CrossRefGoogle Scholar
Lyu, M., Li, F., Xu, G. and Han, S. (2023). Leveraging eye-tracking technologies to promote aviation safety- a review of key aspects, challenges, and future perspectives. Safety Science, 168, 106295.Google Scholar
Mclntosh, C. 2018. Situational Awareness and Decision Making – More than technology. [Online]. Available at: https://www.linkedin.com/pulse/situational-awareness-decision-making-more-than-chris-mcintosh/ [Accessed].Google Scholar
Michel, C., Lehmann, D., Henggeler, B. and Brandeis, D. (1992). Localization of the sources of EEG delta, theta, alpha and beta frequency bands using the FFT dipole approximation. Electroencephalography and Clinical Neurophysiology, 82, 38–44.CrossRefGoogle ScholarPubMed
Ng, K. K. H., Lee, C. K. M., Chan, F. T. S. and Qin, Y. (2017). Robust aircraft sequencing and scheduling problem with arrival/departure delay using the min-max regret approach. Transportation Research Part E: Logistics and Transportation Review, 106, 115–136.CrossRefGoogle Scholar
Ng, K. K. H., Lee, C. K. M., Chan, F. T. S., Chen, C.-H. and Qin, Y. (2020a). A two-stage robust optimisation for terminal traffic flow problem. Applied Soft Computing, 89, 106048.CrossRefGoogle Scholar
Ng, K. K. H., Lee, C. K. M., Zhang, S. Z. and Keung, K. L. (2020b). The impact of heterogeneous arrival and departure rates of flights on runway configuration optimization. Transportation Letters, 14, 215–226.CrossRefGoogle Scholar
Ng, K. K. H., Chen, C.-H., Lee, C. K. M., Jiao, J. and Yang, Z.-X. (2021). A systematic literature review on intelligent automation: Aligning concepts from theory, practice, and future perspectives. Advanced Engineering Informatics, 47, 101246.CrossRefGoogle Scholar
Nguyen, T., Lim, C. P., Nguyen, N. D., Gordon-Brown, L. and Nahavandi, S. (2019). A review of situation awareness assessment approaches in aviation environments. IEEE Systems Journal, 13, 3590–3603.CrossRefGoogle Scholar
Ohneiser, O., De Crescenzio, F., Di Flumeri, G., Kraemer, J., Berberian, B., Bagassi, S., Sciaraffa, N., Aricò, P., Borghini, G. and Babiloni, F. (2018). Experimental simulation set-up for validating out-of-the-loop mitigation when monitoring high levels of automation in air traffic control. International Journal of Aerospace and Mechanical Engineering, 12, 379–390.Google Scholar
Peißl, S., Wickens, C. D. and Baruah, R. (2018). Eye-tracking measures in aviation: A selective literature review. The International Journal of Aerospace Psychology, 28, 98–112.CrossRefGoogle Scholar
Sanei, S. and Chambers, J. A. (2013). EEG Signal Processing. UK: John Wiley & Sons.Google Scholar
Taylor, R. M. 2017. Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Situational Awareness. France (FRA): Routledge, 478, 3.1–3.17.Google Scholar
Trapsilawati, F., Chen, C.-H., Wickens, C. D. and Qu, X. (2021). Integration of conflict resolution automation and vertical situation display for on-ground air traffic control operations. Journal of Navigation, 74, 619–632.CrossRefGoogle Scholar
Vanderhaegen, F., Wolff, M. and Mollard, R. (2020). Non-conscious errors in the control of dynamic events synchronized with heartbeats: A new challenge for human reliability study. Safety Science, 129, 104814.CrossRefGoogle Scholar
van Weelden, E., Alimardani, M., Wiltshire, T. J. and Louwerse, M. M. (2022). Aviation and neurophysiology: A systematic review. Applied Ergonomics, 105, 103838.CrossRefGoogle ScholarPubMed
Wang, Y., Wang, L., Lin, S., Cong, W., Xue, J. and Ochieng, W. (2021). Effect of working experience on air traffic controller eye movement. Engineering, 7, 488–494.CrossRefGoogle Scholar
Weiergraeber, M., Papazoglou, A., Broich, K. and Mueller, R. (2016). Sampling rate, signal bandwidth and related pitfalls in EEG analysis. Journal of Neuroscience Methods, 268, 53–55.CrossRefGoogle Scholar
Yeong Heok, L., Jeong-Dae, J. and Youn-Chul, C. (2012). Air traffic controllers’ situation awareness and workload under dynamic air traffic situations. Transportation Journal, 51, 338–352.Google Scholar
Yoon, S. H. and Ji, Y. G. (2019). Non-driving-related tasks, workload, and takeover performance in highly automated driving contexts. Transportation Research Part F: Traffic Psychology and Behaviour, 60, 620–631.CrossRefGoogle Scholar
Zhang, T., Yang, J., Liang, N., Pitts, B. J., Prakah-Asante, K. O., Curry, R., Duerstock, B. S., Wachs, J. P. and Yu, D. (2020). Physiological measurements of situation awareness: A systematic review. Human Factors, 65(5), 737–758.CrossRefGoogle ScholarPubMed
Zhou, F., Yang, X. J. and de Winter, J. C. (2021). Using Eye-Tracking Data to Predict Situation Awareness in Real Time During Takeover Transitions in Conditionally Automated Driving. IEEE Transactions on Intelligent Transportation Systems.Google Scholar