Automatic Communication Error Detection Using Speech Recognition and Linguistic Analysis for Proactive Control of Loss of Separation

With growing air traffic density, air-traffic controllers (ATCOs) are facing more challenges in interpreting and analyzing air traffic information. As one of the solutions to this problem, automation supports such as tactile human computer interface, interactive 3D radar displays, and conflict resolution aid (CRA) are proposed for the enhancement of the current air traffic control (ATC) systems. To evaluate the proposed ATC systems, questionnaires are commonly used to get the feedback from ATCOs. However, the questionnaires are usually administered upon completion of each ATC simulation task thus provide only overall ratings towards the new ATC systems. In this paper, we propose and implement a novel Electroencephalogram (EEG)-based neurocognitive tools for evaluation of ATC systems. The nature of EEG-based technique is that the brain states can be monitored in a high resolution time, fitting the nature of time-critical ATC tasks. Thus, such EEG-based human factors study allows for real-time monitoring of ATCOs' mental workload during the task performance in ATC systems. We designed and conducted an experiment to evaluate the costs and benefits of the CRA and tactile user interface in future ATC systems. Thirty six participants participated in the experiment and were assigned into three groups of different display modes: Non-Display, Display, and Trajectory Prediction. In each group, three CRA conditions were given: Manual, Reliable and Unreliable. The EEG data were recorded during the tasks, and the traditional workload evaluation method NASA Task Load Index (TLX) was administered at the end of each task. The result shows that the ratings obtained from NASA TLX and from the EEG labeling are highly correlated. Thus, the EEG-based system is reliable to recognize workload during the task performance. With the proposed EEG-based system, we found that 1) relatively high workload was observed at the beginning of the experiment in each group which could be due to participants' unfamiliarity with the interfaces; 2) participants in the trajectory prediction group had much higher workload as compared to vertical display and non-display groups, which could be attributed to its complexity. The changes of workload are due to interaction between the type of display modes and time (p<0.05); 3) workload varied slightly with different CRA setting. The results show that the proposed EEG-based system for human factors study can provide better understanding of real-time mental workload changes during the task performance in new ATC systems, therefore enables the evaluation of current and future ATC systems.

A path object is a set of instructions plus the values of associated parameters that would be used by an aircraft’s flight management system (FMS) or area navigation (RNAV) computer to construct a flight trajectory on the basis of the values of the parameters provided by the pilot or air traffic control (ATC) system. The concept of path objects and its applicability for development of FMS- or RNAV-based flight paths for aircraft routes are described. The use of path objects requires only a small number of parameters for specification of an RNAV route, and the computer then calculates the resulting flight path. Because most RNAV routes are just variations of the same basic shapes, many RNAV routes can be generated from a single path object by changing just one or two parameters. This offers the capability to reduce the size of the navigation database by a significant amount. This also allows the dynamic alteration of three-dimensional FMS-RNAV routes instead of radar vectoring, which represents a significant improvement in ATC procedures. Because of the compact expressions for the path objects, these procedures could be used in a voice communications environment as well as a data link environment. A path object can be thought of as a high-level language with which aircraft and ATC systems communicate flight path intentions. The concept of FMS- or RNAV-stored path objects is a change in thinking about the role of avionics technology. It offers a means of including the precision of the FMS-RNAV technology in the ATC system during the transition from today’s system to the future’s fully automated control system. This is an enabling technology that improves the ability to use and maintain FMS-RNAV in its role as a vital component of the ATC system. It has the potential to enhance future air traffic management-communication navigation surveillance concepts and applications.

Automatic detection of communication errors in conversational systems has been explored extensively in the speech community. However, most previous studies have used only acoustic cues. Visual information has also been used by the speech community to improve speech recognition in dialogue systems, but this visual information is only used when the speaker is communicating vocally. A recent perceptual study indicated that human observers can detect communication problems when they see the visual footage of the speaker during the system's reply. In this paper, we present work in progress towards the development of a communication error detector that exploits this visual cue. In datasets we collected or acquired, facial motion features and head poses were estimated while users were listening to the system response and passed to a classifier for detecting a communication error. Preliminary experiments have demonstrated that the speaker's visual information during the system's reply is potentially useful and accuracy of automatic detection is close to human performance.

Aviation performance depends on education and training of pilots and air traffic controllers (ATCOs) on technical, procedural, crisis management, decision making, leadership and communication skills. Recent incidents and studies, however, show crew members may be caught unaware and errors in human judgement can still occur in air traffic control (ATC) systems even more so in the presence of cyberattacks. This paper focuses on raising situation awareness and decision making of ATCOs with cyberattacks impacting operations in next-generation ATC systems. Our goal is to investigate crew-based cyber security as a layer in making ATC systems resilient to cyber and cyber-physical attacks. Automatic Dependent Surveillance Broadcast (ADS-B) technology is used as the basis of our preliminary investigation due to its widely known security concerns and central role in ATC systems. We present experimental study considerations for assessing cyber readiness and improving training of ATCOs. We propose a Kalman filter based cyber alarm solution approach for a machine learning based aid for ATCOs towards early cyberattack detection and incident response in ATC systems.

Air traffic controllers (ATCos) monitor flight operations and resolve predicted aircraft conflicts to ensure safe flights, making them one of the essential human operators in air traffic control systems. Researchers have been studying ATCos with human subjective approaches to understand their tasks and air traffic managing processes. As a result, models were developed to predict ATCo actions. However, there is a gap between our knowledge and the real-world. The developed models have never been validated against the real-world, which creates uncertainties in our understanding of how ATCos detect and resolve predicted aircraft conflicts. Moreover, we do not know how information from air traffic control systems affects their actions. This Ph.D. dissertation work introduces methods to evaluate existing ATCo action prediction models. It develops a prediction model based on flight contextual information (information describing flight operations) to explain the relationship between ATCo actions and information. Unlike conventional approaches, this work takes data-driven approaches that collect large-scale flight tracking data. From the collected real-world data, ATCo actions and corresponding predicted aircraft conflicts were identified by developed algorithms. Comparison methods were developed to measure both qualitative and quantitative differences between solutions from the existing prediction models and ATCo actions on the same aircraft conflicts. The collected data is further utilized to develop an ATCo action prediction model. A hierarchical structure found from analyzing the collected ATCo actions was applied to build a structure for the model. The flight contextual information generated from the collected data was used to predict the actions. Results from this work found that the collected ATCo actions do not show any preferences on the methods to resolve aircraft conflicts. Results found that the evaluated existing prediction model does not reflect the real-world. Also, a large portion of the real conflicts was to be solved by the model both physically and operationally. Lastly, the developed prediction model showed a clear relationship between ATCo actions and applied flight contextual information. These results suggest the following takeaways. First, human actions can be identified from closed-loop data. It could be an alternative approach to collect human subjective data. Second, the importance of evaluating models before implications. Third, potentials to utilize the flight contextual information to conduct high-end prediction models.

Recently, artificial intelligence and data science have witnessed dramatic progress and rapid growth, especially Automatic Speech Recognition (ASR) technology based on Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). Consequently, new end-to-end Recurrent Neural Network (RNN) toolkits were developed with higher speed and accuracy that can often achieve a Word Error Rate (WER) below 10%. These toolkits can nowadays be deployed, for instance, within aircraft cockpits and Air Traffic Control (ATC) systems in order to identify aircraft and display recognized voice messages related to flight data, especially for airports not equipped with radar. Hence, the performance of air traffic controllers and pilots can ultimately be improved by reducing workload and stress and enforcing safety standards. Our experiment conducted at Tangier’s International Airport ATC aimed to build an ASR model that is able to recognize aircraft call signs in a fast and accurate way. The acoustic and linguistic models were trained on the Ibn Battouta Speech Corpus (IBSC), resulting in an unprecedented speech dataset with approved transcription that includes real weather aerodrome observation data and flight information with a call sign captured by an ADS-B receiver. All of these data were synchronized with voice recordings in a structured format. We calculated the WER to evaluate the model’s accuracy and compared different methods of dataset training for model building and adaptation. Despite the high interference in the VHF radio communication channel and fast-speaking conditions that increased the WER level to 20%, our standalone and low-cost ASR system with a trained RNN model, supported by the Deep Speech toolkit, was able to achieve call sign detection rate scores up to 96% in air traffic controller messages and 90% in pilot messages while displaying related flight information from ADS-B data using the Fuzzy string-matching algorithm.

The identification of backup radio sites, to increase the availability of the future air/ground (A/G) air traffic control (ATC) systems, is investigated, When there is an A/G communications failure, due to a radio site or an ATC controller access failure, another site (or sites) provides the service. The backup site should be capable of providing full coverage of the desired airspace known as the service volume. Meeting the full coverage conditions by one site or a group of sites is analyzed. It is concluded that identification of one site providing full coverage can be determined easily. The identification of a group of sites, each with partial coverage and providing full coverage is analyzed in detail. An algorithm involving both analytical and numerical approaches is described. It determines if two radio sites with given geographical coordinates, and each with partial coverage of a given cylindrical service volume, can provide full coverage. This algorithm can be implemented as a computer program to process a database of radio sites.

Based on a long-term prediction by the International Civil Aviation Organization indicating steady increases in air traffic demand throughout the world, the workloads of air traffic controllers are expected to continuously increase. Air traffic control and management (ATC/M) includes the processing of various unstructured composite data along with the real-time visualization of aircraft data. To prepare for future air traffic, research and development intended to effectively present various complex navigation data to air traffic controllers is necessary. This paper presents a mixed reality-based air traffic control system for the improvement of and support for air traffic controllers’ workflow using mixed reality technology that is effective for the delivery of information such as complex navigation data. The existing control systems involve difficulties in information access and interpretation. Therefore, taking notice of the necessity for the integration of air traffic control systems, this study presents the mixed reality (MR) system, which is a new approach, that enables the control of air traffic in interactive environments. This system is provided in a form usable in actual operational environments with a head-mounted see-through display installed with a controller to enable more structured work support. In addition, since this system can be controlled first-hand by air traffic controllers, it provides a new experience through improved work efficiency and productivity.

We present an overview of the most common techniques used in automatic speech recognition to adapt a general system to a different environment (known as cross-task adaptation) such as in an air traffic control system (ATC). The conditions present in ATC are very specific: very spontaneous, the presence of noise, and high speed speech. So, with a typical speech recognizer the recognition results are unsatisfactory. We have to decide on the best option for the modeling: to develop acoustic models specific to those conditions from scratch using the data available for the new environment, or to carry out cross-task adaptation starting from reliable HMM models (usually requiring less data in the target domain). We begin with a description of the main techniques considered for cross-task adaptation, namely maximum a posteriori (MAP), maximum likelihood linear regression (MLLR), and the two together. We have applied each in two speech recognizers for air traffic control tasks, one for spontaneous speech and the other for a command interface. We show the performance of these techniques and compare them with the development of a new system from scratch. We also show the results obtained for speaker adaptation using a variable amount of adaptation data. The main conclusion is that MLLR can outperform MAP when a large number of transforms is used, and MLLR followed by MAP is the best option. All of these techniques are better than developing a new system from scratch, showing the effectiveness of mean and variance adaptation

The main objective of Air Traffic Control (ATC) systems is to maintain safe separation between aircrafts by providing proper instructions to pilots. ATC automation is a component of ATC systems, which involves with the processing and display of operational information. In order to improve performance of ATC, operational condition evaluation is important. A critical index for evaluating ATC automation system is reliability. In this paper, a safety evaluation index system for ATC automation system considering reliability index is proposed on the basis of the survey of failure records from several airports in recent years. The final evaluation index is calculated using analytic hierarchy process (AHP).

BackgroundRecent advances in natural language processing and artificial intelligence have led to widespread adoption of speech recognition technologies. In consumer health applications, speech recognition is usually applied to support interactions with conversational agents for data collection, decision support, and patient monitoring. However, little is known about the use of speech recognition in consumer health applications and few studies have evaluated the efficacy of conversational agents in the hands of consumers. In other consumer-facing tools, cognitive load has been observed to be an important factor affecting the use of speech recognition technologies in tasks involving problem solving and recall. Users find it more difficult to think and speak at the same time when compared to typing, pointing, and clicking. However, the effects of speech recognition on cognitive load when performing health tasks has not yet been explored.ObjectiveThe aim of this study was to evaluate the use of speech recognition for documentation in consumer digital health tasks involving problem solving and recall.MethodsFifty university staff and students were recruited to undertake four documentation tasks with a simulated conversational agent in a computer laboratory. The tasks varied in complexity determined by the amount of problem solving and recall required (simple and complex) and the input modality (speech recognition vs keyboard and mouse). Cognitive load, task completion time, error rate, and usability were measured.ResultsCompared to using a keyboard and mouse, speech recognition significantly increased the cognitive load for complex tasks (Z=–4.08, P<.001) and simple tasks (Z=–2.24, P=.03). Complex tasks took significantly longer to complete (Z=–2.52, P=.01) and speech recognition was found to be overall less usable than a keyboard and mouse (Z=–3.30, P=.001). However, there was no effect on errors.ConclusionsUse of a keyboard and mouse was preferable to speech recognition for complex tasks involving problem solving and recall. Further studies using a broader variety of consumer digital health tasks of varying complexity are needed to investigate the contexts in which use of speech recognition is most appropriate. The effects of cognitive load on task performance and its significance also need to be investigated.

The human-machine problem in automation systems is generally limited to control tasks in self-contained (i.e., clearly definable) technical processes. Integrated air traffic management and control systems, however, are not definable processes, as the amount and the purpose of traffic vary. Persons charged by different organizations try to override the other claims by means of their automation equipment. In order to avoid dangerous competition, there are some requirements for the design of the automation system. Analyzing the general air traffic management problem, the influence of the system architecture on the verification and validation of the system function is to be shown.

The continuously increasing air traffic density has become a major challenge in air traffic control (ATC) due to the current ATC systems are approaching maximum capacity. To deal with the problem, an automated conflict resolution aid (CRA) and a trajectory prediction aid (TPA) have been proposed to serve as additional safety layers in the ATC systems. However, whether the proposed automation aids are worth to be applied in the current ATC workplace and could better support air traffic controllers (ATCOs) remain unknown. This study aims to investigate the effects of the proposed automations on ATCOs' workload and situation awareness (SA). To do so, twenty-four participants were evenly divided into two groups corresponding to the presence and absence of the TPA. In each condition, participants were instructed to perform simulated ATC tasks with the double of current air traffic load under the presence and absence of the CRA. The results showed that the CRA benefits ATCOs' workload and SA. The application of the CRA alone could benefit ATCOs while the presence of the TPA alone did not offer valuable benefits for ATCOs. Importantly, the CRA could lower the workload substantially when it was integrated with the TPA. The automation design aspect and its application in enhancing safety of future ATC are also highlighted.

This paper presents a simulation system for air traffic control (ATC) training. The structure of the ATC is analyzed to define simulation components to be developed. Requirements that must be satisfied by a training simulator are also identified. The results were reflected in the design of the simulator. The training simulator is composed of three modules same as in real life : air traffic controller, aircraft, radar. Each developed module performs the strict unit tests and combine test based on the scalability, ease, real-time, the operating range of the algorithm derived from the requirements. The simulation system can be connected to an actual ATC, and used to validate the ATC system. At peak load, 1200 aircraft and 30 radars can be operated simultaneously.

Formal methods is an emerging technology that uses mathematical notations to write precise and unambiguous specifications which makes it possible to prove and analyze certain properties of the system so that errors and inconsistencies are identified during early stages of the development process. In this paper formal methods in terms of Z notation is applied for the specification of safety critical system of Air Traffic Control (ATC). Firstly, ATC system model in real world is described. For connectivity of different zones of airspace, the real world ATC system is transformed into a directed graph, which is then used to formalize the major components of formal ATC Model i.e static Topology, Network State, Aircraft and Controller. The whole Formal ATC Model is then presented as encapsulation of formal models of its basic components. Finally, the Formal A TC system Model is checked and analyzed with Z/EVES tool-set.