Multimodal control system for autonomous vehicles using speech and gesture recognition

Nowadays, the development of modern technologies in the transportation area has increased rapidly. The latest technology is focused on autonomous vehicles. The most important aspect in autonomous vehicle control systems is safety, smooth operation and comfort. It can be provided by the natural element which is from human behavior as a reference to the autonomous control system. This paper is briefly describing the human behavior during braking for autonomous electric vehicle (AEV) control system development. To obtain the human behaviour during accelerating and braking, two units of angle sensors were installed to the accelerator and brake pedal of an electric car to measure the angle of both pedals. Real-time speed is recorded by using a GPS unit. The experiment is focused on braking characteristic of the electric vehicle. It was carried out based on four different distances with fixed initial speed before braking action was recorded. A software interface was designed to display and control real-time speed and pedal angle value during operation.

The development of autonomous vehicles has recently become the focus of much research. One area of investigation is the development of systems that will allow people to easily direct autonomous vehicles without any special technical skills or training. Human-machine interface technologies are likely to be used to achieve this objective. In this study, we propose an intuitive, multimodal interface system that uses speech, gesture, and gaze recognition to transmit user commands to the vehicle. We designed the multimodal understanding and dialog control components of the interface system separately, using finite-state transducers (FSTs), a technique which is also used to control conventional automated dialog systems. Our multimodal understanding and dialog control components can be thought of as a cascade of two separate transducers, but since cascading transducers can be combined into one transducer, our system is actually driven by a single FST. During trials, our proposed interface system allowed users to successfully direct and operate an autonomous vehicle using only speech, head movement, and gaze direction.

The rapid advancement of deep learning has significantly transformed the development of next-generation autonomous vehicles, enhancing both safety and efficiency. This paper explores the integration of deep learning techniques, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and reinforcement learning, in perception, decision-making, and control systems of autonomous vehicles. By leveraging vast datasets and real-time processing, deep learning enables precise object detection, path planning, and adaptive driving strategies. Furthermore, the implementation of sensor fusion techniques combining LiDAR, radar, and cameras enhances situational awareness, reducing the risk of accidents. Despite these advancements, challenges such as computational complexity, adversarial robustness, and ethical considerations remain key research areas. This study provides an overview of the current state-of-the-art deep learning applications in autonomous vehicles and discusses future directions toward fully autonomous, safer, and more efficient transportation systems.

This article presents an in‐depth review of path following control strategies that are applicable to a wide range class of marine, ground, and aerial autonomous robotic vehicles. From a control system standpoint, path following can be formulated as the problem of stabilizing a path following error system that describes the dynamics of position and possibly orientation errors of a vehicle with respect to a path, with the errors defined in an appropriate reference frame. In spite of the large variety of path following methods described in the literature we show that, in principle, most of them can be categorized in two groups: stabilization of the path following error system expressed either in the vehicle's body frame or in a frame attached to a “reference point” moving along the path, such as a Frenet‐Serret (F‐S) frame or a Parallel Transport (P‐T) frame. With this observation, we provide a unified formulation that is simple but general enough to cover many methods available in the literature. We then discuss the advantages and disadvantages of each method, comparing them from the design and implementation standpoint. We further show experimental results of the path following methods obtained from field trials testing with under‐actuated and over‐actuated autonomous marine vehicles. In addition, we introduce open‐source Matlab and Gazebo/ROS simulation toolboxes that are helpful in testing path following methods before their integration in the combined guidance, navigation, and control systems of autonomous vehicles.

In this paper a novel method is presented for hierarchical control of autonomous vehicles consisting of high-level (HLC) and low-level (LLC) control systems. The LLC system can be commanded to assure a prescribed velocity and steering profile using PID-type active suspension, velocity and steering control. The outputs of the LLC system are the generalized forces and moments required by the vehicle. Because of its nature, the LLC system cannot eliminate position and orientation errors caused by the unknown centre of gravity point, drifting and time-varying friction. These errors are compensated by the HLC which uses the kinematic model of the vehicle to generate the reference trajectory from the prescribed profiles and employs a linear quadratic optimal controller to compensate the tracking errors. The control algorithm minimizes the perturbations and the optimal solution is realized as a time-varying state-feedback control law. The efficiency of the hierarchical vehicle control system is validated for realistic motion maneuvers by simulation on the nonlinear vehicle model using multibody robotic formalism.

Transportation sustainability is adversely affected by recurring traffic congestions, especially at urban intersections. Frequent vehicle deceleration and acceleration caused by stop-and-go behaviours at intersections due to congestion adversely impacts energy consumption and ambient air quality. Availability of the maturing vehicle technologies such as autonomous vehicles and Vehicle-To-Vehicle (V2V) / Vehicle-To-Infrastructure (V2I) communications provides technical feasibility to develop solutions that can reduce vehicle stops at intersections, hence enhance the sustainability of intersections. This paper presents a next-generation intersection control system for autonomous vehicles, which is named ACUTA. ACUTA employs an enhanced reservation-based control algorithm that controls autonomous vehicles’ passing sequence at an intersection. Particularly, the intersection is divided into n-by-n tiles. An intersection controller reserves certain time-space for each vehicle, and assures no conflict exists between reservations. The algorithm was modelled in microscopic traffic simulation platform VISSIM. ACUTA algorithm modelling as well as enhancement strategies to minimize vehicle intersection stops and eventually emission and energy consumption were discussed in the paper. Sustainability benefits offered by this next-generation intersection were evaluated and compared with traditional intersection control strategies. The evaluation reveals that multi-tile ACUTA reduces carbon monoxide (CO) and Particulate Matter (PM) 2.5 emissions by about 5% under low to moderate volume conditions and by about 3% under high volume condition. Meanwhile, energy consumption is reduced by about 4% under low to moderate volume conditions and by about 12% under high volume condition. Compared with four-way stop control, single-tile ACUTA reduces CO and PM 2.5 emissions as well as energy consumption by about 15% under any prevailing volume conditions. These findings validated the sustainability benefits of employing next-generation vehicle technologies in intersection traffic control. In addition, extending the ACUTA to corridor level was explored in the paper.

Intelligent ground vehicles require automated steering control systems. The complexity of the automated steering systems has increased as the concurrent data receiving, processing, decision-making and monitoring all happen simultaneously in super-fast sampling rates. The research reported in this paper focuses on the neural network training and machine learning approach for an automated steering system. The essential information of the steering controller was trained using artificial neural network (ANN) and pattern recognition algorithm approaches. The objective is to investigate the design of ANN training and deep machine learning on collected data from sensors of steering wheel angular position, speed, steering column torque, and ego vehicle speed and generating acceptable steering commands for the vehicle. The results of the research showed that the proposed network control system had trained and validated more than 96.5% steering system behavior patterns and adapted large random disturbances of the steering controller commands. It is, therefore, necessary to develop artificial intelligence methodologies in automated steering systems of autonomous vehicles with neural network representing the main topology blocks of the control system architecture and utilize ANN abstraction in the control system of autonomous vehicle steering control system.

In this paper, the problem of motion control for small autonomous underwater vehicle (SAUV) is addressed. Firstly, for the position detection problem of SAUV underwater sailing, the dead reckoning method based on Doppler Velocity Log (DVL), compass and depth gauge is proposed. Then, aiming at the problem of controller designing for SAUV trajectory tracking, the segmented control strategy combining PD-sliding mode control with traditional PID control is proposed. And then, the angle priority thrust allocation method is introduced. Finally, the pool experiments are carried out to verify the effectiveness of the proposed method.

For the speed control system of autonomous electric vehicle (AEV), challenge happens with how to determine an appropriate driving speed to satisfy the dynamic environment while resisting uncertainty and disturbance. Therefore, this paper proposes a robust optimal speed control approach based on hierarchical architecture for AEV through combining deep reinforcement learning (DRL) and robust control. In decision‐making layer, a deep maximum entropy proximal policy optimization (DMEPPO) algorithm is presented to obtain an optimal speed via dynamic environment information, heuristic target entropy and adaptive entropy constraint. In motion control layer, to track the learned optimal speed while resisting uncertainty and disturbance, a robust speed controller is designed by the linear matrix inequality (LMI). Finally, simulation experiment results show that the proposed robust optimal speed control scheme based on hierarchical architecture for AEV is feasible and effective.

Robust control for an autonomous underwater vehicle that suppresses pitch and yaw coupling

This paper introduces a robust yet straightforward lane detection and lateral control approach via the deployment of a dual camera based on the look-down strategy for autonomous vehicles. Unlike traditional single-camera systems that rely on the look-ahead methodology and a single front-facing preview, the proposed algorithm leverages two downward-facing cameras mounted beneath the vehicle’s driver and the passenger side mirror, respectively. This configuration captures the road surface, enabling precise detection of the lateral boundaries, particularly during lane changes and in narrow lanes. A Proportional-Integral-Derivative (PID) controller is designed to maintain the vehicle’s position in the center of the road. We compare this system’s accuracy, lateral steadiness, and computational efficiency against (1) a conventional bird’s-eye view lane detection method and (2) a popular deep learning-based lane detection framework. Experiments in the CARLA simulator under varying road geometries, lighting conditions, and lane marking qualities confirm that the proposed look-down system achieves superior real-time performance, comparable lane detection accuracy, and reduced computational overhead relative to both traditional bird’s-eye and advanced neural approaches. These findings underscore the practical benefits of a straightforward, explainable, and resource-efficient solution for robust autonomous vehicle lane-keeping.

With the advancements in utilizing Artificial Intelligence (AI) in embedded safety-critical systems based on Field-Programmable Gate Arrays (FPGAs), assuring that these systems meet their safety requirements is of paramount importance for their revenue service. Based on this context, this paper has two main objectives. The first of them is to present the Safety ArtISt method, developed by the authors to guide the lifecycle of AI-based safety-critical systems, and emphasize its FPGA-oriented tasks and recommended practice towards safety assurance. The second one is to illustrate the application of Safety ArtISt with an FPGA-based braking control system for autonomous vehicles relying on explainable AI generated with High-Level Synthesis. The results indicate that Safety ArtISt played four main roles in the safety lifecycle of AI-based systems for FPGAs. Firstly, it provided guidance in identifying the safety-critical role of activities such as sensitivity analyses for numeric representation and FPGA dimensioning to achieve safety. Furthermore, it allowed building qualitative and quantitative safety arguments from analyses and physical experimentation with actual FPGAs. It also allowed the early detection of safety issues—thus reducing project costs—and, ultimately, it uncovered relevant challenges not discussed in detail when designing safety-critical, explainable AI for FPGAs.

This paper designed a control system of small autonomous underwater vehicle(AUV). A distributed control architecture that is MOOS-IvP is applied in this control system. Using this control architecture, there are distribute and modular characteristic in this control system. It comprises of the sensor module, the propulsion module, the fault detection module, the data log module, the device initialize module, the navigation module, the control module and the intelligent decision module. This control system can realize the constant depth control, the constant heading control, the constant speed control and the waypoint behavior control. We finished these control experiments and verified this control system by the lake experiment in June 2015.

We present a three-dimensional (3D) intersection traffic management platform for small autonomous Unmanned Aerial Vehicles (UAVs), particularly quadcopters, in urban airspace. Assuming many autonomous UAVs are approaching a shared airspace, where UAVs have varying sources and destinations, we propose a system model for a 3D intersection that aims to provide safe and systematic management of UAVs. We also devised a scheduling scheme to ensure that the intersection is efficiently utilized and that there are no collisions among the UAVs in the intersection. The scheduling scheme applies the reservation-based approach, which is sensitive to the sequence of the UAVs in scheduling, thus genetic algorithm is used to determine the best sequence of the UAVs. Simulations were performed to evaluate the efficiency of the system. We also show through the simulations that our scheduling scheme reduces the UAVs’ average time in the system by 27 percent compared with when the UAVs are scheduled in a first-come, first-served manner for the highly crowded intersection.

Our paper bases its research on the subject of deep learning to detect and predict braking events in Autonomous Vehicles. We used a Gradient Boosting (XGBoost) which is trained on data from IMU and GPS sensors from multiple cars. The model achieved high accuracy of 82.48% in classification of braking events by making use of features that were derived using vehicle kinematics. This intelligent framework surpassed old heuristic algorithms, and has a robust and extensible answer to a highly significant AV challenges. This research aims in helping the development of autonomous vehicles by advancing the perception and control system by means of accurate breaking identification method