Tightly-coupled Line Feature-aided Visual Inertial Localization within Lightweight 3D Prior Map for Intelligent Vehicles

In the research reported in this paper, we propose to overcome the unavailability of Global Positioning System (GPS) using combined information obtained from a scanning LADAR rangefinder on an Unmanned Ground Vehicle (UGV) and a LADAR mounted on an Unmanned Aerial Vehicle (UAV) that flies over the terrain being traversed. The approach to estimate and update the position of the UGV involves registering range data from the two LADARs using a combination of a feature-based registration method and a modified version of the well-known Iterative Closest Point (ICP) algorithm. Registration of range data thus guarantees an estimate of the vehicle's position even when only one of the vehicles has GPS information. Additionally, such registration over time (i.e., from sample to sample), enables position information to be maintained even when both vehicles can no longer maintain GPS contact. The approach has been validated by conducting systematic experiments on complex real-world data.

Over the past few decades, the usage of unmanned vehicles has grown exponentially, expanding into applications such as the automation of industrial processes and automobiles. However, their utility has often been limited by operational concerns. Fully controlled unmanned vehicles require multiple human operators, while their fully autonomous counterparts lack the ability to handle the complex maneuvers necessary in natural disaster relief and/or search and rescue situations. Semi-autonomous UAVs offer a feasible compromise between the two extremes. In this scenario, an unmanned aerial vehicle (UAV) sends birds-eye images of the terrain beneath it to a computing cluster, which will identify easily traversable terrain and generate a path of least risk to an unmanned ground vehicle (UGV). If the path's risk is below a certain threshold, then the UGV will be permitted to proceed on its own. Otherwise, a human operator will be notified, so that he or she may control the UGV directly until it exits the most dangerous terrain. This paradigm allows a single operator to manage several UAVs simultaneously.

We study the problem of covering an environment using an Unmanned Aerial Vehicle (UAV) with limited battery capacity. We consider a scenario where the UAV can land on an Unmanned Ground Vehicle (UGV) and recharge the onboard battery. The UGV can also recharge the UAV while transporting the UAV to the next take-off site. We present an algorithm to solve a new variant of the area coverage problem that takes into account this symbiotic UAV and UGV system. The input consists of a set of boustrophedon cells -- rectangular strips whose width is equal to the field-of-view of the sensor on the UAV. The goal is to find a coordinated strategy for the UAV and UGV that visits and covers all cells in minimum time, while optimally finding how much to recharge, where to recharge, and when to recharge the battery. This includes flight time for visiting and covering all cells, recharging time, as well as the take-off and landing times. We show how to reduce this problem to a known NP-hard problem, Generalized Traveling Salesperson Problem (GTSP). Given an optimal GTSP solver, our approach finds the optimal coverage paths for the UAV and UGV. Our formulation models multi-rotor UAVs as well as hybrid UAVs that can operate in fixed-wing and Vertical Take-off and Landing modes. We evaluate our algorithm through simulations and proof-of-concept experiments.

The use of unmanned ground vehicles (mobile robots) and unmanned aerial vehicles (drones) in the civil infrastructure asset management sector: Applications, robotic platforms, sensors, and algorithms

This paper addresses the control of a formation composed of an unmanned ground vehicle (UGV) and an unmanned aerial vehicle (UAV). The formation navigates in a coordinated way, with the UAV following the UGV while keeping a relative pose with respect to the UGV as long as the UGV follows a specified path, so that the UAV can land on the UGV at any desired instant. This application is conceived in a context in which the UAV has finished a package delivery task and comes back to land over a platform used as a base for the package delivery operation. Experimental results are presented, which validate the approach adopted to accomplish such a task.

Ground-Air coordination is a very complex environment for a machine learning algorithm. We focus on the case where an Unmanned Aerial Vehicle (UAV) needs to support a group of Unmanned Ground Vehicles (UGVs). The UAV is required to broadcast an image that contains all UGVs, thus, offering a bird-eye-view on the group as a whole. The source of complexity in this task is twofold. First, coordination needs to occur without communication between the UAV and UGVs. Second, the ability of the UAV to sense the UGVs is coupled with the ability of the UAV to learn how to track laterally the UGVs and adapt its vertical position so that the images of the UGVs are appropriately spaced within the camera field of view. In this paper, we propose using the Deep Actor Network component of an Actor-Critic Deep Reinforcement Learning architecture as a supervised learner. The advantage of this approach is that it offers a step towards autonomous learning whereby the full Actor-Critic model can be utilized in the future. Human demonstrations are collected for the deep Actor network to learn from. The system is built using the Gazebo Simulator, Robot Operating System, and the OpenAI Gym. We show that the proposed setup is able to train the UAV to follow the UGVs while maintaining all UGVs within camera range in situations where UGVs are performing complex maneuvers.

This paper focuses on the real time implementation of cooperative navigation aiding for a small unmanned aerial vehicle (UAV) based on vision systems on board unmanned ground vehicles (UGV). In urban environments the signal quality of global satellite navigation systems (GNSS) often is bad or signals are even lost, so that the navigation solution of a UAV is aected. Especially in such situations the UAV’s geo-referenced position has to be known to ensure a safe guidance. A team of UGVs can overcome the problem of no GNSS position solution by the detection and tracking of the UAV in its on board images and the subsequent geo-localization. The determination of the UAV’s geo-referenced position can be achieved with several algorithms, depending on the number of UGVs. To integrate these position measurements in the UAV’s on board navigation computer the quality of the detection and geo-localization has to be estimated. All parts are implemented on embedded PC platforms. To allow real time usage on board the UGVs special algorithms have to be used for the most tasks, as image processing needs high computational power. The particular parts of this setup as well as the whole system are tested with experimental data.

We consider persistent (long-horizon) surveillance over an environment by using energy-constrained unmanned aerial vehicles (UAVs), which are supported by unmanned ground vehicles (UGVs) serving as mobile charging stations. The goal is to periodically visit a set of monitoring points by the UAVs while minimizing the maximum time between the consecutive visits to any of those points. In general, the optimal planning of UAVs and UGVs in such a persistent surveillance scenario is an NP-hard combinatorial optimization problem. Furthermore, the problem also demands a solution strategy that can successfully handle obstacles, especially on the ground, that are unknown a priori in many real-life scenarios. We present a scalable and robust approximate algorithm that is based on 1) forming uniform UAV-UGV teams, 2) decomposing the environment into maximal partitions that can be covered by the UAVs in a single fuel cycle as long as the UAVs are released sufficiently close to the centers of the partitions, 3) maintaining the teams uniformly distributed over a cyclic path traversing those partitions, and 4) having the UAVs in each team cover their current partition and be transported to the next partition while being recharged by the UGV. We support our proposed algorithm with some theoretical results and simulations.

With the development of technologies related to unmanned vehicles, including unmanned aerial vehicle and unmanned surface vehicle, unmanned vehicles play a pivotal role in maritime management and cruise activities. Meanwhile, lots of evidence shows that a single unmanned vehicle is not qualified for operation under complex environment, whereas the unmanned aerial vehicle–unmanned surface vehicle formation consists of multiple unmanned aerial vehicles and unmanned surface vehicles, which can enhance the performance of the single unmanned vehicle significantly. However, the cooperative operation performance of the unmanned aerial vehicle–unmanned surface vehicle formation is constrained by its communication capacity. Consequently, a cooperative communication framework is designed for the unmanned aerial vehicle–unmanned surface vehicle formation. A distributed dynamic network topology is constructed for fulfilling effective communication for the unmanned aerial vehicle–unmanned surface vehicle formation based on ad hoc network. Experiments are carried out by employing the built unmanned aerial vehicle–unmanned surface vehicle formation platform. Results verify the superiority of the designed cooperative communication framework for the unmanned aerial vehicle–unmanned surface vehicle formation.

Selection of sugar beet (Beta vulgaris L.) cultivars that are resistant to Cercospora Leaf Spot (CLS) disease is critical to increase yield. Such selection requires an automatic, fast, and objective method to assess CLS severity on thousands of cultivars in the field. For this purpose, we compare the use of submillimeter scale RGB imagery acquired from an Unmanned Ground Vehicle (UGV) under active illumination and centimeter scale multispectral imagery acquired from an Unmanned Aerial Vehicle (UAV) under passive illumination. Several variables are extracted from the images (spot density and spot size for UGV, green fraction for UGV and UAV) and related to visual scores assessed by an expert. Results show that spot density and green fraction are critical variables to assess low and high CLS severities, respectively, which emphasizes the importance of having submillimeter images to early detect CLS in field conditions. Genotype sensitivity to CLS can then be accurately retrieved based on time integrals of UGV- and UAV-derived scores. While UGV shows the best estimation performance, UAV can show accurate estimates of cultivar sensitivity if the data are properly acquired. Advantages and limitations of UGV, UAV, and visual scoring methods are finally discussed in the perspective of high-throughput phenotyping.

The autonomous landing of fixed-wing unmanned aerial vehicles (UAVs) on moving unmanned ground vehicles (UGVs) demands precise spatial synchronization of both vehicle systems. To achieve this goal, using individual GNSS receivers and additional sensors for both vehicles and having the vehicles exchange their respective 6D pose via radio communication is a straightforward approach for mutual localization. However, failure of the radio communication or individual sensors can critically affect the safety of the landing maneuver without suitable additional sensors or fallback systems. In this work, we present and compare two methods for estimating the relative 6D pose between a UGV and a UAV using camera systems attached to the UGV. The first approach uses a stereo grayscale camera setup with fiducial markers on the underside of the wings of the UAV. The second approach uses a single RGB camera and a convolutional neural network (CNN) trained on synthetic data, which is combined with a pose tracking filter to improve the update rate. Requirements for the pose estimation system are an average position error below 0.5 m and an orientation angle error below 10 deg, while the update rate should be at least 20 Hz. Different experiments with a demonstrator vehicle setup are presented, showing the performance and advantages of the respective approaches. During flight experiments with eleven flyovers, the CNN-based system was consistently able to detect the UAV pose, while the marker-based system had detections in seven cases. The CNN-based system had RMS position errors between 0.19 m and 1.36 m, and RMS yaw angle errors between 1.2 deg and 7.7 deg, which is an average improvement of 0.33 m and 5.6 deg, respectively, compared to the marker-based system. In future work, the pose estimation will be tested in the control loop with an automated UAV and UGV.

Natural disasters often cause huge and unpredictable losses to human lives and properties. In such an emergency post-disaster rescue situation, unmanned aerial vehicles (UAVs) are effective tools to enter the damaged areas to perform immediate disaster recovery missions, owing to their flexible mobilities and fast deployment. However, UAVs typically have very limited battery and computational capacities, which makes them harder to perform heavy computation tasks during the complicated disaster recovery process. This paper addresses the issue of the battery and computation resource limitation with a fog computing based UAV system. Specifically, we first introduce the vehicular fog computing (VFC) system in which the unmanned ground vehicles (UGVs) perform the computation tasks offloaded from UAVs. To avoid the transmission competitions yet enable cooperations among UAVs and UGVs, a stable matching algorithm is developed to transform the computation task offloading problem into a two-sided matching problem. An iterative algorithm is then developed which matches each UAV with the most suitable UGV for offloading. Finally, extensive simulations are carried out to demonstrate that the proposed scheme can effectively improve utilities of UAVs and reduce average delay through comparison with conventional schemes.

With the advancement of artificial intelligence technology, unmanned vehicle (UV) systems, including unmanned aerial vehicle (UAV) and unmanned ground vehicle (UGV), have been appearing in many scenarios with wide-ranging applications. It is a crucial factor for enhancing human-computer collaboration efficiency to analyze the impact of unmanned vehicle system combinations on operators. The research in this paper presents the effects of different combinations of UAVs and UGVs (1UAV + 1UGV, 1UAV + 2UGVs, 2UAVs + 1UGV, 2UAVs + 2UGVs) on searching performance and operator’s mental workload for accomplishing search tasks. Completion times for tasks and subjective data with operators (N=16) were collected by using Psychopy and questionnaires, respectively. The results in this research indicate that binary growth of controllable unmanned vehicles doesn’t improve the UV utilization rate, but costing longer completion time and increasing operator’s mental workload. Since an excessive number of unmanned vehicles could have a negative impact on task performance, the insights in the paper are helpful for the design of unmanned vehicle systems and the research of human-computer collaboration.

In this paper, we present a novel integrated vehicular system using collaborative unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) for autonomous exploration, mapping, and navigation in GPS-denied 3-dimensional (3-D) unknown environments. The system implements a novel two-layered exploration strategy to decompose the perception task into a coarse exploration layer and a fine mapping layer. The coarse exploration makes use of a UGV to carry out fast autonomous exploration and active 2.5D simultaneous localization and mapping (SLAM) to generate a coarse environment model, which serves as a navigation reference for subsequent complementary 3-D fine mapping conducted by a UAV. The two layers share a novel optimized exploration path planning and navigation framework, which provides optimal exploration paths and integrates the collaborative exploration and mapping efforts through an OctoMap-based volumetric motion planning interface. The proposed system provides an efficient pipeline of fast environment perception taking advantages of the agility of the UAVs as well as the powerful computation resource aboard UGVs, also allowing assistive local perception with augmented object information when necessary. The effectiveness of our system is verified by both simulations and experiments, which demonstrate its capability of implementing heterogeneous UAV and UGV collaborative exploration and structural reconstruction of the environments through active SLAM, providing optimized perception for navigation tasks.

Unmanned aerial vehicles (UAVs) have become increasingly popular in the civil field, and building inspection is one of the most promising applications. In a rural area, the UAVs are assigned to inspect the surface of buildings, and an unmanned ground vehicle (UGV) is introduced to carry the UAVs to reach the rural area and also serve as a charging station. In this paper, the mission planning problem for UAVs and UGV systems is focused on, and the goal is to realize an efficient inspection of buildings in a specific rural area. Firstly, the mission planning problem (MPP) involving UGVs and UAVs is described, and an optimization model is established with the objective of minimizing the total UAV operation time, fully considering the impact of UAV operation time and its cruising capability. Subsequently, the locations of parking points are determined based on the information about task points. Finally, a hybrid ant colony optimization-genetic algorithm (ACO-GA) is designed to solve the problem. The update mechanism of ACO is incorporated into the selection operation of GA. At the same time, the GA is improved and the defects that make GA easy to fall into local optimal and ACO have insufficient searching ability are solved. Simulation results demonstrate that the ACO-GA algorithm can obtain reasonable solutions for MPP, and the search capability of the algorithm is enhanced, presenting significant advantages over the original GA and ACO.