Image Feature Based Smoke Recognition in Mines Using Monocular Camera Mounted on Aerial Vehicles

In this paper, we present a framework to solve the problem of rapidly determining regions of interest (ROIs) from an unknown intensity distribution, especially in radiation fields. The vast majority of existing literature on robotics area coverage does not report the identification of ROIs. In a radiation field, ROIs limit the range of exploration to mitigate the monitoring problem. However, considering the limited resources of Unmanned Aerial Vehicle (UAV) as a mobile measurement system, it is challenging to determine ROIs in unknown radiation fields. Given the target area, we attempt to plan a path that facilitates the localization of ROIs with a single UAV, while minimizing the exploration cost. To reduce the complexity of exploration of large scale environment, initially whole areas are adaptively decomposed by the hierarchical method based on Voronoi based subdivision. Once an informative decomposed sub area is selected by maximizing a utility function, the robot heuristically reaches to contaminated areas and then a boundary estimation algorithm is adopted to estimate the environmental boundaries. Finally, the detailed boundaries are approximated by ellipses, called the ROIs of the target area and whole procedures are iterated to sequentially cover the all areas. The simulation results demonstrate that our framework allows a single UAV to efficiently and explore a given target area to maximize the localization rate of ROIs.

Unmanned Aerial Vehicles (UAVs) are used in various applications, including crowd management, crime prevention, accident detection, and rescue operations. However, since UAVs perform their tasks independently, some UAV applications are dynamic and geographically distributed, which may require extensive real-time processing capabilities. Thus, processing UAV data locally can be challenging due to their limited computing capabilities. To overcome such limitations, fog and cloud computing can facilitate UAV application development by providing additional resource capacities when needed. Despite this, designing sophisticated and efficient UAV task offloading strategies that collaborate with fog and cloud technologies considering their service latency and energy consumption, is rarely addressed in the literature. Therefore, a collaborative offloading strategy for UAV applications is presented in this work, leveraging fog and cloud computing advantages and capabilities. This approach aims to minimize UAVs’ service latency and energy consumption, as well as provide the required resources and services in real time. In addition, task offloading decisions are formulated using the Mixed-Integer Linear Programming (MILP) model to reduce the energy consumption of the entire UAV-fog-cloud system by optimizing the allocation of computation resources and communication requested by each UAV. The simulation results demonstrate that the proposed strategy can significantly reduce UAV service latency by 15.38%, 35.29%, and 59.26%, as well as decrease overall energy consumption (including processing and networking) by 3.3%, 7.37%, and 12% when compared to alternative standalone strategies (namely UAV, fog, and cloud).

Unmanned aerial vehicles (UAVs) are being employed in a rapidly increasing number of applications in mining, including the emerging area of mapping underground void spaces such as stopes, which are otherwise inaccessible to humans, automated ground vehicles and survey technologies. Void mapping can provide both visual rock surface and 3D structural information about stopes, supporting more effective planning of ongoing blast designs. Underground stope mapping by UAVs, however, involves overcoming a number of engineering challenges to allow flights beyond operator line-of-sight where there is no global positioning system (GPS), natural or artificial light, or existing communications infrastructure. This paper describes the construction of a UAV sensor suite that uses sound navigation and ranging (SONAR) data to create a rough 3D model of the underground UAV operational environment in real time to provide operators with high situational awareness for beyond line-of-sight operations. The system also provides a backup when dust obscures visual sensors to provide situation awareness and a coarser, but still informative, 3D model of the underground space. Typically, light detection and ranging (LIDAR) systems have superseded SONAR sensors for similar applications. LIDAR is much more accurate than SONAR, but has several disadvantages. SONAR sensor data is sparse, and therefore much easier to process in real time on-board the UAV than LIDAR. The SONAR sensor hardware is also lighter than current LIDAR systems, which is of importance regarding the constrained payload capacity of UAVs. However, the most important factor that makes SONAR stand out in this application is its ability to operate in dusty or smoke-filled environments. The UAV system was tested both above and below-ground using a predefined path with check point locations for the UAV to follow. Due to the lack of GPS, survey points in combination with photogrammetry allowed the UAV’s location to be estimated. This allowed the system to be tested to determine how accurate the SONAR data is in comparison with 3D modelling via photogrammetry of images from a separate digital single-lens reflex camera. Comparing the shape of void surfaces determined by photogrammetry with that determined by SONAR provides quantifiable accuracy when the photogrammetry models are used as ground truth data. Aboveground and underground pilot studies have determined that SONAR sensors provide acceptable accuracy compared with modelling via photogrammetry, sufficient to provide effective situational awareness for human operation of the UAV beyond line-of-sight.

The proliferation of unmanned aerial vehicles (UAVs) has driven the emergence of urban air mobility (UAM) as a solution to alleviate the increasing demand for UAVs applications in various domains. The concept of UAM envisions a future where UAVs play a pivotal role in short-distance transportation, delivery services, surveillance, emergency rescue, and more. However, the rapid growth of UAVs poses significant challenges arising from the need to ensure the safe coexistence of UAVs with other manned/unmanned aerial vehicles. To ensure the safe integration of UAVs into low-altitude urban airspace and support high-density on-demand mobility, this research focuses on handling the safety-centric challenges related to UAV collision risk assessment, obstacle avoidance, and conflict resolution in complex urban environments.
\nOne of the main challenges is the intrusion of UAVs into airport runways. Airports serve as critical hubs for civil aircraft takeoff and landing operations. However, the increasing presence of UAVs has raised concerns about intruding into airports, leading to flight delays and temporary runway closures. To quantify the potential interference of UAVs on civil aircraft, a collision-course based trajectory planning model is proposed to evaluate the collision risk between manned civil aircraft and intruding UAVs. In this risk assessment model, the trajectory of the intruding UAV is formulated using a collision-course concept, taking into account the limited trajectory information available about intruding UAVs. The planned trajectory of the UAVs is then used to predict its relative position concerning civil aircraft. To account for the possible disturbances of the relative distance, a stochastic position prediction model based on Brownian motion is developed. The collision risk is assessed based on this stochastic kinematic model, providing a quantitative measure of the potential collisions. The proposed model is validated through extensive simulations involving different UAV initial positions, position updates, and different sizes of collision zone tests, as well as comparative tests with the Monte-Carlo algorithm, demonstrating its robustness in assessing the collision risk of the airport restricted area. To facilitate the application, the model is finally used to develop a platform to generate runway heatmaps, accounting for airport runway risk distributions. 
\nAnother safety-centric challenge is ensuring autonomous obstacle avoidance and conflict resolution of UAVs. To enable UAV autonomous safe operations in uncontrolled urban low-altitude airspace, a comprehensive collision-free path planning framework is developed to address the avoidance of both static obstacles and dynamic threats. Firstly, to handle static obstacles, a novel algorithm of 3D voxel jump point research is introduced to generate a global reference path for UAV navigation to avoid static obstacles, particularly for a free flight at different altitudes. Additionally, to optimize the reference path for specific mission requirements, techniques such as trajectory de-diagonalization, reconstruction, and smoothness are proposed to reduce the risk of static collisions, shorten the fly range, and enable UAVs to negotiate turns more efficiently. Secondly, real-time conflict resolution actions based on the Markov decision process are developed to avoid local dynamic threats encountered along the optimized desired path. This allows the host UAV to dynamically adjust its course to a resolution trajectory while maintaining its planned speed without hovering to avoid the potential collision with the approaching intruders. By avoiding near mid-air collisions (NMAC) and preventing the UAV from hovering, the proposed method ensures the timely completion of UAV missions. Simulation results demonstrate that the developed method successfully achieves UAV autonomous global path planning and local real-time conflict resolution in complex urban low-altitude environments, thereby avoiding both static obstacles and dynamic threats. 
\nMoreover, to bridge the research gap of UAV hard conflict resolution in mitigating NMAC risks arising from local dynamic threats, a novel situational NMAC risk assessment approach is developed by integrating the conditional random field (CRF) algorithm with safety metrics and conflict resolution performance to facilitate UAV tiered aerial passage operations. To improve UAV operating environments in urban airspace, a concept of tiered aerial passages is first designed to provide a practical urban airspace planning scheme that addresses UAV heterogeneity and optimizes the utilization of urban airspace. For UAV tiered aerial passage operations, a situational NMAC risk assessment module coupled with a conflict resolution decision-making module is formulated, enabling the host UAV to evaluate and respond to real-time conflicts, ensuring avoidance of potential collisions. To assess NMAC risks, a general risk assessment model is proposed based on the CRF algorithm, considering UAV safety metrics and resolution performance. Then the overtaking, approaching head-on, and converging UAV encounters are analyzed, and situational risk potential scores are created using the Gaussian kernel function in the CRF algorithm, considering advisory, caution, and warning risk alert levels. To resolve conflict, acceptable, tolerable, and unacceptable scenarios are proposed to enhance the flexibility and personalization of the resolution decision-making. The proposed scheme is validated by simulating geometric encounter situations and comparative tests with the previous hard resolution technique. The results indicate that the proposed method outperforms UAV NMAC risk assessment and conflict resolution. Even in the most dangerous unacceptable resolution decision-making scenarios across three conflict encounters, the proposed algorithm achieves NMAC rates of 0.036, 0.070, and 0.043, which are lower than the corresponding NMAC rates associated with the hard resolution approach. Together with the acceptable and tolerable scenarios, the proposed negotiation resolution effectively tailors different risk requirements to accommodate diverse UAV missions. 
\nWith a primary focus on ensuring the safety of the UAM architecture, this research endeavors to assess UAV NMAC risks and efficiently manage potential conflicts. The proposed work are thoroughly validated to demonstrate their effectiveness in real-world scenarios. As a result, this study significantly contributes to the establishment of a robust UAM architecture, facilitating the realization of safe and reliable UAM.

Solar energy is one of the most strategic energy sources for the world’s economic development. This has caused the number of solar photovoltaic plants to increase around the world; consequently, they are installed in places where their access and manual inspection are arduous and risky tasks. Recently, the inspection of photovoltaic plants has been conducted with the use of unmanned aerial vehicles (UAV). Although the inspection with UAVs can be completed with a drone operator, where the UAV flight path is purely manual or utilizes a previously generated flight path through a ground control station (GCS). However, the path generated in the GCS has many restrictions that the operator must supply. Due to these restrictions, we present a novel way to develop a flight path automatically with coverage path planning (CPP) methods. Using a DL server to segment the region of interest (RoI) within each of the predefined PV plant images, three CPP methods were also considered and their performances were assessed with metrics. The UAV energy consumption performance in each of the CPP methods was assessed using two different UAVs and standard metrics. Six experiments were performed by varying the CPP width, and the consumption metrics were recorded in each experiment. According to the results, the most effective and efficient methods are the exact cellular decomposition boustrophedon and grid-based wavefront coverage, depending on the CPP width and the area of the PV plant. Finally, a relationship was established between the size of the photovoltaic plant area and the best UAV to perform the inspection with the appropriate CPP width. This could be an important result for low-cost inspection with UAVs, without high-resolution cameras on the UAV board, and in small plants.

This paper describes the Naval Air Systems Command (NAVAIR) unmanned aerial vehicle (UAV) / unmanned combat aerial vehicle (UCAV) distributed simulation infrastructure. This highly flexible and powerful infrastructure was exercised in a realistic interoperability demonstration. In this effort, the adopted scenario successfully ran as planned from beginning to end with a direct hit on the intended target. This demonstration established the value that NAVAIR brings, through the modeling and simulation tools offered in its facilities, to fully develop and exploit the capabilities and interoperability of UAVs and UCAVs. The exercise provided a baseline for a linked UAV infrastructure. However, some suggested modifications would greatly improve the fidelity and enhance the capabilities of the demonstration. The ultimate product would facilitate studying the interactions between UAVs and UCAVs to enhance our warfighters’ capabilities.

In this paper, the problem of using one active unmanned aerial vehicle (UAV) and four passive UAVs to localize a 3D target UAV in real time is investigated. In the considered model, each passive UAV receives reflection signals from the target UAV, which are initially transmitted by the active UAV. The received reflection signals allow each passive UAV to estimate the signal transmission distance which will be transmitted to a base station (BS) for the estimation of the position of the target UAV. Due to the movement of the target UAV, each active/passive UAV must optimize its trajectory to continuously localize the target UAV. Meanwhile, since the accuracy of the distance estimation depends on the signal-to-noise ratio of the transmission signals, the active UAV must optimize its transmit power. This problem is formulated as an optimization problem whose goal is to jointly optimize the transmit power of the active UAV and trajectories of both active and passive UAVs so as to maximize the target UAV positioning accuracy. To solve this problem, a Z function decomposition based reinforcement learning (ZD-RL) method is proposed. Compared to value function decomposition based RL (VD-RL), the proposed method can find the probability distribution of the sum of future rewards to accurately estimate the expected value of the sum of future rewards thus finding better transmit power of the active UAV and trajectories for both active and passive UAVs and improving target UAV positioning accuracy. Simulation results show that the proposed ZD-RL method can reduce the positioning errors by up to 39.4% and 64.6%, compared to VD-RL and independent deep RL methods, respectively.

Fully automated region of interest segmentation pipeline for UAV based RGB images

Aiming at the problem that the unmanned aerial vehicle (UAV) cannot complete the positioning and ranging in the complex background and the signal is interfered, we proposed a monocular measurement scheme of assigning 4 LED fiducials (also called landmarks) with 3 colors in total on the target UAV. Firstly, feature areas are screened out by using high saturation, high brightness and specific hue range of LED in HSV (Hue, Saturation, Value) color space. After morphological processing, the background that does not meet the features is removed. Then, the mapping relationship between target LED fiducials and image fiducials is established by using the average hue of connected domain and special geometric layout. The improved Lambda-Twist is used to find the distance from the target UAV to the center of the camera, and the performance of the UAV recognition and ranging from 90m to 245m is tested. The experimental results show that the method can quickly and accurately measure the distance to the target, and can be used for UAV auxiliary measurement.

In order to solve the problem of image blur identification caused by high compression ratio in unmanned aerial vehicle (UAV) airborne video images, a singular value decomposition (SVD) image compression method based on the region of interest (ROI) is proposed. Using regional compression method, the experiment firstly contour edge detection was carried out for images collected by UAV to help find out the key information part,and then divided the region of interest ROI on the high-frequency part, segmented it and compressed it to a small extent. Other areas were compressed according to the corresponding compression ratio according to the trans-mission bandwidth requirements, and finally the target area and background area were synthesized to obtain clear image of target with high compression ratio. A conclusion is obtained that the UAV can obtain a large peak signal to noise ratio (PSNR) with this method on the premise of guaranteeing the transmission of important information.

The importance of local intervention capability for the assertion of Canadian Sovereignty in the Northwest Passage is recognized. However, Canada lacks the ability to deploy, on demand, assets to search a wide area for rescue or surveillance purposes in the North. This fact motivated our investigation of the feasibility of a rapid intervention system based on a carrier-scouts design in which a number of unmanned aerial vehicles (UAVs) would be transported, air launched and recovered by a carrier aircraft. These UAVs would have the ability to self-organize in formations that correspond to the task at hand. When searching for a target, they would fly in a linear pattern so that the search area swept per hour and the probability of detecting the target would be considerably increased. A 1973 report by the Tactical Combat Aircraft Programs of the Boeing Aerospace Company for the US Air Force [15] and a 2007 thesis by Chalamont [6] indicate that airborne launch and recovery of many UAVs from a carrier aircraft is feasible and requires only already existing technology. We propose here a solution to the remaining problem of managing simultaneously the many UAVs that are required by the vastness of the areas to be surveyed, with a minimum number of human controllers and communications. Namely, we present algorithms for the self-organization of the UAVs in the required formations. These allow for surveillance operations during which close-up images would be acquired of activities in a region of interest, and searching an area for assets in distress and providing a visual presence for such. We reach the conclusion that our proposed local intervention system with flocking UAVs is feasible and would provide a valuable asset for asserting Canadian Sovereignty in the North.

An evaluative review of the VTOL technologies for unmanned and manned aerial vehicles

Pipeline Oil spill is a major challenge with the Oil and Gas production process as it impacts negatively on the environment, the reputation and the finances of the operator. Real time pressure data from an Oil and Gas Pipeline can be used with specially developed algorithms to create dynamic pressure profiles of the pipeline. This profile shows the pressure map of the entire pipeline and it can be used to detect the onset of a leak. Production management systems can be configured in the exception based reporting mode to generate alerts whenever there is a variation in the pressure at any point on the pipeline. The location of pipeline right of way in remote locations increases the monitoring costs and response time in the event of a leak. This paper presents the deployment of Unmanned Aerial Vehicles (UAVs) as the first line of response to alerts generated by the production management systems and the dynamic pressure profiles to provide real time video feedback of the leak points. This will enable a rapid detection and localization of the leak points enabling the speedy deployment of the response teams. The combination of the dynamic pipeline pressure profile and the UAV will provide a rapid monitoring system for the entire pipeline network and facilitate rapid intervention in the event of a leak or any activity on the pipeline right of way providing real time video feedback of the area under investigation and eliminating response to spurious signals and false alarms

With technological development of multi sensors, UAV (unmanned aerial vehicle) can identify and locate key targets in essential monitoring areas or geological disaster-prone areas by taking video sequence images, and precise positioning of the video sequence images is constantly a matter of great concern. In recent years, precise positioning of aerial images has been widely studied. But it is still a challenge to simultaneously realize precise, robust and dynamic positioning of UAV’s patrolling video sequence images in real time. In order to solve this problem, a visual positioning model for patrolling video sequence images based on DOM rectification is proposed, including a robust block-matching algorithm and a precise polynomial-rectifying algorithm. First, the robust block-matching algorithm is used to obtain the best matching area for UAV’s video sequence image on DOM (Digital Orthophoto Map), a pre-acquired digital orthophoto map covering the whole UAV’s patrolling region. Second, the precise polynomial-rectifying algorithm is used to calculate accurate rectification parameters of mapping UAV’s video sequence image to the best matching area obtained above, and then real time positioning of UAV’s patrolling video sequence images can be realized. Finally, the above two algorithms are analyzed and verified by three practical experiments, and results indicate that even if spatial resolution, surface specific features, illumination condition and topographic relief are significantly different between DOM and UAV’s patrolling video sequence images, proposed algorithms can still steadily realize positioning of each UAV’s patrolling video sequence image with about 2.5 m level accuracy in 1 s. To some extent, this study has improved precise positioning effects of UAV’s patrolling video sequence images in real time, and the proposed mathematical model can be directly incorporated into UAV’s patrolling system without any hardware overhead.

In recent years, unmanned aerial vehicle (UAV) has been widely used in target tracking tasks such as security, anti-terrorism, tracking and monitoring because of its wide and flexible perspective. At the same time, deep learning creates new conditions for UAV visual recognition and motion tracking because it has high recognition accuracy and fast recognition speed. This paper presents an UAV dynamic tracking algorithm based on deep learning. The algorithm mainly includes three parts: the PTZ control, relative position calculation and the UAV control. The main work includes: 1. The tracking target is identified through the deep learning network based on YOLOv3, and the PTZ angle is adjusted in real time by using the control method of combining the PID control with the time-delay control to reduce the impact of image transmission delay; 2. The relative position between the UAV and the target is calculated by combining the UAV flight altitude, UAV attitude, PTZ angle and the coordinate of the target in the image. At the same time, the relative position prediction is added to estimate the relative distance between the UAV and the target at the next frame time as a reference value of the UAV speed, so as to improve the dynamic tracking ability; 3. The relative position information is fed back to the flight control function, and the linear velocity of UAV is output by the PID controller. Through the simulation test in Airsim, it is proved that the proposed algorithm can effectively improve the stability and robustness of the UAV tracking system.