Unreal engine-based photorealistic aerial data generation and unit testing of artificial intelligence algorithms

Timely and accurate extraction of crop planting units plays a critical role in crop yield estimation, soil management, food supplies, and disaster warnings. However, precisely mapping crop types is challenging in smallholder farming systems due to heterogeneous and mixed pixels, where field sizes are small, and crop types are very diverse. In this paper, the crop type system of Huocheng County in northwest China as an example area, crop classification feature variables are constructed using Sentinel-2 remote sensing images, and combining the ReliefF algorithm, three feature selection classification models are established by 3092 crop type sampling points data. The crop planting units are extracted using pixel-based and object-based classification methods, respectively, and unmanned aerial vehicle (UAV) data assess the accuracy as ground truth. The Sentinel-2 image based smallholder crops classification results indicate that: The effective combination of optimal input feature variables selection and classification models significantly improves crop classification accuracy. This is particularly evident when integrating this approach into pixel-based classification, addressing challenges such as low pixel classification accuracy in regions of this type. The validation results from UAV data also confirm these findings. At validation points 2, 3, and 4, pixel-based classification demonstrates significantly higher accuracy compared to object-based classification, with pixel-based classification accuracy improving by 9.16%, 7.83%, and 31.83%, respectively. Although the UAV validation accuracy does not reach a high level, pixel-based classification remains the optimal choice for smallholder crop classification. This method offers new insights and references for research related to the classification of smallholder crops with complex planting structures.

Today, the use of small unmanned aerial vehicles (UAVs) has increased due to technological advances. There has been an interest in using multiple UAVs instead of a single UAV to accomplish a given mission. This is because there are many scenarios where the capabilities of a single UAV are inadequate due to certain constraints (battery capacity, time in the air). For this reason, swarm UAV studies have increased. A swarm UAV consists of a large number of UAVs cooperating to accomplish a specific mission. This study shows three-dimensional formation control of a swarm UAV system in an obstacle environment. A centralized control architecture is used in this process. All task assignments are made from a centralized system. The Artificial potential fields method creates the formation at the target point by avoiding obstacles. The study was carried out using the Robot Operating System (ROS). The methods were tested in Webots simulation environment. Crazyflie robots were used in the experiments. In the simulation environment, square, star and v formations were first tested in two dimensions. Then, as an example of three-dimensional formation, the cubic and pyramid formations were created and observed.

Accurate leaf area index estimation in sorghum using high-resolution UAV data and machine learning models

This work explores the suitability of various deep convolutional neural network (CNN) architectures for semantic segmentation of agricultural crops such as cotton, maize etc. from multi-spectral UAV (unmanned aerial vehicle) data. Initially, the UAV data were preprocessed and training samples for each crop type were manually annotated from multiple UAV scenes. Different CNN architectures such as U-Net, SegNet and PSPNet (Pyramid Scene Parsing Network) were trained with various combinations of input spectral bands along with select band derived indices such as NDVI (Normalized Difference Vegetation Index) and EVI (Enhanced Vegetation Index) as additional features. The experimental results indicated that inclusion of NIR (near-infrared) band and NDVI in the input data yielded high segmentation accuracy of more than 90%. U-Net proved to be the best among the three architectures with 97% overall accuracy while dealing with three classes separation problem. This study demonstrated the scope of deep neural network based semantic segmentation techniques in crop classification from multi-spectral UAV data.KeywordsCrop classificationUAV (unmanned aerial vehicle)Multi-spectral dataDeep learningU-NetSegNetPSPNetImage analysis

In the rapidly evolving technological landscape, the advent of collaborative Unmanned Aerial Vehicle (UAV) inspections represents a revolutionary leap forward in the monitoring and maintenance of power distribution networks. This innovative approach harnesses the synergy of UAVs working together, marking a significant milestone in enhancing the reliability and efficiency of infrastructure management. Despite its promise, current research in this domain frequently grapples with challenges related to efficient coordination, data processing, and adaptive decision-making under complex and dynamic conditions. Intelligent self-organizing algorithms emerge as pivotal in addressing these gaps, offering sophisticated methods to enhance the autonomy, efficiency, and reliability of UAV collaborative inspections. In response to these challenges, we propose the MARL-SOM-GNNs network model, an innovative integration of Multi-Agent Reinforcement Learning, Self-Organizing Maps, and Graph Neural Networks, designed to optimize UAV cooperative behavior, data interpretation, and network analysis. Experimental results demonstrate that our model significantly outperforms existing approaches in terms of inspection accuracy, operational efficiency, and adaptability to environmental changes. The significance of our research lies in its potential to revolutionize the way power distribution networks are inspected and maintained, paving the way for more resilient and intelligent infrastructure systems. By leveraging the capabilities of MARL for dynamic decision-making, SOM for efficient data clustering, and GNNs for intricate network topology understanding, our model not only addresses current shortcomings in UAV collaborative inspection strategies but also sets a new benchmark for future developments in autonomous infrastructure monitoring, highlighting the crucial role of intelligent algorithms in advancing UAV technologies.

This dissertation presents a comprehensive approach to addressing the challenges associated with the control and navigation of Unmanned Aerial Vehicles (UAVs), specifically focusing on robustness under GPS-denied, turbulent, and dark indoor conditions with various disturbances. Initial efforts involved designing and theoretically validating Sliding Mode Controllers (SMC) with Disturbance Observers using MATLAB/SIMULINK, optimized using offline Particle Swarm Optimization (PSO). In order to evaluate their performance, a lightweight simulator was developed to simulate both quadcopter and hexacopter configurations under the PX4 architecture. This simulator served to validate and compare the optimized SMC against PSO-optimized PID controllers. The results of these comparisons demonstrated the superior adaptability of the SMC, especially in environments characterized by significant sensor noise and disturbances. Subsequent practical indoor tests were conducted with real quadcopters, which provided additional verification of the robustness and reliability of the enhanced control theory with the actual UAVs under dynamic disturbances. To further evaluate the compatibility and adaptability of the SMC, this dissertation establishes a framework for interfacing with high-performance STMH7 chips, serving as the UAV microcontroller unit (MCU). This framework facilitates the connection of various sensors and provides guidance for constructing different quadcopter frames. Moreover, it enables the comparison of different flight controllers on distinct air-frames. Concurrently, the optimized SMC allows for remote dynamic parameter tuning during flight. In pursuit of improved localization capacity, the dissertation also explores the integration of an Intel Realsense D435i camera with an Inertial Measurement Unit (IMU) for Visual-Inertial Odometry (VIO) fusion within the Robot Operating System (ROS) environment. This integration employs an onboard companion computer for real-time VIO estimation and flight command. The effectiveness of the VIO algorithm is verified through both simulations and real-world experiments, highlighting its ability to enhance UAV localization and navigation. Additionally, a visual inertial odometry method for UAV Simultaneous Localization and Mapping (vSLAM) is tested and validated in both simulated and real drone environments. Furthermore, the dissertation introduces an innovative hybrid filtered multi-directional radar inertial odometry solution denoted as Hybrid-MRIO. This Error State Extended Kalman Filter (ES-EKF) based Radar Inertial Odometry (RIO) approach is implemented within the open-source PX4 autopilot system, enhancing UAV navigation accuracy and performance, and serves as a robust navigation alternative, particularly in challenging environments where traditional navigation systems prove ineffective, such as dark or smoky conditions. This solution integrates data from multiple software synchronized high-resolution Frequency-Modulated Continuous-Wave (FMCW) mmWave radars with an IMU. A subsequent 4D (x, y, z, doppler) mmWave radar SLAM (rSLAM) system has been developed, including three modules: front-end, loop detection module, and back-end. In the front-end, radar ego-velocity is utilized for states estimation and dynamic object removal, and a point cloud registration-based approach known as APDGICP (Adaptive Probability Distribution-GICP) is employed for keyframe detection. The loop detection module utilizes the intensity scan context to identify potential loop closure candidates. In the back-end, a pose graph is constructed, integrating RIO estimated odometry and identified loop closures for better localization. To develop fully-autonomous UAVs with their decision making modules, this dissertation details the development of a custom AI-powered fully-autonomous quadcopter equipped with companion computers and stereo IR cameras. Additionally, this dissertation discusses a swarm of small Unmanned Aerial Systems (SUAS) interconnected through a software-defined communications network. These drones demonstrated their capacity to independently/collaboratively execute a wide range of tasks, including target search, detection, identification, classification, tracking, and following, both in simulated and real-world scenarios. The system exhibits advanced collision avoidance capabilities and the ability to strategically respond to dynamic scenarios, such as changes in target behavior, highlighting its effectiveness in managing dynamic and challenging situations without the need for continuous human intervention.--Author's abstract

: Unmanned aerial vehicles (UAVs) are critical components of the future naval forces. UAV control and monitoring with autonomous operation will become an absolute necessity and adaptive cooperation of vehicles is the only practical alternative. The objective of this project is to develop and evaluate new methodologies for cooperative (formation) control of multiple unmanned air vehicles. The goal is to have multiple UAVs working together as a group. Instead of separately assigning distinct tasks to each vehicle, the operator would assign tasks to the UAV group, which then determines the best way to accomplish each task, freeing the operator to maintain surveillance over the entire operation. In this project we investigated Path Tracking and obstacle avoidance of UAVs using fuzzy logic method. Algorithms for close formation control of multi-UAVs are developed and simulated. We also investigated fault-tolerant control of single UAVs by neuro-adaptive method. Detailed description of this method is provided in this document. The project has supported S graduate students with 9 technical papers published.

The accurate estimation of tree attributes is essential for sustainable forest management. Terrestrial Laser Scanning (TLS) is a viable remote sensing technology suitable for estimating under canopy structure. However, TLS measurements generally underestimate tree height in taller trees, which leads to the underestimation of other tree attributes (e.g., stem volume). The integration of information derived from TLS and Unmanned Aerial Vehicle (UAV) photogrammetry could potentially improve tree height estimation. This study investigated the applicability of integrating TLS and UAV photogrammetry to estimate individual tree attributes in managed coniferous forests of Japan. Diameter at breast height (DBH), tree height, and stem volume were estimated by (1) TLS data only, (2) integrating TLS and UAV data with TLS tree locations, and (3) integrating TLS and UAV data with treetop detections of the tree canopy. The TLS data only approach achieved high accuracy for DBH estimations with a root mean squared error (RMSE) of 2.36 cm (RMSE% 5.6%); however, tree height was greatly underestimated, with an RMSE of 8.87 m (28.9%) and a bias of −8.39 m. Integrating TLS and UAV photogrammetric data improved tree height estimation accuracy for both the TLS tree location (RMSE of 1.89 m and a bias of −0.46 m) and the treetop detection (RMSE of 1.77 m and a bias of 0.36 m) approaches. Integrating TLS and UAV photogrammetric data also improved the accuracy of the stem volume estimations with RMSEs of 0.21 m3 (10.8%) and 0.21 m3 (10.5%) for the TLS tree location and treetop detection approaches, respectively. Although the tree height of suppressed trees tended to be overestimated by TLS and UAV photogrammetric data integration, a good performance was obtained for dominant trees. The results of this study indicate that the integration of TLS and UAV photogrammetry is beneficial for the accurate estimation of tree attributes in coniferous forests.

As a complicated multi-input, multi-output, and time-varying nonlinear system, flight control system of unmanned aerial vehicle (UAV), which determines the whole system’s performance directly, is crucial for the simulation training system design. This chapter mainly focuses on parameter identification of flight control system based on the modeling of UAVs and a specific controller design for the pendulum-like oscillation in micro aerial vehicle (MAV). A predator–prey particle swarm optimization (PSO) algorithm for identifying parameters of UAV flight control system is presented, with the aim of reducing the workload of the designers during the process of designing complicated UAV control system. Besides, a software environment for UAV controller design was developed based on the UAV model and the proposed method. Then a specific kind of controller design involving the pendulum-like oscillation in the hover and stare state for a MAV in the presence of external disturbances is discussed in detail, since the pendulum-like oscillation caused by uncertainty and external disturbances badly jeopardize the performance of hover and stare, resulting in blurred images or even MAV’s overturning. A novel type of PSO-based linear-quadratic regulator (LQR) controller for stabilizing the pendulum-like oscillation is developed, which can enhance the MAV’s performance efficiently.

Aboveground biomass (AGB) serves as a crucial indicator of the carbon sequestration capacity of coastal wetland ecosystems. Conducting extensive field surveys in coastal wetlands is both time-consuming and labor-intensive. Unmanned aerial vehicles (UAVs) and satellite remote sensing have been widely utilized to estimate regional AGB. However, the mixed pixel effects in satellite remote sensing hinder the precise estimation of AGB, while high-spatial resolution UAVs face challenges in estimating large-scale AGB. To fill this gap, this study proposed an integrated approach for estimating AGB using field sampling, a UAV, and Sentinel-2 satellite data. Firstly, based on multispectral data from the UAV, vegetation indices were computed and matched with field sampling data to develop the Field–UAV AGB estimation model, yielding AGB results at the UAV scale (1 m). Subsequently, these results were upscaled to the Sentinel-2 satellite scale (10 m). Vegetation indices from Sentinel-2 data were calculated and matched to establish the UAV–Satellite AGB model, enabling the estimation of AGB over large regional areas. Our findings revealed the AGB estimation model achieved an R2 value of 0.58 at the UAV scale and 0.74 at the satellite scale, significantly outperforming direct modeling from field data to satellite (R2 = −0.04). The AGB densities of the wetlands in Xieqian Bay, Meishan Bay, and Hangzhou Bay, Zhejiang Province, were 1440.27 g/m2, 1508.65 g/m2, and 1545.11 g/m2, respectively. The total AGB quantities were estimated to be 30,526.08 t, 34,219.97 t, and 296,382.91 t, respectively. This study underscores the potential of integrating UAV and satellite remote sensing for accurately assessing AGB in large coastal wetland regions, providing valuable support for the conservation and management of coastal wetland ecosystems.

The main purpose of this article is to build a general platform for Unmanned Aerial Vehicle (UAV) simulation especially for vision algorithms. This general simulation platform can also connect to realistic UAVs for further tests. The platform enables UAVs controlled by a variety of advanced vision algorithms to accomplish certain tasks. Unlike the previous simulation systems, which can only achieve tasks with certain drones, we are committed to build a platform to support a variety of vision based UAVs simulation platform. And if the simulation test is successful, we can apply vision algorithms to realistic UAVs' tests based on our platform with little adaption. We use open source Gazebo as a simulator and combined with ROS (Robot Operating System) to implement UAVs simulation. Besides, we build a general UAVs link interface to connect realistic UAVs with ROS. All these works construct the general simulation platform.

Nowadays, due to the rapid development of lightweight unmanned aerial vehicles (UAV), remote sensing systems of ultra-high resolution have become available to many researchers. Conventional ground-based measurements for assessing tree stand attributes can be expensive, as well as time- and labor-consuming. Here, we assess whether remote sensing measurements with lightweight UAV can be more effective in comparison to ground survey methods in the case of temperate mixed forests. The study was carried out at the Prioksko-Terrasny Biosphere Nature Reserve (Moscow region, Russia). This area belongs to a coniferous-broad-leaved forest zone. Our field works were carried out on the permanent sampling plot of 1 ha (100×100 m) established in 2016. The coordinates of the plot center are N 54.88876°, E 37.56273° in the WGS 84 datum. All trees with DBH (diameter at breast height) of at least 6 cm (779 trees) were mapped and measured during the ground survey in 2016 (See Fig. 1 and Table 1). Mapping was performed with Laser Technology TruPulse 360B angle and a distance meter. First, polar coordinates of each tree trunk were measured, and then, after conversion to the cartesian coordinates, the scheme of the stand was validated onsite. Species and DBH were determined for each tree. For each living tree, we detected a social status class (according to Kraft). Also for living trees, we measured the tree height and the radii of the crown horizontal projection in four cardinal directions. A lightweight UAV Phantom 4 (DJI-Innovations, Shenzhen, China) equipped with an integrated camera of 12Mp sensor was used for aerial photography in this study. Technical parameters of the camera are available in Table 2. The aerial photography was conducted on October 12, 2017, from an altitude of 68 m. The commonly used mosaic flight mode was used with 90% overlapping both for side and front directions. We applied Agisoft Metashape software for orthophoto mosaic image and dense point cloud building. The canopy height model (CHM) was generated with lidR package in R. We used lasground() function and cloth simulation filter for classification of ground points. To create a normalized dataset with the ground at 0, we used spatial interpolation algorithm tin based on a Delaunay triangulation, which performs a linear interpolation within each triangle, implemented in the lasnormilise() function. CHM was generated according to the pit-free algorithm based on the computation of a set of classical triangulations at different heights. The location and height of individual trees were automatically detected by the function FindTreesCHM() from the package rLIDAR in R. The algorithm implemented in this function is local maximum with fixed window size. Accuracy assessment of automatically detected trees (in QGIS software) was performed through visual interpretation of orthophoto mosaic and comparison with ground survey data. The number of correctly detected trees, omitted by the algorithm and not existing but detected trees were counted. As a result of aerial photography, 501 images were obtained. During these data processing with the Metashape, dense point cloud of 163.7 points / m2 was generated. CHM with 0.5 m resolution was calculated. According to the individual-tree detection algorithm, 241 trees were found automatically (See Fig. 2A). The total accuracy of individual tree detection was 73.9%. Coniferous trees (Pinus sylvestris and Picea abies) were successfully detected (86.0% and 100%, respectively), while results for birch (Betula spp.) required additional treatment. The algorithm correctly detected only 58.2% of birch trees due to false-positive trees (See Fig. 2B and Table 3). These results confirm the published literature data obtained for managed tree stands. Tree heights retrieved from the UAV were well-matched to ground-based method results. The mean tree heights retrieved from the UAV and ground surveys were 25.0±4.8 m (min 8.2 m, max 32.9 m) and 25.3±5.2 m (min 5.9 m, max 34.0 m), respectively (no significant difference, p-value=0.049). Linear regression confirmed a strong relationship between the estimated and measured heights (y=k*x, R2 =0.99, k=0.98) (See Fig. 3A). Slightly larger differences in heights estimated by the two methods were found for birch and pine; for spruce, the differences were smaller (See Fig. 3B and Table 4). We believe that ground measurements of birch and pine height are less accurate than for spruce due to different crown shapes of these trees. So, our results suggested that UAV data can be used for tree stand attributes estimation, but automatically obtained data require validation.

In Indonesia, where the agricultural insurance system has been in full operation since 2016, a new damage assessment estimation formula for rice diseases was created through integrating the current damage assessment method and unmanned aerial vehicle (UAV) multispectral remote sensing data to improve the efficiency and precision of damage assessment work performed for the payments of insurance claims. The new method can quickly and efficiently output objective assessment results. In this study, UAV images and bacterial leaf blight (BLB) rice damage assessment data were acquired during the rainy and dry seasons of 2021 and 2022 in West Java, Indonesia, where serious BLB damage occurs every year. The six-level BLB score (0, 1, 3, 5, 7, and 9) and damage intensity calculated from the score were used as the BLB damage assessment data. The relationship between normalized UAV data, normalized difference vegetation index (NDVI), and BLB score showed significant correlations at the 1% level. The analysis of damage intensities and UAV data for paddy plots in all cropping seasons showed high correlation coefficients with the normalized red band, normalized near-infrared band, and NDVI, similar to the results of the BLB score analysis. However, for paddy plots with damage intensities of 70% or higher, the biased numbering of the BLB score data may have affected the evaluation results. Therefore, we conducted an analysis using an average of 1090 survey points for each BLB score and confirmed a strong relationship, with correlation coefficients exceeding 0.9 for the normalized red band, normalized near-infrared band, and NDVI. Through comparing the time required by the current assessment method with that required by the assessment method integrating UAV data, it was demonstrated that the evaluation time was reduced by more than 60% on average. We are able to propose a new assessment method for the Indonesian government to achieve complete objective enumeration.

This paper investigates a new scenario of spectrum sharing between unmanned\naerial vehicle (UAV) and terrestrial wireless communication, in which a\ncognitive/secondary UAV transmitter communicates with a ground secondary\nreceiver (SR), in the presence of a number of primary terrestrial communication\nlinks that operate over the same frequency band. We exploit the UAV's mobility\nin three-dimensional (3D) space to improve its cognitive communication\nperformance while controlling the co-channel interference at the primary\nreceivers (PRs), such that the received interference power at each PR is below\na prescribed threshold termed as interference temperature (IT). First, we\nconsider the quasi-stationary UAV scenario, where the UAV is placed at a static\nlocation during each communication period of interest. In this case, we jointly\noptimize the UAV's 3D placement and power control to maximize the SR's\nachievable rate, subject to the UAV's altitude and transmit power constraints,\nas well as a set of IT constraints at the PRs to protect their communications.\nNext, we consider the mobile UAV scenario, in which the UAV is dispatched to\nfly from an initial location to a final location within a given task period. We\npropose an efficient algorithm to maximize the SR's average achievable rate\nover this period by jointly optimizing the UAV's 3D trajectory and power\ncontrol, subject to the additional constraints on UAV's maximum flying speed\nand initial/final locations. Finally, numerical results are provided to\nevaluate the performance of the proposed designs for different scenarios, as\ncompared to various benchmark schemes. It is shown that in the quasi-stationary\nscenario the UAV should be placed at its minimum altitude while in the mobile\nscenario the UAV should adjust its altitude along with horizontal trajectory,\nso as to maximize the SR's achievable rate in both scenarios.\n

Manipulating objects using arms mounted to unmanned aerial vehicles (UAVs) is attractive because UAVs may access many locations that are otherwise inaccessible to traditional mobile manipulation platforms such as ground vehicles. Historically, UAVs have been employed in ways that avoid interaction with the environment at all costs. The recent trend of increasing small UAV lift capacity and the reduction of the weight of manipulator components make the realization of mobile manipulating UAVs imminent. Despite recent work, several major challenges remain to be overcome before it will be common practice to manipulate objects from UAVs. Among these challenges, the constantly moving UAV platform and compliance of manipulator arms make it difficult to position the UAV and end-effector relative to an object of interest precisely enough for reliable manipulation. Solving this challenge will bring UAVs one step closer to being able to perform meaningful tasks such as infrastructure repair, disaster response, law enforcement, and personal assistance. Toward a solution to this challenge, this thesis describes a way forward that uses the UAV as a means to crudely position a manipulator within reach of the end-effector's goal position in the world. The manipulator then performs the fine positioning of the end-effector, rejecting position perturbations caused by UAV motions. An algorithm to coordinate the redundant degrees of freedom of an aerial manipulation system is described that allows the motions of the manipulator to serve as inputs to the UAV's position controller. To demonstrate this algorithm, the manipulator's six degrees of freedom are servoed using visual sensing to drive an eye-in-hand camera to a specified pose relative to a target while treating motions of the host platform as perturbations. Simultaneously, the host platform's degrees of freedom are regulated using kinematic information from the manipulator. This ultimately drives the UAV to a position that allows the manipulator to assume a pose relative to the UAV that maximizes reachability, thus facilitating the arm's ability to compensate for undesired UAV motions. Maintaining this loose kinematic coupling between the redundant degrees of freedom of the host UAV and manipulator allows this type of controller to be applied to a wide variety of platforms, including manned aircraft, rather than a single instance of a purpose-built system. As a result of this loose coupling, careful consideration must be given to the manipulator design so that it can achieve useful poses while minimally influencing the stability of the host UAV. Accordingly, the novel application of a parallel manipulator mechanism is described.