Visual and Kinematic Coordinated Control of Mobile Manipulating Unmanned Aerial Vehicles

Due to their ability to navigate in 6 degree of freedom space, Unmanned Aerial Vehicles (UAVs) can access many locations that are inaccessible to ground vehicles. While mobile manipulation is an extremely active field of research for ground traveling host platforms, UAVs have historically been used for applications that avoid interaction with their environment at all costs. Recent efforts have been aimed at equipping UAVs with dexterous manipulators in an attempt to allow these Mobile Manipulating UAVs (MM-UAVs) to perform meaningful tasks such as infrastructure repair, disaster response, casualty extraction, and cargo resupply. Among many challenges associated with the successful manipulation of objects from a UAV host platform include: a) the manipulator's movements and interaction with objects negatively impact the host platform's stability and b) movements of the host platform, even when using highly accurate motion capture systems for position control, translate to poor end effector position control relative to fixed objects. To address these two problems, we propose the use of a hyper-redundant manipulator for MM-UAV applications. The benefits of such a manipulator are that it: a) can be controlled in such a way that links are moved within the arm's free space to help reduce negative impacts on the host platform's stability and b) the redundancy of the arm affords a highly reachable workspace for the end effector, allowing the end effector to track environmental objects smoothly despite host platform motions. This paper describes the design of a hyper-redundant manipulator suitable for studying its applicability to MM-UAV applications and provides preliminary results from its initial testing while mounted on a stationary scaffold.

Manipulating objects using arms mounted to unmanned aerial vehicles (UAVs) is attractive because UAVs may access many locations that are otherwise inaccessible to other mobile manipulation platforms such as ground vehicles. Despite recent work, several major challenges remain to be overcome before it will be practical to manipulate objects from UAVs. Among these challenges are: (a) The constantly moving UAV platform and compliance of manipulator arms make it difficult to position the UAV and end-effector relative to an object of interest precisely enough for manipulation, and (b) The motions of the manipulator impact the stability of the host UAV, further complicating positioning. Solving these challenges will bring UAVs one step closer to being able to perform meaningful tasks such as infrastructure repair, disaster response, casualty extraction, and cargo resupply. Toward solutions to these challenges, this paper describes a hyper-redundant manipulator, manipulator control approaches and system design considerations to position the manipulator relative to objects of interest in such a way that impacts on platform stability are minimized.

In this paper, a novel three dimensional (3D) positioning framework of fluid antenna system (FAS)-enabled unmanned aerial vehicles (UAVs) is developed. In the proposed framework, a set of controlled UAVs including an active UAV and four FAS-enabled passive UAVs cooperatively estimate the real-time 3D position of a target UAV. Here, the active UAV transmits a measurement signal to the passive UAVs via the reflection from the target UAV. Each passive UAV estimates the distance of the active-target-passive UAV link and selects an antenna port to share the distance information with the base station (BS), which calculates the real-time position of the target UAV. As the target UAV is moving due to its task operation, the controlled UAVs must optimize their trajectories and select optimal antenna port for transmitting the positioning information, aiming to estimate the real-time position of the target UAV. We formulate this problem as an optimization problem whose goal is to minimize the target UAV positioning error via optimizing the trajectories of all controlled UAVs and antenna port selection of passive UAVs. To address this problem, an attention-based recurrent multi-agent reinforcement learning (AR-MARL) scheme is proposed, which enables each controlled UAV to use the local Q function to determine its trajectory and antenna port while optimizing the target UAV positioning performance without knowing the trajectories and antenna port selections of other controlled UAVs. Different from current MARL methods that use feedforward neural networks to approximate Q functions, the proposed method uses a recurrent neural network (RNN) that incorporates historical state-action pairs of each controlled UAV, and an attention mechanism to analyze the importance of these historical state-action pairs, thus improving the global Q function approximation accuracy and the target UAV positioning accuracy. Simulation results show that the proposed scheme can reduce the average positioning error by up to 17.5% and 58.5% compared to the value decomposition based-MARL scheme with FAS and the proposed AR-MARL method without FAS.

In recent years, unmanned aerial vehicle (UAV) has been widely used in target tracking tasks such as security, anti-terrorism, tracking and monitoring because of its wide and flexible perspective. At the same time, deep learning creates new conditions for UAV visual recognition and motion tracking because it has high recognition accuracy and fast recognition speed. This paper presents an UAV dynamic tracking algorithm based on deep learning. The algorithm mainly includes three parts: the PTZ control, relative position calculation and the UAV control. The main work includes: 1. The tracking target is identified through the deep learning network based on YOLOv3, and the PTZ angle is adjusted in real time by using the control method of combining the PID control with the time-delay control to reduce the impact of image transmission delay; 2. The relative position between the UAV and the target is calculated by combining the UAV flight altitude, UAV attitude, PTZ angle and the coordinate of the target in the image. At the same time, the relative position prediction is added to estimate the relative distance between the UAV and the target at the next frame time as a reference value of the UAV speed, so as to improve the dynamic tracking ability; 3. The relative position information is fed back to the flight control function, and the linear velocity of UAV is output by the PID controller. Through the simulation test in Airsim, it is proved that the proposed algorithm can effectively improve the stability and robustness of the UAV tracking system.

This paper describes an accurate attitude determination technique using multiple global navigation satellite system (GNSS) receivers and antennas for small unmanned aerial vehicles (UAVs). A laser survey using a small UAV needs highly accurate position and attitude information for reconstructing 3D models. The proposed technique uses the multiple GNSS antenna's positions computed by the GNSS carrier phase measurements to estimate the UAV's position and attitude. To solve the GNSS carrier phase ambiguity, the reliable ambiguity search is performed with a couple of constraints for the base line vector. The performance of the proposed technique is evaluated in the static test. From the test, the proposed technique could estimate UAV's attitude in 0.1 degree accuracy. Furthermore, the accurate UAV's position could be estimated in almost 100 % availability.

With technological development of multi sensors, UAV (unmanned aerial vehicle) can identify and locate key targets in essential monitoring areas or geological disaster-prone areas by taking video sequence images, and precise positioning of the video sequence images is constantly a matter of great concern. In recent years, precise positioning of aerial images has been widely studied. But it is still a challenge to simultaneously realize precise, robust and dynamic positioning of UAV’s patrolling video sequence images in real time. In order to solve this problem, a visual positioning model for patrolling video sequence images based on DOM rectification is proposed, including a robust block-matching algorithm and a precise polynomial-rectifying algorithm. First, the robust block-matching algorithm is used to obtain the best matching area for UAV’s video sequence image on DOM (Digital Orthophoto Map), a pre-acquired digital orthophoto map covering the whole UAV’s patrolling region. Second, the precise polynomial-rectifying algorithm is used to calculate accurate rectification parameters of mapping UAV’s video sequence image to the best matching area obtained above, and then real time positioning of UAV’s patrolling video sequence images can be realized. Finally, the above two algorithms are analyzed and verified by three practical experiments, and results indicate that even if spatial resolution, surface specific features, illumination condition and topographic relief are significantly different between DOM and UAV’s patrolling video sequence images, proposed algorithms can still steadily realize positioning of each UAV’s patrolling video sequence image with about 2.5 m level accuracy in 1 s. To some extent, this study has improved precise positioning effects of UAV’s patrolling video sequence images in real time, and the proposed mathematical model can be directly incorporated into UAV’s patrolling system without any hardware overhead.

In the unmanned aerial vehicle (UAV) based localization of slow-moving radio frequency (RF) sources with unknown transmitted strength of signal, such as a person with a cell phone in a search and rescue mission, the UAV navigation errors are significant sources of localization error. Although the use of a global positioning system (GPS) can reduce the UAV’s localization error significantly resulting in more accurate RF source localization, if the GPS signal is lost temporarily or permanently, the accuracy of the UAV-based localization decreases rapidly. In this paper, a simultaneous localization and mapping (SLAM)-inspired approach for simultaneous localization of UAV and RF sources (SLUS) is proposed. The proposed approach solves these two connected problems, i.e. the UAV localization and RF source localization, simultaneously to decrease the error of the UAV position estimation and the error of the RF source localization. In the proposed approach, beside the UAV position prediction, the RF source position prediction is also performed. Then the predicted states are augmented and the augmented predicted state information is corrected using range-ratio and bearing observations, i.e. RF source features, considering the unknown transmitted power. The proposed approach is simulated and the results show that the normal divergence of a target localization and the divergence of the UAV navigation in latitude and longitude channels have been eliminated using this approach. In other words, simultaneous localization of the UAV and RF sources uses the RF sources, as features in the environment, to aid the navigation system. Although the approach is similar to the mapping of RF sources in the environment, the created map would be useless after finding the RF sources. That is why SLUS has been used instead of SLAM. The main contributions of this work are: 1) performing simultaneous localization of a UAV and targets using RF signals, especially in a non-line of sight (NLOS) condition; 2) using difference of signal strength, i.e. differential received strength signal indicator (DRSSI), to eliminate the impact of unknown target signal power; and 3) simultaneous multi-target localization and tracking.

Unmanned aerial vehicles (UAVs) have been considered as a solution to extend coverage for disaster-resilient communication networks in a disaster area when the communication infrastructure which is known as the base station (BS) cannot afford the service as usual. However, when the link between the UAV and the serving BS is constrained, the UAV must keep the appropriate distance for reliable transmission. In addition, the altitude of the UAV may significantly affect the link quality between the UAV and the BS, as well as the link between the UAV and the users/rescuers. In this paper, we analyze the coverage of the disaster-resilient communication network with a UAV for guaranteeing the reliability of all links. We show that the position of UAV can be a function of the BS's and user's elevation angle. The simulation results show that the maximum radius of coverage in the high-rise urban environment can be 5.25 km and 2.46 km for downlink and uplink, respectively. Moreover, this paper can provide a principle for the central emergency operation center (CEOC) to immediately deploy the disaster-resilient communication network.

In this paper, the problem of using one active unmanned aerial vehicle (UAV) and four passive UAVs to localize a 3D target UAV in real time is investigated. In the considered model, each passive UAV receives reflection signals from the target UAV, which are initially transmitted by the active UAV. The received reflection signals allow each passive UAV to estimate the signal transmission distance which will be transmitted to a base station (BS) for the estimation of the position of the target UAV. Due to the movement of the target UAV, each active/passive UAV must optimize its trajectory to continuously localize the target UAV. Meanwhile, since the accuracy of the distance estimation depends on the signal-to-noise ratio of the transmission signals, the active UAV must optimize its transmit power. This problem is formulated as an optimization problem whose goal is to jointly optimize the transmit power of the active UAV and trajectories of both active and passive UAVs so as to maximize the target UAV positioning accuracy. To solve this problem, a Z function decomposition based reinforcement learning (ZD-RL) method is proposed. Compared to value function decomposition based RL (VD-RL), the proposed method can find the probability distribution of the sum of future rewards to accurately estimate the expected value of the sum of future rewards thus finding better transmit power of the active UAV and trajectories for both active and passive UAVs and improving target UAV positioning accuracy. Simulation results show that the proposed ZD-RL method can reduce the positioning errors by up to 39.4% and 64.6%, compared to VD-RL and independent deep RL methods, respectively.

U-turn is a common and typical turn process in area coverage of Unmanned Aerial Vehicle (UAV) in which the flying path of UAV projection and motion path of camera footprint center do not coincide with each other. This paper investigates the U-turn optimization for three-dimensional area coverage of UAV. Firstly, the relationship between UAV attitude and camera footprint is studied. And some related formulas concerning the camera footprint of UAV are given for different maneuvers which can be used to obtain the parameters of camera footprint and coordinate transformation between UAV centroid and footprint center. Then the differences paths of UAV projection and camera footprint are discussed in line sweep coverage approach. Finally, The U-turn optimization of UAV which contains the distances and duration is analyzed in three different cases.

This paper considers unmanned aerial vehicle (UAV)-assisted infrastructure-to-vehicle (I2V) communication employing UAVs as relays to increase the throughput between a roadside unit (RSU) and a vehicular user equipment (VUE). We investigate the UAV position planning problem under both single UAV and multiple cooperative UAVs scenarios while considering the mobility of the VUE, aiming to maximize the data rate of the system. We first consider using a single UAV and prove that the single UAV position planning can be formulated as a convex optimization problem, and then obtain the optimal position of the UAV. Next, we investigate the multiple cooperative UAVs scenario and formulate the joint power control and position planning problem to improve the data rate of the system under a fixed total power consumption. Numerical simulations are provided to verify our theoretical results. Our findings highlight the effects of important system parameters, such as height, transmit power, and the number of UAVs, on the optimal UAV positioning and system performance.

소형 및 저가형 CCD 카메라의 성능은 소형 쿼드콥터의 정밀 추적기능을 구현하는데 있어서 충분한 성능을 갖추고 있지 못하는데 본 연구에서는 덜 정확한 GPS 보다 CCD 카메라를 이용한 보행자와 같은 대상물의 상공에서 강건한 호버링을 유지시키기 위한 방법을 제시하였다. 기존의 연구 대상이었던 고정된 물체가 아닌 보행자를 타깃으로 이용한 UAV의 절대 위치를 추정하는 방법을 제시하였다. 이는 UAV가 산악이나 사람들이 붐비는 공공지역에서 이동할 때 UAV의 절대위치를 인식할 수 있는 방법이 없을 경우 UAV 주변에서 움직이는 물체의 정보를 활용하여 UAV의 절대위치를 보정하는 방법으로 매우 유용하다. 연구를 위해서 보행자의 위치를 알고 있는 것으로 가정하나 실제적인 상황 속에서는 영상매칭을 통하여 그 정보를 수신하는 것으로 해석한다. 본 연구를 위하여 UAV의 위치 추정 불확실성을 정량적으로 나타내었으며, 좌표계 변환을 통한 영상기반의 기하학적 구속 식을 유도하여, 칼만 필터를 적용하여 로봇의 위치를 보정하여 위치 추정 불확실성을 줄일 수 있음을 보였다. The accuracy of small and low cost CCD camera is insufficient to provide data for precisely tracking unmanned aerial vehicles(UAVs). This study shows how UAV can hover on a human targeted tracking object by using CCD camera rather than imprecise GPS data. To realize this, UAVs need to recognize their attitude and position in known environment as well as unknown environment. Moreover, it is necessary for their localization to occur naturally. It is desirable for an UAV to estimate of his attitude by environment recognition for UAV hovering, as one of the best important problems. In this paper, we describe a method for the attitude of an UAV using image information of a maker on the floor. This method combines the observed position from GPS sensors and the estimated attitude from the images captured by a fixed camera to estimate an UAV. Using the a priori known path of an UAV in the world coordinates and a perspective camera model, we derive the geometric constraint equations which represent the relation between image frame coordinates for a marker on the floor and the estimated UAV's attitude. Since the equations are based on the estimated position, the measurement error may exist all the time. The proposed method utilizes the error between the observed and estimated image coordinates to localize the UAV. The Kalman filter scheme is applied for this method. its performance is verified by the image processing results and the experiment.

As a complicated multi-input, multi-output, and time-varying nonlinear system, flight control system of unmanned aerial vehicle (UAV), which determines the whole system’s performance directly, is crucial for the simulation training system design. This chapter mainly focuses on parameter identification of flight control system based on the modeling of UAVs and a specific controller design for the pendulum-like oscillation in micro aerial vehicle (MAV). A predator–prey particle swarm optimization (PSO) algorithm for identifying parameters of UAV flight control system is presented, with the aim of reducing the workload of the designers during the process of designing complicated UAV control system. Besides, a software environment for UAV controller design was developed based on the UAV model and the proposed method. Then a specific kind of controller design involving the pendulum-like oscillation in the hover and stare state for a MAV in the presence of external disturbances is discussed in detail, since the pendulum-like oscillation caused by uncertainty and external disturbances badly jeopardize the performance of hover and stare, resulting in blurred images or even MAV’s overturning. A novel type of PSO-based linear-quadratic regulator (LQR) controller for stabilizing the pendulum-like oscillation is developed, which can enhance the MAV’s performance efficiently.

: Unmanned aerial vehicles (UAVs) are critical components of the future naval forces. UAV control and monitoring with autonomous operation will become an absolute necessity and adaptive cooperation of vehicles is the only practical alternative. The objective of this project is to develop and evaluate new methodologies for cooperative (formation) control of multiple unmanned air vehicles. The goal is to have multiple UAVs working together as a group. Instead of separately assigning distinct tasks to each vehicle, the operator would assign tasks to the UAV group, which then determines the best way to accomplish each task, freeing the operator to maintain surveillance over the entire operation. In this project we investigated Path Tracking and obstacle avoidance of UAVs using fuzzy logic method. Algorithms for close formation control of multi-UAVs are developed and simulated. We also investigated fault-tolerant control of single UAVs by neuro-adaptive method. Detailed description of this method is provided in this document. The project has supported S graduate students with 9 technical papers published.

Wireless communication with unmanned aerial vehicle (UAV) has attracted much attention due to its flexibility and low-cost in providing wireless connectivity. In this article, we investigate the spectrum sharing between UAV and terrestrial wireless communication. Two scenarios are considered in this work. Scenario 1: the UAV is energy sufficient to transmit the required data. In this case, our objective is to minimize the time that is needed to complete data transmission. Scenario 2: the battery of the UAV is energy insufficient. In this case, the goal becomes maximizing the size of data that can be transmitted. Different scenarios require different placement of the UAV. When the UAV flies towards the primary user (PU), the sensing result will be more accurate and more opportunities can be utilized to transmit data. However, the interference to primary receiver (PR) and the propulsion energy due to UAV's movement will be increased. In this article, the sensing performance, the power allocation and the UAV's position are jointly optimized to satisfy different requirements of the UAV system. To efficiently utilize the licensed spectrum, a hybrid transmission mode is proposed for the UAV system, in which sensing and power control are jointly employed to protect the PR. Then, efficient iterative algorithms are proposed to solve the optimization problems. Simulations results show that proposed designs can significantly reduce the time that is required to complete data transmission in energy sufficient scenario and enhance the size of data that can be transferred in energy insufficient scenario.