Manipulating Camera Gimbal Positioning by Deep Deterministic Policy Gradient Reinforcement Learning for Drone Object Detection

In this study, the trend of unmanned aerial vehicle (UAV) technology and its applications in the mining industry were reviewed. To understand the trend of the UAV technology, the definition and classification of the UAVs, their major applications in each field, the characteristics of the sensors of the UAV were investigated. Some examples that used the UAV technology for planning, work management, facility management, safety management, and other works in the mining industry were reviewed. By introducing the UAV technology in the mining site, it is expected that the work efficiency related to mining operations would be increased. Therefore, the application of the UAV technology in the mining industry would be expanded further.

In recent years, as the cost of Unmanned Aerial Vehicle (UAV) technology has steadily decreased and advancements in data processing speed and image recognition have accelerated, UAVs have become increasingly favored in civil engineering surveying and mapping. The purpose of this paper is to study the development of UAV technology in the field of civil engineering surveying and mapping and its current application methods. By analysing the above contents, the problems still existing in the current practical use of UAV technology are studied. Through a review of relevant literature, newly developed technologies will be integrated with current UAV capabilities to propose recommendations for future improvements. This article employs literature analysis and case study methodologies to investigate the utilization of UAV technology. It delineates the current state of development and the challenges that UAV technology faces in the surveying field by examining extant academic research materials and legal regulations. It also provides a concise overview of the distinctive benefits of UAV technology. Simultaneously, the deficiencies of UAV technology in field environments are investigated, particularly focusing on data collection and accuracy in engineering surveying projects. UAV technology has obvious advantages compared with traditional methods in the field of civil engineering surveying and mapping. It can complete surveying and mapping tasks efficiently and quickly in many application scenarios. The high flexibility of UAVs and the high-precision scanning of their sensors are enough to meet the needs of current engineering surveying and mapping. The wide application of UAV technology still faces problems such as imperfect laws and regulations, insufficient surveying and mapping scope, and large surveying and mapping data. Addressing these issues requires the development of new laws and the creation of algorithms to enhance data processing capabilities.

Abstract. This paper looks into the latest achievement in the low-cost Unmanned Aerial Vehicle (UAV) technology in their capacity to map the semi-development areas. The objectives of this study are to establish a new methodology or a new algorithm in image registration during interior orientation process and to determine the accuracy of the photogrammetric products by using UAV images. Recently, UAV technology has been used in several applications such as mapping, agriculture and surveillance. The aim of this study is to scrutinize the usage of UAV to map the semi-development areas. The performance of the low cost UAV mapping study was established on a study area with two image processing methods so that the results could be comparable. A non-metric camera was attached at the bottom of UAV and it was used to capture images at both sites after it went through several calibration steps. Calibration processes were carried out to determine focal length, principal distance, radial lens distortion, tangential lens distortion and affinity. A new method in image registration for a non-metric camera is discussed in this paper as a part of new methodology of this study. This method used the UAV Global Positioning System (GPS) onboard to register the UAV image for interior orientation process. Check points were established randomly at both sites using rapid static Global Positioning System. Ground control points are used for exterior orientation process, and check point is used for accuracy assessment of photogrammetric product. All acquired images were processed in a photogrammetric software. Two methods of image registration were applied in this study, namely, GPS onboard registration and ground control point registration. Both registrations were processed by using photogrammetric software and the result is discussed. Two results were produced in this study, which are the digital orthophoto and the digital terrain model. These results were analyzed by using the root mean square errors and mean absolute error to determine the level of accuracy and the precision of photogrammetric products. It can be concluded that the new method of image registration by using the GPS onboard of the UAV produces medium accuracy result compared to the method that uses the ground control point. This new method can be used for the medium accuracy requirements. Unmanned Aerial Vehicle can be used for several applications, which requires a medium accuracy.

This study aims to propose a technology forecasting approach based on hierarchical S-curves. The proposed approach uses holistic forecasting by evaluating the S-curves of sub-technologies as well as the main technology under concern. A case study of unmanned aerial vehicle (UAV) technologies is conducted to demonstrate how the proposed approach works in practice. This is the first study that applies hierarchical S-curves to technology forecasting of unmanned aerial vehicle technologies in the literature. The future trend of the UAV technologies is analysed in detail through a hierarchical S-curve approach. Hierarchical S-curves are also utilised to investigate the sub-technologies of the UAV. In addition, the technology development life cycle of technology is assessed by using the three indexes namely, (1) the current technological maturity ratio (TMR), (2) estimating the number of potential patents that could be granted in the future (PPA), and (3) forecasting the expected remaining life (ERL). The results of this study indicate that the UAV technologies and their sub-technologies are at the growth stage in the technology life cycle, and most of the developments in UAV technology will have been completed by 2048. Hence, these technologies can be considered emerging technologies.

This paper underscores the significance of safety and reliability in the realm of unmanned aerial vehicle (UAV) technologies, and how regulations play a pivotal role in ensuring their responsible use. We have analyzed safety incidents and trends both in Canada and globally, noting a decline in incidents attributed to enhanced regulations. Our comparative analysis of different UAV technologies identified batteries as the most reliable power supply, Global Navigation Satellite System as the most effective navigation system, and light detection and ranging as the optimal optical sensor due to regulatory compliance and system redundancies. We also examined the regulatory framework in Canada, comparing it with the risk-based approach of the European Union Aviation Safety Agency and the efforts of Joint Authorities for Rule-making on Unmanned Systems towards global harmonization. Furthermore, we highlighted emerging trends in automation and flight control technologies, with a focus on European regulations shaping UAV automation trends. In conclusion, by adhering to best practices from other regulatory bodies, embracing emerging trends, and adopting a risk-based approach, Canada can promote the growth of the UAV industry while ensuring safety and reliability in UAV technologies.

Unmanned aerial vehicles (UAVs) are increasingly deployed in missions that require high autonomy and reliable decision-making; however, many operational concepts still assume access to GNSS and stable communication with a human operator. In contested environments, this assumption may no longer hold because GNSS degradation, radio-frequency interference, and intentional jamming can disrupt positioning and communication, thereby reducing mission effectiveness and safety. Recent surveys show that operation in GNSS-denied environments remains a major challenge and often requires alternative perception, localization, and control strategies. In response, this article investigates a reinforcement learning (RL)-based decision-support system for the autonomous control of a quadrotor UAV in a three-dimensional simulated environment. Rather than following pre-programmed waypoints, the UAV learns a control policy through interaction with the environment and reward-driven adaptation. The proposed system is designed for mission execution under uncertainty, limited external guidance, and partial observability. Two policy-gradient approaches are implemented and compared: classical REINFORCE and Proximal Policy Optimization (PPO) with an Actor-Critic architecture. The study presents the simulation environment, state and action representation, reward formulation, staged training procedure, and comparative evaluation. The results indicate that, within the considered unseen test scenario, the PPO-based configuration achieved higher mission effectiveness than REINFORCE in the final unseen test scenario, supporting the practical relevance of structured deep reinforcement learning for UAV operation in GPS-denied and communication-constrained environments.

Abstract—These technological advances make researchers around the world develop many new things to support the harmony of life. Humans think a lot, including in terms of technology for the advancement of the times and to maintain life, as well as for handling the unprecedented outbreak of the corona virus that spread in the end of 2019, referred to as COVID-19 by the World Health Organization (WHO), and has put the whole world in a precarious position. The impact of the COVID-19 outbreak, which was witnessed only by Chinese citizens, has become a problem of great concern for almost every country around the world makes the COVID-19 outbreak is spreading tremendous fear, overburdened healthcare system has forced most countries around the world are in a state of partial or complete lockdown. Researchers have begun to develop studies in various fields to help control the virus. Technology is also forced to adapt and develop faster to deal with the existing situation. In response to this action, with this review paper we use trusted sources to present a detailed review of specific aspects related to technological developments during the COVID-19 pandemic, especially UAV or drone technology. In this study we explore the use of technologies such as Unmanned Aerial Vehicles (UAVs) or drones that are modified for handling pandemic situations, and how these technologies are also combined with other trending technologies such as the Internet of Things (IoT), Artificial Intelligence (AI), and 5G, among others, to help reduce their impact from the COVID-19 outbreak and the challenges for the post COVID-19 period. Where UAV technology can be modified and developed according to the needs of the pandemic and development for the post-pandemic and endemic period

Information acquisition and rescue operation management can be facilitated by the use of unmanned aerial vehicles (UAVs) equipped with cameras and sensors that are capable of aerial photography and acquire important information from disaster-stricken areas. UAV technology is a relatively low-cost, flexible, and safe solution in missions that are difficult for humans or other systems. UAV usage in pre-and post-disaster assessment, response, and management is an active area of research. There are two important technical aspects of UAV applications in disaster situations. The first is the limited communication range. Information acquired and photography produced by UAVs are only transmitted in real-time to the ground control station (GCS) within its communication range. Remote real-time monitoring of distant areas beyond the UAV's communication range is not possible using a single UAV. It is only possible through communication among several UAVs. The second is the relatively short flight time due to limited battery capacity. UAV replacement is required in order to carry out a persistent monitoring mission. In this paper, we propose a new remote real-time monitoring system of a distant disaster area using multiple UAVs. Wireless communications between multiple UAVs constitutes a UAV network in our proposed system. Moreover, the proposed system replaces the UAVs in-flight with standby UAVs according to the remaining energy levels. We compare the performance of the proposed system with an existing UAV system. Our simulation results show that the proposed system can achieve efficient remote real-time monitoring.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Unmanned Aerial Vehicle (UAV) technology is being used increasingly for military and civilian purposes. The primary reason for this increase is that UAVs eliminate the risk to human life in difficult and dangerous missions, are cost effective, and easily are deployed. Developments in UAV technology and decreasing costs have increased UAV usage. However, when multiple UAVs are deployed, inter UAV communication becomes complicated. For this reason, communication in multi-UAV systems is the most important problem that needs to be solved. To enable communication among UAVs without infrastructure support, a Flying Ad Hoc Network (FANET) is used. A FANET provides UAVs to fly in tandem without colliding. To ensure coordinated flight, UAVs require the location information of other UAVs. In this study, we developed a common channel multi-token circulation protocol to share location information in multi-UAV systems that communicate using a FANET. The proposed method ensures that UAVs in multi-UAV systems know each other's coordinate information with minimum error.DOI: 10.5755/j01.eie.25.1.22738

A survey on cellular-connected UAVs: Design challenges, enabling 5G/B5G innovations, and experimental advancements

Unmanned aerial vehicle (UAV) technology is a rapidly growing field with tremendous opportunities for research and applications. To achieve true autonomy for UAVs in the absence of remote control, external navigation aids like global navigation satellite systems and radar systems, a minimum energy trajectory planning that considers obstacle avoidance and stability control will be the key. Although this can be formulated as a constrained optimization problem, due to the complicated non-linear relationships between UAV trajectory and thrust control, it is almost impossible to be solved analytically. While deep reinforcement learning is known for its ability to provide model free optimization for complex system through learning, its state space, actions and reward functions must be designed carefully. This paper presents our vision of different layers of autonomy in a UAV system, and our effort in generating and tracking the trajectory both using deep reinforcement learning (DRL). The experimental results show that compared to conventional approaches, the learned trajectory will need 20% less control thrust and 18% less time to reach the target. Furthermore, using the control policy learning by DRL, the UAV will achieve 58.14% less position error and 21.77% less system power.

Unmanned Aerial Vehicles (UAVs) are considered an important element in wireless communication networks due to their agility, mobility, and ability to be deployed as mobile base stations (BSs) in the network to improve the communication quality and coverage area. UAVs can be used to provide communication services for ground users in different scenarios, such as transportation systems, disaster situations, emergency cases, and surveillance. However, covering a specific area under a dynamic environment for a long time using UAV technology is quite challenging due to its limited energy resources, short communication range, and flying regulations and rules. Hence, a distributed solution is needed to overcome these limitations and to handle the interactions among UAVs, which leads to a large state space. In this paper, we introduced a novel distributed control solution to place a group of UAVs in the candidate area in order to improve the coverage score with minimum energy consumption and a high fairness value. The new algorithm is called the state-based game with actor–critic (SBG-AC). To simplify the complex interactions in the problem, we model SBG-AC using a state-based potential game. Then, we merge SBG-AC with an actor–critic algorithm to assure the convergence of the model, to control each UAV in a distributed way, and to have learning capabilities in case of dynamic environments. Simulation results show that the SBG-AC outperforms the distributed DRL and the DRL-EC3 in terms of fairness, coverage score, and energy consumption.

In recent years, the use of unmanned aerial vehicle (UAV) platforms in civil and military applications has surged, highlighting the critical role of artificial intelligence (AI) embedded UAV systems in the future. This study introduces the Autonomous Drone (Vechür-SIHA), a novel AI-embedded UAV system designed for real-time detection and tracking of other UAVs during flight sequences. Leveraging advanced object detection algorithms and an LSTM-based tracking mechanism, our system achieves an impressive 80% accuracy in drone detection, even in challenging conditions like varying backgrounds and adverse weather. Our system boasts the capability to simultaneously track multiple drones within its field of view, maintaining flight for up to 35 minutes, making it ideal for extended missions that require continuous UAV tracking. Moreover, it can lock onto and track other UAVs in mid-air for durations of 4-10 seconds without losing contact, a feature with significant potential for security applications. This research marks a substantial contribution to the development of AI-embedded UAV systems, with broad implications across diverse domains such as search and rescue operations, border security, and forest fire prevention. These results provide a solid foundation for future research, fostering the creation of similar systems tailored to different applications, ultimately enhancing the efficiency and safety of UAV operations. The novel approach to real-time UAV detection and tracking presented here holds promise for driving innovations in UAV technology and its diverse applications.

An evaluative review of the VTOL technologies for unmanned and manned aerial vehicles