Hierarchical markov decision processes based distributed data fusion and collaborative sensor management for multitarget multisensor tracking applications

Bayesian method is a method that synthesizes the prior information about unknown parameters with sample information, then obtains the posterior information according to Bayesian formula, and then deduces the unknown parameters according to the posterior information. Markov decision process is the optimal decision process of stochastic dynamic system based on Markov process theory. In this paper, we establish a two-stage hierarchical Markov decision process, which simulates the sequential marketing decision under the fluctuation of pigsty price. The decision-making process is based on market information such as pork, piglets and feed prices. In addition, the bayesian method is used to update the information and embed it into the hierarchical Markov decision process, and the optimal strategy under different price fluctuation patterns is analyzed. We also evaluated the value of incorporating price information into the model. In the manufacture of fattening pigs, pig sales refer to a series of selection decisions until the production unit is emptied. The profit of a production unit depends on the price of pork, the cost of raising and the cost of buying piglets. The price fluctuation in the market will affect the profit, and the optimal marketing decision may also change under different price conditions. Most studies have considered pig marketing at constant prices. However, because price fluctuation can not have no effect on marketing decision, it is necessary to consider pig marketing under price fluctuation. In this paper, we establish a two-level hierarchical markov decision-making process that simulates sequential marketing decisions under price fluctuations in pigsty. The decision-making process is based on market information such as pork, piglets and feed prices. In addition, the bayesian method is used to update the information and embed it into the hierarchical markov decision process to analyze the optimal strategy in different price fluctuation modes. we also evaluated the value of incorporating price information into the model.

In the manufacturing of fattening pigs, pig marketing refers to a sequence of culling decisions until the production unit is empty. The profit of a production unit is highly dependent on the price of pork, the cost of feeding and the cost of buying piglets. Price fluctuations in the market consequently influence the profit, and the optimal marketing decisions may change under different price conditions. Most studies have considered pig marketing under constant price conditions. However, because price fluctuations have an influence on profit and optimal marketing decisions it is relevant to consider pig marketing under price fluctuations. In this paper we formulate a hierarchical Markov decision process with two levels which model sequential marketing decisions under price fluctuations in a pig pen. The state of the system is based on information about pork, piglet and feed prices. Moreover, the information is updated using a Bayesian approach and embedded into the hierarchical Markov decision process. The optimal policy is analyzed under different patterns of price fluctuations. We also assess the value of including price information into the model.

A hierarchical Markov decision process modeling feeding and marketing decisions of growing pigs

This article presents an algorithmic framework for distributed data fusion using a network of multiple unmanned aerial vehicles (UAVs). Each UAV in the network is capable of sensing signals, performing computational tasks, and exchanging information with its immediate neighbors. Based on the limited local measurement and exchanged information only from neighbors, we aim to propose a fully distributed algorithm to integrate information from various measurements of individual UAVs. As an important building block of the distributed data fusion algorithm, a novel sum consensus scheme is also developed with only assuming row-stochastic weight matrices. Consequently, through the proposed distributed algorithm, each UAV is able to obtain a more reliable fused sensing result with higher accuracy, compared to the local measurement collected by the UAV itself. The effectiveness of the proposed algorithmic framework is validated by both theoretical analysis and numerical simulations.

Network-centric multisensor-multitarget tracking has numerous advantages over single-sensor or single-platform tracking. In this paper, we present a solution to one of the main problems of network-centric tracking, namely, decentralized information sharing among the platforms participating in the distributed data fusion. This paper presents a decision mechanism that provides each platform with the required data for the distributed data fusion process while reducing redundancy in the information flow in the overall system. We consider a distributed data fusion system consisting of platforms that are decentralized, heterogeneous, and potentially unreliable. The proposed approach, which is based on Markov decision processes and decentralized lookup substrate, will control the information exchange process based, among the other parameters, on tracking performance metrics of individual platforms, thereby enhancing the whole distributed system's reliability as well as that of each participating platform. Simulation examples demonstrate the operation and the performance results of the system

Electric vehicle (EV) is emerging as an effective choice to reduce carbon emissions in the modern transportation system. However, the large access of EVs brings tremendous passive influence on the stability and performance of power grid operation. Thus, it is valuable to investigate the charging power management method to improve charging experience of EV drivers while maintaining normal bus voltage of the power system. This paper builds the charging power control problem of multiple EVs as a hierarchical Markov decision process (MDP) model and propose a hierarchical multi-objective reinforcement learning method (Hamlet) to obtain the real-time charging power control decisions. Specifically, by treating EVs with the same remaining charging time as the same agent, this paper addresses the dynamic state and action spaces issue and significantly reduce the input dimension of both state and action spaces. Besides, this paper quantifies drivers' anxiety about the battery volume and distributes the comfort reward over the entire charging interval, which overcomes the delayed comfort reward issue. In order to alleviate the influence of multiple objectives on training stability, this paper develops a multi-objective learning method to dynamically adjust the optimization direction. Simulation results on IEEE 33 bus and 69 bus test feeders prove the validity of the proposed method in mitigating voltage fluctuation, minimizing bill payment and maximizing battery comfort.

Markov decision processes (MDPs) have proven useful as models of stochastic planning and decision problems. To try to propose practical implementation of MDPs, hierarchical methods are often used in MDPs or reinforcement learning to delegate the optimization of the total problem to simpler hierarchical sub-problems. The goal of the paper is to propose a generic discrete-event based software Framework allowing to use hierarchical MDPs and reinforcement learning to solve planning or decision problems. The proposed approach has been validated using the "grid world" typical MDP use case.

Clustering algorithms have been recently used in multitarget multisensor tracking (MMT) problems in order to reduce the size of the data association problem. This paper derives an equivalence relation (ER) clustering algorithm used in a MMT problem and briefly compares it to other clustering schemes such as the nearest neighbor method. The main contribution of this work is the analytical evaluation of ER clustering performance, in the context of multitarget multisensor tracking, as a function of the distance between targets, measurement probability density function, and cluster parameter.

This study focuses on the problem of multitarget tracking. To address the existing problems of current tracking algorithms, as manifested by the time consumption of subgroup separation and the uneven group size of unmanned aerial vehicles (UAVs) for target tracking, a multitarget tracking control algorithm under local information selection interaction is proposed. First, on the basis of location, number, and perceived target information of neighboring UAVs, a temporary leader selection strategy is designed to realize the local follow-up movement of UAVs when the UAVs cannot fully perceive the target. Second, in combination with the basic rules of cluster movement and target information perception factors, distributed control equations are designed to achieve a rapid gathering of UAVs and consistent tracking of multiple targets. Lastly, the simulation experiments are conducted in two- and three-dimensional spaces. Under a certain number of UAVs, clustering speed of the proposed algorithm is less than 3s, and the equal probability of the UAV subgroup size after group separation is over 78%.

The Joint Probabilistic Data Association (JPDA) solves single sensor multi-target tracking in clutter, but it cannot be used directly in multi-sensor multi-target tracking (MMT) and has high computational complexity with the number of targets and the number of returns. This paper presents a hybrid method to implement MMT by combing Maximum Likelihood Estimation (MLE) with Adaptive Neuro-Fuzzy Inference System (ANFIS). The MLE is applied to classify the same source observations at one time into the same set, then the cheap JPDA(CJPDA) approach is used to calculate the data association probability, and ANFIS is used to realize the MMT. The computer simulations indicate that this scheme achieves MMT perfectly with higher precision and easy realization.

A graph-theoretic metric for distributed data fusion in multi-target tracking

In the unmanned aerial vehicle (UAV) based localization of slow-moving radio frequency (RF) sources with unknown transmitted strength of signal, such as a person with a cell phone in a search and rescue mission, the UAV navigation errors are significant sources of localization error. Although the use of a global positioning system (GPS) can reduce the UAV’s localization error significantly resulting in more accurate RF source localization, if the GPS signal is lost temporarily or permanently, the accuracy of the UAV-based localization decreases rapidly. In this paper, a simultaneous localization and mapping (SLAM)-inspired approach for simultaneous localization of UAV and RF sources (SLUS) is proposed. The proposed approach solves these two connected problems, i.e. the UAV localization and RF source localization, simultaneously to decrease the error of the UAV position estimation and the error of the RF source localization. In the proposed approach, beside the UAV position prediction, the RF source position prediction is also performed. Then the predicted states are augmented and the augmented predicted state information is corrected using range-ratio and bearing observations, i.e. RF source features, considering the unknown transmitted power. The proposed approach is simulated and the results show that the normal divergence of a target localization and the divergence of the UAV navigation in latitude and longitude channels have been eliminated using this approach. In other words, simultaneous localization of the UAV and RF sources uses the RF sources, as features in the environment, to aid the navigation system. Although the approach is similar to the mapping of RF sources in the environment, the created map would be useless after finding the RF sources. That is why SLUS has been used instead of SLAM. The main contributions of this work are: 1) performing simultaneous localization of a UAV and targets using RF signals, especially in a non-line of sight (NLOS) condition; 2) using difference of signal strength, i.e. differential received strength signal indicator (DRSSI), to eliminate the impact of unknown target signal power; and 3) simultaneous multi-target localization and tracking.

This paper studies optimal powered dynamic soaring flights of unmanned aerial vehicles (UAVs) that utilize low‐altitude wind gradients for reducing fuel consumptions. Three‐dimensional point‐mass UAV equations of motion are used, and linear wind gradients are assumed. Fundamental UAV performance parameters are identified through the normalization of the equations of motion. In particular, a single wind condition parameter is defined that represents the combined effect of air density, UAV wing loading, and wind gradient slope on UAV flight. An optimal control problem is first used to determine bounds on wind conditions over which optimal powered dynamic soaring is meaningful. Then, powered UAV dynamic soaring flights through wind gradients are formulated as non‐linear optimal control problems. For a jet‐engined UAV, performance indices are selected to minimize the average thrust required per cycle of powered dynamic soaring that employs either variable or constant thrust. For a propeller‐driven UAV, in comparison, performance indices are selected to minimize the average power required per cycle of powered dynamic soaring with either variable or constant power. All problem formulations are subject to UAV equations of motion, UAV operational constraints, proper initial conditions, and terminal conditions that enforce a periodic flight. These optimal control problems are converted into parameter optimization with a collocation method and solved numerically using the parameter optimization software NPSOL. Analytical gradient expressions are derived for the numerical solution process. Extensive numerical solutions are obtained for a wide range of wind conditions and UAV performance parameters. Results reveal basic features of powered dynamic soaring flights through linear wind gradients. Copyright © 2004 John Wiley & Sons, Ltd.

U NMANNED Aerial Vehicles (UAVs) hold good promise for autonomously carrying out complex civilian and military operations. However, many of these missions require them to fly at low altitudes, making them vulnerable to collision with both stationary as well as moving obstacles. Hence, it is vital that UAVs are equipped with autonomous capability to sense and avoid collisions, especially for the pop-up threats. When such a threat is sensed and a collision is predictedwithin a short time ahead, theUAV should be able to react and maneuver away quickly so that the collision is avoided. An algorithmwhich can assure such amaneuver is called a “reactive collision avoidance algorithm.” Since the available reaction time in such a scenario is usually is small and UAVs are usually limited by computational resources, such an algorithm should also be computationally very efficient (it should preferably be noniterative). It is also required that whilemaneuvering away, it should notmaneuver toomuch away from the obstacle either. This is both to avoid collision from other nearby obstacles and not to compromise on the overall mission objective. There are various attempts in the literature to develop algorithms for collision avoidance purpose, many of which are inspired from global path planning algorithms. The artificial potential field method is such an approach where the motion of the vehicle is guided under the influence of a potential field. The potential field (which is essentially a cost function) is designed in such a way that obstacles have repulsive fields while the destination has an attractive field. The safe path of the UAV is then found by optimizing the carefully selected cost function. To tune this basic philosophy for reactive collision avoidance, a model predictive control-based algorithm has been proposed in the literature. This algorithm essentially assures path following under safe conditions (i.e. if no collision is predicted in the near future) and invokes the potential field function when new collisions are sensed. However, in potential field based techniques the associated optimization process is typically done in an iterative manner. Because of this they are usually computationally intensive and hence are not suitable for reactive collision avoidance of airborne UAVs in general. A promising algorithm in collision avoidance and global path planning is the philosophy of rapidly-exploring random tree (RRT) [1], which has also been used for reactive collision avoidance. However, there are many concerns about the RRT approach, which can largely be attributed to the random nature of the algorithm. For example, the path predicted by RRT is usually a sting of connected straight lines that does not reflect the path followed by a vehicle with nonholonomic constraints. More important, it is a probabilistic approach and hence there is no guarantee of finding a feasible path within a limited finite time. Other graph search algorithms such as best-first search are also implemented for reactive collision avoidance [2]. However, this is not systematic approach and could result in the algorithm searching far too many nodes under some conditions. Moreover, precomputing motion primitives and saving them in a lookup table is infeasible for UAVs, which are usually resource-limited. An interesting perspective to collision avoidance problem is the minimum effort guidance [3], where an optimal control-based approach has been proposed after applying the collision cone philosophy to detect collisions. This method is computationally nonintensive as a closed form solution has been proposed. Even though this is an interesting idea, by minimizing the lateral acceleration, perhaps it imposes unwanted extra constraint on the problem formulation as reactive collision avoidance problems do not necessarily have to be carried out with minimum lateral acceleration. More important, one can observe that this formulation only assures position guarantee and no constraint is imposed on the velocity vector. Hence, even though it guides the vehicle to a carefully selected target point on the safety boundary (we call it the “aiming point”), it causes the vehicle to maneuver until this point. This can be risky as the vehicle may enter the safety ball before reaching the aiming point. Even though the collision cone based aiming point philosophy is a very good idea, the authors of this Note strongly believe that instead of only position guarantee, rather the velocity vector should be aligned towards the aiming point as soon as a collision is detected (which will automatically lead to position guarantee as well). Towards this objective, two new nonlinear guidance laws are proposed in this Note, which are named as nonlinear geometric guidance (NGG) and differential geometric guidance (DGG). These guidance laws are inspired by the philosophy of “aiming point guidance” (APG) [4], which has been proposed in missile guidance literature. It turns out that the APG is a simplified case of the NGG where the associated since function is replaced by its linear approximation (hence, for a systematic discussion, it is renamed as linear geometric guidance (LGG) in this Note). Both of the guidance algorithms proposed in this Note quickly align the velocity vector of the UAValong the aiming point within a part of the available time-togo, which ensures quick reaction and hence safety of the vehicle. The main feature of this philosophy is that they effect high maneuvering at the beginning, causing the velocity vector of the UAV to align with the aiming point direction quickly and then settling along it. Therefore there is no need to maneuver all the way until the aiming point is reached and hence the chance of the UAV entering into the safety ball is minimized. Using the point of closest approach (PCA) [5], the proposed NGG andDGGalgorithms have also been extended for collision avoidance with moving obstacles in both cooperative as well as ignorant scenarios. Mathematical correlations between the guidance laws have also been established, which show that the NGG and DGG are exactly correlated to each other with appropriate gain selections, while the LGG is an approximation of DGG. A “sphere-tracking algorithm” is also proposed in this Note where the UAV is guided to track the surface of the safety spherewhenever a brief violation of the safety boundary occurs after reaching the aiming point because of the location of the next aiming point (which may include the target in Presented as Paper 2010-8315 at the AIAA Guidance, Navigation and Control, Toronto, 2–5August 2010; received 26May 2010; revision received 4 October 2010; accepted for publication 6 October 2010. Copyright © 2010 by Radhakant Padhi. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission. Copies of this paper may be made for personal or internal use, on condition that the copier pay the $10.00 per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923; include the code 0731-5090/11 and $10.00 in correspondence with the CCC. ∗Former Project Assistant, Department of Aerospace Engineering; anushamujumdar87@gmail.com. Associate Professor, Department of Aerospace Engineering; padhi@ aero.iisc.ernet.in. JOURNAL OF GUIDANCE, CONTROL, AND DYNAMICS Vol. 34, No. 1, January–February 2011

To address the challenges of real-time decision-making and resource optimization in multi-agent cooperative interception tasks within dynamic environments, this paper proposes a hierarchical framework for reinforcement learning-based interception algorithm (HFRL-IA). By constructing a hierarchical Markov decision process (MDP) model based on dynamic game equilibrium theory, the complex interception task is decomposed into two hierarchically optimized stages: dynamic task allocation and distributed path planning. At the high level, a sequence-to-sequence reinforcement learning approach is employed to achieve dynamic bipartite graph matching, leveraging a graph neural network encoder–decoder architecture to handle dynamically expanding threat targets. At the low level, an improved prioritized experience replay multi-agent deep deterministic policy gradient algorithm (PER-MADDPG) is designed, integrating curriculum learning and prioritized experience replay mechanisms to effectively enhance the interception success rate against complex maneuvering targets. Extensive simulations in diverse scenarios and comparisons with conventional task assignment strategies demonstrate the superiority of the proposed algorithm. Taking a typical scenario of 10 agents intercepting as an example, the HFRL-IA algorithm achieves a 22.51% increase in training rewards compared to the traditional end-to-end MADDPG algorithm, and the interception success rate is improved by 26.37%. This study provides a new methodological framework for distributed cooperative decision-making in dynamic adversarial environments, with significant application potential in areas such as maritime multi-agent security defense and marine environment monitoring.