Approaches to Multisensor Data Fusion in Target Tracking: A Survey

In the recent era of the Internet of Things, the dominant role of sensors and the Internet provides a solution to a wide variety of real-life problems. Such applications include smart city, smart healthcare systems, smart building, smart transport and smart environment. However, the real-time IoT sensor data include several challenges, such as a deluge of unclean sensor data and a high resource-consumption cost. As such, this paper addresses how to process IoT sensor data, fusion with other data sources, and analyses to produce knowledgeable insight into hidden data patterns for rapid decision-making. This paper addresses the data processing techniques such as data denoising, data outlier detection, missing data imputation and data aggregation. Further, it elaborates on the necessity of data fusion and various data fusion methods such as direct fusion, associated feature extraction, and identity declaration data fusion. This paper also aims to address data analysis integration with emerging technologies, such as cloud computing, fog computing and edge computing, towards various challenges in IoT sensor network and sensor data analysis. In summary, this paper is the first of its kind to present a complete overview of IoT sensor data processing, fusion and analysis techniques.

Multi-sensor data fusion has found widespread applications in industrial and research sectors. The purpose of real time multi-sensor data fusion is to dynamically estimate an improved system model from a set of different data sources, i.e., sensors. This paper presented a systematic and unified real time sensor validation and fusion framework (RTSVFF) based on distributed autonomous sensors. The RTSVFF is an open architecture which consists of four layers - the transaction layer, the process fusion layer, the control layer, and the planning layer. This paradigm facilitates distribution of intelligence to the sensor level and sharing of information among sensors, controllers, and other devices in the system. The openness of the architecture also provides a platform to test different sensor validation and fusion algorithms and thus facilitates the selection of near optimal algorithms for specific sensor fusion application. In the version of the model presented in this paper, confidence weighted averaging is employed to address the dynamic system state issue noted above. The state is computed using an adaptive estimator and dynamic validation curve for numeric data fusion and a robust diagnostic map for decision level qualitative fusion. The framework is then applied to automatic monitoring of a gas-turbine engine, including a performance comparison of the proposed real-time sensor fusion algorithms and a traditional numerical weighted average.

Thermal/power issues have become increasingly important with more and more transistors being placed on a single chip. Many dynamic thermal/power management techniques have been proposed to address such issues but they all depend heavily on accurate knowledge of the chip's thermal state during runtime. In this paper, we describe a unified statistical framework for designing an on-chip thermal sensing infrastructure that can be used to track the chip's thermal state at runtime. Specifically, we address the following problems in this statistical framework: 1) sensor placement; 2) sensor data compression; 3) sensor data fusion; and 4) overall interplay. Our methods exploit the correlations between temperatures in different parts of the chip to drive sensor placement, data compression, and data fusion in both noiseless and noisy sensor cases. Our framework is also capable of choosing the appropriate degree of compression for each sensor while accounting for their local space constraints during deployment. The experimental results show that the root-mean-square error of the thermal estimates produced by our sensing infrastructure is on average 35% better than an equivalent system that uses a range-based placement scheme and a uniform compression scheme. It took our methods at most about 9 s to decide the overall solution for placement, compression, and data fusion at the design stage. This demonstrates the effectiveness and applicability of our unified statistical design methodology.

This tutorial paper provides a short introduction to selected aspects of sensor data fusion by discussing characteristic examples. We consider three cases when fusion of sensor data is important: when emphasis is placed on data produced at different instants of time (i.e., target tracking), when data being collected from different sensor sources are important, and when we have data with background information on the sensor performance as well as data with nonsensor context information. The feedback from data processing to the data acquisition process is illustrated by a sensor management example.

Unmanned Aerial System (UAS)-based phenotyping of soybean using multi-sensor data fusion and extreme learning machine

The problem of sensor integration and data fusion is addressed. We consider the problem of combining information from diversified sources in a coherent fashion. We assume that at the fusion, the information from various sensors may be available in different forms. For example, data from infrared (IR) sensors may be combined with range radar (RR) data, and further combined with visual images. In each case, the data and information from the different sensors are presented in a different format which may not be directly compatible for all sensors. Furthermore, the available information may be in the form of attributes and not dynamical measurements. A theory for sensor integration and data fusion that accommodates diversified sources of information is presented. Data (or, more generically, information) fusion may proceed at different levels, like the level of dynamics, the level of attributes, and the level of evidence. All different levels are considered and several practical examples of real world data fusion problems are discussed.

Since the potential of soft computing for driver assistance systems has been recognized, much effort has been spent in the development of appropriate techniques for robust lane detection, object classification, tracking, and representation of task relevant objects. For such systems in order to be able to perform their tasks the environment must be sensed by one or more sensors. Usually a complex processing, fusion, and interpretation of the sensor data is required and imposes a modular architecture for the overall system. In this paper, we present specific approaches considering the main components of such systems. We concentrate on image processing as the main source of relevant object information, representation and fusion of data that might arise from different sensors, and behavior planning and generation as a basis for autonomous driving. Within our system components most paradigms of soft computing are employed; in this article we focus on Kalman filtering for sensor fusion, neural field dynamics for behavior generation, and evolutionary algorithms for optimization of parts of the system.

Deep reinforcement learning (DRL) is widely used for autonomous systems including autonomous driving, robots, and drones. DRL training is essential for human-level control and adaptation to rapidly changing environments in mobile autonomous systems. However, acceleration of DRL training has three challenges: 1) large memory access, 2) various data patterns, 3) complex data dependency due to utilization of multiple DNNs. Two CMOS DRL accelerators have been proposed to support high speed, high energy-efficiency DRL training in mobile autonomous systems. One accelerator handles different data patterns with transposable PE architecture and reduces large feature map memory access with top-3 experience compression. The other accelerator supports group-sparse training for weight compression and integrates the on-line DRL task scheduler to support multi-DNNs operations.

This chapter reviews architectures for efficient multi-sensor data fusion (MSDF). The following definitions are used: Data Fusion: This is the process of combining sensed and contextual data through increasing levels of abstraction in order to detect, classify and track objects, and to form a situation assessment and infer its probable impact The chapter focuses only on sensed data fusion and specifically on track fusion. Architecture: This refers to the structural organisation of an MSDF system with respect to its sensing, communications and data processing sub-systems. There are two fundamentally different types of MSDF architecture: centralised and decentralised systems. The chapter emphasizes decentralised systems, as these are generally the more efficient architecture. Efficiency: This is a measure of the performance of an MSDF system as a function of its sensing, computational and communications resource requirements. Information theory is a principled framework for decentralised data fusion and systems resource management.

This research presents an integrated approach for object detection and tracking in autonomous perception systems, combining deep learning techniques for object detection with sensor fusion and field programmable gate array (FPGA-based) hardware implementation of the Kalman filter. This approach is suitable for applications like autonomous vehicles, robotics, and augmented reality. The study explores the seamless integration of pre-trained deep learning models, sensor data from a depth camera, real-sense D435, and FPGA-based Kalman filtering to achieve robust and accurate 3D position and 2D size estimation of tracked objects while maintaining low latency. The object detection and feature extraction are implemented on a central processing unit (CPU), and the Kalman filter sensor fusion with universal asynchronous receiver transmitter (UART) communication is implemented on a Basys 3 FPGA board that performs 8 times faster compared to the software approach. The experimental result provides the hardware resource utilization of about 29% of look-up tables, 6% of lookup table RAMs (LUTRAM), 15% of Flip-flops, 32% of Block-RAM, 38% of DSP blocks operating at 100 MHz, and 230400 baud rates for the UART. The whole FPGA design executes at 2.1 milliseconds, the Kalman filter executes at 240 microseconds, and the UART at 1.86 milliseconds.

Detection and tracking of on-road moving objects are crucial for vehicle perception. Data Fusion of perception sensors like cameras, lidar, and radar, allows the automotive perception to achieve a certain level of robustness and performance. The efficiency of vehicle perception determines its accuracy to take better decisions related to road safety. In this article, we propose two approaches for the detection and tracking of on-road moving objects by combining information from lidar and radar sensors. In this work, we propose to use a probabilistic occupancy grid for sensor fusion of radar and lidar point cloud, and a modified YOLO DeepSORT approach to detect, classify, and track the moving objects. Using both traditional and deep learning approaches, data fusion allows us to improve the robustness of the automotive perception pipeline. We finally used the nuScenes dataset to evaluate and compare the accuracy of our approaches with state-of-the-art trackers and detectors. We achieved an AMOTA score of 68.3% and a mAP score of 68.7% for the tracking and detection tasks respectively.

Energy Optimization is key challenge in Wireless Sensor Network. This tutorial addresses the challenges in energy optimization and discusses the methodologies available to increase the network lifetime. Wireless Sensor Network is widely used for object detection and tracking application. Usage of single sensor may not provide accurate results for detection and tracking. Multi Sensor Data Fusion is a technique that combines data from multiple sensors (homogeneous or heterogeneous) to derive accurate inferences. This tutorial presents the usage multisensor data and image fusion schemes along with Complex Event Processing (CEP) for object detection and tracking.

This paper addresses the impact of data fusion on engineering and manufacturing information systems. Data fusion is the integration and analysis of data from multiple sensors to develop a more accurate understanding of a situation and determine how to respond to it. (Sensor fusion, a related term, has a somewhat different, more implementation oriented focus. The difference between these two concepts is addressed specifically in Section 1.) Although data fusion can be applied in many situations, this paper focuses on its application to manufacturing and how it changes some of the more traditional, less adaptive information models that support the design and manufacturing functions. Data fusion requires changes, primarily extensions, to these traditional information models. Engineering models normally address geometry, features, and performance characteristics of a part, while manufacturing models address machine tool characteristics and how they interact with the workpiece and operations scheduling. For data fusion and adaptive control, these models must include the dynamic behavior and interactions between the workpiece and the machine tool. In addition, these models must specify how these dynamic behaviors can be seen and interpreted by various types of sensors (e.g. temperature, pressure, and vibration). On a broader level it must also consider how the machine tools and robots interact within a work cell or production line. The paper consists of four parts. The first section explains what data fusion is and its impact on manufacturing. The second section describes what an information system architecture is and explains the natural language-based information modeling methodology used by this research project. The third section identifies the major design and manufacturing functions, reviews the information models required to support them, and then shows how these models must be extended to support data fusion. The fourth section discusses the future directions of this work.

This text is the latest version of Data and Sensor Fusion Concepts and Applications (SPIE Vol. TT35). In this edition, Dr. Klein has substantially expanded and updated the information to incorporate recent approaches to data and sensor fusion. For example, classical inference is a new topic, and the chapter on Dempster-Shafer theory is more comprehensive. The book also provides a summary of the information required to implement each of the data fusion methods.

Sensor Data Fusion (SDT) algorithms and models have been widely used in diverse applications. One of the main challenges of SDT includes how to deal with heterogeneous and complex datasets with different formats. The present work utilised both homogenous and heterogeneous datasets to propose a novel SDT framework. It compares data mining-based fusion software packages such as RapidMiner Studio, Anaconda, Weka, and Orange, and proposes a data fusion framework suitable for in-home applications. A total of 574 privacy-friendly (binary) images and 1722 datasets gleaned from thermal and Radar sensing solutions, respectively, were fused using the software packages on instances of homogeneous and heterogeneous data aggregation. Experimental results indicated that the proposed fusion framework achieved an average Classification Accuracy of 84.7% and 95.7% on homogeneous and heterogeneous datasets, respectively, with the help of data mining and machine learning models such as Naïve Bayes, Decision Tree, Neural Network, Random Forest, Stochastic Gradient Descent, Support Vector Machine, and CN2 Induction. Further evaluation of the Sensor Data Fusion framework based on cross-validation of features indicated average values of 94.4% for Classification Accuracy, 95.7% for Precision, and 96.4% for Recall. The novelty of the proposed framework includes cost and timesaving advantages for data labelling and preparation, and feature extraction.