The ArosDyn project: Robust analysis of dynamic scenes

Thermal/power issues have become increasingly important with more and more transistors being placed on a single chip. Many dynamic thermal/power management techniques have been proposed to address such issues but they all depend heavily on accurate knowledge of the chip's thermal state during runtime. In this paper, we describe a unified statistical framework for designing an on-chip thermal sensing infrastructure that can be used to track the chip's thermal state at runtime. Specifically, we address the following problems in this statistical framework: 1) sensor placement; 2) sensor data compression; 3) sensor data fusion; and 4) overall interplay. Our methods exploit the correlations between temperatures in different parts of the chip to drive sensor placement, data compression, and data fusion in both noiseless and noisy sensor cases. Our framework is also capable of choosing the appropriate degree of compression for each sensor while accounting for their local space constraints during deployment. The experimental results show that the root-mean-square error of the thermal estimates produced by our sensing infrastructure is on average 35% better than an equivalent system that uses a range-based placement scheme and a uniform compression scheme. It took our methods at most about 9 s to decide the overall solution for placement, compression, and data fusion at the design stage. This demonstrates the effectiveness and applicability of our unified statistical design methodology.

In the recent era of the Internet of Things, the dominant role of sensors and the Internet provides a solution to a wide variety of real-life problems. Such applications include smart city, smart healthcare systems, smart building, smart transport and smart environment. However, the real-time IoT sensor data include several challenges, such as a deluge of unclean sensor data and a high resource-consumption cost. As such, this paper addresses how to process IoT sensor data, fusion with other data sources, and analyses to produce knowledgeable insight into hidden data patterns for rapid decision-making. This paper addresses the data processing techniques such as data denoising, data outlier detection, missing data imputation and data aggregation. Further, it elaborates on the necessity of data fusion and various data fusion methods such as direct fusion, associated feature extraction, and identity declaration data fusion. This paper also aims to address data analysis integration with emerging technologies, such as cloud computing, fog computing and edge computing, towards various challenges in IoT sensor network and sensor data analysis. In summary, this paper is the first of its kind to present a complete overview of IoT sensor data processing, fusion and analysis techniques.

Vehicular ad-hoc network (VANET) is constructed of networks using direct radio communication among moving vehicles without fixed infrastructure. Network topologies change frequently due to high mobility and communication disconnections particularly occur due to interruptions of message propagation caused by obstacles such as buildings in urban traffic environment. Therefore, a VANET routing protocol approach to distributing messages among highly mobile vehicles in urban environment is needed. We suggest a scheme to improve the transmission performance at intersections using an approach to the distance-based broadcasting which utilizes the vehicle's position. It determines the stable route decision using the relative velocity among vehicles and zone-based priorities in the intersection. We analyze the performance of schemes under simulating environment implemented with the Manhattan model. The results show that the performance is significantly improved compared with the previous study when using the suggested scheme in the urban traffic environment.

The problem of sensor integration and data fusion is addressed. We consider the problem of combining information from diversified sources in a coherent fashion. We assume that at the fusion, the information from various sensors may be available in different forms. For example, data from infrared (IR) sensors may be combined with range radar (RR) data, and further combined with visual images. In each case, the data and information from the different sensors are presented in a different format which may not be directly compatible for all sensors. Furthermore, the available information may be in the form of attributes and not dynamical measurements. A theory for sensor integration and data fusion that accommodates diversified sources of information is presented. Data (or, more generically, information) fusion may proceed at different levels, like the level of dynamics, the level of attributes, and the level of evidence. All different levels are considered and several practical examples of real world data fusion problems are discussed.

In response to the issue of low positioning accuracy and insufficient robustness in small UAVs (unmanned aerial vehicle) caused by sensor noise and cumulative motion errors during flight in complex environments, this paper proposes a multisource, multimodal data fusion method. Initially, it employs a multimodal data fusion of various sensors, including GPS (global positioning system), an IMU (inertial measurement unit), and visual sensors, to complement the strengths and weaknesses of each hardware component, thereby mitigating motion errors to enhance accuracy. To mitigate the impact of sudden changes in sensor data, a high-fidelity UAV model is established in the digital twin based on the real UAV parameters, providing a robust reference for data fusion. By utilizing the extended Kalman filter algorithm, it fuses data from both the real UAV and its digital twin, and the filtered positional information is fed back into the control system of the real UAV. This enables the real-time correction of UAV positional deviations caused by sensor noise and environmental disturbances. The multisource, multimodal fusion Kalman filter method proposed in this paper significantly improves the positioning accuracy of UAVs in complex scenarios and the overall stability of the system. This method holds significant value in maintaining high-precision positioning in variable environments and has important practical implications for enhancing UAV navigation and application efficiency.

Along with the fast developments in VANET, a large number of attractive applications have emerged, which make routing of data in VANETs a great challenging task, especially in the complex urban traffic environment. The urban environment place restrictions on the packets that they should better be relayed along the road. Moreover, the increasing number of applications intensifies the traffic load condition of the whole network. However, most of the current routing protocols do not take into account the geographic information and traffic load together. In this article, we present a novel routing strategy based on using Geographic and Traffic Load information for VANET in urban traffic environment, GTLR (Geographic and Traffic Load Based Routing). In addition, we compare this new strategy with GPSR and GSR by means of delivering simulation in realistic environment, and the simulation results prove that GTLR outperforms them in terms of packet delivery ratio, and average packet end-to-end delay.

The recognition of dairy cow behavior is essential for enhancing health management, reproductive efficiency, production performance, and animal welfare. This paper addresses the challenge of modality loss in multimodal dairy cow behavior recognition algorithms, which can be caused by sensor or video signal disturbances arising from interference, harsh environmental conditions, extreme weather, network fluctuations, and other complexities inherent in farm environments. This study introduces a modality mapping completion network that maps incomplete sensor and video data to improve multimodal dairy cow behavior recognition under conditions of modality loss. By mapping incomplete sensor or video data, the method applies a multimodal behavior recognition algorithm to identify five specific behaviors: drinking, feeding, lying, standing, and walking. The results indicate that, under various comprehensive missing coefficients (λ), the method achieves an average accuracy of 97.87 % ± 0.15 %, an average precision of 95.19 % ± 0.4 %, and an average F1 score of 94.685 % ± 0.375 %, with an overall accuracy of 94.67 % ± 0.37 %. This approach enhances the robustness and applicability of cow behavior recognition based on multimodal data in situations of modality loss, resolving practical issues in the development of digital twins for cow behavior and providing comprehensive support for the intelligent and precise management of farms. • Modality Mapping Completion Network addresses multimodal data loss challenges. • Achieves 97.87 % accuracy and high robustness under varying data loss conditions. • Uses Transformers for effective cross-modal fusion of image and sensor data. • Adapts to real-world challenges like interference, occlusion, and harsh environments.

To enable successful automated driving, precise behavior prediction of surrounding vehicles is indispensable in urban traffic scenarios. Furthermore, given that a vehicle’s behavior is influenced by the movements of other road users, it becomes crucial to estimate their intentions to anticipate precise future motion. However, the elevated complexity resulting from interdependencies among traffic participants and the uncertainty arising from the object recognition errors present additional challenges. Despite extensive research on inferring intentions, many studies have concentrated on estimating intentions from interactions, resulting in a lack of practicality in urban traffic environments due to low computational efficiency and low robustness against recognition failure of strongly interacting road users. In this paper, we introduce a practical stochastic model for intention estimation and trajectory prediction of surrounding vehicles in automated driving under urban traffic environments. The trajectory is forecasted based on hierarchically computed and probabilistically estimated intentions, which represent an interpretation of vehicle behavior, utilizing only the kinematic state of the focal vehicle and HD maps to ensure real-time performance and enhance robustness. The evaluated results demonstrate that the proposed model surpasses straightforward methods in terms of accuracy while maintaining computational efficiency and exhibits robustness against the recognition failure of traffic participants which strongly influence the focal vehicle.

To take the advantages of both stereo cameras and radar, this paper proposes a fusion approach to accurately estimate the location, size, pose and motion information of a threat vehicle with respect to the host from observations obtained by both sensors. To do that, we first fit the contour of a threat vehicle from stereo depth information, and find the closest point on the contour from the vision sensor. Then the fused closest point is obtained by fusing radar observations and the vision closest point. Next by translating the fitted contour to the fused closest point, the fused contour is obtained. Finally the fused contour is tracked by using the rigid body constraints to estimate the location, size, pose and motion of the threat vehicle. Experimental results from both the synthetic data and the real world road test data demonstrate the success of the proposed algorithm.

The development of unmanned logistics vehicles is the general trend, but the right of road for this type of vehicle has not been clarified in China so far, which is not conducive to the healthy development of the unmanned delivery vehicle industry. This paper compares the key parameters of unmanned delivery vehicles in the urban traffic environment and the N1 type vehicles specified by the existing national standards on outer size, performance, and application scenario in order to provide a reference for the supervision of related industries.

We propose a range estimation method for vision-based forward collision warning systems with a monocular camera. To solve the problem of variation of camera pitch angle due to vehicle motion and road inclination, the proposed method estimates virtual horizon from size and position of vehicles in captured image at run-time. The proposed method provides robust results even when road inclination varies continuously on hilly roads or lane markings are not seen on crowded roads. For experiments, a vision-based forward collision warning system has been implemented and the proposed method is evaluated with video clips recorded in highway and urban traffic environments. Virtual horizons estimated by the proposed method are compared with horizons manually identified, and estimated ranges are compared with measured ranges. Experimental results confirm that the proposed method provides robust results both in highway and in urban traffic environments.

Research and/or Engineering Questions/Objective The vehicle of the future will support its driver by advising him regarding potential hazards. Essential prerequisite therefore is the sensor based perception of the traffic situation. For the recognition of traffic related objects, camera based sensors, deepness cameras, vehicle sensors as well as radar and lidar sensors are used. For the future development of ADAS the fusion of multiple sensor data to a consistent environmental picture will play a key role. The evaluation approach of real world driving tests will no longer be sufficient due to the complexity of the system interactions. New simulation methods are needed to evaluate ADAS by using virtual test driving with realistic vehicle behavior and complex traffic environment. Methodology Therefore it is important to integrate camera based components in a “closed loop”-simulation platform to be able to test sensor data fusion technologies under realistic conditions. To test new driver assistance systems in a simulation environment today animation data is filmed, subsequently this data is used to test an image processing algorithm or a fusion algorithm. But this method cannot be applied if wide-angle cameras such as cameras with fisheye lenses will be used. Within a research frame work for autonomous driving functions a new simulation technology was developed to integrate virtual cameras beside the well know environment sensor in the vehicle dynamic simulation CarMaker. For this purpose the real-time animation was extended with a sophisticated virtual camera model so called “VideoDataStream” to generate simultaneous video data (also PMD for 3D images). The camera positions as well as the camera properties could be applied individually. Additionally it is possible to freely define the type of the camera lens (e.g. fisheye) with lens settings like opening angle and the typical lens failures (e.g. distortion and vignetting). With this new technology it is possible that e.g. camera and radar data can be provided time and place synchronal for the fusion algorithm which should be tested! Results The video data could be used for evaluating image processing and sensor data fusion in Model-/Software-/Hardware-in-the-Loop applications within virtual test driving conditions. Here the created method and examples of image based perception of the vehicle environment as well as sensor data fusion algorithms shall be presented. Among others this covers first of all the recognition of traffic lanes, traffic signs and other traffic partners as well as the fusion of the single information up to a comprehensive environment picture. A further field of application will be the conjunction with navigation systems and digital maps, by which the virtual vehicle supports the navigation system with related GPS position and gets back the “MPP—Most Probable Path” with the “electronic horizon”, which is a type of predictive sensor, with all related preview information in front of the vehicle which are defined in the ADASIS protocol. Conclusion By using the introduced method the capability and efficiency of function development and testing in the area of Advanced Driver Assistant Systems will significantly be improved. Due to a powerful simulation environment a broad range of validation tests can be shifted into simulation because also complex test scenarios can be replicated and the tests are reproducible. The simulation data can be provided time and place synchronal, which is absolutely important, e.g. for a fusion algorithm which should be tested.

Multi-sensor data fusion has found widespread applications in industrial and research sectors. The purpose of real time multi-sensor data fusion is to dynamically estimate an improved system model from a set of different data sources, i.e., sensors. This paper presented a systematic and unified real time sensor validation and fusion framework (RTSVFF) based on distributed autonomous sensors. The RTSVFF is an open architecture which consists of four layers - the transaction layer, the process fusion layer, the control layer, and the planning layer. This paradigm facilitates distribution of intelligence to the sensor level and sharing of information among sensors, controllers, and other devices in the system. The openness of the architecture also provides a platform to test different sensor validation and fusion algorithms and thus facilitates the selection of near optimal algorithms for specific sensor fusion application. In the version of the model presented in this paper, confidence weighted averaging is employed to address the dynamic system state issue noted above. The state is computed using an adaptive estimator and dynamic validation curve for numeric data fusion and a robust diagnostic map for decision level qualitative fusion. The framework is then applied to automatic monitoring of a gas-turbine engine, including a performance comparison of the proposed real-time sensor fusion algorithms and a traditional numerical weighted average.

We present an approach to a general decision support system. The aim is to cover the complete process for automatic target recognition, from sensor data to the user interface. The approach is based on a query-based information system, and include tasks like feature extraction from sensor data, data association, data fusion and situation analysis. Currently, we are working with data from laser radar, infrared cameras, and visual cameras, studying target recognition from cooperating sensors on one or several platforms. The sensors are typically airborne and at low altitude. The processing of sensor data is performed in two steps. First, several attributes are estimated from the (unknown but detected) target. The attributes include orientation, size, speed, temperature etc. These estimates are used to select the models of interest in the matching step, where the target is matched with a number of target models, returning a likelihood value for each model. Several methods and sensor data types are used in both steps. The user communicates with the system via a visual user interface, where, for instance, the user can mark an area on a map and ask for hostile vehicles in the chosen area. The user input is converted to a query in ΣQL, a query language developed for this type of applications, and an ontological system decides which algorithms should be invoked and which sensor data should be used. The output from the sensors is fused by a fusion module and answers are given back to the user. The user does not need to have any detailed technical knowledge about the sensors (or which sensors that are available), and new sensors and algorithms can easily be plugged into the system.

The combined action of several sensing systems, so that they are able to compensate the technical flaws of each other, is common in robotics. Monte Carlo Localization (MCL) is a popular technique used to estimate the pose of a mobile robot, which allows the fusion of heterogeneous sensor data. Several sensor fusion schemes have been proposed which include sensors like GPS to improve the performance of this algorithm. In this paper, an Adaptive MCL algorithm is used to combine data from wheel odometry, an inertial measurement unit, a global positioning system and laser scanning. A particle weighting model which integrates GPS measurements is proposed, and its performance is compared with a particle generation approach. Experiments were conducted on a real robotic car within an urban environment.