A Traffic Information Awareness Approach Based on Video Data and Millimeter Wave Radar Data Fusion

For many automated driving functions, a highly accurate perception of the vehicle environment is a crucial prerequisite. Modern high-resolution radar sensors generate multiple radar targets per object, which makes these sensors particularly suitable for the 2D object detection task. This work presents an approach to detect 2D objects solely depending on sparse radar data using PointNets. In literature, only methods are presented so far which perform either object classification or bounding box estimation for objects. In contrast, this method facilitates a classification together with a bounding box estimation of objects using a single radar sensor. To this end, PointNets are adjusted for radar data performing 2D object classification with segmentation, and 2D bounding box regression in order to estimate an amodal 2D bounding box. The algorithm is evaluated using an automatically created dataset which consist of various realistic driving maneuvers. The results show the great potential of object detection in high-resolution radar data using PointNets.

Using ultra-wideband (UWB) impulse radar for detecting and tracking fast-moving small targets over the ocean surface has been considered before with limited applications. The challenges of deploying such radar sensors on small, unmanned marine platforms are addressed in this study. The first challenge is the stringent size and weight requirement to allow a tracking radar sensor to be fitted into the payload of a small unmanned surface vehicle (USV). For the first time, we implemented a design that is based on a single chip UWB radar sensor operating at X-band, which effectively achieves the size and weight requirement for a small USV payload. The second challenge is range extension and range ambiguity resolution. With the UWB radar operating various high-PRF modes, we developed a novel approach that stitches together range profiles from multiple PRFs, to extend the effective non-ambiguous range at the cost of scan speed. The third challenge is developing a lowcost, ultra-wideband planar antenna and front-end, which is also part of the USV payload and needs to be able to perform either sector scanning, or even electronic scanning, with a very low profile. We have successfully designed and implemented one such antenna using a dipole array design. By integrating the solutions into a complete system, we have performed a series of lab and outdoor tests of the UWB radar sensor and obtained some promising target data. Simulations are also being developed for testing the potential target signatures and tracking effectiveness of moving targets over ocean surface clutter environments.

Target tracking using multiple sensors can provide better performance than using a single sensor. One approach to multiple target tracking with multiple sensors is to first perform single sensor tracking and then fuse the tracks from the different sensors. Two processing architectures for track fusion are presented: sensor to sensor track fusion, and sensor to system track fusion. Technical issues related to the statistical correlation between track estimation errors are discussed. Approaches for associating the tracks and combining the track state estimates of associated tracks that account for this correlation are described and compared by both theoretical analysis and Monte Carlo simulations.

Radar and communication system (RadComm) spectrum sharing has received considerable attention from the research community in recent years. This paper considers the distributed radars present in the surveillance region with multiple in-band wireless communication transmitters (IWCTs). A new measurement model is proposed by considering both radar returns and returns due to IWCTs. The tracking performance is evaluated using the global nearest neighbor (GNN) tracker with an extended Kalman filter (EKF) for the received measurement set. A single radar case is considered, where near geometry scenario (IWCTs are placed near the radar and target) and far-geometry scenario (IWCTs are placed far from the radar and target) are considered to evaluate the tracking performance. It is observed that a large number of tracks are resulted due to IWCTs, and identifying the actual target track is ambiguous in a single radar case. Therefore, in the second case, multiple radars are placed to investigate the problem comprehensively. The track-to-track association (T2TA) is performed to identify the true target track on multiple tracks produced owing to the presence of IWCTs and the resulting tracks from all radars pertaining to the true targets. Once the true target tracks from each radar are identified, using the T2TA, the track-to-track fusion (T2TF) is carried out to improve the estimates of the true target. The simulation results are quantified with position root mean square error (PRMSE). The posterior Cramer–Rao lower bounds (PCRLBs) quantifying the achievable estimation accuracies are also presented. The simulation results reveal that the association and fusion of tracks from multiple radars identify the true target track with good accuracy and overcome the inability to determine the true track, as in the case of a single radar. Further, the results disclose that, as the number of radars increases, the T2TA and fusion improved the PRMSE.

The heartbeat detection using the scaling function of Wavelet transform is proposed for a Doppler radar sensor. The conventional methods such as the fast‐Fourier transform and the autocorrelation show the respiration rate and the heartbeat from the raw data of the radar sensors acquiring for a sufficient sampling time. The methods have the limit to detect the biometric information that varies with real‐time because they only show the overall statistical information of the sampled data. In the proposed method, the scaling function in the Daubechies wavelet transform can be used to accurately find out the periodicity of radar signals for detecting heartbeat varying in real‐time. The results of the signal processing using the radar signals acquired for 3 min results show that the proposed method lowered a mean error rate of 2.5% and a SD of 2.0% compared with the method using the wavelet function. The proposed method in the measurement for 1 minute using the radar sensor also showed the lowest mean error rate of 3.8% and the low SD of 3.2% using the contact sensor as the reference among various signal processing methods including auto‐correlation and peak detection with filtering.

In this paper, based on the fusion tracking of infrared and radar, a federated kalman filtering algorithm based on adaptive fault tolerance is designed. According to the observation equation of infrared and radar, a distributed fusion federated kalman filtering structure is established by using three infrared sensors and three radar sensors. The sub nodes filter their detection information respectively, and the estimation results are obtained. After that, the sub nodes filter results are sent into their part of fault-tolerant judgment respectively, then we can get the weight information of each sub node, thus it can determine the degree of dependence of each node, and the filtering results of all sub nodes are fused into calculation by their weight. The final simulation shows that the algorithm can isolate the interference in the multi-mode fusion tracking process effectively , and get a stable state estimation output.

In this paper, we classify digits written in mid-air using hand gestures. Impulse radio ultrawideband (IR-UWB) radar sensors are used for data acquisition, with three radar sensors placed in a triangular geometry. Conventional radar-based gesture recognition methods use whole raw data matrices or a group of features for gesture classification using convolutional neural networks (CNNs) or other machine learning algorithms. However, if the training and testing data differ in distance, orientation, hand shape, hand size, or even gesture speed or the radar setup environment, these methods become less accurate. To develop a more robust gesture recognition method, we propose not using raw data for the CNN classifier, but instead employing the hand's mid-air trajectory for classification. The hand trajectory has a stereotypical shape for a given digit, regardless of the hand's orientation or speed, making its classification easy and robust. Our proposed method consists of three stages: signal preprocessing, hand motion localization, and tracking and transforming the trajectory data into an image to classify it using a CNN. Our proposed method outperforms conventional approaches because it is robust to changes in orientation, distance, and hand shape and size. Moreover, this method does not require building a huge training database of digits drawn by different users in different orientations; rather, we can use training databases already available in the image processing field. Overall, the proposed mid-air handwritten digit recognition system provides a user-friendly and accurate mid-air handwriting modality that does not place restrictions on users.

Today, a lot of research on autonomous driving technology is being conducted, and various vehicles with autonomous driving functions, such as ACC (adaptive cruise control) are being released. The autonomous vehicle recognizes obstacles ahead by the fusion of data from various sensors, such as lidar and radar sensors, including camera sensors. As the number of vehicles equipped with such autonomous driving functions increases, securing safety and reliability is a big issue. Recently, Mobileye proposed the RSS (responsibility-sensitive safety) model, which is a white box mathematical model, to secure the safety of autonomous vehicles and clarify responsibility in the case of an accident. In this paper, a method of applying the RSS model to a variable focus function camera that can cover the recognition range of a lidar sensor and a radar sensor with a single camera sensor is considered. The variables of the RSS model suitable for the variable focus function camera were defined, the variable values were determined, and the safe distances for each velocity were derived by applying the determined variable values. In addition, as a result of considering the time required to obtain the data, and the time required to change the focal length of the camera, it was confirmed that the response time obtained using the derived safe distance was a valid result.

Radar sensors are an attractive technology for different kinds of applications, especially, in the case of safety driving systems. This research deals with designing the target return signal detection algorithm for radar sensors using the generalized detector (GD) with adaptive detection threshold based on a sliding window technique to estimate the noise power. The general GD structure is introduced, and the proposed method to define the adaptive detection threshold is assumed to be employed by 24 GHz frequency modulation continuous wave (FMCW) radar sensor and the simulation results show the better detection performance in comparison with the cell averaging constant false alarm rate (CA-CFAR) detector one which is widely used by radar sensor systems for car applications.

Introduction. An important part of an automotive unmanned vehicle (UV) control system is the environment analysis module. This module is based on various types of sensors, e.g. video cameras, lidars and radars. The development of computer and video technologies makes it possible to implement an environment analysis module using a single video camera as a sensor. This approach is expected to reduce the cost of the entire module. The main task in video image processing is to analyse the environment as a 3D scene. The 3D trajectory of an object, which takes into account its dimensions, angle of view and movement vector, as well as the vehicle pose in a video image, provides sufficient information for assessing the real interaction of objects. A basis for constructing a 3D trajectory is vehicle pose estimation.Aim. To develop an automatic method for estimating vehicle pose based on video data analysis from a single video camera.Materials and methods. An automatic method for vehicle pose estimation from a video image was proposed based on a cascade approach. The method includes vehicle detection, key points determination, segmentation and vehicle pose estimation. Vehicle detection and determination of its key points were resolved via a neural network. The segmentation of a vehicle video image and its mask preparation were implemented by transforming it into a polar coordinate system and searching for the outer contour using graph theory.Results. The estimation of vehicle pose was implemented by matching the Fourier image of vehicle mask signatures and the templates obtained based on 3D models. The correctness of the obtained vehicle pose and angle of view estimation was confirmed by experiments based on the proposed method. The vehicle pose estimation had an accuracy of 89 % on an open Carvana image dataset.Conclusion. A new approach for vehicle pose estimation was proposed, involving the transition from end-to-end learning of neural networks to resolve several problems at once, e.g., localization, classification, segmentation, and angle of view, towards cascade analysis of information. The accuracy level of end-to-end learning requires large sets of representative data, which complicates the scalability of solutions for road environments in Russia. The proposed method makes it possible to estimate the vehicle pose with a high accuracy level, at the same time as involving no large costs for manual data annotation and training.

Vision-based traffic sign detection plays a crucial role in intelligent transportation systems. Recently, many approaches based on deep learning for traffic sign detection have been proposed and showed better performance compared with traditional approaches. However, due to difficult conditions in driving environment and the size of traffic signs in traffic scene images, the performance of deep learning-based methods on small traffic sign detection is still limited. In addition, the inference speed of current state-of-the-art approaches on traffic sign detection is still slow. This paper proposes a deep learning-based approach to improve the performance of small traffic sign detection in driving environments. First, a lightweight and efficient architecture is adopted as the base network to address the issue of the inference speed. To enhance the performance on small traffic sign detection, a deconvolution module is adopted to generate an enhanced feature map by aggregating a lower-level feature map with a higher-level feature map. Then, two improved region proposal networks are used to generate proposals from the highest-level feature map and the enhanced feature map. The proposed improved region proposal network is designed for fast and accuracy proposal generation. In the experiments, the German Traffic Sign Detection Benchmark dataset is used to evaluate the effectiveness of each enhanced module, and the Tsinghua-Tencent 100K dataset is used to compare the effectiveness of the proposed approach with other state-of-the-art approaches on traffic sign detection. Experimental results on Tsinghua-Tencent 100K dataset show that the proposed approach achieves competitive performance compared with current state-of-the-art approaches on traffic sign detection while being faster and simpler.

As the junction between highways and other urban roads, the divergence and convergence sections cause vehicles to perform frequent and complex acceleration, deceleration and lane changes. This poses a challenge for traffic radar to accurately estimate the position of vehicles. The classical algorithm based depends too much on the prior model and cannot accurately track complex vehicle movements. This paper proposes an improved method to achieve accurate tracking of vehicle targets in the divergence and convergence sections. Firstly, based on the unscented Kalman filter framework, we derive the filtering algorithm framework after introducing model errors. Secondly, we design a gated recurrent neural network to learn the implicit target motion state information in the dataset through the memory of the recurrent neural network, and converted it into model error through the gating structure. Then the model error is compensated in the state estimation. In the simulation, the position and velocity errors of the proposed algorithm are reduced by 16.83% and 17.45% compared with the baseline filter.

Automotive radar systems are currently under development for various applications to increase comfort and safety. In this paper a network of distributed radar sensors integrated behind the front bumper of a car is considered and described. Each pulse radar sensor measures the target range with extremely high accuracy and resolution. The two-dimensional target positioning is realized by signal processing techniques applied to the range measurements. Target tracking is most important in multiple-target situations. Therefore, three different tracking algorithms are described, tested and compared in this paper.

Automated, effective and efficient sleep‐stage monitoring and structure analysis is an essential enabling procedure for healthcare automation. Sleep diagnosis by polysomnography is a golden standard but expensive procedure involving huge effort from patients. There remain challenges for smart devices to precisely identify sleep stage and minimize intrusive effect on sleep progression. Herein, a novel noncontact sleep structure prediction system (NSSPS) using a single radar sensor is presented to analyze sleep structure without any tethered unit. The NSSPS is realized through training a convolutional recurrent neural network and neural conditional random fields using reflected radio frequency (RF) waves acquired by radar antennas. By capturing implicit temporal information in RF signals and transitions of sleep progression, high accuracy of sleep‐stage prediction is achieved and characteristics of sleep structure are extracted. The performance of the NSSPS is validated by transfer learning between radar signals with different frequency bands and crossvalidation among different subjects. Moreover, the NSSPS is demonstrated to estimate overnight parameters that are critical for sleep diagnosis. Benefiting from its low cost, convenient setup, and accurate prediction capability of sleep‐stage identification, the NSSPS can be widely deployed in “smart” homes and exploited to conduct daily sleep structure analysis.

In this paper, the foliage penetration measurement data is provided by Air Force Office of Scientific Research (AFOSR). When radar echoes are in good quality, the detection of target can be achieved by applying our differential based technology on received single UWB radar waveform. We compared our approach in case of no target as well as with target against the scheme in which 2-D image was created via adding voltages with the appropriate time offset. Results show that our approach can work much better. When radar echoes are in poor condition and single radar is unable to carry out the detection, we employ both radar sensor networks (RSN) and RAKE structure to combine the echoes from different radar members and successfully detect the target.