Fast and Accurate Real Time Pedestrian Detection Using Convolutional Neural Network

We present a unified multi-task learning architecture for fast and accurate pedestrian detection. Different from existing methods which often focus on either a new loss function or architecture, we propose an improved multi-task convolutional neural network learning architecture to effectively and efficiently interfuse the task of pedestrian detection and semantic segmentation. To achieve this, we integrate a lightweight semantic segmentation branch to Faster R-CNN detection framework that enables end-to-end hard parameter sharing in order to boost the detection performance and maintain computational efficiency as follows. Firstly, a Semantic Segmentation to Feature Module (SS2FM) refines the convolutional features in RPN stage by integrating the features generated from the semantic segmentation branch. Secondly, a Semantic Segmentation to Confidence Module (SS2CM) refines the classification confidence in RPN stage by fusing it with the semantic segmentation confidence. We also introduce an effective anchor matching point transform to alleviate the problem of feature misalignment for heavily occluded pedestrians. The proposed unified multi-task learning architecture lends itself well to more robust pedestrian detection in diverse scenarios with negligible computation overhead. In addition, the proposed architecture can achieve high detection performance with low resolution input images, which significantly reduces the computational complexity. Experiment results on CityPersons and Caltech datasets show that our method is the fastest among all state-of-the-art pedestrian detection methods while exhibiting competitive detection performance.

In recent years, the fall detection system has become an important topic in the homecare system. Compared with the traditional fall detection algorithm, the method used by neural network is more robust and has higher accuracy. However neural network consumes a large amount of energy due to a huge number of computations, and needs more memory to store parameters as compared to traditional algorithms. In this paper, we propose a fall detection system in combination of the traditional algorithm with the neural network. First, we propose a skeleton information extraction algorithm, which transforms depth information into skeleton information and extracts the important joints related to fall activity. Also we have modified the skeleton-based method with seven highlight feature points. Second, we propose a highly robust deep convolution neural network architecture, which uses a pruning method to reduce parameters and calculations in the network. The low number of parameters and calculations makes the system suitable for the implementation on an embedded system. The experiment results show the high accuracy and robustness on the popular benchmark dataset NTU RGB+D. The proposed system has been implemented on NVIDIA Jetson Tx2 platform with real-time processing.

In this paper, we propose the dense disparity map-based pedestrian detection method for intelligent vehicle. The dense disparity map is utilized to improve the pedestrian detection performance. Our method consists of several steps namely, obstacle area detection using road feature information and column detection, pedestrian area detection using dense disparity map-based segmentation, and pedestrian detection using optimal feature. The first step is to detect all obstacle areas using column detection and pedestrian height information. However, there are many objects in single obstacle area. Thus each obstacle area needs to be separated into single object for improving pedestrian detection performance. Thus, the second step is performed to segment the detected obstacle area. The last step is to detect only pedestrian using classifier trained by optimal feature. The optimal feature is extracted by positive and negative training images. ETH database is utilized to evaluate our proposed pedestrian detection method.

本文在传统的依靠RGB通道模板和卷积神经网络的行人检测框架下，提出了结合多通道模板和卷积神经网络的行人检测方法。主要工作是：一是采用运动侦测方法初步检测感兴趣区域，提高行人检测效率和降低虚警。二是采用多通道模板和最大值融合策略，降低单纯依靠RGB通道模板可能存在的行人漏检现象。 In this paper, based on the traditional pedestrian detection framework of RGB channel template and convolution neural network, a pedestrian detection method combining multi-channel tem-plate and convolution neural network is proposed. The main contribution is: firstly, the motion detection method is used to detect the interest area, improve pedestrian detection efficiency and reduce the false alarm. Secondly, multi-channel template and maximum fusion strategy are adopted to reduce the possible pedestrian leakage detection based on RGB channel template.

Gait recognition is a noncontact biometric procedure that determines the identity or health status of a person by analyzing his or her walking posture and habits, including skeletal and joint movements. The most remarkable feature of this method is the possibility of conducting recognition without demanding much cooperation from participants. Therefore, this recognition technique has attracted much attention from scholars. Additionally, because of the rapid development of graphics processing unit technology, related hardware and computation performance, the applications of deep-learning technology are considerably enhanced. The objective of this study was to apply a deep neural network (DNN), which employs deep-learning technology, to achieve gait-based automatic pedestrian detection and recognition. In contrast to using wearable devices to precisely capture skeletal and joint movements, pedestrian color-image sequences were used as input in this study. Subsequently, a pretraining convolutional neural network (CNN) was employed to capture pedestrian location and extract pedestrian dense optical flow to serve as concrete low-level feature inputs. Then, a finely-tuned DNN based on the wide residual network was employed to extract high-level abstract features. In addition, to overcome the difficulty of obtaining local temporal features by using a 2D CNN, part of the 3D convolutional structure was introduced into the CNN. This design enabled use of limited memory to acquire more effective features and enhance the DNN performance. The experimental results show that the proposed method has exceptional performance for pedestrian detection and recognition.

In order to solve the problem that the traditional pedestrian recognition and ranging are not satisfactory, a monocular visual nighttime pedestrian recognition and fast ranging method suitable for vehicle infrared night vision images is proposed. Based on the non-contact thermal imaging principle of far-infrared night vision goggles and mathematical morphology method, an improved pedestrian detection and recognition method for three-frame differential monocular vision and geometric motion rate is established. On the basis of pedestrian recognition, monocular camera fast ranging algorithm is used for fast ranging pedestrians. The experimental results show that this method has the characteristics of high pedestrian recognition accuracy and high real-time ranging accuracy.

Recently, neural networks have shown good results for pattern detection. In our previous papers by El-Bakry et al., a fast algorithm for pattern detection using neural networks was presented. Such algorithm was designed based on cross correlation in the frequency domain between the input image and the weights of neural networks. In practical implementation using MATLAB, image conversion into symmetric shape was established so that fast neural networks can give the same results as conventional neural networks. Another configuration of symmetry was suggested to improve the speed up ratio. In this paper, our previous algorithm for fast neural networks is developed. The frequency domain cross correlation is modified in order to compensate for the symmetric condition which is required by the input image. Two new ideas are introduced to modify the cross correlation algorithm. Both methods accelerate the speed of the fast neural networks as there is no need for converting the input image into symmetric one as previous. Theoretical and practical results show that both approaches provide faster speed up ratio than the previous algorithm.

Vision-based pedestrian detection technology is widely used in security surveillance, such as transportation, prisons, and important places. Security monitoring requires pedestrian detection not only to be accurate and reliable, but also to meet the needs of real-time monitoring, and the false alarm rate is low. The existing pedestrian learning method based on deep learning has high reliability, but the timeliness cannot meet the monitoring requirements. Traditional pedestrian detection methods are time-sensitive, but false alarm rates tend to be higher. This paper proposes a self-learning pedestrian detection method, combined with motion detection, HOG feature extraction, and SVM classification for preliminary pedestrian detection. The false alarms caused during the pedestrian detection process are manually confirmed, and then the system automatically detects the false alarm image at a specific time. Self-learning, retraining the classifier, so as to continuously reduce the pedestrian detection false alarm rate and improve the pedestrian detection performance index through self-learning during the system operation, and the timeliness can meet the needs of video surveillance. Experiments show that the false alarm rate of this method is low, timeliness is high, and the recognition rate is also high.

While remarkable progress has been made to pedestrian detection in recent years, robust pedestrian detection in the wild e.g., under surveillance scenarios with occlusions, remains a challenging problem. In this paper, we present a novel approach for joint pedestrian and body part detection via semantic relationship learning under unconstrained scenarios. Specifically, we propose a Body Part Indexed Feature (BPIF) representation to encode the semantic relationship between individual body parts (i.e., head, head-shoulder, upper body, and whole body) and highlight per body part features, providing robustness against partial occlusions to the whole body. We also propose an Adaptive Joint Non-Maximum Suppression (AJ-NMS) to replace the original NMS algorithm widely used in object detection, leading to higher precision and recall for detecting overlapped pedestrians. Experimental results on the public-domain CUHK-SYSU Person Search Dataset show that the proposed approach outperforms the state-of-the-art methods for joint pedestrian and body part detection in the wild.

This paper presents a real-time pedestrian situation detection framework for autonomous mobility platforms. The proposed approach extracts pedestrians from images acquired by a camera mounted on an autonomous mobility system, classifies their postures, tracks their trajectories, and subsequently detects pedestrian situations. A convolutional neural network (CNN) is employed for pedestrian detection and posture classification, where the YOLOv12 model is fine-tuned via transfer learning for this purpose. To improve detection and classification performance, a region of interest (ROI) is defined using camera calibration data, enabling robust detection of small-scale pedestrians over long distances. Using a custom-labeled dataset, the proposed method achieves a precision of 96.6% and a recall of 97.0% for pedestrian detection and posture classification. The detected pedestrians are tracked using the DeepSORT algorithm, and their situations are inferred through a rule-based analysis module. Experimental results demonstrate that the proposed system operates at an execution speed of 58.11 ms per frame, corresponding to 17.2 fps, thereby satisfying the real-time requirements for autonomous mobility applications. These results confirm that the proposed framework enables reliable real-time pedestrian extraction and situation awareness in real-world autonomous mobility environments.

Autonomous vehicles are highly complex systems, required to function reliably\nin a wide variety of situations. Manually crafting software controllers for\nthese vehicles is difficult, but there has been some success in using deep\nneural networks generated using machine-learning. However, deep neural networks\nare opaque to human engineers, rendering their correctness very difficult to\nprove manually; and existing automated techniques, which were not designed to\noperate on neural networks, fail to scale to large systems. This paper focuses\non proving the adversarial robustness of deep neural networks, i.e. proving\nthat small perturbations to a correctly-classified input to the network cannot\ncause it to be misclassified. We describe some of our recent and ongoing work\non verifying the adversarial robustness of networks, and discuss some of the\nopen questions we have encountered and how they might be addressed.\n

Convolutional neural networks (CNNs) have shown vulnerable to adversarial examples which contain imperceptible perturbations. In this paper, we propose an approach to defend against adversarial examples with soft decision trees embedding. Firstly, we extract the semantic features of adversarial examples with a feature extraction network. Then, a specific soft decision tree is trained and embedded to select the key semantic features for each feature map from convolutional layers and the selected features are fed to a light-weight classification network. To this end, we use the probability distributions of each tree node to quantify the semantic features. In this way, some small perturbations can be effectively removed and the selected features are more discriminative in identifying adversarial examples. Moreover, the influence of adversarial perturbations on classification can be reduced by migrating the interpretability of soft decision trees into the black-box neural networks. We conduct experiments to defend the state-of-the-art adversarial attacks. The experimental results demonstrate that our proposed approach can effectively defend against these attacks and improve the robustness of deep neural networks.

Pedestrian detection is considered one of the most challenging problems in computer vision, as it involves the combination of classification and localization within a scene. Recently, convolutional neural networks (CNNs) have been demonstrated to achieve superior detection results compared to traditional approaches. Although YOLOv3 (an improved You Only Look Once model) is proposed as one of state-of-the-art methods in CNN-based object detection, it remains very challenging to leverage this method for real-time pedestrian detection. In this paper, we propose a new framework called SA YOLOv3, a scale-aware You Only Look Once framework which improves YOLOv3 in improving pedestrian detection of small scale pedestrian instances in a real-time manner. Our network introduces two sub-networks which detect pedestrians of different scales. Outputs from the sub-networks are then combined to generate robust detection results. Experimental results show that the proposed SA YOLOv3 framework outperforms the results of YOLOv3 on public datasets and run at an average of 11 fps on a GPU.

A number of studies have been conducted to enhance the pedestrian detection accuracy of intelligent surveillance systems. However, detecting pedestrians under outdoor conditions is a challenging problem due to the varying lighting, shadows, and occlusions. In recent times, a growing number of studies have been performed on visible light camera-based pedestrian detection systems using a convolutional neural network (CNN) in order to make the pedestrian detection process more resilient to such conditions. However, visible light cameras still cannot detect pedestrians during nighttime, and are easily affected by shadows and lighting. There are many studies on CNN-based pedestrian detection through the use of far-infrared (FIR) light cameras (i.e., thermal cameras) to address such difficulties. However, when the solar radiation increases and the background temperature reaches the same level as the body temperature, it remains difficult for the FIR light camera to detect pedestrians due to the insignificant difference between the pedestrian and non-pedestrian features within the images. Researchers have been trying to solve this issue by inputting both the visible light and the FIR camera images into the CNN as the input. This, however, takes a longer time to process, and makes the system structure more complex as the CNN needs to process both camera images. This research adaptively selects a more appropriate candidate between two pedestrian images from visible light and FIR cameras based on a fuzzy inference system (FIS), and the selected candidate is verified with a CNN. Three types of databases were tested, taking into account various environmental factors using visible light and FIR cameras. The results showed that the proposed method performs better than the previously reported methods.

Recurrent neural networks (RNNs) are powerful and robust neural networks that capture time dynamics via cycles in the graph. Compared to Convolutional Neural Networks (CNNs), RNNs consider both the current and the historical inputs which leads them to the standard approach for practitioners to address machine learning tasks involving sequential data. Compared to the achievements in these sequential learning tasks, RNNs pale in presenting a promising performance on human action recognition. Do we miss something? In this paper, we will conduct a detailed investigation of the RNNs for the human action recognition task. Several alternative recurrent architectures are designed and compared. Even simply, we find that adding the supervision signal at the end of an RNN can provide us with a comparable if not better performance than the previous state-of-the-art. We further analyze the different combinations of modalities and the effect of the length of the sequence. Compared to CNN-based recognition architecture, RNNs focus more on the action itself. Going deeper, we also find that when modeling long and difficult actions, RNNs perform much better than CNNs. It means a complex video action dataset may be required.