Pedestrian detection based on multi-convolutional features by feature maps pruning

Gait recognition is a noncontact biometric procedure that determines the identity or health status of a person by analyzing his or her walking posture and habits, including skeletal and joint movements. The most remarkable feature of this method is the possibility of conducting recognition without demanding much cooperation from participants. Therefore, this recognition technique has attracted much attention from scholars. Additionally, because of the rapid development of graphics processing unit technology, related hardware and computation performance, the applications of deep-learning technology are considerably enhanced. The objective of this study was to apply a deep neural network (DNN), which employs deep-learning technology, to achieve gait-based automatic pedestrian detection and recognition. In contrast to using wearable devices to precisely capture skeletal and joint movements, pedestrian color-image sequences were used as input in this study. Subsequently, a pretraining convolutional neural network (CNN) was employed to capture pedestrian location and extract pedestrian dense optical flow to serve as concrete low-level feature inputs. Then, a finely-tuned DNN based on the wide residual network was employed to extract high-level abstract features. In addition, to overcome the difficulty of obtaining local temporal features by using a 2D CNN, part of the 3D convolutional structure was introduced into the CNN. This design enabled use of limited memory to acquire more effective features and enhance the DNN performance. The experimental results show that the proposed method has exceptional performance for pedestrian detection and recognition.

This paper proposes a novel multispectral data fusion method for pedestrian detection. For all-day vision, a fusion of CCD and Infrared (IR) sensors are inevitable, and fusion of heterogeneous data based on a Convolutional neural network (CNN) is only based on the feature map concatenation of parallel CNN architecture. However, concatenation that is simply applied has a problem in that it can not fully utilize multispectral data in a deep network architecture. Therefore, this paper proposes a method called Feature Map Swap (FMS) of swapping feature maps in addition to concatenation. The proposed method can use multispectral data more efficiently by facilitating learning, by swapping the different domain weights of feature maps besides to concatenation. Also, the proposed method does not require any modifications, such as adding or removing layers to an already configured network architecture. Experimental results show that the performance of the KAIST multispectral pedestrian dataset is improved by about 7-10% based on log-average miss rate compared to simple concatenation.

In recent electroencephalograph (EEG)-based emotion recognition, the differential entropy (DE) features extracted from multiple electrodes are organized as a 2D feature map for convolutional neural network (CNN) in order to utilize the information hidden in the electrodes. In this study, we attempt to investigate the influence of different feature maps on the recognition performance. Six different 2D feature maps (M1-M4: baseline feature maps without sparsity and location relationship, M5-M6: pre-defined feature maps with sparsity and location relationship) are used to organize the DE features for the traditional CNN model. Evaluation study on the DEAP dataset finds that the 2D feature map configuration exhibits statistically significant effect on the classification performance of the traditional CNN model in classifying the high/low arousal and high/low valence, respectively. However, the differences are rather limited, e.g., only 1% improvement can be resulted from selecting the optimal 2D feature map among 6 feature maps. This implies that the feature map may not be a critical issue when applying the DE features to classifying the emotion states in a CNN.

Deep learning has been successfully deployed to a broad range of applications with its outstanding performance. Supporting an efficient hardware architecture is critical to making effective use of a deep learning approach with proven algorithm performance. One challenge in implementation of deep learning algorithm is to reduce memory bandwidth because a single memory access normally consumes 100* more energy than an arithmetic operation. To reduce the memory bandwidth, deep learning data could be compressed and decompressed before memory write/read operations. Especially, feature maps, which account for a significant portion of the convolutional neural network (CNN), could be compressed further by reducing the correlations between feature map planes. This paper proposes a compression method for feature maps in CNN that adaptively exploits the varying correlation between feature map planes. For every feature map plane, the proposed method searches the most similar plane among nearby planes in the same layer, and compresses the residual of the two planes instead of the plane itself. Experimental results show that the average bit length to store feature maps is reduced by 14.2% compared to the compression without correlation reduction, and the CNN accuracy does not change and additional training is also not required because the proposed method applies lossless compression.

.Significance: Automatic and accurate classification of three-dimensional (3-D) retinal optical coherence tomography (OCT) images is essential for assisting ophthalmologist in the diagnosis and grading of macular diseases. Therefore, more effective OCT volume classification for automatic recognition of macular diseases is needed.Aim: For OCT volumes in which only OCT volume-level labels are known, OCT volume classifiers based on its global feature and deep learning are designed, validated, and compared with other methods.Approach: We present a general framework to classify OCT volume for automatic recognizing macular diseases. The architecture of the framework consists of three modules: B-scan feature extractor, two-dimensional (2-D) feature map generation, and volume-level classifier. Our architecture could address OCT volume classification using two 2-D image machine learning classification algorithms. Specifically, a convolutional neural network (CNN) model is trained and used as a B-scan feature extractor to construct a 2-D feature map of an OCT volume and volume-level classifiers such as support vector machine and CNN with/without attention mechanism for 2-D feature maps are described.Results: Our proposed methods are validated on the publicly available Duke dataset, which consists of 269 intermediate age-related macular degeneration (AMD) volumes and 115 normal volumes. Fivefold cross-validation was done, and average accuracy, sensitivity, and specificity of 98.17%, 99.26%, and 95.65%, respectively, are achieved. The experiments show that our methods outperform the state-of-the-art methods. Our methods are also validated on our private clinical OCT volume dataset, consisting of 448 AMD volumes and 462 diabetic macular edema volumes.Conclusions: We present a general framework of OCT volume classification based on its 2-D feature map and CNN with attention mechanism and describe its implementation schemes. Our proposed methods could classify OCT volumes automatically and effectively with high accuracy, and they are a potential practical tool for screening of ophthalmic diseases from OCT volume.

Due to the weak structural features of pipelines and underwater light attenuation, the complex and harsh environment of the seabed greatly increases the possibility of an underwater autonomous remotely operated vehicle losing a detected seabed-contacting segment during pipe-laying operations. To address this problem, we propose a cascade attention module and a prefusion module with a convolutional neural network. The cascade attention module samples the feature maps in a non-convolutional form to realize the interaction between structure and channels, and the attention map is generated by cascading attention. The prefusion module pre-fuses the three layers of feature maps from different stages in the backbone, and the delicate features of the shallow feature maps are fused with the deeper feature maps to generate richer feature maps with space location and semantic classification information. We conduct experiments to verify our modules, both on the underwater pipeline dataset and in a tank test. The results show that our modules can improve the performance of different neural network models for seabed-contacting segment detection. The target detection and instance segmentation performance of the best model is improved through a 6.3% increase in AP and a 3.8% increase in mean intersection over union compared with the baseline model.

Automatic target recognition has been one of the hottest research in synthetic aperture radar (SAR) data processing. Noticing that popular recognition methods cannot utilize multiple features of SAR complex data, a method fused scattering center feature and deep convolutional neural network (CNN) feature is proposed in this letter. This method contains three key parts, namely, scattering center extraction and reconstruction block, CNN feature extraction block, and final feature fusion and classification block. In this process, the scattering center feature and CNN feature are fused at the level of feature maps, which retain the space information of 2-D feature maps. What is more, the proposed half end-to-end strategy realizes the automatic update of weighting parameters in feature extraction network and subnetwork, which promotes a better recognition efficiency. Experimental results on measured SAR data show that the proposed method can achieve better accuracy than other single feature-based methods and feature fusion methods.

Semantic segmentation is a fundamental task in remote sensing image processing. The large appearance variations of ground objects make this task quite challenging. Recently, deep convolutional neural networks (DCNNs) have shown outstanding performance in this task. A common strategy of these methods (e.g., SegNet) for performance improvement is to combine the feature maps learned at different DCNN layers. However, such a combination is usually implemented via feature map summation or concatenation, indicating that the features are considered indiscriminately. In fact, features at different positions contribute differently to the final performance. It is advantageous to automatically select adaptive features when merging different-layer feature maps. To achieve this goal, we propose a gated convolutional neural network to fulfill this task. Specifically, we explore the relationship between the information entropy of the feature maps and the label-error map, and then a gate mechanism is embedded to integrate the feature maps more effectively. The gate is implemented by the entropy maps, which are generated to assign adaptive weights to different feature maps as their relative importance. Generally, the entropy maps, i.e., the gates, guide the network to focus on the highly-uncertain pixels, where detailed information from lower layers is required to improve the separability of these pixels. The selected features are finally combined to feed into the classifier layer, which predicts the semantic label of each pixel. The proposed method achieves competitive segmentation accuracy on the public ISPRS 2D Semantic Labeling benchmark, which is challenging for segmentation by only using the RGB images.

SummaryIn recent years, researchers have made great efforts in computer vision task (e.g., object detection) with the widely use of convolutional neural networks (CNNs). However, object detection algorithms based on CNNs suffer from high computation cost even on the high‐performance computers. In addition, with the development of high‐resolution videos, the deployment of object detection algorithms becomes more and more difficult because of the large amount of data, let alone the portable platforms, such as unmanned aerial vehicles (UAVs). In this paper, we research a lightweight network on portable platform for outdoor tiny pedestrian detection. Concretely, we first set up a training dataset manually for lack of tiny pedestrian samples in common datasets. We provide a lightweight network, and then, parallel computing is introduced to make the most of the advantage of GPU. Finally, our method can achieve real‐time performance on Jetson TX2. Experimental results verify that the proposed model has promising performance in tiny pedestrian detection designed for portable GPU platforms.

In the process of autonomous driving, there will be missed detections and false detections caused by dense crowds and occlusions during pedestrian target detection. This paper proposes a pedestrian object detection network model that combines Swin Transformer and YOLOv3. First use the lightweight Swin Transformer Tiny to replace the original Darknet53 as the backbone network of YOLOv3. The multi-scale detection is realized through the self-attention hierarchical network, which optimizes the detection effect in the case of dense pedestrians. Secondly, to deal with the occlusion in the crowd, Focal-EIoU Loss is used as a new loss function. I Introduce edge length loss and Focal L1 loss to increase the loss and gradient of IoU, thereby improving the regression accuracy. Finally, experiments are performed on the Caltech dataset. The experimental results show that the precision on the Caltech dataset reaches 95.23% and the recall rate reaches 89.57%. Compared with the original YOLOv3 algorithm, the precision is increased by 3.22%, and the recall rate is increased by 4.35%. The effectiveness of the algorithm is verified, and the performance of pedestrian detection is greatly improved.

Change detection methods aim to identify significantly changed areas in co-registered bitemporal images taken of the same area. Since not only do bitemporal images usually have different environmental conditions ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> different weather conditions, noises, and seasonal changes) but also changes irrelevant to the purpose of change detection ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.,</i> road changes when detecting building change), which should not be detected as changed areas, change detection methods often suffer from the problem of pseudo-change detection. To alleviate this problem, we propose an encoder-decoder-based Siamese network with a channel-wise co-attention module that considers the channel-wise correlations between a feature map in one image and all feature maps in the other image. By comparing the feature map in one image with the revised feature map in the other image considering the correlations, we are able to reduce the differences between the feature maps when pseudo-changes exist, thereby rendering the proposed method more robust to pseudo-changes. In addition, we apply a contrastive loss function that encourages the pairs of feature maps corresponding to unchanged regions to be similar, which can help improve the performance of change detection. We verified the performance of the proposed method through experiments using datasets such as the change detection dataset (CDD) and building change detection dataset (BCDD). In the experiment, the proposed method achieved significantly improved performance compared with existing methods in terms of recall, precision, f1-score, and overall accuracy.

Convolutional Neural Networks (CNN) are widely used in different artificial intelligence (AI) applications. Major part of the computation of a CNN involves 2D convolution. In this paper, we propose novel fast convolution algorithms for both 1D and 2D to remove the redundant multiplication operations in convolution computations at the cost of controlled increase of addition operations. For example, when the 2D processing block size is 3×3, our algorithm has multiplication saving factor as high as 3.24, compared to direct 2D convolution computation scheme. The proposed algorithm can also process input feature maps and generate output feature maps with the same flexible block sizes that are independent of convolution weight kernel size. The memory access efficiency is also largely improved by the proposed method. These structures can be applied to different CNN layers, such as convolution with stride > 1, pooling and deconvolution by exploring flexible feature map processing tile sizes. The proposed algorithm is suitable for both software and hardware implementation.

Big Model and Small Model : Remote modeling and local information extraction module for medical image segmentation

In deep convolutional neural networks (CNNs), local pooling operation is a key building block to effectively downsize feature maps for reducing computation cost as well as increasing robustness against input variation. There are several types of pooling operation, such as average/max-pooling, from which one has to be manually selected for building CNNs. The optimal pooling type would be dependent on characteristics of features in CNNs and classification tasks, making it hard to find out the proper pooling module in advance. In this paper, we propose a flexible pooling method which adaptively tunes the pooling functionality based on input features without manually fixing it beforehand. In the proposed method, the parameterized pooling form is derived from a probabilistic perspective to flexibly represent various types of pooling and then the parameters are estimated by means of global statistics in the input feature map. Thus, the proposed local pooling guided by global features effectively works in the CNNs trained in an end-to-end manner. The experimental results on image classification tasks demonstrate the effectiveness of the proposed pooling method in various deep CNNs.

Handwritten digit recognition is a well-researched area in the field of pattern recognition that is used for distinguishing the pre-segmented handwritten digits. Deep learning is a recent research trend in this field and architectures like convolutional neural network (CNN) are being used. CNN is a computationally expensive architecture that is normally used to provide high accuracy in complex classification problems which require tuning of millions of parameters. On the contrary, less complex classification problems can be solved using considerably less number of parameters—hence using CNN on them leads to unnecessary wastage of resources. In the present work, we have proposed to reduce the feature maps that are used to train the CNN to reduce storage space and computation time. Experimental results show that the time required to train the CNN decreases with reducing the number of feature maps without notably affecting the accuracy.