Effective use of convolutional neural networks and diverse deep supervision for better crowd counting

Accurately estimate the crowd count from a still image with arbitrary perspective and arbitrary crowd density is one of the difficulties of crowd analysis in surveillance videos. Conventional methods are scene-specific and subject to occlusions. In this paper, we propose a Multi-task Multi-column Convolutional Neural Network (MMCNN) architecture for crowd counting and crowd density estimation in still images of surveillance scenes. The MMCNN architecture is an end-to-end system which is robust for images with different perspective and different crowd density. By promoting MCNN with \(3\times 3\) filter, the MMCNN could utilize local spatial features from each column. Furthermore, the ground truth density map is generated based on Perspective-Adaptive Gaussian kernels which can better represent the heads of pedestrians. Finally, we use an iterative switching process in our deep crowd model to alternatively optimize the crowd density map estimation task and crowd counting task. We conduct experiments on the WorldExpo’10 dataset and our method achieves better results.

In this paper, we propose a method called Convolutional Neural Network-Markov Random Field (CNN-MRF) to estimate the crowd count in a still image. We first divide the dense crowd visible image into overlapping patches and then use a deep convolutional neural network to extract features from each patch image, followed by a fully connected neural network to regress the local patch crowd count. Since the local patches have overlapping portions, the crowd count of the adjacent patches has a high correlation. We use this correlation and the Markov random field to smooth the counting results of the local patches. Experiments show that our approach significantly outperforms the state-of-the-art methods on UCF and Shanghaitech crowd counting datasets.

Crowd counting on still images is very challenging due to heavy occlusions and scale variations. In this paper, we aim to develop a method that can accurately estimate the crowd count from a still image. Recently, convolutional neural networks have been shown effective in many computer vision tasks including crowd counting. To this end, we propose a fully convolutional network (FCN) architecture to map the input image of arbitrary size or resolution to its density map. In order to address the perspective and scale variation issues, Inception-like modules with multiple kernel size filters are used to capture multi-scale features, which is necessary for higher crowd counting performance. We test our model on challenging ShanghaiTech dataset, the results show that our method outperforms the state-of-the-art methods.

The task of crowd counting and density map estimation is riddled with many challenges, such as occlusions, non-uniform density, intra-scene and inter-scene variations in scale and perspective. Due to the development of deep learning and large crowd datasets in recent years, most crowd counting methods have achieved notable success. This paper aims to solve crowd density estimation problem for both sparse and dense conditions. To this end, we make two contributions: (1) a network named Patch Scale Discriminant Regression Network (PSDR). Given an input crowd image, it divides the image into patches and sends image patches of different density levels into different regression networks to get the corresponding density maps. It combines all patch density maps to predict the entire density map as the output. (2) A person classification activation map (CAM) method. CAM provides person location information and guides the generation of the entire density map in the final stage. Experiment confirms that CAM allows PSDR to gain another round of performance boost. For instance, on the SmartCity dataset, we achieve (8.6–1.1) MAE and (11.6–1.4) MSE. Our method combining above two methods performs better than state-of-the-art methods.

This paper aims to utilize the deep learning architecture to break through the limitations of camera perspective, image background, uneven crowd density distribution and pedestrian occlusion to estimate crowd density accurately. In this paper, we proposed a new neural network called Deep Scale-Adaptive Convolutional Neural Network (DSA-CNN), which can convert a single crowd image to density map for crowd counting directly. For a crowd image with any size and resolution, our algorithm can output the density map of the crowd image by end-to-end method and finally estimate the number of the crowd in the image. The proposed DSA-CNN consists of two parts: the seven layers CNN network structure and DSA modules. In order to ensure the proposed method is robust to camera perspective effect, DSA-CNN has adopted different sizes of filters in the network and combines them ingeniously. In order to reduce the depth of the data to increase the speed of training, the proposed method utilized 1 × 1 filter in DSA module. To validate the effectiveness of the proposed model, we conducted comparative experiments on four popular public datasets (ShanghiTech dataset, UCF_CC_50 dataset, WorldExpo’10 dataset and UCSD dataset). We compare the proposed method with other well-known algorithms on the MAE and MSE indicators, such as MCNN, Switching-CNN, CSRNet, CP-CNN and Cascaded-MTL. Experimental results show that the proposed method has excellent performance. In addition, we found that the proposed model is easily trained, which further increases the usability of the proposed model.

Due to the rise in coronavirus disease (CoVID-19) cases, it has been a significant concern for governments all over the world to control the spread of the virus. We aim to aid in controlling the spread of the CoVID-19 virus. The face mask detection and crowd counting model with the Internet of things (IoT) platform (Thingspeak) can be deployed in situations where there is a lack of workforce, and additional assistance is required. Using a camera, an area can be remotely monitored, restricting contact between people and the frontline workers, promoting their safety. The research conducted to date about this topic is solely on detecting facemask or doing crowd counting, respectively. We included both face mask detection and crowd counting, connecting both to the IoT platform, enabling us to access the results online. In previous research work, traditional/wired cameras were used. But in our research work, we have used the concept of a Wi-Fi camera, i.e., wireless camera, by which the camera can be positioned in places where the placement of wires is difficult. For face mask detection, we have created two data sets which include images of faces with masks and without a mask, and this data set is used to train the model. For training and testing the model, we have included TensorFlow, Keras and SKLearn libraries. We have included different deep learning layers in training the model like Dense, Dropout and AveragePooling2D. In the test model, python code is connected to a wireless camera, which can capture live video feed and detect whether the person is wearing a mask or not. We have used OpenCV Histogram of oriented gradients (HOG) descriptor in crowd counting, often used to extract features from image data. Using an OpenCV HOG Descriptor, we can count the number of people present in a crowd. Face mask and crowd count is then sent to the IoT platform. Whenever the person is found wearing a mask, a green box appears around the face, and whenever the person is not wearing a mask, a red box appears around the face. When lots of people are detected by a crowd counting model, a green color box enclosing the entire body appears. This study helps in getting face mask detection and crowd size estimation on a single IoT platform, which helps monitor the data with great ease.

Crowd counting is of great importance to many applications. Conventional vision-based approaches require line of sight and pose privacy concerns, while most radio-based approaches involve high deployment cost. In this paper, we propose to utilize WiFi channel state information (CSI) to infer crowd count in a device-free way, with only one pair of WiFi transmitter and receiver. The proposed method establishes the statistical relationship between the variation of CSI and the number of people with deep neural networks (DNN) and thereafter estimates the people count according to the real-time CSI through the trained DNN model. Evaluations demonstrate the effectiveness of the method. For the crowd size of 6, the counting error was within 1 person for 100% of the cases. For the crowd size of 34, the counting error was within 1 person for 97.7% of the cases and within 2 persons for 99.3% of the cases.

Counting the number of person has received much attention in recent years. Most of the existing crowd counting methods adopted density map regression pipeline, which formulates the crowd counting problem to two fragmented part: density map regression and integration of the overall counting. To solve this problem, this paper presents a multi-task deep learning scheme to enhance the counting performance. More specifically, we firstly build a multi-scale deep convolutional neural network, based on combining the feature maps of conv layers with different filters, to solve the multi-scale problem in crowd counting. Secondly, we develop the multi-task structure that can simultaneously learn the density map and the global counting. Experiments on large scale crowd counting datasets, Shanghaitech and WorldExpo10, demonstrate that the proposed method achieves much reduction in counting error respectively

We formulate counting as a sequential decision problem and present a novel crowd counting model solvable by deep reinforcement learning. In contrast to existing counting models that directly output count values, we divide one-step estimation into a sequence of much easier and more tractable sub-decision problems. Such sequential decision nature corresponds exactly to a physical process in reality—scale weighing. Inspired by scale weighing, we propose a novel ‘counting scale’ termed LibraNet where the count value is analogized by weight. By virtually placing a crowd image on one side of a scale, LibraNet (agent) sequentially learns to place appropriate weights on the other side to match the crowd count. At each step, LibraNet chooses one weight (action) from the weight box (the pre-defined action pool) according to the current crowd image features and weights placed on the scale pan (state). LibraNet is required to learn to balance the scale according to the feedback of the needle (Q values). We show that LibraNet exactly implements scale weighing by visualizing the decision process how LibraNet chooses actions. Extensive experiments demonstrate the effectiveness of our design choices and report state-of-the-art results on a few crowd counting benchmarks, including ShanghaiTech, UCF_CC_50 and UCF-QNRF. We also demonstrate good cross-dataset generalization of LibraNet. Code and models are made available at https://git.io/libranet.KeywordsCrowd countingReinforcement learning

Counting the number of people in public places has received much attention, and researchers have devoted much effort to the task. However, the existing crowd counting approaches are mainly trained and tested in similar scenarios. The performance of crowd counting approaches degrades sharply when the test scenarios of the models are of different types from its training scenes. In practice, the crowd scenes are highly variable, and the lack of cross-scene capability could seriously limit the application of the existing approaches. We attribute the improvement in cross-scene crowd counting capability to the necessity of accommodating large changes in the scale of individuals and the ability to suppress the interference of cluttered backgrounds. To this end, we propose a multi-scale feature fusion network (MFFNet) with cascaded supervision. The multi-scale features extracted from the crowd images are upsampled and then combined into several feature blocks, followed by convolution and deconvolution operations on the feature blocks to derive feature matrices of different resolutions. The feature matrices are fused from bottom to top. In the process of feature fusion, the crowd density maps corresponding to the feature matrices of different resolutions are predicted separately. We devise cascaded supervision to synchronously optimize the network of different resolution density map prediction during training. The cross-scene crowd counting experiments are conducted on four types of scenes: ShanghaiTech Part_A (SHT A) with high-density crowd scenes and small-scale individuals, ShanghaiTech Part_B (SHT B) with sparse crowd distribution and medium-scale individuals, UCF_CC_50 dataset with extremely dense scenes and tiny scale individuals, and UCF-QNRF dataset with extreme variations. MFFNet exhibits the strongest scene adaptability relative to the state-of-the-art approaches, with an average decrease of 17.1% and 8.4% in mean absolute error (MAE) and mean square error (mse), respectively. The contributions of different components in our method are verified in the ablation study using the devised evaluation metrics. Our implementation will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/learnsharing/MFFNet</uri> .

During the past five years, there has been an increasing trend of weakly supervised crowd counting methods being developed since such methods just rely on count-level annotations and avoid a laborious labeling process. But, the existing weakly supervised methods usually fail to achieve comparable counting performance to the fully supervised methods. To improve the accuracy of crowd counting tasks, we propose to combine the convolutional neural network (CNN) and Transformer frameworks. Since CNN focuses on capturing local detail information and Transformer can effectively extract global context information, we believe that the combination of CNN and Transformer could learn more efficient feature representations for crowd images. Our proposed framework is named CrowdCCT (Crowd Counting via CNN and Transformer), and it is composed of a CNN feature extraction part, a Transformer feature extraction part, and a counting regression part. In the CNN part, we utilize DenseNet121 to learn rich semantic features with its inherent dense connection structure. In the Transformer part, we introduce two attention modules, Multi-Scale Dilated Attention (MSDA) and Location-Enhanced Attention (LEA), working together to extract more expressive features. The output features are then fed into the regression part to generate the predicted counting results. Experiments on four crowd counting benchmark datasets demonstrate that our proposed CrowdCCT can achieve superior performance. Also, the experimental results validate the feasibility and effectiveness of combining CNN and Transformer for weakly supervised counting tasks. Our work could be expected to promote further combination research on CNN and Transformer.

Abstract. Most of the existing crowd counting research methods are based on convolutional neural network (CNN), which has a strong ability to extract local features, but is limited by the size of the receptive field, making it difficult to model the global context. At the same time, the background of crowd images is complex, the targets are densely distributed, and they are easily disturbed by external conditions such as occlusion and lighting. Therefore, it is extremely cumbersome and error-prone to label the heads of pedestrians in the image. In response to the above problems, this paper proposes a weakly supervised crowd counting method based on Swin Transformer. First, Swin Transformer is used as the backbone network for feature extraction to capture global context information and realize the modeling of feature interactions between targets. Secondly, attention-based multi-scale feature fusion module is designed to aggregate global spatial position features on multiple scales to improve the detection effect of small objects. Finally, global average pooling is used for feature dimensionality reduction and a regression layer is designed to predict the number of people. Tests were carried out on three crowd datasets including Shanghai Tech, UCF_CC_50 and UCF_QNRF. The experimental results show that the overall performance of the proposed method is better than other common crowd counting methods.

Crowd counting in still images is a challenging problem in practice due to huge crowd-density variations, large perspective changes, severe occlusion, and variable lighting conditions. The state-of-the-art patch rescaling module (PRM) based approaches prove to be very effective in improving the crowd counting performance. However, the PRM module requires an additional and compromising crowd-density classification process. To address these issues and challenges, the paper proposes a new multi-resolution fusion based end-to-end crowd counting network. It employs three deep-layers based columns/branches, each catering the respective crowd-density scale. These columns regularly fuse (share) the information with each other. The network is divided into three phases with each phase containing one or more columns. Three input priors are introduced to serve as an efficient and effective alternative to the PRM module, without requiring any additional classification operations. Along with the final crowd count regression head, the network also contains three auxiliary crowd estimation regression heads, which are strategically placed at each phase end to boost the overall performance. Comprehensive experiments on three benchmark datasets demonstrate that the proposed approach outperforms all the state-of-the-art models under the RMSE evaluation metric. The proposed approach also has better generalization capability with the best results during the cross-dataset experiments.

This paper aims to propose a convolutional neural network that can accurately estimate the density of pedestrians in crowd images, and analyze the primary factors that affect the performance of the network employed in crowd counting. In contrast to the multi-column convolutional neural network (MCNN) which is mainly used in previous works, the proposed network is designed by using the improved Inception-ResNet-A module in the manner of end-to-end and therefore it is convenient for training and easy to be employed. Experiments have been conducted on four datasets: Mall dataset, UCSD dataset, ShanghaiTech dataset and UCF_CC_SO dataset. Besides, the performances of different convolutional neural network (CNN) architectures have been compared to prove the effectiveness of the proposed network. The experimental results show that the proposed network is more accurate and efficient against the existing state-of-art methods, overcoming the problems generated by scale variation, pedestrian occlusion and appearance change. To the best of our knowledge, the proposed network is the first network to use the Inception-ResNet-A structure for crowd counting.

Crowd behavior prediction (CBP) and crowd counting (CC) are the essential functions of vision-based crowd analysis (CA), which play a crucial role in controlling crowd disasters. The CA using different models for the CBP and the CC will increase computational overheads and have synchronization issues. The state-of-the-art approaches utilized deep convolutional architectures to exploit spatial-temporal features to accomplish the objective, but such models suffer from computational complexities during convolution operations. Thus, to sort out the issues as mentioned earlier, this paper develops a single deep model which performs two functionalities of CA: CBP and CC. The proposed model uses multilayers of depth-wise separable CNN (DSCNN) to extract fine-grained spatial-temporal features from the scene. The DSCNN can minimize the number of matrix multiplications during convolution operation compared to traditional CNN. Further, the existing datasets are available to accomplish the single functionality of CA. In contrast, the proposed model needs a dual-tasking CA dataset which should provide the ground-truth labels for CBP and CC. Thus, a dual functionality CA dataset is prepared using a benchmark crowd behavior dataset, i.e. MED. Around 41 000 frames have been manually annotated to obtain ground-truth crowd count values. This paper also demonstrates an experiment on the proposed multi-functional dataset and outperforms the state-of-the-art methods regarding several performance metrics. In addition, the proposed model processes each test frame at 3.40 milliseconds, and thus is easily applicable in real-time.