Recurrent DETR: Transformer-Based Object Detection for Crowded Scenes

SummaryObject detection in crowded scenes involves various difficulties, such as small objects, occluded objects, and insufficient features. Existing models for crowded object detection often focus on only one detection difficulty, and they are too large to be applied in practice. To address the diverse challenges of object detection in crowded scenes, we construct a lightweight crowded object detector called MBB‐YOLO, which contains several modules for comprehensive improvement. To improve the network's ability to extract fine‐grained features, we use SPD‐Conv and the proposed MS‐Conv to replace the strided convolution in the network. An bi‐branch multi‐scale convolution attention (BMCA) module is proposed to aggregate multi‐scale contextual information. We also propose boundary‐NMS to better identify proposal boxes from different objects, which reduces suppression errors caused by object occlusion. MBB‐YOLO achieves 87.6% AP and an inference speed of 78.8 FPS on the CrowdHuman dataset, which surpasses other mainstream lightweight object detectors.

A novel framework for anomaly detection in crowded scenes is presented. Three properties are identified as important for the design of a localized video representation suitable for anomaly detection in such scenes: (1) joint modeling of appearance and dynamics of the scene, and the abilities to detect (2) temporal, and (3) spatial abnormalities. The model for normal crowd behavior is based on mixtures of dynamic textures and outliers under this model are labeled as anomalies. Temporal anomalies are equated to events of low-probability, while spatial anomalies are handled using discriminant saliency. An experimental evaluation is conducted with a new dataset of crowded scenes, composed of 100 video sequences and five well defined abnormality categories. The proposed representation is shown to outperform various state of the art anomaly detection techniques.

Real-time video event detection in crowded scenes using MPEG derived features: A multiple instance learning approach

Modelling events in densely crowded environments remains challenging, due to the diversity of events and the noise in the scene. We propose a novel approach for anomalous event detection in crowded scenes using dynamic textures described by the Local Binary Patterns from Three Orthogonal Planes (LBP-TOP) descriptor. The scene is divided into spatio-temporal patches where LBP-TOP based dynamic textures are extracted. We apply hierarchical Bayesian models to detect the patches containing unusual events. Our method is an unsupervised approach, and it does not rely on object tracking or background subtraction. We show that our approach outperforms existing state of the art algorithms for anomalous event detection in UCSD dataset.

A crowd is a gathering of a huge number of individuals in a confined area. Early identification and detection of unusual behaviors in terms of panic occurring in crowded scenes are very important. Panic detection comprises of formulating normal scene behaviors and detecting and identifying non-matching behaviors. However, panic detection and recognition is a very difficult problem, especially when considering diverse scenes. Many methods proposed to cope with these problems have limited robustness as the density of the crowd varies. In order to handle this challenge, this paper proposes the integration of different features into a unified model. Discriminant binary patterns and neighborhood information are used to model complex and unique motion patterns in order to characterize different levels of features for diverse types of crowd scenes, focusing in particular on the detection of panic and non-pedestrian entities. The proposed method was evaluated considering two benchmark datasets and outperformed five existing methods.

Pedestrian detection in crowded scenes is widely used in computer vision. However, it still has two difficulties: 1) eliminating repeated predictions (multiple predictions corresponding to the same object); 2) false detection and missing detection due to the high scene occlusion rate and the small visible area of detected pedestrians. This paper presents a detection framework based on DETR (detection transformer) to address the above problems, and the model is called AD-DETR (asymmetrical relation detection transformer). We find that the symmetry in a DETR framework causes synchronous prediction updates and duplicate predictions. Therefore, we propose an asymmetric relationship fusion mechanism and let each query asymmetrically fuse the relative relationships of surrounding predictions to learn to eliminate duplicate predictions. Then, we propose a decoupled cross-attention head that allows the model to learn to restrict the range of attention to focus more on visible regions and regions that contribute more to confidence. The method can reduce the noise information introduced by the occluded objects to reduce the false detection rate. Meanwhile, in our proposed asymmetric relations module, we establish a way to encode the relative relation between sets of attention points and improve the baseline. Without additional annotations, combined with the deformable-DETR with Res50 as the backbone, our method can achieve an average precision of 92.6%, MR$ ^{-2} $ of 40.0% and Jaccard index of 84.4% on the challenging CrowdHuman dataset. Our method exceeds previous methods, such as Iter-E2EDet (progressive end-to-end object detection), MIP (one proposal, multiple predictions), etc. Experiments show that our method can significantly improve the performance of the query-based model for crowded scenes, and it is highly robust for the crowded scene.

In this paper, we propose a fast and robust framework for anomaly detection in crowed scenes. In our method, anomaly is adaptively modeled as a deviation from the normal behavior of crowd observed in the scene. For this purpose, we extract motion features by repeatedly initializing a grid of particles over a temporal window. These features are exploited in a real-time anomaly detection system. In order to model the ordinary behavior of the people moving in the crowd, we use the Gaussian mixture model (GMM) technique, which is robust enough to capture the scene dynamics. As opposed to explicitly modeling the values of all the pixels as a mixture of Gaussians, we adopted the GMM to learn the behavior of the motion features extracted from the particles. Based on the persistence and the variance of each Gaussian distribution, we determine which Gaussians can be associated to the normal behavior of the crowd. Particles with motion features that do not fit the distributions representing normal behavior are signaled as anomaly, until there is a Gaussian able to include them with sufficient evidence supporting it. Experiments are extensively conducted on publically available benchmark dataset, and also on a challenging dataset of video sequences we captured. The experimental results revealed that the proposed method performs effectively for anomaly detection.

We propose a simple yet effective proposal-based object detector, aiming at detecting highly-overlapped instances in crowded scenes. The key of our approach is to let each proposal predict a set of correlated instances rather than a single one in previous proposal-based frameworks. Equipped with new techniques such as EMD Loss and Set NMS, our detector can effectively handle the difficulty of detecting highly overlapped objects. On a FPN-Res50 baseline, our detector can obtain 4.9\% AP gains on challenging CrowdHuman dataset and 1.0\% $\text{MR}^{-2}$ improvements on CityPersons dataset, without bells and whistles. Moreover, on less crowed datasets like COCO, our approach can still achieve moderate improvement, suggesting the proposed method is robust to crowdedness.

Different approaches have been proposed in the literature for anomaly detection in image/video area. Traditional methods such as trajectory or spatio-temporal based techniques rely on hand-crafted features. The occlusion problems and high complexity in crowded scenes are the vital drawbacks behind using these methodologies. Deep learning structures proved recently to be useful for defining effective solutions for anomaly detection where the high level features are learnt and selected automatically. However, the block-wise methods such as CNNs are computationally slow. On the other hand, they are totally supervised learning methodologies, while the video-based anomaly detection is an unsupervised problem. Auto- Encoders (Convolutional AE, vibrational AE, etc.) can be considered as an alternative option. This paper presents a stateof- the-art deep learning algorithm to be applied in such an unsupervised problem. Using the basic concepts behind the Auto-Encoders as a well-known unsupervised learning algorithm, we propose a novel methodology to detect and localize the anomalies in a video scene. The presented network is trained based on the normal patterns during training phase. The proposed structure enables the system to capture the 2D structure in image sequences during the learning process. The working hypothesis is that a deep network is able to learn normal events in videos, and, therefore, the difference of normal and anomalous frames can be used for devising an anomaly score. The simulation results on well-known data sets such as UCSD confirm that the proposed methodology achieves high performance results in case of accuracy and total processing time compared with counterpart approaches.

This paper presents a new inlier-based outlier detection scheme for the analysis of abnormal behavior in crowded scenes. First, we segment the video into a set of cubes, and then extract the three feature descriptors from each cube, including the histogram of oriented gradients (HOG), the histogram of motion directions, and the motion magnitude descriptors. Thereafter, for each feature descriptor the Kullback-Leibler importance estimation procedure (KLIEP) is employed to compute the ratio of test and training densities, referred to as the importance, instead of computing these two densities separately. This allows us to avoid the difficulty in estimating complex probability density estimation. The importance denotes an inlier score which represents the degree of similarity between the test and training data. Based on the importance of each descriptor and a prespecified threshold in each cube, we can then identify if a cube contains an anomaly event. Through computer simulations, we find that the new approach provides high accuracy of localization rate based on the widespread UCSD datasets compared with previous works.

We present a new method for detection of abnormal behaviors in crowded scenes. Based on statistics of low-level feature—optical flow, which describes human movement efficiently, the motion energy model is proposed to represent the local motion pattern in the crowd. The model stresses the difference between normal and abnormal behaviors by considering sum of square differences (SSD) metric of motion information in the center block and its neighboring blocks. Meanwhile, data increasing rate is introduced to filter outliers to achieve boundary values between abnormal and normal motion patterns. In this model, an abnormal behavior is detected if the occurrence probability of anomaly is higher than a preset threshold, namely the motion energy value of its corresponding block is higher than that of the normal one. We evaluate the proposed method on two public available datasets, showing competitive performance with respect to state-of-the-art approaches not only in detection accuracy, but also in computational efficiency.

In this paper, we propose a novel method for global abnormal events detection in crowded scenes. Each video is described as the set of overlapping space-time cubes. The histogram of optical flow orientation and motion magnitude are used as global feature descriptor to capture the motion magnitude and orientation of the normal and abnormal events. The motion-rich space-time cubes are selected to enhance the computational speed of the algorithm and omit background subtraction step, forms the proposed approach novel and unique. Simple and cost-effective one-class SVM classifier is used to distinguish between normal and abnormal events in a video. The one-class SVM classifier is trained with normal events from the training data. The proposed method is tested on benchmark UMN and PETS2009 dataset. The improved results are reported to evaluate the performance of the proposed method.

Crowd-SAM: SAM as a Smart Annotator for Object Detection in Crowded Scenes

People head detection in crowded scenes is challenging due to the large variability in clothing and appearance, small scales of people, and strong partial occlusions. Traditional bottom-up proposal methods and existing region proposal network approaches suffer from either poor recall or low precision. In this paper, we propose to improve both the recall and precision of head detection of region proposal models by integrating the local head information. In specific, we first use a region proposal network to predict the bounding boxes and corresponding scores of multiple instances in the region. A local head classifier network is then trained to score the bounding box generated from the region proposal model. After that, we propose an adaptive fusion method by optimally combining both the region and local scores to obtain the final score of each candidate bounding box. Furthermore, our fusion models can automatically learn the optimal hyper-parameters from data. Our algorithm achieves superior people head detection performance on the crowded scenes data set, which significantly outperforms several recent state-of-the-art baselines in the literature.

In this paper we present a novel framework for abnormal behavior detection in crowded scenes. For this purpose, the theory of topological simplification on the dense field is extended to the sparse particle motion field, which is used to describe the dynamics of the crowd. We propose two new methods for analysis of boundary point structure and extraction of critical point from the particle motion field. Both methods can be used to describe the global topological structure of the crowd motion, which is the kernel idea of our work. Various types of abnormal behaviors, including crowd formation/dispersal, crowds splitting/merging, can be detected by monitoring the changes of the topological structure. The advantage of our method is that each kind of abnormal event can be described as a specific topological structure change, therefore we do not need a complex classifier to detect these anomalies. Experiments are conducted on known datasets and results show that our method is effective in detecting and locating these kinds of abnormal behaviors.