Abnormal behavior detection using hybrid agents in crowded scenes

We propose a method to detect abnormal crowd behavior using integrated multiple behavior models. Traditional abnormal detection methods are based only on personal behavior models. However, a single behavior model cannot accurately reflect complex crowd behavior. To solve this problem, we introduce an integrated multiple behavior model for accurate abnormal behavior detection in a complex crowd scene. We use not only the personal behavior model, but also multiple social behavior models. The experimental results show that our proposed method efficiently detects the abnormal behavior in a crowded scene.

As society develops, the demand for security is rapidly increasing. Accordingly, there is growing interest in research on methods to detect and prevent abnormal behavior using surveillance cameras in public places and private spaces such as shopping malls and airports for human safety. Many detection techniques based on deep learning models have been researched in the field of abnormal behavior detection. However, due to the lack of labeled abnormal behavior data, there are significant difficulties in developing an effective detection system. This paper surveys methods for deep learning methods to detect abnormal human behavior in surveillance video and presents recent techniques. First, I will introduce popular datasets that have often been used in previous research. After that, we categorized the existing methods for detecting abnormal behavior using deep learning into three types: supervised learning, unsupervised learning, and partially-supervised learning. We then explained the basic concepts and advantages of each method and summarized their shortcomings. We also briefly describe future research directions based on the advantages and disadvantages of each method. Based on this, it is expected that the technology of video surveillance systems that apply abnormal behavior detection will further develop.

ABSTRACTThe detection of abnormal behaviours with fast and automatic recognising is significant in crowded areas to provide higher security to the public. The adoption of deep learning and machine learning‐based abnormal behaviour detection models enhances the influential detection and real‐time security monitoring in crowds. The researchers have remotely evaluated the heart rate based on physiological information to detect abnormal activities in various years. Over the past few years, several progress have been made, and there are still some issues concerning processing time, accuracy, and computational complexity. The developed approaches detects the activities of anomalies like traffic rule violations, riots, fighting, and stampede, in addition, several anomalous entities such as abandoned luggage and weapons at the sensitive place automatically in time. However, the identification of video anomalies methods poses several challenges because of various environmental conditions, the ambiguous nature of the anomaly, lack of proper datasets, and the complex nature of human characteristics. In recent days, there have been only a few devoted surveys associated with deep learning related video anomaly identification as the research domain is in its initial stages. In this review work, the abnormal behaviour analysis models using deep learning are reviewed in depth in for security applications. Based on the traditionally used abnormal behaviour analysis models in crowded scenes, we widely categorised the methods into classification using object tracking, classification using handcrafted extracted features, classification using non‐contact heart rate variability and blood pressure, analysing motion patterns from the visual frames, and classification using face images. We also discuss the comparative analysis of the previous methods with respect to datasets, computational infrastructure, and performance measures for both qualitative and quantitative analysis. In addition, the open and trending research challenges are analysed for future research.

In this paper, we target on the problem of estimating the statistic of pedestrian travel time within a period from an entrance to a destination in a crowded scene. Such estimation is based on the global distributions of crowd densities and velocities instead of complete trajectories of pedestrians, which cannot be obtained in crowded scenes. The proposed method is motivated by our statistical investigation into the correlations between travel time and global properties of crowded scenes. Active regions are created for each source-destination pair to model the probable walking regions over the corresponding source-destination traffic flow. Two sets of scene features are specially designed for modeling moving and stationary persons inside the active regions and their influences on pedestrian travel time. The estimation of pedestrian travel time provides valuable information for both crowd scene understanding and pedestrian behavior analysis, but was not sufficiently studied in literature. The effectiveness of the proposed pedestrian travel time estimation model is demonstrated through several surveillance applications, including dynamic scene monitoring, localization of regions blocking traffics, and detection of abnormal pedestrian behaviors. Many more valuable applications based on our method are to be explored in the future.

Abnormal crowd behavior detection has recently attracted increasing attention due to its wide applications in computer vision research areas. However, it is still an extremely challenging task due to the great variability of abnormal behavior coupled with huge ambiguity and uncertainty of video contents. To tackle these challenges, we propose a new probabilistic framework named variational abnormal behavior detection (VABD), which can detect abnormal crowd behavior in video sequences. We make three major contributions: (1) We develop a new probabilistic latent variable model that combines the strengths of the U-Net and conditional variational auto-encoder, which also are the backbone of our model; (2) We propose a motion loss based on an optical flow network to impose the motion consistency of generated video frames and input video frames; (3) We embed a Wasserstein generative adversarial network at the end of the backbone network to enhance the framework performance. VABD can accurately discriminate abnormal video frames from video sequences. Experimental results on UCSD, CUHK Avenue, IITB-Corridor, and ShanghaiTech datasets show that VABD outperforms the state-of-the-art algorithms on abnormal crowd behavior detection. Without data augmentation, our VABD achieves 72.24% in terms of AUC on IITB-Corridor, which surpasses the state-of-the-art methods by nearly 5%.

We present a new method for detection of abnormal behaviors in crowded scenes. Based on statistics of low-level feature—optical flow, which describes human movement efficiently, the motion energy model is proposed to represent the local motion pattern in the crowd. The model stresses the difference between normal and abnormal behaviors by considering sum of square differences (SSD) metric of motion information in the center block and its neighboring blocks. Meanwhile, data increasing rate is introduced to filter outliers to achieve boundary values between abnormal and normal motion patterns. In this model, an abnormal behavior is detected if the occurrence probability of anomaly is higher than a preset threshold, namely the motion energy value of its corresponding block is higher than that of the normal one. We evaluate the proposed method on two public available datasets, showing competitive performance with respect to state-of-the-art approaches not only in detection accuracy, but also in computational efficiency.

Abnormal behaviour detection is crucial given the reliance on video surveillance for public safety. However, most vision-based methods fail to adapt to the variability and complexity of the crowd scenes inherent in real-world scenarios. This study introduced an abnormal crowd behaviour detection algorithm based on lightweight Graphic Neural Networks (GNNs) using trajectory similarities and spatial-temporal attributes. Integrating Graph Attention Networks (GATs) to encode the graph structure, creating latent embeddings that capture both local and global relationships within crowds. Also introducing an autoencoder (AE) structure to identify anomalies by comparing true movement with the decoded output. Anomaly is detected when the reconstruction error is high. Experiments on UCSD Pedestrians and ShanghaiTech campus datasets show the successful use of trajectory similarities as graph features for GNNs to generate logical embeddings of crowd movement and highlight the impact of time windows on anomaly detection precision as reflected in F1-scores. The Area Under the Curve (AUC) metric shows stability, highlighting the model’s balanced performance. Qualitative analysis of the ShanghaiTech campus dataset illustrates the model’s ability to align detection with visual anomalies. Overall, this study demonstrates the effectiveness of the proposed graph-based anomaly detection framework for crowd monitoring through a lightweight model suitable for CCTV.

Problem of detection and localization of abnormal behaviors in crowded scenes is considered in this paper. To extract various activities of crowd, spatiotemporal Laplacian eigenmap method is proposed. Here, spatiotemporal variations of local motions are learned in an embedded space. Regular behavior of a crowd is modeled using representatives of various activities. To detect abnormal behavior in local as well as global contexts and to localize regions showing abnormality, model of regular crowd behavior is used. Computational simplicity is the feature of this method.

In this paper; we propose a novel crowd behavior representation method to detect abnormal behaviors in videos. An adaptive optical flow filtering method is proposed to utilize low-level optical flow informations. Furthermore, a simple framework is developed to detect and to localize abnormal crowd behavior using adaptive optical flow filtering result. The proposed method is more robust than other modeling methods in representing different behaviors. In this model, a normal behavior is presented by the general value. Some outliers in the temporal domain or spatial domain are presented by a higher value. Spatio-temporal cuboids are extracted from the filtering result to present the likelihood of anomaly in the frame. Experimental evaluations are performed on two public datasets with comparison to the provisos abnormal behavior detection methods in the literature. Experimental results show that the proposed methods outperform previous abnormal behavior detection techniques in the literature.

This paper presents a novel video descriptor, referred to as Histogram of Oriented Track lets, for recognizing abnormal situation in crowded scenes. Unlike standard approaches that use optical flow, which estimates motion vectors only from two successive frames, we built our descriptor over long-range motion trajectories which is called track lets in the literature. Following the standard procedure, we divided video sequences in spatio-temporal cuboids within which we collected statistics on the track lets passing through them. In particular, we quantized orientation and magnitude in a 2-dimensional histogram which encodes the motion patterns expected in each cuboid. We classify frames as normal and abnormal by using Latent Dirichlet Allocation and Support Vector Machines. We evaluated the effectiveness of the proposed descriptors on three datasets: UCSD, Violence in Crowds and UMN. The experiments demonstrated (i) very promising results in abnormality detection, (ii) setting new state-of-the-art on two of them, and (iii) outperforming former descriptors based on the optical flow, dense trajectories and the social force model.

This paper proposes a new algorithm for abnormal behavior detection in surveillance video. Firstly, the motion information image of each frame is constructed by calculating the optical flow size and the angle difference between the optical flow vectors between consecutive frames, and then the obtained motion image information is input into the convolutional neural network (CNN) for training, and used for video abnormal behavior detection. In the algorithm, the motion information image generated based on optical flow information can provide the motion information features in the video image more accurately, which makes it easier to distinguish the normal behavior and abnormal behavior of the video. The experiment of this algorithm is carried out on the commonly used data set PETS 2009. Experimental results show that the proposed method and other algorithms have a significant improvement in the accuracy of abnormal behavior detection.

Abnormal behavior analysis in crowded scenes is a challenging task in the computer vision field. Determining how to quickly and accurately locate abnormal behavior in a crowd, especially in airports, shopping malls, stations, and other practical application scenarios, has become the key to the timely handling of mass incidents. The authors have developed an embedded video analysis robot capable of local processing for crowd anomaly detection in application scenarios. In recent years, many methods of crowd anomaly detection have been proposed by scholars, but the performance of these methods when applied to video service robots is unsatisfactory. In this paper, an anomaly analysis method based on improved k-means is proposed. This method combines mean shift and the k-means classification method to achieve rapid and accurate crowd anomaly detection. A quantitative experimental evaluation was carried out on a video service robot, and the results demonstrate that the method is effective for the detection of abnormal behavior on multiple and publicly available video sequences.

This study presents a velocity-based egress model, which takes into account different aspects of human behavior in an emergency situation, for the evacuation analysis on passenger ships. It was supposed that the egress model consists of three behaviors: individual, crowd, and emergency behavior. The individual behavior was represented by the body shape, walking speed, walking direction, and rotation of each passenger. The basic walking direction of each passenger was obtained as a solution to the shortest distance route to a destination using a visibility graph. The crowd behavior of the passengers was composed of two components: one is a flock behavior, a form of collective behavior of a large number of interacting passengers with a common group objective, and the other is a leader- following behavior, which causes one or more passengers to follow another moving passenger who is designated as the leader. The emergency behavior of the passengers was represented by a counterflow-avoiding behavior to avoid collision with other passengers walking in the opposite direction. Eleven basic tests and 2 examples specified in International Maritime Organization Maritime Safety Committee/Circulation 1238 were conducted, and it was confirmed that all the requirements of such tests had been met.

With the development of track and field, people pay more and more attention to the quality of classroom teaching of track and field technology, and the evaluation of teaching quality plays a key role in it. In today's educational reform, teaching evaluation plays an important role as an important method to test teachers' teaching and students' learning. With the rapid development of machine learning, especially deep learning, image-based individual abnormal behavior detection technology is becoming more and more mature, but there are still many difficulties to be solved in video-based group abnormal behavior detection technology. Therefore, it is necessary to study the detection algorithm. Based on machine vision theory, image processing theory and video analysis technology, this paper studies three key technologies involved in human abnormal behavior detection in video: moving target detection, moving target tracking and abnormal behavior detection.

Amid rapid advancements in artificial intelligence, the detection of abnormal human behaviors in complex traffic environments has garnered significant attention. However, detection errors frequently occur due to interference from complex backgrounds, small targets, and other factors. Therefore, this paper proposes a research methodology that integrates the anomaly detection YOLO-SGCF algorithm with the tracking BoT-SORT-ReID algorithm. The detection module uses YOLOv8 as the baseline model, incorporating Swin Transformer to enhance global feature modeling capabilities in complex scenes. CBAM and CA attention are embedded into the Neck and backbone, respectively: CBAM enables dual-dimensional channel-spatial weighting, while CA precisely captures object location features by encoding coordinate information. The Neck layer incorporates GSConv convolutional modules to reduce computational load while expanding feature receptive fields. The loss function is replaced with Focal-EIoU to address sample imbalance issues and precisely optimize bounding box regression. For tracking, to enhance long-term tracking stability, ReID feature distances are incorporated during the BoT-SORT data association phase. This integrates behavioral category information from YOLO-SGCF, enabling the identification and tracking of abnormal pedestrian behaviors in complex environments. Evaluations on our self-built dataset (covering four abnormal behaviors: Climb, Fall, Fight, Phone) show mAP@50%, precision, and recall reaching 92.2%, 90.75%, and 86.57% respectively-improvements of 3.4%, 4.4%, and 6% over the original model-while maintaining an inference speed of 328.49 FPS. Additionally, generalization testing on the UCSD Ped1 dataset (covering six abnormal behaviors: Biker, Skater, Car, Wheelchair, Lawn, Runner) yielded an mAP score of 92.7%, representing a 1.5% improvement over the original model and outperforming existing mainstream models. Furthermore, the tracking algorithm achieved an MOTA of 90.8% and an MOTP of 92.6%, with a 47.6% reduction in IDS, demonstrating superior tracking performance compared to existing mainstream algorithms.