MemATr: An Efficient and Lightweight Memory-augmented Transformer for Video Anomaly Detection

Anomaly detection in crowd surveillance videos is a critical task for ensuring public safety and security. In this research paper, we propose a comprehensive framework for anomaly detection using deep learning techniques, specifically Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Generative Adversarial Networks (GANs). Our objective is to narrow the gap between computational complexity and detection effectiveness while satisfying the demand for explainability in anomaly detection systems. The proposed framework leverages the UCSD Ped2 dataset and focuses on incorporating spatial constraints to enhance anomaly detection performance. We introduce novel approaches for feature extraction and anomaly detection using GANs, aiming to capture essential spatial and temporal information while reducing computational complexity. Specifically, we develop encoders within the GAN framework to map input data to a lower-dimensional latent space and perform anomaly detection based on the similarity between latent codes. Through extensive experimentation and evaluation, we compare the performance of RNNs, CNNs, and GANs in detecting anomalies in crowd surveillance videos. Various evaluation metrics, including precision, recall, F1 score, ROC-AUC, and PR-AUC, is considered to assess the effectiveness of each approach. Additionally, they studied hyperparameters such as learning rate, batch size, network architecture, and GAN-specific parameters to optimize anomaly detection performance. Our results demonstrate the usefulness of the proposed framework in achieving robust and explainable anomaly detection in crowd surveillance videos. By combining advanced deep learning techniques with spatial constraints and explainability considerations, our research contributes to the development of more reliable and interpretable anomaly detection systems for real-world applications.

Anomaly detection in videos is challenging due to the scarcity and variance in positive samples. Current anomaly detection methods can be categorized into reconstruction models and future frame prediction-based models. However, reconstruction models might be exceptionally adapted to abnormal events due to the learning capacity and generalization ability of deep neural networks, whereas prediction-based methods can be sensitive to noise. In this study, we propose an anomaly detection model based on the latent feature space, which combines advantages from both sides. We argue that the constraints in the latent feature space can promote reconstruction; moreover, the optical flow is also considered to introduce temporal information. We use SPyNet for accurate and efficient optical flow estimation. We extensively validate our method on the UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets. The results demonstrated the feasibility of the proposed method and the benefit of utilizing information in the latent feature space.

Anomaly detection in videos is the task of identifying frames from a video sequence that depict events that do not conform to expected behavior, which is an extremely challenging task due to the ambiguous and unbounded properties of anomalies. With the development of deep learning, video anomaly detection methods based on deep neural networks have made great progress. The existing methods mainly follow two routes, namely, frame reconstruction and frame prediction. Due to the powerful generalization ability of neural networks, the application of reconstruction-based methods is limited. Recently, anomaly detection methods based on prediction have achieved advanced performance. However, their performance suffers when they cannot guarantee lower prediction errors for normal events. In this paper, we propose a novel future frame prediction model based on a bidirectional retrospective generation adversarial network (BR-GAN) for anomaly detection. To predict a future frame with higher quality for normal events, first, we propose a bidirectional prediction combined with a retrospective prediction method to fully mine the bidirectional temporal information between the predicted frame and the input frame sequence. Then, the intensity and gradient loss between the predicted frame and the actual frame together with an adversarial loss are used for appearance (spatial) constraints. In addition, we propose a sequence discriminator composed of a 3-dimensional (3D) convolutional neural network to capture the long-term temporal relationships between frame sequences composed of predicted frames and input frames; this network plays a crucial role in maintaining the motion (temporal) consistency of the predicted frames for normal events. Such appearance and motion constraints further facilitate future frame prediction for normal events, and thus, the prediction network can be highly capable of distinguishing normal and abnormal patterns. Extensive experiments on benchmark datasets demonstrate that our method outperforms most existing state-of-the-art methods, validating the effectiveness of our method for anomaly detection.

Anomaly detection in surveillance videos is a challenging task due to the diversity of anomalous video content and duration. In this paper, we consider video anomaly detection as a regression problem with respect to anomaly scores of video clips under weak supervision. Hence, we propose an anomaly detection framework, called Anomaly Regression Net (ARNet), which only requires video-level labels in training stage. Further, to learn discriminative features for anomaly detection, we design a dynamic multiple-instance learning loss and a center loss for the proposed AR-Net. The former is used to enlarge the inter-class distance between anomalous and normal instances, while the latter is proposed to reduce the intra-class distance of normal instances. Comprehensive experiments are performed on a challenging benchmark: ShanghaiTech. Our method yields a new state-of-the-art result for video anomaly detection on ShanghaiTech dataset.

Anomaly detection in video is a challenging computer vision problem. Due to the lack of anomalous events at training time, anomaly detection requires the design of learning methods without full supervision. In this paper, we approach anomalous event detection in video through self-supervised and multi-task learning at the object level. We first utilize a pre-trained detector to detect objects. Then, we train a 3D convolutional neural network to produce discriminative anomaly-specific information by jointly learning multiple proxy tasks: three self-supervised and one based on knowledge distillation. The self-supervised tasks are: (i) discrimination of forward/backward moving objects (arrow of time), (ii) discrimination of objects in consecutive/intermittent frames (motion irregularity) and (iii) reconstruction of object-specific appearance information. The knowledge distillation task takes into account both classification and detection information, generating large prediction discrepancies between teacher and student models when anomalies occur. To the best of our knowledge, we are the first to approach anomalous event detection in video as a multi-task learning problem, integrating multiple self-supervised and knowledge distillation proxy tasks in a single architecture. Our lightweight architecture outperforms the state-of-the-art methods on three benchmarks: Avenue, ShanghaiTech and UCSD Ped2. Additionally, we perform an ablation study demonstrating the importance of integrating self-supervised learning and normality-specific distillation in a multi-task learning setting.

We propose a novel unsupervised approach based on a two-stage object-centric adversarial framework that only needs object regions for detecting frame-level local anomalies in videos. The first stage consists in learning the correspondence between the current appearance and past gradient images of objects in scenes deemed normal, allowing us to either generate the past gradient from current appearance or the reverse. The second stage extracts the partial reconstruction errors between real and generated images (appearance and past gradient) with normal object behaviour, and trains a discriminator in an adversarial fashion. In inference mode, we employ the trained image generators with the adversarially learned binary classifier for outputting region-level anomaly detection scores. We tested our method on four public benchmarks, UMN, UCSD, Avenue and ShanghaiTech and our proposed object-centric adversarial approach yields competitive or even superior results compared to state-of-the-art methods.

Reducing the inference time of Deep Neural Networks (DNNs) is critical when running time sensitive applications on mobile devices. Existing research has shown that partitioning a DNN and offloading a part of its computation to cloud servers can reduce the inference time. The single DNN partition problem has been extensively investigated recently. However, in real-world applications, a mobile device usually generates multiple DNN inference jobs simultaneously, and little attention has been paid to this case. We aim to minimize the makespan of multiple DNNs by jointly optimizing their partitioning and scheduling. Our observations show that the local computation time on a mobile device follows an increasing function, while the communication workload for offloading is usually decreasing as more DNN layers are computed. Based on this, we first relax our problem on continuous domain and show that partitioning all line-structure DNNs at the same layer is sufficient for makespan optimization. Then, for the discrete domain, two types of partitions are sufficient when the time difference between two adjacent partition layers is not drastic, subject to a given condition. An algorithm based on the binary search that efficiently finds optimal partition layers is illustrated. We also extend our approach to general-structure DNNs and offer a heuristic solution. Experiments have been conducted to evaluate the performance of different partition and scheduling methods on sample DNNs. Results validate the optimality of our theoretical results.

Intelligent video surveillance systems with anomaly detection capabilities are indispensable for outdoor security. Video anomaly detection (VAD) is usually performed by learning patterns representing normal events and declaring an anomaly when an abnormal pattern is encountered. However, the features of normal patterns in a video often vary with time as real-world videos are non-stationary in nature, which makes its handling essential during VAD. To this end, we propose an approach for anomaly detection in videos, where a novel deep temporally recursive differencing network (DDN) diminishes the adverse effects of the non-stationary nature on VAD. The DDN consists of multiple layers of differencing operators of optimized orders, where every two consecutive layers are separated by a suitable nonlinearity. Spatial and temporal features are extracted from nonoverlapping blocks in video frames and fed to the DDN. While the spatial feature is obtained using a pretrained network, our temporal feature computation involves the use of FlowNetS with a new training strategy that does not require ground truth. The features at the output of DDN are used in a predictor based on autoregression and moving average of the regression errors. Then, the predictor's output estimates are compared to the corresponding actual values for anomaly detection, which also involves block-level selection and consistency check. Qualitative evaluation and quantitative comparison with several existing approaches on multiple standard datasets demonstrate the effectiveness of the proposed VAD approach. An ablation study highlighting the significance of the various components of our approach and a hyperparameter analysis are also provided.

Anomaly detection in surveillance footage is crucial for ensuring protection and safety standards, as it enables the timely identification of unusual or suspicious activities. Recent literature has shown the emergence of graph and hypergraph (HG)-based matching algorithms for object tracking in video frames, facilitating anomaly detection. While these techniques incorporate sampling methods to enhance pace, the task of balancing accuracy and efficiency remains unresolved, particularly when detecting anomalies during simultaneous object tracking. This paper addresses this gap by proposing a unified framework that integrates Game-Theoretic Hypergraph Matching (GTHG) with a Convolutional Autoencoder (CAE). Unlike existing methods that treat tracking and detection separately, the proposed approach combines structural consistency and appearance reconstruction to improve both detection accuracy and computational performance. The proposed method has been tested with a chain of benchmarked films and video clips, and a detailed account of matching between successive frames has been provided. Evaluation metrics, including the Regularity Score, Receiver Operating Characteristics (ROC), and Area Under the Curve (AUC), assess the accuracy of anomaly detection across multiple datasets. Our method achieves an area under the curve (AUC) of 88.7%, 91.2%, and 86.6% on the UCSD Ped1, UCSD Ped2, and CUHK Avenue Datasets, respectively, surpassing the performance of many existing models.

Detecting abnormal activities in real-world surveillance videos is an important yet challenging task as the prior knowledge about video anomalies is usually limited or unavailable. Despite that many approaches have been developed to resolve this problem, few of them can capture the normal spatio-temporal patterns effectively and efficiently. Moreover, existing works seldom explicitly consider the local consistency at frame level and global coherence of temporal dynamics in video sequences. To this end, we propose Convolutional Transformer based Dual Discriminator Generative Adversarial Networks (CT-D2GAN) to perform unsupervised video anomaly detection. Specifically, we first present a convolutional transformer to perform future frame prediction. It contains three key components, i.e., a convolutional encoder to capture the spatial information of the input video clips, a temporal self-attention module to encode the temporal dynamics, and a convolutional decoder to integrate spatio-temporal features and predict the future frame. Next, a dual discriminator based adversarial training procedure, which jointly considers an image discriminator that can maintain the local consistency at frame-level and a video discriminator that can enforce the global coherence of temporal dynamics, is employed to enhance the future frame prediction. Finally, the prediction error is used to identify abnormal video frames. Thoroughly empirical studies on three public video anomaly detection datasets, i.e., UCSD Ped2, CUHK Avenue, and Shanghai Tech Campus, demonstrate the effectiveness of the proposed adversarial spatio-temporal modeling framework.

The growing interest in deep learning approaches to video surveillance raises concerns about the accuracy and efficiency of neural networks. However, fast and reliable detection of abnormal events is still a challenging work. Here, we introduce a two-stream approach that offers an autoencoder-based structure for fast and efficient detection to facilitate anomaly detection from surveillance video without labeled abnormal events. Furthermore, we present post hoc interpretability of feature map visualization to show the process of feature learning, revealing uncertain and ambiguous decision boundaries in the video sequence. Experimental results on Avenue, UCSD Ped2, and Subway datasets show that our method can detect abnormal events well and explain the internal logic of the model at the object level.

Facing the problems of low working efficiency, poor accuracy and insufficient stability of traditional motion detection algorithms and machine learning algorithms in pedestrian detection, this paper will take artificial intelligence technology as the core, adopt R-CNN series deep learning models, combine Support Vector Machine (SVM) classifier, and rely on TensorFlow deep learning framework to realize pedestrian detection and recognition in video and picture files with the support of OpengCV Open Source Visual Library. Combined with ASP.NET framework, C# language is used to complete the design and development of each functional module, improve the deployment of the corresponding API interface, and build a front-end interactive interface to form a pedestrian detection system based on Web. The overall design of the system chooses B/S architecture, which allows users to access the Web Server through simple request operation to complete the detection of video image data in the database and return the detection results to the front page for display. The system will greatly improve the precision and accuracy of pedestrian detection and make a positive attempt to further expand the application scenarios of pedestrian detection in video images.

Video anomaly detection is of great significance due to its wide applications in video surveillance. Recently, there is a trend of using a video prediction framework to tackle this problem. This kind of methods detects anomalies according to the difference between a predicted frame and its ground truth. However, existing prediction methods lack the consideration of multi-scale temporal constraints on generating future frames. Therefore, based on the prediction framework, this paper proposes a temporal enhanced anomaly detection approach, which designs a generative adversarial network with dual discriminator (frame discriminator and sequence discriminator) to predict future frames. In order to obtain more realistic predictions, other than commonly used spatial constraints and the adversarial penalty from the frame discriminator, we also consider both short-range and long-range motions to impose constraints. Specifically, for short-range motion modeling, we utilize the optical flow loss to ensure temporal continuity over two adjacent frames, while for long-range motion modeling, we design a sequence discriminator to identify fake contained sequences from real sequences, making frames more consistent with their previous consecutive frames. Experiments on three datasets, UCSD Ped1, UCSD Ped2 and Avenue, demonstrate the effectiveness of our method in terms of various evaluation criteria for video anomaly detection.

To tackle the issue of low detection accuracy in insulator images caused by intricate backgrounds and small defect sizes, as well as the requirement for real-time detection on embedded and mobile devices, this research introduces the YOLOv5s-KE model. Integrating multiple strategies, YOLOv5s-KE aims to boost detection accuracy significantly. Initially, an enhanced anchor generation method utilizing the K-means++ algorithm is proposed to generate more appropriate anchor boxes for insulator defects. Moreover, an attention mechanism is integrated into both the backbone and neck networks to enhance the model’s capacity to focus on defect features and resist interference. To improve the detection of small defects, the EIoU loss function is implemented in place of the original CIoU loss function. In order to meet the real-time detection needs on embedded and mobile devices, the model is further refined through the integration of Ghost convolution for lightweight feature extraction and a linear transformation to reduce the computational burden of standard convolution. A channel pruning strategy is deployed to optimize the sparsely trained network, diminishing redundancy, and improving model generalization. Additionally, the CARAFE operator replaces the original upsampling operator to minimize model parameters and elevate detection speed. Experimental outcomes demonstrate that YOLOv5s-KE achieves a detection accuracy of 92.3% on the Chinese transmission line insulator dataset, marking a 5.2% enhancement over the original YOLOv5s. The streamlined version of YOLOv5s-KE achieves a detection speed of 94.3 frames per second, indicating an improvement of 30.1 frames per second compared to the original model. Model parameters are condensed to 9.6 M, resulting in a detection accuracy of 91.1%. This study underscores the precision and efficiency of the proposed approach, suggesting that the advanced strategies explored introduce novel possibilities for insulator defect detection.

This study focuses on the application of neural networks for anomaly detection in microservice architectures. The increasing complexity of modern microservice systems makes traditional anomaly detection methods less effective, as they struggle to adapt to dynamic environments and large volumes of data. Neural networks offer a promising solution due to their ability to model complex patterns and detect subtle deviations that may indicate system failures or security threats. The aim of this work is to compare the effectiveness of different neural network-based approaches to anomaly detection, evaluate their strengths and weaknesses, and determine the most suitable method for practical implementation. To achieve this, several key tasks were outlined: an in-depth review of contemporary anomaly detection techniques applied to microservices; an analysis of neural network-based models, with a particular focus on autoencoders and recurrent neural networks (RNNs); a comparative assessment of these methods in terms of accuracy, computational efficiency, and robustness to noise; an exploration of hybrid models that combine different techniques to improve detection performance; and the formulation of recommendations for selecting the optimal approach based on specific application scenarios. This research examines the use of deep neural networks for detecting behavioral anomalies in microservice environments. Various architectures are tested and compared to identify trade-offs between detection accuracy, processing speed, and resistance to false positives. The findings indicate that deep neural networks, particularly autoencoders and recurrent models, achieve high precision in identifying anomalies, but their efficiency varies depending on the architecture and the nature of the dataset. The study highlights that while autoencoders effectively capture deviations in data distribution, recurrent networks excel in detecting temporal anomalies in sequential microservice interactions. The conclusions confirm that neural networks provide a powerful tool for anomaly detection in microservice monitoring, significantly outperforming conventional statistical approaches. However, challenges remain, including the need for adaptive models capable of learning in real time and optimizing computational resources. Further research should focus on hybrid approaches that integrate multiple neural architectures to enhance detection accuracy and scalability. The development of self-learning and auto-tuning models could also improve adaptability to evolving microservice behaviors, ensuring robust and efficient anomaly detection in complex distributed systems.