Synthetic Temporal Anomaly Guided End-to-End Video Anomaly Detection

As surveillance technology is continuously improving, an ever-increasing number of cameras are being deployed everywhere. Relying on manual detection of anomalies through cameras may be unreliable and untimely. Therefore, the application of deep learning in video anomaly detection is being extensively studied. Anomaly Detection (AD) refers to identifying events that deviate from the desired actions. This article discusses representative unsupervised and weakly-supervised learning methods applied to various data types. In these machine learning methods, Generative Adversarial Network, Auto Encoder, Recurrent Neural Network, etc. are broadly adopted for AD. Some renowned and new datasets are reviewed. Furthermore, we also proposed several future directions of research in video anomaly detection.

We propose a novel unsupervised approach based on a two-stage object-centric adversarial framework that only needs object regions for detecting frame-level local anomalies in videos. The first stage consists in learning the correspondence between the current appearance and past gradient images of objects in scenes deemed normal, allowing us to either generate the past gradient from current appearance or the reverse. The second stage extracts the partial reconstruction errors between real and generated images (appearance and past gradient) with normal object behaviour, and trains a discriminator in an adversarial fashion. In inference mode, we employ the trained image generators with the adversarially learned binary classifier for outputting region-level anomaly detection scores. We tested our method on four public benchmarks, UMN, UCSD, Avenue and ShanghaiTech and our proposed object-centric adversarial approach yields competitive or even superior results compared to state-of-the-art methods.

Detecting abnormal activities in real-world surveillance videos is an important yet challenging task as the prior knowledge about video anomalies is usually limited or unavailable. Despite that many approaches have been developed to resolve this problem, few of them can capture the normal spatio-temporal patterns effectively and efficiently. Moreover, existing works seldom explicitly consider the local consistency at frame level and global coherence of temporal dynamics in video sequences. To this end, we propose Convolutional Transformer based Dual Discriminator Generative Adversarial Networks (CT-D2GAN) to perform unsupervised video anomaly detection. Specifically, we first present a convolutional transformer to perform future frame prediction. It contains three key components, i.e., a convolutional encoder to capture the spatial information of the input video clips, a temporal self-attention module to encode the temporal dynamics, and a convolutional decoder to integrate spatio-temporal features and predict the future frame. Next, a dual discriminator based adversarial training procedure, which jointly considers an image discriminator that can maintain the local consistency at frame-level and a video discriminator that can enforce the global coherence of temporal dynamics, is employed to enhance the future frame prediction. Finally, the prediction error is used to identify abnormal video frames. Thoroughly empirical studies on three public video anomaly detection datasets, i.e., UCSD Ped2, CUHK Avenue, and Shanghai Tech Campus, demonstrate the effectiveness of the proposed adversarial spatio-temporal modeling framework.

Anomaly detection in surveillance videos is a challenging task due to the diversity of anomalous video content and duration. In this paper, we consider video anomaly detection as a regression problem with respect to anomaly scores of video clips under weak supervision. Hence, we propose an anomaly detection framework, called Anomaly Regression Net (ARNet), which only requires video-level labels in training stage. Further, to learn discriminative features for anomaly detection, we design a dynamic multiple-instance learning loss and a center loss for the proposed AR-Net. The former is used to enlarge the inter-class distance between anomalous and normal instances, while the latter is proposed to reduce the intra-class distance of normal instances. Comprehensive experiments are performed on a challenging benchmark: ShanghaiTech. Our method yields a new state-of-the-art result for video anomaly detection on ShanghaiTech dataset.

Anomaly detection in videos is the task of identifying frames from a video sequence that depict events that do not conform to expected behavior, which is an extremely challenging task due to the ambiguous and unbounded properties of anomalies. With the development of deep learning, video anomaly detection methods based on deep neural networks have made great progress. The existing methods mainly follow two routes, namely, frame reconstruction and frame prediction. Due to the powerful generalization ability of neural networks, the application of reconstruction-based methods is limited. Recently, anomaly detection methods based on prediction have achieved advanced performance. However, their performance suffers when they cannot guarantee lower prediction errors for normal events. In this paper, we propose a novel future frame prediction model based on a bidirectional retrospective generation adversarial network (BR-GAN) for anomaly detection. To predict a future frame with higher quality for normal events, first, we propose a bidirectional prediction combined with a retrospective prediction method to fully mine the bidirectional temporal information between the predicted frame and the input frame sequence. Then, the intensity and gradient loss between the predicted frame and the actual frame together with an adversarial loss are used for appearance (spatial) constraints. In addition, we propose a sequence discriminator composed of a 3-dimensional (3D) convolutional neural network to capture the long-term temporal relationships between frame sequences composed of predicted frames and input frames; this network plays a crucial role in maintaining the motion (temporal) consistency of the predicted frames for normal events. Such appearance and motion constraints further facilitate future frame prediction for normal events, and thus, the prediction network can be highly capable of distinguishing normal and abnormal patterns. Extensive experiments on benchmark datasets demonstrate that our method outperforms most existing state-of-the-art methods, validating the effectiveness of our method for anomaly detection.

Intelligent video surveillance systems with anomaly detection capabilities are indispensable for outdoor security. Video anomaly detection (VAD) is usually performed by learning patterns representing normal events and declaring an anomaly when an abnormal pattern is encountered. However, the features of normal patterns in a video often vary with time as real-world videos are non-stationary in nature, which makes its handling essential during VAD. To this end, we propose an approach for anomaly detection in videos, where a novel deep temporally recursive differencing network (DDN) diminishes the adverse effects of the non-stationary nature on VAD. The DDN consists of multiple layers of differencing operators of optimized orders, where every two consecutive layers are separated by a suitable nonlinearity. Spatial and temporal features are extracted from nonoverlapping blocks in video frames and fed to the DDN. While the spatial feature is obtained using a pretrained network, our temporal feature computation involves the use of FlowNetS with a new training strategy that does not require ground truth. The features at the output of DDN are used in a predictor based on autoregression and moving average of the regression errors. Then, the predictor's output estimates are compared to the corresponding actual values for anomaly detection, which also involves block-level selection and consistency check. Qualitative evaluation and quantitative comparison with several existing approaches on multiple standard datasets demonstrate the effectiveness of the proposed VAD approach. An ablation study highlighting the significance of the various components of our approach and a hyperparameter analysis are also provided.

Video anomaly detection is of great significance due to its wide applications in video surveillance. Recently, there is a trend of using a video prediction framework to tackle this problem. This kind of methods detects anomalies according to the difference between a predicted frame and its ground truth. However, existing prediction methods lack the consideration of multi-scale temporal constraints on generating future frames. Therefore, based on the prediction framework, this paper proposes a temporal enhanced anomaly detection approach, which designs a generative adversarial network with dual discriminator (frame discriminator and sequence discriminator) to predict future frames. In order to obtain more realistic predictions, other than commonly used spatial constraints and the adversarial penalty from the frame discriminator, we also consider both short-range and long-range motions to impose constraints. Specifically, for short-range motion modeling, we utilize the optical flow loss to ensure temporal continuity over two adjacent frames, while for long-range motion modeling, we design a sequence discriminator to identify fake contained sequences from real sequences, making frames more consistent with their previous consecutive frames. Experiments on three datasets, UCSD Ped1, UCSD Ped2 and Avenue, demonstrate the effectiveness of our method in terms of various evaluation criteria for video anomaly detection.

Video anomaly detection is an essential task in computer vision which attracts massive attention from academia and industry. The existing approaches are implemented in diverse deep learning frameworks and settings, making it difficult to reproduce the results published by the original authors. Undoubtedly, this phenomenon is detrimental to the development of Video Anomaly detection and community communication. In this paper, we present a PyTorch-based video anomaly detection toolbox, namely PyAnomaly that contains high modular and extensible components, comprehensive and impartial evaluation platforms, a friendly manageable system configuration, and the abundant engineering deployment functions. To make it easy-to-use and easy-to-extend, we implement the architecture by hooks and registers functionality. Remarkably, we have reproduced the comparable experimental results of six representative methods as those published by the original authors, and we will release these pre-trained models with more rich configurations. To our best knowledge, the PyAnomaly is the first open-source tool in video anomaly detection and is available at https://github.com/YuhaoCheng/PyAnomaly.

Video surveillance systems are essential for an intelligent and smart traffic monitoring system. Detecting and recognizing traffic anomalies is the foremost task for the safety and security of human lives. The advancement of computer vision technology, especially in deep learning, has paved the way for accurate anomaly detection. This paper presents a novel attention-based variational autoencoder (A-VAE) architecture for detecting anomalies in traffic videos. A-VAE is designed with the help of 2D CNN and BiLSTM layers with an attention mechanism for representation learning. In addition, a shortcut connection is employed between the spatial encoder and the spatial decoder, which helps for better decoding. A-VAE is trained end-to-end on regular video sequences in an unsupervised manner. To check the effectiveness of the proposed A-VAE, two challenging real-world traffic datasets (Crossroad1 and Pedestrian) have been utilized. A-VAE delivers the AUC and EER of 87.4% and 27% in Crossroad1, and 82.2% and 31% on the Pedestrian dataset, respectively. During testing, an improved runtime of 0.0149s (∼ 67 fps) and 0.0192s (∼ 52 fps) has been achieved for the Crossroad1 and Pedestrian datasets, respectively.

Training-free video anomaly detection (VAD) has recently emerged as a scalable alternative to supervised approaches, yet existing methods largely rely on static prompting and geometry-agnostic feature fusion. As a result, anomaly inference is often reduced to shallow similarity matching over Euclidean embeddings, leading to unstable predictions and limited interpretability, especially in complex or hierarchically structured scenes. We introduce MM-VAD, a geometry-aware semantic reasoning framework for training free VAD that reframes anomaly detection as adaptive test-time inference rather than fixed feature comparison. Our approach projects caption-derived scene representations into hyperbolic space to better preserve hierarchical structure and performs anomaly assessment through an adaptive question answering process over a frozen large language model. A lightweight, learnable prompt is optimised at test time using an unsupervised confidence-sparsity objective, enabling context-specific calibration without updating any backbone parameters. To further ground semantic predictions in visual evidence, we incorporate a covariance-aware Mahalanobis refinement that stabilises cross-modal alignment. Across four benchmarks, MM-VAD consistently improves over prior training-free methods, achieving 90.03% AUC on XD-Violence and 83.24%, 96.95%, and 98.81% on UCF-Crime, ShanghaiTech, and UCSD Ped2, respectively. Our results demonstrate that geometry-aware representation and adaptive semantic calibration provide a principled and effective alternative to static Euclidean matching in training-free VAD.

Autoencoders (AEs) have attracted much attention in one-class classification (OCC) based unsupervised anomaly detection. The AEs aim to learn the unity features on targets without involving anomalies and thus the targets are expected to obtain smaller reconstruction errors than anomalies. However, AE-based OCC algorithms may suffer from the overgeneralization of AE and fail to detect anomalies that have similar distributions to target data. To address these issues, a novel within-class constraint based multi-task AE (WC-MTAE) is proposed in this paper. WC-MTAE consists of two different task: one for reconstruction and the other for the discrimination-based OCC task. In this way, the encoder is compelled by the OCC task to learn the more compact encoded feature distribution for targets when minimizing OCC loss. Meanwhile, the within-class scatter based penalty term is constructed to further regularize the encoded feature distribution. The aforementioned two improvements enable the unsupervised anomaly detection by the compact encoded features, thereby addressing the issue of the overgeneralization in AEs. Comparisons with several state-of-the-art (SOTA) algorithms on several non-image datasets and an image dataset CIFAR10 are provided where the WC-MTAE is conducted on 3 different network structures including the multilayer perception (MLP), LeNet-type convolution network and full convolution neural network. Extensive experiments demonstrate the superior performance of the proposed WC-MTAE. The source code would be available in future.

In recent years, weakly supervised multimodal video anomaly detection, which leverages RGB, optical flow, and audio modalities, has garnered significant attention from researchers, emerging as a vital subfield within video anomaly detection. However, previous studies have inadequately explored the role of text modality in this domain. With the proliferation of large-scale text-annotated video datasets and the advent of video captioning models, obtaining text descriptions from videos has become increasingly feasible. Text modality, carrying explicit semantic information, can more accurately characterize events within videos and identify anomalies, thereby enhancing the model's detection capabilities and reducing false alarms. However, text feature extraction challenges anomaly detection. Pre-trained large language models often struggle to effectively capture the nuances associated with anomalies, as their training is based on generalized datasets. Directly fine-tuning the text feature extractor is also challenging, as anomaly-related text descriptions are sparse. Furthermore, due to the varying amounts of information carried by different modalities, issues such as modality redundancy and modality imbalance arise during feature fusion. To address the challenges of text feature extraction and the issues of modality redundancy and imbalance, we propose a novel text-guided weakly supervised multimodal video anomaly detection framework. Specifically, we introduce an in-context learning based multi-stage text augmentation mechanism to generate high-quality anomaly text samples. These high-quality samples are then used to fine-tune the text feature extractor, aiming to obtain a more effective text feature extractor for anomaly detection. Additionally, we present a multi-scale bottleneck Transformer fusion module to enhance multimodal integration, utilizing a set of reduced bottleneck tokens to progressively transmit compressed information across modalities, aiming to address the issues of modality redundancy and imbalance. Experimental results on large-scale datasets UCF-Crime and XD-Violence demonstrate that our proposed approach achieves state-of-the-art performance. This project is publicly available at https://shengyangsun.github.io/TGMVAD.

Video anomaly detection (VAD) is of great importance for a variety of real-time applications in video surveillance. Most deep learning-based anomaly detection algorithms adopt a one-class learning scheme to train a classifier using only normal data to distinguish between normal and abnormal events during the test phase. However, these methods, whether they are reconstruction or prediction models, commonly face the challenge of the model’s overly strong representation capability, which leads to excessive fitting of abnormal events and thus limits the performance of the model in diverse scenarios. To address these challenges, this work develops a novel residual autoencoder and generative adversarial network-based 3-D convolutional network, called READ3D-net, for anomaly detection. An adaptive multimodal pseudoanomaly generator is developed to simulate and generate diverse pseudoanomalies, aiming to enhance the model’s ability to extract the features with regard to “abnormal” behaviors while reducing the interference of background on the detection performance. In addition, residual structures are incorporated into the design of a reconstruction-based autoencoder model to enhance its feature extraction and discriminative capabilities. To further improve the model’s reconstruction ability on normal data, a dynamic generative adversarial strategy is proposed for effective feature learning. Extensive experiments conducted on public benchmark datasets demonstrate that the proposed model is more competitive than the state-of-the-art methods in video anomaly detection tasks, fully validating the effectiveness and practicality of the proposed approach.

Video anomaly detection plays a crucial role in various fields such as surveillance, health monitoring, and industrial quality control. This research paper introduces a novel contribution to the field by presenting MaskedConv3D layers within a modified UNet architecture. These MaskedConv3D layers provide a unique approach to information propagation in three-dimensional video data by selectively masking temporal regions of convolutional kernels. By incorporating these layers into the skip connections of the UNet, the model gains the ability to infer missing information in the temporal domain based on the surrounding context. This innovative mechanism enhances the preservation of spatial and temporal details, addressing the challenge of effectively detecting anomalies in video data. The proposed methodology is evaluated on popular video datasets, showcasing its effectiveness in capturing intricate patterns and contexts. The results highlight the superiority of the modified UNet with MaskedConv3D layers compared to traditional approaches. Overall, this research introduces a novel technique for information propagation in video data and demonstrates its potential for advancing video anomaly detection.

Video anomaly detection (VAD) is a challenging computer vision task that aims at identifying the events that do not conform to expected behaviors in a video. Training an unsupervised learning model on normal data is one of the most prevalent VAD paradigms owning to rare abnormal snippets. However, anomaly related and irrelated information sharing feature space will degrade the representation ability of the model for abnormal events. In this article, we propose a novel unsupervised video anomaly detection method, named Cross-Modal Two-Stream Target Focused Network (CMTSTFN) to focus on the anomaly related target and alleviate the impact of anomaly irrelated information on the model. Firstly, we use the global-local optical flow as a priori knowledge to blur the anomaly irrelated information in the image domain by Gaussian blur kernel. More importantly, conditioned on the blurred video frame, we then employ a dual cross-modal attention module to reconstruct the fused image feature map and optical flow feature map, respectively. Therefore, both spatial and temporal cues of abnormal events could further deteriorate the quality of the final reconstructed frame, making the anomalies more detectable. Experimental results on UCSD Pred1 and UCSD Pred2 demonstrate the effectiveness of the proposed CMTSTFN.