Real Time Abnormal Event Detection in Pedestrain Pathways Using Deep Learning

Abnormal event detection, also known as anomaly detection, is one challenging task in security video surveillance. It is important to develop effective and robust movement representation models for global and local abnormal event detection to fight against factors such as occlusion and illumination change. In this paper, a new algorithm is proposed. It can locate the abnormal events on one frame, and detect the global abnormal frame. The proposed algorithm employs a sparse measurement matrix designed to represent the movement feature based on optical flow efficiently. Then, the abnormal detection mission is constructed as a one-class classification task via merely learning from the training normal samples. Experiments demonstrate that our algorithm performs well on the benchmark abnormal detection datasets against state-of-the-art methods.

With the booming development of tourism, travel security problems are becoming more and more prominent. Congestion, stampedes, fights and other tourism emergency events occurred frequently, which should be a wake-up call for tourism security. Therefore, it is of great research value and application prospect to real-time monitor tourists and detect abnormal events in tourism surveillance video by using computer vision and video intelligent processing technology, which can realize the timely forecast and early warning of tourism emergencies. At present, although most of the video-based abnormal event detection methods work well in simple scenes, there are often problems such as low detection rate and high false positive rate in complex motion scenarios, and the detection of abnormal events can’t be processed in real time. To tackle these issues, we propose an abnormal event detection model in tourism video based on salient spatio-temporal features and sparse combination learning, which has good robustness and timeliness in complex motion scenarios and can be adapted to real-time anomaly detection in practical applications. Specifically, spatio-temporal gradient model is combined with foreground detection to extract 3D gradient features on the foreground target of video sequence as the salient spatio-temporal features, which can eliminate the interference of the background. Sparse combination learning algorithm is used to establish the abnormal event detection model, which can realize the real-time detection of abnormal events. In addition, we construct a new ScenicSpot dataset with 18 video clips (5964 frames) containing both normal and abnormal events. The experimental results on ScenicSpot dataset and two standard benchmark datasets show that our method can realize the automatic detection and recognition of tourists’ abnormal behavior, and has better performance compared with the classical methods.

With the huge success of applying deep learning (DL) methodologies to image recognition and natural language processing in recent years, researchers are now keen to use them in the machine condition monitoring (MCM) context. There are numerous papers in applying various DL techniques, such as auto-encoder, restricted Boltzmann machine, convolutional neural network and recurrent neural network, etc., to MCM problems ranging from component-level condition monitoring (machine tool wear prediction, bearing fault diagnosis and classification and hydraulic pump fault diagnosis) to system-level health management (aircraft and spacecraft diagnosis). In this paper, we give a brief overview in the area of DL for MCM with a focus on reviewing the most recent papers published since 2019. In Part 1, we present some critical views regarding whether any breakthrough has been achieved from an MCM domain expert perspective, with the main conclusion that DL has great potential for MCM applications, and a major breakthrough could come soon since the shortfalls lie more in data than in the DL methodologies. Our overall impression is that (a) DL models are not really showing their great potentials with only a small training data; (b) faulty-condition data is hard to come by for training DL, but normal condition data is abundant, so anomaly detection makes more sense; (c) applying DL only to the Case Western Reserve University (CWRU) bearing fault dataset is not sufficient for real world industrial applications as it was from a very simple test rig, and applying DL to data from complex systems like helicopter gearbox data may deliver much more convincing results. In Part 2, we enhance the main conclusion of the critical review with supplement views and a case study on analysing Bell-206B helicopter main gearbox planet bearing failure data using some traditional MCM techniques in contrast to applying the long short-term memory (LSTM) DL method. We can conclude from the case study that the DL-based methods are not necessarily always superior to the traditional MCM techniques for dataset from moderately complex machinery.

We investigated cognitively demanding tasks on patterns of human gait in healthy adults with a deep learning methodology that learns from raw gait data. Age-related differences were analyzed in dual-tasks in a cohort of 69 cognitively healthy adults organized in stratified groups by age. A novel spatio-temporal deep learning methodology was introduced to effectively classify dual-tasks from spatio-temporal raw gait data, obtained from a unique tomography floor sensor. The approach outperformed traditional machine learning approaches. The most favorable classification F-score obtained was of 97.3% in dual-tasks in a young age group experiment for a total of 12 users. The deep machine learning methodology outperformed classical machine learning methodologies by 63.5% in the most favorable case. Finally, a 2D manifold representation was obtained from trained deep learning models' data, to visualize and identify clusters from features learned by the deep learning models. This study demonstrates a novel approach to dual-task research by proposing a data-driven learning methodology with stratified age-groups.

In this paper, we propose a joint learning of spatio-temporal representation based on 3D deep convolutional neural network for simultaneous representation of appearance and motion information in 3D volumes which are extracted from the multiple consecutive frames, and an end-to-end learning framework to detect abnormal events in surveillance scenes. By using the joint learning approach, the proposed framework can detect various abnormal events which can appear with diverse motion and appearance patterns. The proposed framework detects abnormal events in each volume by analyzing the spatio-temporal representation trained by the joint learning method. This volume-level event detection approach makes it possible to localize an abnormal event. We verify the proposed joint learning and the framework on the publicly available abnormal event datasets containing UMN dataset, UCSD dataset, and subway dataset, by comparing it with existing state-of-the-art methods. The experimental results demonstrate that the proposed joint learning and event detection method not only detect various abnormal events more efficiently but also localize anomalous regions more accurately.

In order to recognize the abnormal event, such as emergency or panic, happened in public scenes timely, an algorithm based on features extraction and BP Adaboost to detect abnormal frame event from surveillance video of complex situation is proposed. The proposed method detects an abnormal event where people are running, and this panic situation is simulated by the frame in a video. Experiments show that the method can distinguish and detect the abnormal event effectively and efficiently, which has potentiality to be used in the real public monitoring.

These days consciousness about public safety increases and CCTV cameras are installed at almost all public places. But generally automatic smart surveillance systems are not available. In this manuscript, emphasis is given to detect and classify abnormal events in surveillance video especially in crowd environments. Abnormal event detection is a challenging task because the definition of abnormality is subjective. A normal event in one situation can be considered an abnormal event in another case. In the surveillance video with a dense crowd, automatic anomaly detection becomes very difficult because of clutter and severe occlusion. This manuscript represents CNN (Convolutional Neural Network) and CNN-LSTM (Convolutional Neural Network-Long Short-Term Memory) based approaches for detection and classification of abnormal events. The CNN architecture is developed from scratch and can be used for spatial domains. LSTM architecture is developed for the temporal domain. Feature sequences are generated using CNN model and given as input to LSTM model. Experiments are carried out using five different publicly available benchmark datasets. The performance is measured by accuracy and area under the ROC (receiver operating characteristic) curve (AUC). CNN-LSTM approach works better than only CNN.

Intelligent Video Investigation is on nice interest in trade applications because of increasing demand to scale back the force of analyzing the large-scale video information. Sleuthing the abnormal events from crowded video scenes offer varied difficulties. Initially, an oversized variety of moving persons will simply distract the native anomaly detector. Secondly it’s tough to model the abnormal events in real time. Thirdly, the inaccessibility of ample samples of coaching information for abnormal events ends up in problem in sturdy detection of abnormal events. Our planned system provides a peculiar approach to find anomaly in crowded video scenes. We are initially divide the video frame into patches and apply the Difference-of-Gaussian (DoG) filter to extract edges. Then we work out Multiscale Histogram of Optical Flow (MHOF) and Edge directed bar chart (EOH) for every patch. Then exploitation of Normalized Cuts (NCuts) and Gaussian Expectation-Maximization (GEM) techniques, and to cluster the similar patches into cluster and assign the motion context. Finally exploitation of k-Nearest neighbor (k-NN) search, and establish the abnormal activity at intervals in the crowded scenes. Our spatio-temporal anomaly search system helps to boost the accuracy and computation time for detection of irregular patterns. This technique is helpful for investigation, trade specific and market applications like public transportation, enforcement, etc.KeywordsMultiscale Histogram of Optical FlowGaussian Expectation MaximizationSpatio-temporal anomaly search

Alzheimer's Disease (AD) is a serious neurological illness that causes memory loss gradually by destroying brain cells. This deadly brain illness primarily strikes the elderly, impairing their cognitive and bodily abilities until brain shrinkage occurs. Modern techniques are required for an accurate diagnosis of AD. Machine learning has gained attraction in the medical field as a means of determining a person's risk of developing AD in its early stages. One of the most advanced soft computing neural network-based Deep Learning (DL) methodologies has garnered significant interest among researchers in automating early-stage AD diagnosis. Hence, a comprehensive review is necessary to gain insights into DL techniques for the advancement of more effective methods for diagnosing AD. This review explores multiple biomarkers associated with Alzheimer's Disease (AD) and various DL methodologies, including Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), The k-nearest-neighbor (k-NN), Deep Boltzmann Machines (DBM), and Deep Belief Networks (DBN), which have been employed for automating the early diagnosis of AD. Moreover, the unique contributions of this review include the classification of ATN biomarkers for Alzheimer's Disease (AD), systemic description of diverse DL algorithms for early AD assessment, along with a discussion of widely utilized online datasets such as ADNI, OASIS, etc. Additionally, this review provides perspectives on future trends derived from critical evaluation of each variant of DL techniques across different modalities, dataset sources, AUC values, and accuracies.

Health state estimation of cutting tools is a key component for maintaining the reliability and stability of the manufacturing process. Although deep learning (DL) based tool wear indirect monitoring methods have been presented, most of them require a substantial amount of data to avoid high misclassification rates. To this end, this article proposes a novel estimation approach based on data-enhanced deep transfer learning (TL) methodology, which integrates multivariate data enhancements, deep convolution network, and generative adversarial learning for state assessment of the tool wear under small sample size. The proposed method includes three innovative points: 1) a multi-step data enhancement method is employed to augment the wavelet scalogram data as model input; 2) a novel training process is adopted to obtain optimal hyperparameters of TL-AlexNet and TL-VGGNet-16 model; 3) a deep transfer learning model for the enhanced data is presented to adaptively estimate the health state of the cutting tools. The proposed approach only requires a small quantity of data to achieve satisfactory estimation performance compared with other DL methods through experimental validation, and the average prediction accuracy for each stage is greater than 95%, which demonstrates the effectiveness and superiority of the presented model.

Event detection plays an essential role in video content analysis. On the other hand, according to our analysis, the coding structures in new video coding standard High Efficient Video Coding (HEVC) have a high correlation with video contents. Hence there is large potential to identify events by reusing coding structures in HEVC, which can save a huge amount of computational resources. In this paper, we proposed a new compressed-domain feature for abnormal event detection, namely Motion Intensity Count (MIC), which makes use of motion vectors, coding unit and prediction unit modes in HEVC with little computational cost. MIC can well predict the normal paths of moving objects, which enables us to identify motions in unexpected locations where abnormal events are likely to happen. Our experiments show that MIC can correctly detect abnormal events at about 1250 fps.

Abnormal event detection with the lowest latency is an indispensable function for safety-critical systems, such as cyber defense systems. However, as systems become increasingly complicated, conventional sequential event detection methods become less effective, especially when we need to define indicator metrics from complicated data manually. Although deep neural networks (DNNs) have been used to handle heterogeneous data, the theoretic assurability and explainability are still insufficient. This article provides a holistic framework for the quickest and sequential detection of abnormalities and time-dependent abnormal events. We explore the latent space characteristics of zero-bias neural networks considering the classification boundaries and abnormalities. We then provide a novel method to convert zero-bias DNN classifiers into performance-assured binary abnormality detectors. Finally, we provide a sequential quickest detection (QD) scheme that provides the theoretically assured lowest abnormal event detection delay under false alarm constraints using the converted abnormality detector. We verify the effectiveness of the framework using real massive signal records in aviation communication systems and simulation. Codes and data are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/pcwhy/AbnormalityDetectionInZbDNN</uri> .

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.

Existing methods have addressed the issue of detecting abnormal events at a smart home for medical care or security monitoring services extensively in the past decades. However, most of approaches use wearable sensors that require users to be equipped with the sensor devices at every moment. If the monitored users stop or pause the sensors, any abnormal events are not able to be detected. The use of non-wearable and non-intrusive sensors (e.g., IP cameras) is necessary for providing better user experiences and achieving sustainable and reliable detection model. However, it is still very challenging to analyze such non-wearable sensor data with a high accuracy. In this work, we propose an event detection model using a Bayesian Network. We first obtain the features by analyzing the daily videos and audios captured from different angles by multiple IP cameras at a smart home. These features are then used to construct a Bayesian network. We propose a probabilistic graph model where the dependence relations are defined in the graph as opposed to the naive Bayesian network. The experiments are presented to demonstrate the performance and utility of our model.

We address the problem of abnormal event detection from trajectory data. In this paper, a new adversarial approach is proposed for building a deep neural network binary classifier, trained in an unsupervised fashion, that can distinguish normal from abnormal trajectory-based events without the need for setting manual detection threshold. Inspired by the generative adversarial network (GAN) framework, our GAN version is a discriminative one in which the discriminator is trained to distinguish normal and abnormal trajectory reconstruction errors given by a deep autoencoder. With urban traffic videos and their associated trajectories, our proposed method gives the best accuracy for abnormal trajectory detection. In addition, our model can easily be generalized for abnormal trajectory-based event detection and can still yield the best behavioural detection results as demonstrated on the CAVIAR dataset.