Object Detection based Approach for an Efficient Video Summarization with System Statistics over Cloud

An efficient video summarization for surveillance system using normalized k-means and quick sort method

During past few years’ significant usage of video monitoring has been observed in several places. Video monitoring has always helped in improvement of security specific aspects. These systems provide major helping hand in monitoring through multiple object detection methods. However, there are certain challenges related to weather conditions which needs to be addressed. In view of this in this paper, we study multiple object detection in surveillance videos through hybrid deep gaussian mixture ensembles. This method integrates multiple deep gaussian mixture components to form ensembles which helps to develop multiple object detection solutions in surveillance videos. The computational system consists of ensemble of three major deep gaussian mixture components. The ensemble pipeline is formed with augmenting, bagging and stacking methods. The ensemble consists of hybrid object detection paradigm with steps background analysis, video pre-processing, data integration, object detection, foreground detection and post-processing. The computational pipeline is successfully experimented with ViSOR and CDnet 2014 benchmarked datasets. All results are validated with accuracy, precision, sensitivity or recall, specificity, F1-Score and RMSE metrics. Several comparative studies are performed with state-of-the-art methods as well as baselines. The experimental results demonstrate superiority of this method in comparison with other methods. Ablation studies have also been performed here with superior results. This system presents robustness in multiple object detection for real life challenges such as sudden illumination variation, shadow presence, long term occlusion and formation of dynamic backgrounds. It provides cost-effective, more profitable, efficient and sustainable real time multiple object detection solutions

Video summarization focuses on creating concise versions of lengthy videos by preserving essential content. This study reveals a multimodal machine learning strategy that integrates visual and auditory information by using pretrained Gated Recurrent Neural Network architectures referred to as GARNN, combining Gated Recurrent Units (GRUs) and AlexNet, to extract audio, image, and visual features. The keyframe detection is improved by removing redundant frames and applying motion-compensated feature reduction, followed by optional PCA-based dimensionality reduction. An adversarial encoder-based Long Short-Term Memory (AE-LSTM) classifier is employed for temporal modeling by achieving high accuracy in summarization. The results got evaluated by using sensitivity, F-scores, and positive predictive values, and the method attained an average F-score of 0.985. A gated AlexNet is introduced in a multimodal GARNN-AE-LSTM framework, where motion-compensated PCA-based reduction eliminates redundancy, GRUs record temporal progression, and gating fine-tunes spatial feature selection, all of which contribute to a more accurate and efficient video summarization system. The improved F1 score demonstrates the model's effectiveness in generating accuracy in video summaries by creating a meaningful video. This approach highlights the potential of multimodal feature extraction and advanced deep learning techniques for robust video analysis and compression.

Many televiewers and soccer fans choose to watch summaries of football games as watching a full soccer match requires a lot of time. Generally, the number of soccer matches broadcasted every weekend and during the week is very important. Taking the example of the most famous soccer leagues (Spain, England, Italy, Germany, French), we've ended up with fifty of soccer matches released every week (75 hours of soccer broadcasting): hence the need for developing a system of video summarization. We note that soccer video summarization is complete in the sense that it contains important events (significant action) extracted from soccer matches. Habitually, the soccer match summarization is done manually, however, this requires a considerable amount of time. For that reason, it is necessary to have a means for doing the soccer match summarization automatically. Several works have been developed for video soccer summarization. However, the previous approaches use only the video content to create a summary of the soccer video. Owing to the wide semantic disparity between low-level features and high-level events, it is not evident to come up with a generic model to achieve a high accuracy of video soccer summarization. In this paper, we present a novel approach for soccer video summarization based on video content analysis and social media streams. Social media streams such as Twitter generate important volume of content for most sport events on a daily basis. The mining of the tweets can be used for the detection of events, which can be qualified as soccer match highlights. Incorporating social media events detection into video content analysis significantly improves the quality of the soccer video summarization. Results of experiment applications are performed over a database, consisting of more than 30 hours of soccer video.

Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.

In video object detection, the deterioration of an object’s appearance in a single frame brings challenges for recognition; therefore, it is natural to exploit temporal information to boost the robustness of video object detection. Existing methods usually utilize temporal information to enhance features, often ignoring the information in label assignments. Label assignment, which assigns labels to anchors for training, is an essential part of object detection. It is also challenged in video object detection and can be improved by temporal information. In this work, a temporal-guided label assignment framework is proposed for the learning task of a region proposal network (RPN). Specifically, we propose a feature instructing module (FIM) to establish the relation model among labels through feature similarity in the temporal dimension. The proposed video object detection framework was evaluated on the ImageNet VID benchmark. Without any additional inference cost, our work obtained a 0.8 mean average precision (mAP(%)) improvement over the baseline and achieved a mAP(%) of 82.0. The result was on par with the state-of-the-art accuracy without using any post-processing methods.

We present a new approach for video summarization and browsing of MPEG-2 compressed video based on the growing cell structures (GCS) neural algorithms. It first applies GCS to select keyframes for each shot and then clusters them using TreeGCS to form a hierarchical view of the video for efficient browsing. The keyframe selection is based on histogram features of the dc-images for T frames. It captures well the video content and outperforms two other approaches. The main advantage of the TreeGCS module is the ability to form dynamically a flexible hierarchy depending on the video content.

This paper addresses the video summarization problem. For the given video goal is to find the subset of frames that capture the important events of the input video and produce a small concise summary. We formulate video summarization as a sequence labeling problem, where for a given input video a subset of frames are selected as a summary video. Based on the principle of semantic segmentation, here each pixel within a frame is assigned to one of the labels, where each frame is assigned a binary label indicating whether it will be included in the summary video or not. We propose a SegNet sequence network (SegNetSN) for video summarization and further extend the work by applying various feature fusion techniques to enhance the input. We performed experiments on the benchmark dataset TVSum.

Considering the challenges of low multi‐object detection accuracy and difficulty in identifying small targets caused by challenging environmental conditions including irregular lighting patterns and ambient noise levels in the mining environment with autonomous electric locomotives. A new network model based on SOD−YOLOv5s−4L has been proposed to detect multi‐objects for autonomous electric locomotives in underground coal mines. Improvements have been applied in YOLOv5s to construct the SOD−YOLOv5s−4L model, by introducing the SIoU loss function to address the mismatch between real and predicted bounding box directions, facilitating the model to learn target position information more efficiently. This research introduces a decoupled head to enhance feature fusion and improve the positioning precision of the network model, enabling rapid capture of multi‐scale target features. Furthermore, the detection capability of the model has been increased by introducing the small target detection layer which is developed by increasing the number of detection layers from three to four. The experimental results on multiple object detection dataset show that the proposed model achieves significant improvement in mean average precision (mAP) of almost 98% for various types of targets and an average precision (AP) of nearly 99% for small targets on the other hand it achieves 5.19% (mAP) and 9.79% (AP) compared to the YOLOv5s model. Furthermore, comparative analysis with other models like YOLOv7 and YOLOv8 shows that the proposed model has superior performance in terms of object detection.

This paper presents an effective method for summarizing content of the video with plot preserved. The proposed method provides static video summary and consists of three major procedures (1) extracting keyframes regarding temporal information, (2) estimating Region of Interest (ROI) from the extracted frames, and (3) assembling the ROI into one image by arranging them according to the temporal order and their size, while they blend each other smoothly. In each process, we make the method concise for computational complexity. Experiment on various types of video (e.g. movie, animation, home video) demonstrates that the proposed method generates expressive video summarization and conserves the plot information. The main contribution relies on reflecting temporal information in video summarization.

Advancements in technology resulted in a sharp growth in the number of digital cameras at people's disposal all across the world. Consequently, the huge storage space consumed by the videos from these devices on video repositories make the job of video processing and analysis to be time-consuming. Furthermore, this also slows down the video browsing and retrieval. Video summarization plays a very crucial role in solving these issues. Despite the number of video summarization approaches proposed up to the present time, the goal is to take a long video and generate a video summary in form of a short video skim without losing the meaning or the message transmitted by the original lengthy video. This is done by selecting the important frames called key-frames. The approach proposed by this work performs automatic summarization of digital videos based on detected objects' deep features. To this end, we apply sparse subspace clustering with an automatically estimated number of clusters to the objects' deep features. The summary generated from our scheme will store the meta-data for each short video inferred from the clustering results. In this paper, we also suggest a new video dataset for video summarization. We evaluate the performance of our work using the TVSum dataset and our video summarization dataset.

Video summarisation is the process of creating a meaningful and self-explanatory summary of a given video by automatically selecting keyframes and thus creating a short and concise video summary of the original video clip. Video captioning refers to the task of automatically generating natural language description of a given video to provide some additional or interpretive information. Video summarisation and video captioning are often considered as two different tasks, each with a diverse application field. Thus, we propose an approach to jointly combine these two tasks and present a model which generates a short video summary along with a relevant caption. We make use of self-attention based transformer network combined with Multi Layer Perceptron (MLP) with multiple hidden layers for video captioning and video summarisation. We aim to demonstrate that the joint model can attain better performance than many of the previous approaches in both of the individual tasks. Our proposed model is based on the fact that both of the above mentioned tasks of captioning and summarisation will improve the performance of the other one as both the tasks are complementary to each other

Unmanned Aerial Vehicles (UAVs) have gained significant traction for real-time surveillance applications such as object tracking and search-and-rescue missions. A key enabler of these capabilities is the integration of a real-time, highly accurate object detection system. While available two-stage detectors like R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN achieve high detection accuracy and precise localization by dividing the image into regions and classifying each region, they are often slow, complex, and resource-intensive, making them unsuitable for real-time applications. To address this limitation, this research utilizes a Darknet-based YOLOv3 (You Only Look Once, Version 3), a state-of-the-art algorithm for real-time object detection in videos, live feeds, and images. Leveraging a deep Convolutional Neural Network (CNN), YOLOv3 efficiently learns features and predicts object locations and class probabilities in a single pass, ensuring high-speed and reliable detection. A real-time aerial surveillance system for quadcopter application using machine vision is proposed. The training dataset, obtained from the internet and self-taken images from a camera, was manually annotated into three critical categories: suspect, bandit, and weapon. The dataset was subsequently divided into training and testing subsets. Experimental results demonstrate that the proposed system achieves outstanding detection accuracy, with an overall mean average precision (mAP@0.5) of 93.82%, precision of 94%, and recall of 83%. Compared to a ResNet-50-based Faster R-CNN model, the YOLOv3-based approach outperformed, achieving success rates of 0.81 for the Bandit class and 1.00 for both Suspect and Weapon classes. This research improves drone-based AI for real-time object detection in security systems, enhancing efficiency and adaptability. It also creates specialized datasets, including a tailored bandit dataset, to support future UAV security research.

Rapid development in video technologies and easy availability of video capturing devices produces voluminous data which need to handle efficiently to increase the usability of it. Automatic video summarization is a solution to traverse through this data and gives a concise view of it, which is useful in many applications. However, traditional video summarization produces succinct videos without notably qualitative and quantitative loss of information contained in videos. It considers only the important information from the video while generating a summary independent of user interest. Video summarization is a highly subjective task which is not handled by traditional methods and people are more interested in personalized Summary. The solution to handle this problem is multi-model video summarization which helps to produce user-interested summaries by taking two inputs i.e. video and user queries. Different techniques have been explored in previous work based on conventional video summarization. This study presents the reviews of the state-of-the-art query focused video summarization methods to generate the personalized video summary which has not been investigated before. This paper discusses the demand of query-based video summarization in various applications, performance of existing methods and put forward future directions to help the research community to work in this domain.KeywordsVideo summarizationDeep learningQuery-focused video summarizationMulti-model

With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from original lengthy videos. Though supervised video summarization approaches have demonstrated the state-of-the-art performance, unsupervised methods are still highly demanded due to resourcefully expensive human annotations and the subjectiveness of video summarization tasks. In this paper, a novel unsupervised-based Deep Self-attention Recurrent summarization network with Reinforcement Learning (DSR-RL) for video summarization is proposed. The model can learn the input video sequence and suggest the key-shot summary without additional human annotations by integrating self-attention, BRNN, and reinforcement learning mechanisms. The DSR-RL improves not only importance score through the attention map vector of self-attention network but also the diversity of summaries via the reward function of reinforcement learning. Our method outperforms the state-of-the-art unsupervised video summarization methods on both SumMe and TVSum datasets. The source code is available at https://github.com/phaphuang/DSR-RL.