Automatic Variable Length Summarized Video using Motion of Object

Rapid development in video technologies and easy availability of video capturing devices produces voluminous data which need to handle efficiently to increase the usability of it. Automatic video summarization is a solution to traverse through this data and gives a concise view of it, which is useful in many applications. However, traditional video summarization produces succinct videos without notably qualitative and quantitative loss of information contained in videos. It considers only the important information from the video while generating a summary independent of user interest. Video summarization is a highly subjective task which is not handled by traditional methods and people are more interested in personalized Summary. The solution to handle this problem is multi-model video summarization which helps to produce user-interested summaries by taking two inputs i.e. video and user queries. Different techniques have been explored in previous work based on conventional video summarization. This study presents the reviews of the state-of-the-art query focused video summarization methods to generate the personalized video summary which has not been investigated before. This paper discusses the demand of query-based video summarization in various applications, performance of existing methods and put forward future directions to help the research community to work in this domain.KeywordsVideo summarizationDeep learningQuery-focused video summarizationMulti-model

According to the literature, automatic video summarization techniques can be classified in two parts, following the output nature: "video skims", which are generated using portions of the original video and "key-frame sets", which correspond to the images, selected from the original video, having a significant semantic content. The difference between these two categories is reduced when we consider automatic procedures. Most of the published approaches are based on the image signal and use either pixel characterization or histogram techniques or image decomposition by blocks. However, few of them integrate properties of the Human Visual System (HVS). In this paper, we propose to extract keyframes for video summarization by studying the variations of salient information between two consecutive frames. For each frame, a saliency map is produced simulating the human visual attention by a bottom-up (signal-dependent) approach. This approach includes three parallel channels for processing three early visual features: intensity, color and temporal contrasts. For each channel, the variations of the salient information between two consecutive frames are computed. These outputs are then combined to produce the global saliency variation which determines the key-frames. Psychophysical experiments have been defined and conducted to analyze the relevance of the proposed key-frame extraction algorithm.

With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from original lengthy videos. Though supervised video summarization approaches have demonstrated the state-of-the-art performance, unsupervised methods are still highly demanded due to resourcefully expensive human annotations and the subjectiveness of video summarization tasks. In this paper, a novel unsupervised-based Deep Self-attention Recurrent summarization network with Reinforcement Learning (DSR-RL) for video summarization is proposed. The model can learn the input video sequence and suggest the key-shot summary without additional human annotations by integrating self-attention, BRNN, and reinforcement learning mechanisms. The DSR-RL improves not only importance score through the attention map vector of self-attention network but also the diversity of summaries via the reward function of reinforcement learning. Our method outperforms the state-of-the-art unsupervised video summarization methods on both SumMe and TVSum datasets. The source code is available at https://github.com/phaphuang/DSR-RL.

Advancements in technology resulted in a sharp growth in the number of digital cameras at people's disposal all across the world. Consequently, the huge storage space consumed by the videos from these devices on video repositories make the job of video processing and analysis to be time-consuming. Furthermore, this also slows down the video browsing and retrieval. Video summarization plays a very crucial role in solving these issues. Despite the number of video summarization approaches proposed up to the present time, the goal is to take a long video and generate a video summary in form of a short video skim without losing the meaning or the message transmitted by the original lengthy video. This is done by selecting the important frames called key-frames. The approach proposed by this work performs automatic summarization of digital videos based on detected objects' deep features. To this end, we apply sparse subspace clustering with an automatically estimated number of clusters to the objects' deep features. The summary generated from our scheme will store the meta-data for each short video inferred from the clustering results. In this paper, we also suggest a new video dataset for video summarization. We evaluate the performance of our work using the TVSum dataset and our video summarization dataset.

Watching sports videos over streaming sites and television network is one of the most entertaining ways to engage with sports activities. Sports videos like cricket has been viewed by larger audiences than viewing in person. The pandemic since 2020 has changed the world of sports viewing to a larger extent. Some of the sports events are even streamed live through YouTube. The most interesting part of any sports videos is watching the highlights or events of great interest. This is because of the lack of time to watch the entire length of the game. Automatic video summarization is the solution to this. Some of the day long sports like cricket needs the summarization to be very precise and bring the content within few minutes to the audience. There are several attempts in the literature to automatically summarize the sports videos, particularly the game of cricket. In this paper, an attempt has been made to review some of the latest developments in creating the video summary of cricket sports. A brief review of existing methods of video summarization that addresses many sports including soccer, cricket, tennis, and basketball are reviewed at the beginning. Later, the methods that are developed based on latest machine learning and high-performance algorithms are discussed in detail. Towards the end of this paper, a comparison of these methods is presented. The goal is to lead the prospective researchers in the direction where the methods have open avenues and scope to strengthen.

Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.

The Kalman filter has long been regarded as the optimal solution to many applications in computer vision for example the tracking objects, prediction and correction tasks. Its use in the analysis of visual motion has been documented frequently, we can use in computer vision and open cv in different applications in reality for example robotics, military image and video, medical applications, security in public and privacy society, etc. In this paper, we investigate the implementation of a Matlab code for a Kalman Filter using three algorithm for tracking and detection objects in video sequences (block-matching (Motion Estimation) and Camshift Meanshift (localization, detection and tracking object)). The Kalman filter is presented in three steps: prediction, estimation (correction) and update. The first step is a prediction for the parameters of the tracking and detection objects. The second step is a correction and estimation of the prediction parameters. The important application in Kalman filter is the localization and tracking mono-objects and multi-objects are given in results. This works presents the extension of an integrated modeling and simulation tool for the tracking and detection objects in computer vision described at different models of algorithms in implementation systems.

Video summarization is a keenly intellective video compression technique to select a subset of keyframes or keyshots which are combined to represent shorter and compendious summary of the original input video without losing the contextual semantics of the same. Previous work has shown that extracting rich contextual information from the input video frames is imperative for generating summary that is closer to human interpretation of the original input video. However, recent convolutional architectures were unable to account for the same. In this paper, we institute a novel relation between image super resolution and video summarization and introduce a novel architecture by adapting residual dense network (RDN), which fully exploits both local and global structural context. Experimental results indicate that introducing a modified RDN (SUM-RDN) unit significantly improves the performance over standard convolutional networks.Parihar, Anil Singh Mittal, Ritvik Himanshu Jain, Prashuk

Video summarization and captioning condense content by selecting keyframes and generating language descriptions, integrating both visual and textual perspectives. Existing video-and-language learning models typically select multiple frames as proxies rather than analyzing all frames, which improves computational efficiency but may not adequately represent the original content without redundancy. In this paper, we propose an adaptive dual video summarization framework and demonstrate its effectiveness within the context of video captioning. Given the video frames, we extract visual representations using a video-domain fine-tuned ViT model to narrow the domain shift. The keyframes are summarized based on the frame-level scores. To minimize the number of keyframes while ensuring captioning quality, we introduce a cross-modal video summarizer that selects the most semantically consistent frames according to pseudo score labels. Furthermore, we incorporate an adaptive keyframe selector that determines the optimal number of keyframes based on the video's complexity and content, enhancing the framework's adaptability and generalization. The proposed adaptive keyframe selector enables the framework to handle diverse video content, making it more generalizable and applicable to real-world scenarios.We designed a ranking scheme to assess the video's static appearance and temporal dynamics from score-based and time-based perspectives. To conclude, we use a lightweight LSTM decoder to generate descriptions. Experimental results on the MSR-VTT, MSVD and VATEX benchmarks demonstrate that our adaptive dual video summarization framework can effectively convey the same semantic information as the original video while using a significantly reduced number of keyframes, leading to improved video captioning performance.

A novel approach for long-term secure storage of domain independent videos

The aim of video summarization is to refine a video into concise form without losing its gist. In general, a summary with similar semantics to the original video can represent the original well. Unfortunately, most existing methods focus more on the diversity and representation content of the video, and few of them take video’s semantic into consideration. In addition, most methods related to semantic pursue their own description of the video and this way will learn a biased mode. In order to solve these issues, we propose a novel semantic-consistent unsupervised framework termed ScSUM which is able to extract the essence of the video via obtaining the greatest semantic similarity, and requires no manual description. In particular, ScSUM consists of a frame selector, and a video descriptor which not only predict the description of summary, but also produce the description of original video used as targets. The main goal of our propose is to minimize the distance between the summary and original video in the semantic space. Finally, experiments on two benchmark datasets validate the effectiveness of the proposed methods and demonstrate that our method achieves competitive performance.

Real time object tracking is an essential part of surveillance and robot applications. The performance of any object tracking system depends on its accuracy and its ability to deal with various sizes of objects, for better results the tracking should happen at a high speed. Object tracking can be defined as the process of segmenting the region of interest from the video sequence and keeping track of the motion in order to extract useful information. The basic fundamentals of video surveillance such as object detection, tracking and recognition using multiple cameras. Whenever an object enters a video frame, it is detected using background subtraction and the detected objects are tracked with a novel Bayesian Kalman filter with simplified Gaussian mixture (BKF-SGM). Object recognition algorithm is applied using Gait method which is employed to support multiple object tracking with its robust performance. Processing of image is done in MATLAB to get detection, tracking and recognized results.

Vision sensors in Internet of Things (IoT)-connected smart cities play a vital role in the exponential growth of video data, thereby making its analysis and storage comparatively tough and challenging. Those sensors continuously generate data for 24 hours, which requires huge storage resources, dedicated networks for sharing with data centers, and most importantly, it makes browsing, retrieval, and event searching a difficult and time-consuming job. Video summarization (VS) is a promising direction toward a solution to these problems by analyzing the visual contents acquired from a vision sensor and prioritizing them based on events, saliency, person's appearance, and so on. However, the current VS literature still lacks focus on resource-constrained devices that can summarize data over the edge and upload it to data repositories efficiently for instant analysis. Therefore, in this article, we carry out a survey of functional VS methods to understand their pros and cons for resource-constrained devices, with the ambition to provide a compact tutorial to the community of researchers in the field. Further, we present a novel saliency-aware VS framework, incorporating 5G-enabled IoT devices, which keeps only important data, thereby saving storage resources and providing representative data for immediate exploration. Keeping privacy of data as a second priority, we intelligently encrypt the salient frames over resource-constrained devices before transmission over the 5G network. The reported experimental results show that our proposed framework has additional benefits of faster transmission (1.8~13.77 percent frames of a lengthy video are considered for transmission), reduced bandwidth, and real-time processing compared to state-of-the-art methods in the field.

The paper presents an automatic video summarization technique based on graph theory methodology and the dominant sets clustering algorithm. The large size of the video data set is handled by exploiting the connectivity information of prototype frames that are extracted from a down-sampled version of the original video sequence. The connectivity information for the prototypes which is obtained from the whole set of data improves video representation and reveals its structure. Automatic selection of the optimal number of clusters and hereafter keyframes is accomplished at a next step through the dominant set clustering algorithm. The method is free of user-specified modeling parameters and is evaluated in terms of several metrics that quantify its content representational ability. Comparison of the proposed summarization technique to the Open Video storyboard, the Adaptive clustering algorithm and the Delaunay clustering approach, is provided.

Modern video summarization methods are based on deep neural networks that require a large amount of annotated data for training. However, existing datasets for video summarization are small-scale, easily leading to over-fitting of the deep models. Considering that the annotation of large-scale datasets is time-consuming, we propose a multimodal self-supervised learning framework to obtain semantic representations of videos, which benefits the video summarization task. Specifically, the self-supervised learning is conducted by exploring the semantic consistency between the videos and text in both coarse-grained and fine-grained fashions, as well as recovering masked frames in the videos. The multimodal framework is trained on a newly-collected dataset that consists of video-text pairs. Additionally, we introduce a progressive video summarization method, where the important content in a video is pinpointed progressively to generate better summaries. Extensive experiments have proved the effectiveness and superiority of our method in rank correlation coefficients and F-score <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .