A deep audio-visual model for efficient dynamic video summarization

With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from original lengthy videos. Though supervised video summarization approaches have demonstrated the state-of-the-art performance, unsupervised methods are still highly demanded due to resourcefully expensive human annotations and the subjectiveness of video summarization tasks. In this paper, a novel unsupervised-based Deep Self-attention Recurrent summarization network with Reinforcement Learning (DSR-RL) for video summarization is proposed. The model can learn the input video sequence and suggest the key-shot summary without additional human annotations by integrating self-attention, BRNN, and reinforcement learning mechanisms. The DSR-RL improves not only importance score through the attention map vector of self-attention network but also the diversity of summaries via the reward function of reinforcement learning. Our method outperforms the state-of-the-art unsupervised video summarization methods on both SumMe and TVSum datasets. The source code is available at https://github.com/phaphuang/DSR-RL.

The goal of video summarization is to distill a raw video into a more compact form without losing much semantic information. However, previous methods mainly consider the diversity and representation interestingness of the obtained summary, and they seldom pay sufficient attention to semantic information of resulting frame set, especially the long temporal range semantics. To explicitly address this issue, we propose a novel technique which is able to extract the most semantically relevant video segments (i.e., valid for a long term temporal duration) and assemble them into an informative summary. To this end, we develop a semantic attended video summarization network (SASUM) which consists of a frame selector and video descriptor to select an appropriate number of video shots by minimizing the distance between the generated description sentence of the summarized video and the human annotated text of the original video. Extensive experiments show that our method achieves a superior performance gain over previous methods on two benchmark datasets.

We propose HiSum, a novel Hierarchical and temporally-enhanced framework for the task of long video Summarization. A significant challenge in this domain is generating concise yet informative summaries for long videos, which requires a nuanced understanding of both spatial and temporal information. Existing methods are often limited, struggling with temporally complex summaries and supporting videos typically shorter than one hour. To address these limitations, we propose HiSum, a training-free framework for long video summarization that is hierarchical, scene-aware, and temporally enhanced. HiSum will segment a video based on scene boundaries and then construct a summary hierarchically, effectively overcoming context length constraints. A dedicated temporal enhancement component is employed to mitigate information loss between chunks. Experimental results demonstrate that HiSum outperforms not only in resource consumption, but also in summary generation quality. Evaluated on the InfiniBench dataset, HiSum attained a GPT-5 score of 7.51 (on a 0-10 scale), significantly outperforming prevailing methods like Qwen2.5-VL, which scored 3.47.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.

This paper focuses on the query-focused video summarization, which is an extended task of video summarization and aims to automatically generate user-oriented summary by highlighting frames/shots relevant to the query. This task is different from traditional video summarization in paying attention to users' subjectivity through queries. Diversity is a recognized important property in video summarization. However, existing methods only consider diversity as the dissimilarity between frames/shots which is far from user-oriented summarization. Users' different understandings of video should be an important source of diversity, reflected in the process of eliminating query-unrelated redundancy. To this end, this paper explores user-diversified & query-focused video summarization via a well-devised hierarchical variational network called HVN. HVN has three distinctive characteristics: (i) it has a hierarchical structure to model query-related long-range temporal dependency; (ii) it employs diverse attention mechanisms to encode query-related and context-important information and makes them balanced; (iii) it employs a multilevel self-attention module and a variational autoencoder module to add user-oriented diversity and stochastic factors. Experimental results demonstrate that HVN not only outperforms the state-of-the-arts but also improves the user-oriented diversity to some extent.

Sigmoid and hyperbolic activation functions in long short-term memory (LSTM) and gated recurrent unit (GRU) based models used in recent studies on video summarization, may cause gradient decay over layers. Moreover, interpreting and developing network models are difficult because of entanglement of neurons on recurrent neural network (RNN). To solve these issues, in this study, we propose a method that uses deep reinforcement learning together with independently recurrent neural networks (IndRNN) for unsupervised video summarization. In this method, Leaky Rectified Linear Unit (Leaky ReLU) is used as an activation function to deal with decaying gradient and dying neuron problems. The model, which does not rely on any labels or user interaction, is designed with a reward function that jointly accounts for uniformity, diversity and representativeness of generated summaries. In this way, our model can create summaries as uniform as possible, has more layers and can be trained with more steps without having any problem related to gradients. Based on the experiments conducted on two benchmark datasets, we observe that, compared to the state-of-the-art methods on video summarization task, better summarization performance can be obtained.

An efficient video summarization for surveillance system using normalized k-means and quick sort method

Rapid development in video technologies and easy availability of video capturing devices produces voluminous data which need to handle efficiently to increase the usability of it. Automatic video summarization is a solution to traverse through this data and gives a concise view of it, which is useful in many applications. However, traditional video summarization produces succinct videos without notably qualitative and quantitative loss of information contained in videos. It considers only the important information from the video while generating a summary independent of user interest. Video summarization is a highly subjective task which is not handled by traditional methods and people are more interested in personalized Summary. The solution to handle this problem is multi-model video summarization which helps to produce user-interested summaries by taking two inputs i.e. video and user queries. Different techniques have been explored in previous work based on conventional video summarization. This study presents the reviews of the state-of-the-art query focused video summarization methods to generate the personalized video summary which has not been investigated before. This paper discusses the demand of query-based video summarization in various applications, performance of existing methods and put forward future directions to help the research community to work in this domain.KeywordsVideo summarizationDeep learningQuery-focused video summarizationMulti-model

This paper addresses the video summarization problem. For the given video goal is to find the subset of frames that capture the important events of the input video and produce a small concise summary. We formulate video summarization as a sequence labeling problem, where for a given input video a subset of frames are selected as a summary video. Based on the principle of semantic segmentation, here each pixel within a frame is assigned to one of the labels, where each frame is assigned a binary label indicating whether it will be included in the summary video or not. We propose a SegNet sequence network (SegNetSN) for video summarization and further extend the work by applying various feature fusion techniques to enhance the input. We performed experiments on the benchmark dataset TVSum.

Advancements in technology resulted in a sharp growth in the number of digital cameras at people's disposal all across the world. Consequently, the huge storage space consumed by the videos from these devices on video repositories make the job of video processing and analysis to be time-consuming. Furthermore, this also slows down the video browsing and retrieval. Video summarization plays a very crucial role in solving these issues. Despite the number of video summarization approaches proposed up to the present time, the goal is to take a long video and generate a video summary in form of a short video skim without losing the meaning or the message transmitted by the original lengthy video. This is done by selecting the important frames called key-frames. The approach proposed by this work performs automatic summarization of digital videos based on detected objects' deep features. To this end, we apply sparse subspace clustering with an automatically estimated number of clusters to the objects' deep features. The summary generated from our scheme will store the meta-data for each short video inferred from the clustering results. In this paper, we also suggest a new video dataset for video summarization. We evaluate the performance of our work using the TVSum dataset and our video summarization dataset.

Video summarisation is the process of creating a meaningful and self-explanatory summary of a given video by automatically selecting keyframes and thus creating a short and concise video summary of the original video clip. Video captioning refers to the task of automatically generating natural language description of a given video to provide some additional or interpretive information. Video summarisation and video captioning are often considered as two different tasks, each with a diverse application field. Thus, we propose an approach to jointly combine these two tasks and present a model which generates a short video summary along with a relevant caption. We make use of self-attention based transformer network combined with Multi Layer Perceptron (MLP) with multiple hidden layers for video captioning and video summarisation. We aim to demonstrate that the joint model can attain better performance than many of the previous approaches in both of the individual tasks. Our proposed model is based on the fact that both of the above mentioned tasks of captioning and summarisation will improve the performance of the other one as both the tasks are complementary to each other

Sparse dictionary selection (SDS) has demonstrated to be an effective solution for keyframe based video summarization (VS), which generally assumes a linear relation among similar video frames. However, such a linear assumption is not always true for videos. In this paper, the nonlinearity among frames is taken into consideration and a nonlinear SDS model is formulated for VS, in which the nonlinearity is transformed to linearity by projecting a video to a high dimensional feature space induced by a kernel function. Moreover, a kernel simultaneous orthogonal matching pursuit (KSOMP) is proposed to solve the problem. In order to achieve an intuitive and flexible configuration of the VS process, an adaptive criterion is devised to produce video summaries with different lengths for different video content. Experimental results on benchmark video datasets demonstrate that the proposed algorithm outperforms several state-of-the-art VS algorithms.

The tremendous volume of video data generated by industrial surveillance networks presents a number of difficulties when examining such videos for a variety of purposes, including video summarization (VS), analysis, indexing and retrieval. The task of creating video summaries is extremely difficult because of the huge amount of data, redundancy, interleaved views and light variations. Multiple object detection and identification in video is difficult for machines to recognize and classify. To address all such issues, multiple low-feature and clustering-based machine learning strategies that fail to completely exploit VS are recommended. In this work, we achieved VS by embedding deep neural network-based soft computing methods. Firstly, the objects in extracted frames are detected using YOLOv5, and then the frames without objects (useless frames) are removed. Video summary generation occurs with the help of frames containing Objects. To check the quality of the proposed work Summary length, precision, recall, PR curve, and mean average precision (mAP) are used and system resource utilization during the model training are also tracked. As a result, the proposed work was able to identify the most effective video summarization framework with best summary length under varying conditions.

Modern video summarization methods are based on deep neural networks that require a large amount of annotated data for training. However, existing datasets for video summarization are small-scale, easily leading to over-fitting of the deep models. Considering that the annotation of large-scale datasets is time-consuming, we propose a multimodal self-supervised learning framework to obtain semantic representations of videos, which benefits the video summarization task. Specifically, the self-supervised learning is conducted by exploring the semantic consistency between the videos and text in both coarse-grained and fine-grained fashions, as well as recovering masked frames in the videos. The multimodal framework is trained on a newly-collected dataset that consists of video-text pairs. Additionally, we introduce a progressive video summarization method, where the important content in a video is pinpointed progressively to generate better summaries. Extensive experiments have proved the effectiveness and superiority of our method in rank correlation coefficients and F-score <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

Video summarization via minimum sparse reconstruction