Hierarchical Variational Network for User-Diversified & Query-Focused Video Summarization

With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from original lengthy videos. Though supervised video summarization approaches have demonstrated the state-of-the-art performance, unsupervised methods are still highly demanded due to resourcefully expensive human annotations and the subjectiveness of video summarization tasks. In this paper, a novel unsupervised-based Deep Self-attention Recurrent summarization network with Reinforcement Learning (DSR-RL) for video summarization is proposed. The model can learn the input video sequence and suggest the key-shot summary without additional human annotations by integrating self-attention, BRNN, and reinforcement learning mechanisms. The DSR-RL improves not only importance score through the attention map vector of self-attention network but also the diversity of summaries via the reward function of reinforcement learning. Our method outperforms the state-of-the-art unsupervised video summarization methods on both SumMe and TVSum datasets. The source code is available at https://github.com/phaphuang/DSR-RL.

A deep audio-visual model for efficient dynamic video summarization

Rapid development in video technologies and easy availability of video capturing devices produces voluminous data which need to handle efficiently to increase the usability of it. Automatic video summarization is a solution to traverse through this data and gives a concise view of it, which is useful in many applications. However, traditional video summarization produces succinct videos without notably qualitative and quantitative loss of information contained in videos. It considers only the important information from the video while generating a summary independent of user interest. Video summarization is a highly subjective task which is not handled by traditional methods and people are more interested in personalized Summary. The solution to handle this problem is multi-model video summarization which helps to produce user-interested summaries by taking two inputs i.e. video and user queries. Different techniques have been explored in previous work based on conventional video summarization. This study presents the reviews of the state-of-the-art query focused video summarization methods to generate the personalized video summary which has not been investigated before. This paper discusses the demand of query-based video summarization in various applications, performance of existing methods and put forward future directions to help the research community to work in this domain.KeywordsVideo summarizationDeep learningQuery-focused video summarizationMulti-model

Sigmoid and hyperbolic activation functions in long short-term memory (LSTM) and gated recurrent unit (GRU) based models used in recent studies on video summarization, may cause gradient decay over layers. Moreover, interpreting and developing network models are difficult because of entanglement of neurons on recurrent neural network (RNN). To solve these issues, in this study, we propose a method that uses deep reinforcement learning together with independently recurrent neural networks (IndRNN) for unsupervised video summarization. In this method, Leaky Rectified Linear Unit (Leaky ReLU) is used as an activation function to deal with decaying gradient and dying neuron problems. The model, which does not rely on any labels or user interaction, is designed with a reward function that jointly accounts for uniformity, diversity and representativeness of generated summaries. In this way, our model can create summaries as uniform as possible, has more layers and can be trained with more steps without having any problem related to gradients. Based on the experiments conducted on two benchmark datasets, we observe that, compared to the state-of-the-art methods on video summarization task, better summarization performance can be obtained.

We propose HiSum, a novel Hierarchical and temporally-enhanced framework for the task of long video Summarization. A significant challenge in this domain is generating concise yet informative summaries for long videos, which requires a nuanced understanding of both spatial and temporal information. Existing methods are often limited, struggling with temporally complex summaries and supporting videos typically shorter than one hour. To address these limitations, we propose HiSum, a training-free framework for long video summarization that is hierarchical, scene-aware, and temporally enhanced. HiSum will segment a video based on scene boundaries and then construct a summary hierarchically, effectively overcoming context length constraints. A dedicated temporal enhancement component is employed to mitigate information loss between chunks. Experimental results demonstrate that HiSum outperforms not only in resource consumption, but also in summary generation quality. Evaluated on the InfiniBench dataset, HiSum attained a GPT-5 score of 7.51 (on a 0-10 scale), significantly outperforming prevailing methods like Qwen2.5-VL, which scored 3.47.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>

Modern video summarization methods are based on deep neural networks that require a large amount of annotated data for training. However, existing datasets for video summarization are small-scale, easily leading to over-fitting of the deep models. Considering that the annotation of large-scale datasets is time-consuming, we propose a multimodal self-supervised learning framework to obtain semantic representations of videos, which benefits the video summarization task. Specifically, the self-supervised learning is conducted by exploring the semantic consistency between the videos and text in both coarse-grained and fine-grained fashions, as well as recovering masked frames in the videos. The multimodal framework is trained on a newly-collected dataset that consists of video-text pairs. Additionally, we introduce a progressive video summarization method, where the important content in a video is pinpointed progressively to generate better summaries. Extensive experiments have proved the effectiveness and superiority of our method in rank correlation coefficients and F-score <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .

With the proliferation of wearable cameras, the number of videos of users documenting their personal lives using such devices is rapidly increasing. Since such videos may span hours, there is an important need for mechanisms that represent the information content in a compact form (i.e., shorter videos which are more easily browsable/sharable). Motivated by these applications, this paper focuses on the problem of egocentric video summarization. Such videos are usually continuous with significant camera shake and other quality issues. Because of these reasons, there is growing consensus that direct application of standard video summarization tools to such data yields unsatisfactory performance. In this paper, we demonstrate that using gaze tracking information (such as fixation and saccade) significantly helps the summarization task. It allows meaningful comparison of different image frames and enables deriving personalized summaries (gaze provides a sense of the camera wearer's intent). We formulate a summarization model which captures common-sense properties of a good summary, and show that it can be solved as a submodular function maximization with partition matroid constraints, opening the door to a rich body of work from combinatorial optimization. We evaluate our approach on a new gaze-enabled egocentric video dataset (over 15 hours), which will be a valuable standalone resource.

While videos contain long-term temporal information with diverse contents, existing approaches to video understanding usually focus on a short trimmed video clip with a specific content such as a particular action or object. For comprehensive understanding of untrimmed videos, we address an integrated video task of video summarization with scene and action recognition. We propose a novel convolutional neural network architecture for handling untrimmed videos with multiple contents. The proposed architecture is an encoder-decoder structure where the encoder captures long-term temporal dynamics from an entire video and the decoder predicts detailed temporal information of multiple contents of the video. Two-stream processing is adopted for obtaining feature representations, one for focusing on the spatial information and the other for the temporal information. We evaluate the proposed method on the benchmark of the Challenge on Comprehensive Video Understanding in the Wild (CoVieW 2019), and the experimental results demonstrate that our method achieves outstanding performance

Dynamic graph convolutional network for multi-video summarization

In today's society where the enormous amount of documents are overflowing, the demand for automatic text summarization has been increasing. In particular, when considering the practical use, it is expected that the query-oriented text summarization, which generates summaries focusing on the given specific queries, will be more important rather than the generic summarization that simply summarizes the entire document. In the automatic summarization methods based on the deep neural model, Long-Short Term Memory (LSTM) which enables longterm information storage, and attention mechanism which allows emphasis on specific time steps from the vectors encoded by the Recurrent Neural Network (RNN), are essential. However, Koehn et al. [1] report that the accuracy is degraded in the encoding of texts longer than 60 tokens entered into the LSTM. Even in the summarization task, it is considered to be a problem that the encoding of a long sentence fails or causes the disappearance of the relation of the sentences, as a factor that greatly deteriorates the quality of the summary result. In this paper, we propose a method to generate a summary by introducing a sentence unit vector in addition to the word unit vector of the original document. We aim to generate summaries considering the importance degree of a sentence unit and the relations between sentences, by learning both the attention mechanism of word unit and the attention mechanism of sentence unit. In the experiment, we examine the summarization model which added the mechanism of sentence unit vector to the latest query-focused summarization model, and clarify the influence on the summary generated by presence or absence of the sentence vector.

The exponential increase in video consumption has created new difficulties for browsing and navigating through video more effectively and efficiently. Researchers are interested in video summarization because it offers a brief but instructive video version that helps users and systems save time and effort when looking for and comprehending relevant content. Key frame extraction is a method of video summarization that only chooses the most important frames from a given video. In this article, a novel supervised learning method ‘TC‐CLSTM Auto Encoder with Mode‐based Learning’ using temporal and spatial features is proposed for automatically choosing keyframes or important sub‐shots from videos. The method was able to achieve an average F‐score of 84.35 on TVSum dataset. Extensive tests on benchmark data sets show that the suggested methodology outperforms state‐of‐the‐art methods.

Video summarization is essential for the user to understand the main theme of video sequences in a short period, especially when the volume of the video is huge and the content is highly redundant. In this paper, we present a video summarization system, built for the rushes summarization task in TRECVID 2008. The goal is to create a video excerpt including objects and events in the video with minimum redundancy and duration (up to 2% of the original video). We first segment a video into shots and then apply a multi-stage clustering algorithm to eliminate similar shots. Frame importance values that depend on both the temporal content variation and the spatial image salience are used to select the most interesting video clips as part of the summarization. We test our system with two output configurations - a dynamic playback rate and at the native playback rate - as a tradeoff between ground truth inclusion rate and ease of browsing. TRECVID evaluation results show that our system achieves a good inclusion rate and verify that the created video summarization is easy to understand.

Video summarization via minimum sparse reconstruction

Topic-aware video summarization using multimodal transformer

Exploiting the inner-shot and inter-shot dependencies is essential for key-shot based video summarization. Current approaches mainly devote to modeling the video as a frame sequence by recurrent neural networks. However, one potential limitation of the sequence models is that they focus on capturing local neighborhood dependencies while the high-order dependencies in long distance are not fully exploited. In general, the frames in each shot record a certain activity and vary smoothly over time, but the multi-hop relationships occur frequently among shots. In this case, both the local and global dependencies are important for understanding the video content. Motivated by this point, we propose a reconstructive sequence-graph network (RSGN) to encode the frames and shots as sequence and graph hierarchically, where the frame-level dependencies are encoded by long short-term memory (LSTM), and the shot-level dependencies are captured by the graph convolutional network (GCN). Then, the videos are summarized by exploiting both the local and global dependencies among shots. Besides, a reconstructor is developed to reward the summary generator, so that the generator can be optimized in an unsupervised manner, which can avert the lack of annotated data in video summarization. Furthermore, under the guidance of reconstruction loss, the predicted summary can better preserve the main video content and shot-level dependencies. Practically, the experimental results on three popular datasets (i.e., SumMe, TVsum and VTW) have demonstrated the superiority of our proposed approach to the summarization task.