HiSum: A Hierarchical Framework for Long Video Summarization with Temporal Enhancement

Entirely watching separate video segments of interest or their summary might not be smooth enough nor comprehensible for viewers since contextual information between those segments may be lost. A unified framework for context-preserving video retrieval and summarization is proposed in order to solve this problem. Given a video database and ontologies specifying relationships among concepts used in MPEG-7 annotations, the objective is to identify according to a user query relevant segments together with summaries of contextual segments. Two types of contextual segments are defined: intra-contextual segments intended to form semantically coherent segments, and inter-contextual segments intended to semantically link together two separate segments. Relationships among verbs [3] are exploited to identify contextual segments as the relationships can provide the knowledge about events, causes and effects of actions over time. A query model and context-preserving video summarization are also presented.

While videos contain long-term temporal information with diverse contents, existing approaches to video understanding usually focus on a short trimmed video clip with a specific content such as a particular action or object. For comprehensive understanding of untrimmed videos, we address an integrated video task of video summarization with scene and action recognition. We propose a novel convolutional neural network architecture for handling untrimmed videos with multiple contents. The proposed architecture is an encoder-decoder structure where the encoder captures long-term temporal dynamics from an entire video and the decoder predicts detailed temporal information of multiple contents of the video. Two-stream processing is adopted for obtaining feature representations, one for focusing on the spatial information and the other for the temporal information. We evaluate the proposed method on the benchmark of the Challenge on Comprehensive Video Understanding in the Wild (CoVieW 2019), and the experimental results demonstrate that our method achieves outstanding performance

Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.

An efficient video summarization for surveillance system using normalized k-means and quick sort method

A deep audio-visual model for efficient dynamic video summarization

The tremendous volume of video data generated by industrial surveillance networks presents a number of difficulties when examining such videos for a variety of purposes, including video summarization (VS), analysis, indexing and retrieval. The task of creating video summaries is extremely difficult because of the huge amount of data, redundancy, interleaved views and light variations. Multiple object detection and identification in video is difficult for machines to recognize and classify. To address all such issues, multiple low-feature and clustering-based machine learning strategies that fail to completely exploit VS are recommended. In this work, we achieved VS by embedding deep neural network-based soft computing methods. Firstly, the objects in extracted frames are detected using YOLOv5, and then the frames without objects (useless frames) are removed. Video summary generation occurs with the help of frames containing Objects. To check the quality of the proposed work Summary length, precision, recall, PR curve, and mean average precision (mAP) are used and system resource utilization during the model training are also tracked. As a result, the proposed work was able to identify the most effective video summarization framework with best summary length under varying conditions.

This paper addresses the video summarization problem. For the given video goal is to find the subset of frames that capture the important events of the input video and produce a small concise summary. We formulate video summarization as a sequence labeling problem, where for a given input video a subset of frames are selected as a summary video. Based on the principle of semantic segmentation, here each pixel within a frame is assigned to one of the labels, where each frame is assigned a binary label indicating whether it will be included in the summary video or not. We propose a SegNet sequence network (SegNetSN) for video summarization and further extend the work by applying various feature fusion techniques to enhance the input. We performed experiments on the benchmark dataset TVSum.

Video summarisation is the process of creating a meaningful and self-explanatory summary of a given video by automatically selecting keyframes and thus creating a short and concise video summary of the original video clip. Video captioning refers to the task of automatically generating natural language description of a given video to provide some additional or interpretive information. Video summarisation and video captioning are often considered as two different tasks, each with a diverse application field. Thus, we propose an approach to jointly combine these two tasks and present a model which generates a short video summary along with a relevant caption. We make use of self-attention based transformer network combined with Multi Layer Perceptron (MLP) with multiple hidden layers for video captioning and video summarisation. We aim to demonstrate that the joint model can attain better performance than many of the previous approaches in both of the individual tasks. Our proposed model is based on the fact that both of the above mentioned tasks of captioning and summarisation will improve the performance of the other one as both the tasks are complementary to each other

There is context-related temporal and spatial information in video. How to make better use of these information is the key to improving the accuracy and speed of video object detection. The method of processing spatial and temporal information through 3D convolution is better than that of processing spatial information first and then processing time dimension information. There are two reasons for this assertion. One is that the 3D convolution can process spatial information and temporal information at the same time which will not divide them into two parts in feature extraction. The other is that the human visual derivation mechanism needs to memorize multi frames and process a video as a whole. After the training test on NFS dataset, it is verified that 3D convolution has stronger ability to extract temporal and spatial information. In addition, this paper also proposes a downward sampling method by connecting the low-resolution layer and the high-resolution layer, which can effectively improve the FPN model and enrich its high-level feature representation. After training and testing with different number of downward sample modules on MOT16 dataset, the improvement of model performance by down-sampling module is verified. Finally, this paper combines the downward sample module with the 3D network, and obtains the best test results on the NFS dataset.

This paper presents an effective method for summarizing content of the video with plot preserved. The proposed method provides static video summary and consists of three major procedures (1) extracting keyframes regarding temporal information, (2) estimating Region of Interest (ROI) from the extracted frames, and (3) assembling the ROI into one image by arranging them according to the temporal order and their size, while they blend each other smoothly. In each process, we make the method concise for computational complexity. Experiment on various types of video (e.g. movie, animation, home video) demonstrates that the proposed method generates expressive video summarization and conserves the plot information. The main contribution relies on reflecting temporal information in video summarization.

Rapid development in video technologies and easy availability of video capturing devices produces voluminous data which need to handle efficiently to increase the usability of it. Automatic video summarization is a solution to traverse through this data and gives a concise view of it, which is useful in many applications. However, traditional video summarization produces succinct videos without notably qualitative and quantitative loss of information contained in videos. It considers only the important information from the video while generating a summary independent of user interest. Video summarization is a highly subjective task which is not handled by traditional methods and people are more interested in personalized Summary. The solution to handle this problem is multi-model video summarization which helps to produce user-interested summaries by taking two inputs i.e. video and user queries. Different techniques have been explored in previous work based on conventional video summarization. This study presents the reviews of the state-of-the-art query focused video summarization methods to generate the personalized video summary which has not been investigated before. This paper discusses the demand of query-based video summarization in various applications, performance of existing methods and put forward future directions to help the research community to work in this domain.KeywordsVideo summarizationDeep learningQuery-focused video summarizationMulti-model

With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from original lengthy videos. Though supervised video summarization approaches have demonstrated the state-of-the-art performance, unsupervised methods are still highly demanded due to resourcefully expensive human annotations and the subjectiveness of video summarization tasks. In this paper, a novel unsupervised-based Deep Self-attention Recurrent summarization network with Reinforcement Learning (DSR-RL) for video summarization is proposed. The model can learn the input video sequence and suggest the key-shot summary without additional human annotations by integrating self-attention, BRNN, and reinforcement learning mechanisms. The DSR-RL improves not only importance score through the attention map vector of self-attention network but also the diversity of summaries via the reward function of reinforcement learning. Our method outperforms the state-of-the-art unsupervised video summarization methods on both SumMe and TVSum datasets. The source code is available at https://github.com/phaphuang/DSR-RL.

This paper focuses on the query-focused video summarization, which is an extended task of video summarization and aims to automatically generate user-oriented summary by highlighting frames/shots relevant to the query. This task is different from traditional video summarization in paying attention to users' subjectivity through queries. Diversity is a recognized important property in video summarization. However, existing methods only consider diversity as the dissimilarity between frames/shots which is far from user-oriented summarization. Users' different understandings of video should be an important source of diversity, reflected in the process of eliminating query-unrelated redundancy. To this end, this paper explores user-diversified & query-focused video summarization via a well-devised hierarchical variational network called HVN. HVN has three distinctive characteristics: (i) it has a hierarchical structure to model query-related long-range temporal dependency; (ii) it employs diverse attention mechanisms to encode query-related and context-important information and makes them balanced; (iii) it employs a multilevel self-attention module and a variational autoencoder module to add user-oriented diversity and stochastic factors. Experimental results demonstrate that HVN not only outperforms the state-of-the-arts but also improves the user-oriented diversity to some extent.

In the context of automotive driving assistance, this paper describes a generic (i.e. applicable to both vehicle interior and exterior scenes) vision based approach for scene content analysis. It makes use of temporal and spatial information from a stereoscopic sequence of images to localize objects and estimate their position and motion. The proposed method is divided into three steps. First, image features are selected, tracked and reconstructed in the 3D world space. Second, a clustering step is processed in the 5D space made of the positions and 2D motions parameters. The last step is devoted to clusters interpretation: it is out of the scope of the paper, however orientations are given to illustrate the capabilities of the proposed approach. The paper is organized as follows: first, the use of temporal and spatial information from a stereoscopic sequence is investigated. A state of the art of existing methods is presented. Then, a generic approach for object segmentation is proposed. Lastly, experimental results are presented.

Short videos on the Internet are growing exponentially, and the number of videos uploaded every day is huge; people also involve a lot of video data in real life. People can retrieve and view all kinds of videos, but it also brings a lot of problems. On the one hand, the accumulation of a large number of videos makes people unable to find the videos they want quickly, and the repeated scenes in the videos will also waste people's time and energy; on the other hand, a large amount of video data also brings enormous pressure to storage. Aiming at the problems of inaccurate selection of key frames and how to select video frame features in existing video summarization models, this paper proposes a multi-feature-based video summarization generation model (DME-VSNet), which extracts multiple features of video frames. Including importance score, image memory strength and image entropy. Aiming at the problem of inaccurate video shot segmentation, this model proposes a video shot segmentation algorithm based on TransNet network, which divides the original video into several short shots through shot boundaries; the model inputs the above three features into the proposed The video frame score is obtained in the MLP architecture, and the key frame is selected by the score to generate a video summary. The effectiveness of the video shot segmentation method based on TransNet network and the overall model based on convolutional neural network is verified by comparative experiments. The experimental results show that the evaluation results of the video summaries generated by the three features are better.