A Video Segmentation and Key Frame Extraction Method for Compressed Video Stream

In an interactive multimedia information system, users should have the flexibility to browse and choose various scenarios they want to see. This means that two-way communications should be captured by the conceptual model. Digital video has gained increasing popularity in many multimedia applications. Instead of sequential access to the video contents, the structuring and modeling of video data so that users can quickly and easily browse and retrieve interesting materials has become an important issue in designing multimedia information systems. An abstract semantic model called the augmented transition network (ATN), which can model video data and user interactions, is proposed in this paper. An ATN and its subnetworks can model video data based on different granularities, such as scenes, shots and key frames. Multimedia input strings are used as inputs for ATNs. The details of how to use multimedia input strings to model video data are also discussed. Key frame selection is based on the temporal and spatial relations of semantic objects in each shot. These relations are captured from our proposed unsupervised video segmentation method, which considers the problem of partitioning each frame as a joint estimation of the partition and class parameter variables. Unlike existing semantic models, which only model multimedia presentation, multimedia database searching or browsing, ATNs together with multimedia input strings can model these three in one framework.

A new algorithm to distinguish the foreground from compressed videos is proposed in this paper. Local motion, which is estimated from the residual between the original motion and the global motion, is one of the strongest influences on visual attention. Global motion is modeled by four parameters related to camera pan, tilt, zoom and rotation. The initial parameters are obtained from the least-squares method and updated iteratively using the Levenberg-Marquardt algorithm. Temporal and spatial filters are also introduced to revise the final global motion. In addition, DC coefficients are employed to refine the result based on the local motion. Experiments show that the proposed algorithm can segment foreground effectively with a largely reduced computational complexity, as DC coefficients and motion vectors can easily be extracted from compressed videos.

Key frame selection, which aims to choose a small number of frames that well represent the entire video contents, is one of the important research issues in video content analysis. Many existing methods developed so far use some prior information about the contents of video frames based on high-level semantic features, and thus they are applicable only in particular problems. Without any prior information, approaches based on low-level index features such as clusters, correlation, and consecutive frame difference are useful. Key frames found by these approaches are expected to be as much different from each other as possible. Therefore, accurately evaluating the difference among candidate key frames is a key challenge. However, a standard similarity metric such as the squared error and the cross correlation tends to provide redundant key frames. To cope with this problem, we propose to use quadratic mutual information (QMI) as an alternative similarity measure, which allows us to capture higher-order correlation for more than two variables simultaneously. Through experiments, we demonstrate the usefulness of the proposed QMI-based key frame selection method that uses an accurate and computationally efficient least-squares QMI estimator.

A Video Segmentation and Key Frame Extraction Method for Compressed Video Stream

Due to video data exponential growth supported by quick advances in multimedia technology, it became difficult for the user to retrieve information from a large videos collection. Key frame extraction consists in providing an abstract of the entire video, containing the most informative frames. In this paper, we present an efficient key frame extraction method. This method is based on local interest points description and repeatability graph clustering via approaching modularity value. To minimize the data to be treated, the process will be applied only on a set candidate frames selected with a windowing rule. Indeed, the first step consists in extracting interest points on a set of candidate frames. After that, we compute repeatability values between each two frames from the candidate set. These values are represented by a repeatability direct graph. The selection of key frames is performed using graph clustering by approaching modularity principle. The experiments performed showed that the proposed method succeeds in extracting key frames that preserve the relevant video content without redundancy.

We present a method for motion-based video segmentation and segment classification as a step towards video summarization. The sequential segmentation of the video is performed by detecting changes in the dominant image motion, assumed to be related to camera motion and represented by a 2D affine model. The detection is achieved by analysing the temporal variations of some coefficients of the 2D affine model (robustly) estimated. The obtained video segments supply reasonable temporal units to be further classified. For the second stage, we adopt a statistical representation of the residual motion content of the video scene, relying on the distribution of temporal co-occurrences of local motion-related measurements. Pre-identified classes of dynamic events are learned off-line from a training set of video samples of the genre of interest. Each video segment is then classified according to a Maximum Likelihood criterion. Finally, excerpts of the relevant classes can be selected for video summarization. Experiments regarding the two steps of the method are presented on different video genres leading to very encouraging results while only low-level motion information is considered.

In video processing, a common first step is to segment the videos into physical units, generally called shots. A shot is a video segment that consists of one continuous action. In general, these physical units need to be clustered to form more semantically significant units, such as scenes, sequences, programs, etc. This is the so-called story-based video structuring. Automatic video structuring is of great importance for video browsing and retrieval. The shots or scenes are usually described by one or several representative frames, called key-frames. Viewed from a higher level, key frames of some shots might be redundant in terms of semantics. In this paper, we propose automatic solutions to the problems of: (i) video partitioning, (ii) key frame computing, (iii) key frame pruning. For the first problem, an algorithm called “net comparison” is devised. It is accurate and fast because it uses both statistical and spatial information in an image and does not have to process the entire image. For the last two problems, we develop an original image similarity criterion, which considers both spatial layout and detail content in an image. For this purpose, coefficients of wavelet decomposition are used to derive parameter vectors accounting for the above two aspects. The parameters exhibit (quasi-) invariant properties, thus making the algorithm robust for many types of object/camera motions and scaling variances. The novel “seek and spread” strategy used in key frame computing allows us to obtain a large representative range for the key frames. Inter-shot redundancy of the key-frames is suppressed using the same image similarity measure. Experimental results demonstrate the effectiveness and efficiency of our techniques.

In this paper we carry out a survey on key frame extraction methods for Video Summary. We also discuss the summary evaluation criteria and compare the approaches based on the method, data set and the results. Video Summary is a process of presenting an abstract of entire video within a short period of time. It aims to provide a compact video representation, while preserving the essential activities of the original video. It is an essential task in video analysis and indexing applications. Most of the video summaries are based on selection of key frames within the shots of a video. Many of them use motion features and few use visual features for extracting the key frames. The video summary quality assessment methods are based more on subjective and less on objective measures. Tong wei Ren et al has provided a framework to assess the quality of the video against a given reference summary using both subjective and objective measures. Ciocca et al used the objective measures for evaluation of summary and most of them evaluate by taking the subjective opinion of experts. A framework for automatic evalution is needed based on both subjective and objective measures without the reference summary.

There has been remarkable progress in the field of Semantic segmentation in recent years. Yet, it remains a challenging problem to apply segmentation to the video-based applications. Videos usually involve significantly larger volume of data compared to images. Particularly, a video contains around 30 frames per second. Segmentation of the similar frames unnecessarily adds to the time required for segmentation of complete video. In this paper, we propose a contour detection-based approach for detection of salient frames for faster semantic segmentation of videos. We propose to detect the salient frames of the video and pass only the salient frames through the segmentation block. Then, the segmented labels of the salient frames are mapped to the non-salient frames. The salient frame is defined by the variation in the pixel values of the background subtracted frames. The background subtraction is done using MOG2 background subtractor algorithm for background subtraction in various lighting conditions. We demonstrate the results using the Pytorch model for semantic segmentation of images. We propose to concatenate the semantic segmentation model to our proposed framework. We evaluate our result by comparing the time taken and the mean Intersection over Union (mIoU) for segmentation of the video with and without passing the video input through our proposed framework. We evaluate the results of Saliency Detection Block using Retention and Condensation ratio as the quality metrics.

Surface hardness monitoring of laser shock Peening: Acoustic emission and key frame selection

Rapid increase in video databases has forced the industry to have efficient and effective frameworks for video retrieval and indexing. Video segmentation into scenes is widely used for video summarization, partitioning, indexing and retrieval. In this paper, we propose a framework for scene detection mainly based on entropy and Speeded Up Robust Features (SURF) features. First, we detect the fade and abrupt boundaries based on frame entropy analysis and SURF features matching. Fade boundaries are smart indication of scenes beginning or ending in many videos and dramas, and are detected by frame entropy analysis. Before abrupt boundary detection, unnecessary frames which are obviously not abrupt boundaries, such as blank screens, high intensity influenced images, sliding credits, are removed. Candidate boundaries are detected to make SURF features efficient for abrupt boundary detection, and SURF features between candidate boundaries and their adjacent frames are used to detect the abrupt boundaries. Second, key frames are extracted from abrupt shots. We evaluate our key frame extraction with other famous algorithms and show the effectiveness of the key frames. Finally, scene boundaries are detected using sliding window of size K over the key frames in temporal order. In experimental evaluation on the TRECVID-2007 shot boundary test set, the algorithm for shot boundary achieves substantial improvements over state-of-the-art methods with the precision of 99% and the recall of 97.8%. Experimental results for video segmentation into scenes are also promising, compared to famous state-of-the-art techniques.

A key challenge in self-supervised video representation learning is how to effectively capture motion information besides context bias. While most existing works implicitly achieve this with video-specific pretext tasks (e.g., predicting clip orders, time arrows, and paces), we develop a method that explicitly decouples motion supervision from context bias through a carefully designed pretext task. Specifically, we take the key frames and motion vectors in compressed videos (e.g., in H.264 format) as the supervision sources for context and motion, respectively, which can be efficiently extracted at over 500 fps on CPU. Then we design two pretext tasks that are jointly optimized: a context matching task where a pairwise contrastive loss is cast between video clip and key frame features; and a motion prediction task where clip features, passed through an encoderdecoder network, are used to estimate motion features in a near future. These two tasks use a shared video backbone and separate MLP heads. Experiments show that our approach improves the quality of the learned video representation over previous works, where we obtain absolute gains of 16.0% and 11.1% in video retrieval recall on UCF101 and HMDB51, respectively. Moreover, we find the motion prediction to be a strong regularization for video networks, where using it as an auxiliary task improves the accuracy of action recognition with a margin of 7.4% ∼ 13.8%.

Shot reconstruction degree: a novel criterion for key frame selection

In this paper, we present a fast method of key frame selection and an accurate feature point matching method for 3D reconstruction through the control of quadrotor. The quadrotor is controlled to fly at a fixed altitude and the gimbal camera has always been a downward direction. Therefore, the additional information of the quadrotor could be easily added to the 3D reconstruction process. As a result, we find a fast method to select key frames and an optimized matching method through the geometric constraints of the flight path and direction. The method is mainly to improve the efficiency of key frame selection and to get more accurate matching points. The advantage of our approach is to combine the two processes between video capture and video processing by adding flight control to the 3D scene reconstruction method. Therefore, it can use less time to complete the entire reconstruction task. The proposed approach is tested by quadrotor platform. Experiment results show that our method can greatly reduce the key frame selection time and get more matching points at the same accuracy.

Efficient, interactive foreground/background segmentation in video is of great practical importance in video editing. This paper proposes an interactive and unsupervised video object segmentation algorithm named E-GrabCut concentrating on achieving both of the segmentation quality and time efficiency as highly demanded in the related filed. There are three features in the proposed algorithms. Firstly, we have developed a powerful, non-iterative version of the optimization process for each frame. Secondly, more user interaction in the first frame is used to improve the Gaussian Mixture Model (GMM). Thirdly, a robust algorithm for the following frame segmentation has been developed by reusing the previous GMM. Extensive experiments demonstrate that our method outperforms the state-of-the-art video segmentation algorithm in terms of integration of time efficiency and segmentation quality.