Integrated video shot segmentation algorithm

Shot boundary detection is essentially to detect the position of frames where the shot changes. It has been actively studied in video analysis and management for convenience, which becomes a key technique with the rapid proliferation of rich and diverse videos. With respect to the complex characteristics of different shots in varying length and content variation property, in this paper we present a two stage method for shot boundary detection (TSSBD) which distinguishes abrupt shot by fusing color histogram and deep features, and locate gradual shot changes with C3D-based deep analysis. Abrupt shot changes are detected firstly as it occurs between two frames, which divides the complete video into segments containing gradual transitions; Over these video segments, gradual shot change detection is implemented using 3D-convolutional neural network, which classifies clips into specific gradual shot change types; Finally, an effective merging strategy is proposed to locate positions of gradual shot transitions. The experimental analysis illustrates that the proposed progressive method is capable of detecting both abrupt shot transitions and gradual shot transitions accurately.

Temporal segmentation of a video sequence into different shots is fundamental to a number of video retrieval and analysis applications. Motion estimation has been wildly used in many applications of video processing, since it provides the most essential information for an image sequence. In this paper, we explore the possibility to exploit motion and illumination estimation in a video sequence to detect various types of shot changes. Optical flow is the motion vector computed at each pixel in an image sequence from intensity variation. Traditionally, optical flow computational algorithms were derived from the brightness constancy assumption. In this paper, we employ a generalized optical flow constraint that includes an illumination parameter to model local illumination changes. An iterative optical flow and illumination estimation algorithm is developed in this paper to refine the generalized optical flow constraints step by step, thus leading to a very accurate estimation of the optical flow and illumination parameters. Two robust measures are defined from the mean and standard deviation of the estimated intensity compensation values for all the blocks in the same image. Either of these two measures corresponds significantly to various types of shot changes. We show the usefulness of these two measures through experiments.

The temporal segmentation of a video into shots is a fundamental prerequisite for video retrieval. There are two types of shot boundaries: abrupt shot changes (“cuts”) and gradual transitions. Several high-quality algorithms have been proposed for detecting cuts, but the successful detection of gradual transitions remains a surprisingly difficult problem in practice. In this paper, we present an unsupervised approach for detecting gradual transitions. It has several advantages. First, in contrast to alternative approaches, no training stage and hence no training data are required. Second, no thresholds are needed, since the used clustering approach separates classes of gradual transitions and non-transitions automatically and adaptively for each video. Third, it is a generic approach that does not employ a specialized detector for each transition type. Finally, the issue of removing false alarms caused by camera motion is addressed: in contrast to related approaches, it is not only based on low-level features, but on the results of an appropriate algorithm for camera motion estimation. Experimental results show that the proposed approach achieves very good performance on TRECVID shot boundary test data.KeywordsVideo indexinggradual transition detectionshot boundary detectiongradual video shot changesvideo retrieval

<p>Segmentation of the video sequence by detecting shot changes is essential for video analysis, indexing and retrieval. In this context, a shot boundary detection algorithm is proposed in this paper based on the scale invariant feature transform (SIFT). The first step of our method consists on a top down search scheme to detect the locations of transitions by comparing the ratio of matched features extracted via SIFT for every RGB channel of video frames. The overview step provides the locations of boundaries. Secondly, a moving average calculation is performed to determine the type of transition. The proposed method can be used for detecting gradual transitions and abrupt changes without requiring any training of the video content in advance. Experiments have been conducted on a multi type video database and show that this algorithm achieves well performances.</p>

Shot change detection is the main technique for automatic temporal segmentation of video. Until now, there were little research reports about real-time shot change detection for applying to hardware terminals with low performance such as PMPs (Portable Multimedia Player) and cellular phones. In this paper, we propose an adaptive real-time shot change detection algorithm using sub-sampling, weighting variance, and an adaptive threshold. We use weighting variance of frame difference for dividing feature extraction at a changed frame. We set an adaptive threshold using the average of variance from the next one of a last shot change frame to the previous one of a current frame. In the experiments, we obtained the detection rate of about 94.4% in overall accuracy. We verified real-time operation of shot change detection by implementing our algorithm on the TVUS HM-900 PMP from the company of Homecast. Therefore, our proposing algorithm will be helpful in searching video data on portable media player such as PMPs and cellular phones.

This paper deals with the problem of shot-change detection on H.264/AVC compressed video. As H.264/AVC employs several new coding tools, the statistic information of the macroblock is different from the former MPEG video. How to detect shot-changes efficiently on H.264/AVC compressed video is a challenging work. In this paper, we propose a novel scheme, a macroblock type analysis plus the intra-mode statistical constraint rule, to detect shot-changes for P-frame and B-frame coded video of H.264/AVC. For detecting shot-changes in I-frame coded video, we propose an intra mode histogram and use the weighted city block distance to measure the similarities among I-frames. Experimental results show that the proposed algorithms achieve satisfactory performance and very fast detection.

Shot change detection is a main technique for automatic temporal segmentation of video, which requires adaptive adjustment of thresholds and real-time operation in actual applications. To date, few researches have been conducted on real-time shot change detection to be applied to hardware terminals with low performance such as PMPs (portable media player) and cellular phones. In this paper, we propose an adaptive real-time shot change detection algorithm and implement it on an actual PMP. The proposed algorithm consists of the following elements: sub-sampling, weighting variance, and adaptive thresholds. Our experiments obtained the detection rate of about 94.4% in overall accuracy, demonstrating a higher detection rate and a computational amount 1/256 less than the conventional algorithms which have optimal fixed thresholds adjusted by humans for each video. We verified the real-time operation of shot change detection by implementing our algorithm on the PMP of Homecast, a PMP company. The proposed algorithm will be helpful in searching video data on portable media players such as PMPs and cellular phones.

A new method for detecting shot boundaries in video sequences by fusing features obtained by singular value decomposition (SVD) and mutual information (MI) is proposed. The first method relies on performing singular value decomposition on a matrix created from 3D color histograms of single frames. The method can detect cuts and gradual transitions, such as dissolves, fades and wipes. The second method relies on evaluating mutual information between two consecutive frames. It can detect abrupt cuts, fade-ins and fade-outs with very high accuracy. A combination of features derived from these methods and subsequent processing through a clustering procedure results in very efficient detection of abrupt cuts and gradual transitions, as demonstrated by experiments on the TRECVID2004 video test set containing different types of shots with significant object and camera motion inside the shots.

ABSTRACTWith the advent of a digital era, people have encountered some difficulty in using and absorbing overwhelming information generated by technological advances in multimedia. Thus, the development of video summarization enables people to catch a general idea about videos in a short time. In this paper, we focus on the shot change, a part of the video summarization, to conduct an experimental sample on news programs. Moreover, a high-level fuzzy Petri net model is presented to describe the frame combination which indicates a shot boundary used for a video frame sequence in order to detect both cut transitions and gradual transitions. This study has used feature functions to estimate the direct shot change in consideration of video shot boundary detection which adopts the HLFPN model to find a threshold value. The experimental results manifest that the proposed system saves a lot of time and reduces the occurrence of improper shot changes caused by the motions of objects and cameras when comparing the proposed approach with other existing ones.

Almost every shot change detection algorithm detects abrupt transition (hard cut) without difficulty, but gradual transitions such as fades, dissolves, wipes are left as hard-to-detect problems. Dissolve effect, among the various gradual transition effects, is one of the most frequently used shot transition methods with special semantic meaning such as scene transition. Information of the shot change type can also be the basis for the shot clustering algorithms. In this paper, we present a fast and effective dissolve-transition detection algorithm based on the spatio-temporal distribution of the macro block pipes in MPEG-compressed video. In the proposed algorithm, the ratio of forward macro blocks in the B-type frames and the spatial distribution of forward/backward macro blocks are utilized for detecting dissolve transition. After finding such sequence of frames, we apply 2 heuristic rules: (1) The global color distributions of the frames at which dissolve starts and terminates are very different, and (2) The duration of dissolve transition is typically more than 0.3 second.

In this paper, to detect the moving objects between frames in compressed video and to obtain the best compression video and the noiseless video. We describe a video in which frames by classifying macroblocks (MB), and describe motion estimation (ME), motion vector field (MV) and motion compensation (MC). we propose to classify Macroblocks of each video frame into different classes and use this class information to describe the frame content based on the motion vector. MB class information video applications such as shot change detection, motion discontinuity detection, Outlier rejection for global motion estimation. To reduce the noise and to improve the clarity of the compressed video by using contrast limited adaptive histogram equalization (CLAHE) Algorithm.

We consider the problem of referring segmentation in images and videos with natural language. Given an input image (or video) and a referring expression, the goal is to segment the entity referred by the expression in the image or video. In this paper, we propose a cross-modal self-attention (CMSA) module to utilize fine details of individual words and the input image or video, which effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the visual input. We further propose a gated multi-level fusion (GMLF) module to selectively integrate self-attentive cross-modal features corresponding to different levels of visual features. This module controls the feature fusion of information flow of features at different levels with high-level and low-level semantic information related to different attentive words. Besides, we introduce cross-frame self-attention (CFSA) module to effectively integrate temporal information in consecutive frames which extends our method in the case of referring segmentation in videos. Experiments on benchmark datasets of four referring image datasets and two actor and action video segmentation datasets consistently demonstrate that our proposed approach outperforms existing state-of-the-art methods.

Shot boundary detection is a pre-requisite process for content-based video indexing and retrieval, an application domain that has attracted much research and consumer attentions due to the explosive growth of video data available and accessible to users worldwide. Currently, a number of edge-based techniques have been proposed in the literature for detecting abrupt shot boundaries to avoid the influence of flashlights common in many video types, such as sports, news, entertainment, and interviews videos. However, these techniques are susceptible to miss and false detections of abrupt shot changes. Our study shows that one main reason for these errors is due to the presence of superimposed texts that are common in various video genres. To address this problem, we present in this paper an efficient method that utilizes the edge type - text-edge (edge in text region) or non-text-edge (edge in non-text region) - to reduce erroneous detection due to the sudden appearance and disappearance of superimposed text. Extensive experiments have been conducted and the results show that our proposed method, as compared with other existing methods, can obtain better performance in detecting abrupt shot boundaries and differentiating them from the effects of superimposed text.

In the European project SMASH mass-market storage systems for domestic use are under study. Besides the storage technology that is developed in this project, the related objective of user-friendly browsing/query of video data is studied as well. Key issues in developing a user-friendly system are (1) minimizing the user-intervention in preparatory steps (extraction and storage of representative information needed for browsing/query), (2) providing an acceptable representation of the stored video content in view of a higher automation level, (3) the possibility for performing these steps directly on the incoming stream at storage time, and (4) parameter-robustness of algorithms used for these steps. This paper proposes and validate novel approaches for automation of mentioned preparatory phases. A detection method for abrupt shot changes is proposed, using locally computed threshold based on a statistical model for frame-to-frame differences. For the extraction of representative frames (key frames) an approach is presented which distributes a given number of key frames over the sequence depending on content changes in a temporal segment of the sequence. A multimedia database is introduced, able to automatically store all bibliographic information about a recorded video as well as a visual representation of the content without any manual intervention from the user.

Adaptive fractional motion and disparity estimation skipping in MV-HEVC