Two Stage Shot Boundary Detection via Feature Fusion and Spatial-Temporal Convolutional Neural Networks

Video segmentation is fundamental to a number of applications related to video retrieval and analysis. Shot change detection is the initial step of video segmentation and indexing. There are two basic types of shot changes. One is the abrupt change or cut, and the other is the gradual shot transition. The smooth variations of the video feature values in a gradual transition produced by the editing effects are often confused with those caused by camera or object motions. To overcome this difficulty, it is reasonable to estimate the motions and suppress the disturbance caused by them. In this thesis, we explore the possibility to exploit motion and illumination estimation in a video sequence to detect both abrupt and gradual shot changes. A generalized optical flow constraint that includes an illumination parameter to model local illumination changes is employed in the motion and illumination estimation. An iterative process is used to refine the generalized optical flow constraints step by step. A robust measure that is the likelihood ratio of the corresponding motion-compensated blocks in the consecutive frames is used for detecting abrupt changes. For the detection of gradual shot transitions, we compute the average monotony of intensity variations on the stable pixels in the images in a twin-comparison framework. We test the proposed algorithm on a number of video sequences in TREC 2001 and compare the detection results with the best results reported in the TREC 2001 benchmark. The comparisons indicate that the proposed shot change detection algorithm is competitive against the best existing algorithms.

The temporal segmentation of a video into shots is a fundamental prerequisite for video retrieval. There are two types of shot boundaries: abrupt shot changes (“cuts”) and gradual transitions. Several high-quality algorithms have been proposed for detecting cuts, but the successful detection of gradual transitions remains a surprisingly difficult problem in practice. In this paper, we present an unsupervised approach for detecting gradual transitions. It has several advantages. First, in contrast to alternative approaches, no training stage and hence no training data are required. Second, no thresholds are needed, since the used clustering approach separates classes of gradual transitions and non-transitions automatically and adaptively for each video. Third, it is a generic approach that does not employ a specialized detector for each transition type. Finally, the issue of removing false alarms caused by camera motion is addressed: in contrast to related approaches, it is not only based on low-level features, but on the results of an appropriate algorithm for camera motion estimation. Experimental results show that the proposed approach achieves very good performance on TRECVID shot boundary test data.KeywordsVideo indexinggradual transition detectionshot boundary detectiongradual video shot changesvideo retrieval

Video shot boundary detection (SBD) is the first step of video analysis, summarization, indexing, and retrieval. In SBD process, videos are segmented into basic units called shots. In this paper, a new SBD method is proposed using color, edge, texture, and motion strength as vector of features (feature vector). Features are extracted by projecting the frames on selected basis vectors of Walsh-Hadamard transform (WHT) kernel and WHT matrix. After extracting the features, based on the significance of the features, weights are calculated. The weighted features are combined to form a single continuity signal, used as input for Procedure Based shot transition Identification process (PBI). Using the procedure, shot transitions are classified into abrupt and gradual transitions. Experimental results are examined using large-scale test sets provided by the TRECVID 2007, which has evaluated hard cut and gradual transition detection. To evaluate the robustness of the proposed method, the system evaluation is performed. The proposed method yields F1-Score of 97.4% for cut, 78% for gradual, and 96.1% for overall transitions. We have also evaluated the proposed feature vector with support vector machine classifier. The results show that WHT-based features can perform well than the other existing methods. In addition to this, few more video sequences are taken from the Openvideo project and the performance of the proposed method is compared with the recent existing SBD method.

Shot boundary detection is a pre-requisite process for content-based video indexing and retrieval, an application domain that has attracted much research and consumer attentions due to the explosive growth of video data available and accessible to users worldwide. Currently, a number of edge-based techniques have been proposed in the literature for detecting abrupt shot boundaries to avoid the influence of flashlights common in many video types, such as sports, news, entertainment, and interviews videos. However, these techniques are susceptible to miss and false detections of abrupt shot changes. Our study shows that one main reason for these errors is due to the presence of superimposed texts that are common in various video genres. To address this problem, we present in this paper an efficient method that utilizes the edge type - text-edge (edge in text region) or non-text-edge (edge in non-text region) - to reduce erroneous detection due to the sudden appearance and disappearance of superimposed text. Extensive experiments have been conducted and the results show that our proposed method, as compared with other existing methods, can obtain better performance in detecting abrupt shot boundaries and differentiating them from the effects of superimposed text.

We propose a method of shot boundary detection based on the co-occurrence of global motion in video stream. In addition to the conventional features based on appearance and local motion, we apply ST (Space-Time) patch analysis for detecting global motion in video stream. And then we perform shot boundary detection by constructing AdaBoost classifiers which represent the co-occurrence of global motion and the conventional features. Experimental results show that our method had 3.8% higher F-measure value than that of the conventional method for gradual shot boundary detection.

ABSTRACTWith the advent of a digital era, people have encountered some difficulty in using and absorbing overwhelming information generated by technological advances in multimedia. Thus, the development of video summarization enables people to catch a general idea about videos in a short time. In this paper, we focus on the shot change, a part of the video summarization, to conduct an experimental sample on news programs. Moreover, a high-level fuzzy Petri net model is presented to describe the frame combination which indicates a shot boundary used for a video frame sequence in order to detect both cut transitions and gradual transitions. This study has used feature functions to estimate the direct shot change in consideration of video shot boundary detection which adopts the HLFPN model to find a threshold value. The experimental results manifest that the proposed system saves a lot of time and reduces the occurrence of improper shot changes caused by the motions of objects and cameras when comparing the proposed approach with other existing ones.

Many recent cases of illegal activities have been reported regarding the video and all the unlawful activities designed to destroy the originality of the video data. A slight modification can alter the meaning and intent of the video. Nowadays, many devices have gained the power of tapping or capturing digital video, and all these videos can be approved over the internet and many other unsafe channels. Therefore, the detection o of video authentication proving is a widespread and critical problem in various areas such as criminal, forensic investigations, and court cases. The preceding study proposed numerous techniques to ensure the authentication of video data and structure. This study emphasized retrieving a video structure to detect an alteration made in the video using shot boundary detection to prove authentication. The detection of shot boundary is a complicated task and is usually utilized for structured video because these techniques can identify modifications made with a fundamental shot change frequency. This study focuses on retrieving a video structure to detect an alteration made in the video by detecting shot boundary to prove authentication. The detection of shot boundary is a complicated task and is usually utilized for structured video because these techniques can identify modifications made with a regular shot change frequency. Furthermore, this study aims to develop a new model for video authentication detection using the shot boundary detection method, which is considered the practical tool to track down the originality based on their fundamental structure.

In content-based video analysis, shot boundary detection is a common first step to segment video data into fundamental units of shots, each composing consecutive frames filmed with a single camera act. Many methods have been proposed in the literature for detection of shot boundaries. In this paper, we propose a new and effective post-refinement method on the detected shot boundaries by performing sequential detection of abrupt change in two underlying distributions. Experimental results show that the proposed method can eliminate most false detections and also recover many missed detections from the original detected shot boundaries, attaining better detection performance.

One of the essential pre-processing steps of semantic video analysis is the video shot boundary detection (SBD). It is the primary step to segment the sequence of video frames into shots. Many SBD systems using supervised learning have been proposed for years; however, the training process still remains its principal limitation. In this paper, a multi-modal visual features-based SBD framework is employed that aims to analyze the behaviors of visual representation in terms of the discontinuity signal. We adopt a candidate segment selection that performs without the threshold calculation but uses the cumulative moving average of the discontinuity signal to identify the position of shot boundaries and neglect the non-boundary video frames. The transition detection is structurally performed to distinguish candidate segment into a cut transition and a gradual transition, including fade in/out and logo occurrence. Experimental results are evaluated using the golf video clips and the TREC2001 documentary video data set. Results show that the proposed SBD framework can achieve good accuracy in both types of video data set compared with other proposed SBD methods.

At present, content-based video retrieval (CBVR) is the most mainstream video retrieval method, using the video features of its own to perform automatic identification and retrieval. This method involves a key technology, i.e. shot segmentation. In this paper, the method of automatic video shot boundary detection with K-means clustering and improved adaptive dual threshold comparison is proposed. First, extract the visual features of every frame and divide them into two categories using K-means clustering algorithm, namely, one with significant change and one with no significant change. Then, as to the classification results, utilize the improved adaptive dual threshold comparison method to determine the abrupt as well as gradual shot boundaries.Finally, achieve automatic video shot boundary detection system.

Video Shot Boundary Detection (SBD) is the fundamental process towards video summarization and retrieval. A fast and efficient SBD algorithm is necessary for real-time video processing applications. Extensive work has focused on accurate shot boundary detection at the expense of demanding computational costs. In this paper, we propose a fast SBD approach that reduces the computation pixel-wise and frame-wise while still giving satisfactory accuracy. The proposed approach substantially speeds up the computation through reducing both detection region and scope. Color histogram and mutual information are used together to measure the difference between frames. Corner distribution of frames is utilized to exclude most of false boundaries. We conduct extensive experiments to evaluate the proposed approach, and the results show that our approach can not only speed up SBD, but also detect shot boundaries with high accuracy in both Cut (CUT) and Gradual Transition (GT) boundaries.

Recent advances in computing, communication, and sensor technology are pushing the development of many new applications. This trend is especially evident in pervasive computing. The content-based video analysis is one example of this innovation. Shot boundary detection servers as an important step to structure the contents of videos, many research efforts have been devoted to this field. A novel approach is proposed to detect cuts and gradual shots. An effective feature selection mechanism is introduced using shot boundary coefficient (SBC) combined with HSV divisional histogram. SBC is derived from a model called average sliding-window difference. It is a novel way to enlarge frame difference. SBC can remove the factors affected by camera/object motion, illumination and other artifacts on the curve of frame-wise histogram difference. Each area of the frame has different weight according to its respective importance. In order to improve the speed, at the beginning of the program, we adopt n rank Newton interpolation formula to deflate the frame size. We use Self-Organizing Map (SOM) Network to detect the abrupt shot. It can avoid the threshold completely. Gauss Model is selected to detect gradual shots. Based on the set of abrupt shots, we qualify the candidate set of the gradual shot furthermore. The experimental results have shown that a good compromise between speed and accuracy is achieved. It also has good noise tolerance characteristics.

Shot boundary detection has attracted much more research interesting in recent years, but many of the algorithms proposed in the literatures are not robust to flash lamp and motion of camera and object. This paper present a novel shot boundary detection algorithm based on graph theory. It can detect cut and gradual shot and distinguish them, and it is not intensive to all of those interference factors. At first the feature of color is extracted, the dissimilarity of video frames is defined. Then the video frames are divided into several different groups through performing graph-theoretical algorithm. According to cut and gradual changes have the different characters on the two successive frames belong to different groups, it detect cut and gradual shot. The efficiency of the proposed algorithm is extensively tested on movie, sports and other videos. The experiments results indicate the method had a high accurate rate in both cut shot detection and gradual shot detection.

An adaptive threshold shot detection algorithm based on improved block color features is proposed in this paper. This paper adopts an improved block color feature extraction method based on equal area of rectangular ring. Sub-block accumulative color histogram is extracted as color features and different weight for different rectangle rings is set in order to highlight the central part of frame. Then, adaptive threshold of detecting abrupt shot and gradual shot is calculated, and different detection modules is used according to the distance of the characteristics between frames. In the abrupt shots detection, several frames' frame difference and the edge shape features between adjacent frames are calculated to detect the flash. In the gradual shots detection, the discontinuous frame differences between the current frame and back frames are used to detect the boundary of the gradual shot. The experimental results show that this method has better effect to different types of video.

In the European project SMASH mass-market storage systems for domestic use are under study. Besides the storage technology that is developed in this project, the related objective of user-friendly browsing/query of video data is studied as well. Key issues in developing a user-friendly system are (1) minimizing the user-intervention in preparatory steps (extraction and storage of representative information needed for browsing/query), (2) providing an acceptable representation of the stored video content in view of a higher automation level, (3) the possibility for performing these steps directly on the incoming stream at storage time, and (4) parameter-robustness of algorithms used for these steps. This paper proposes and validate novel approaches for automation of mentioned preparatory phases. A detection method for abrupt shot changes is proposed, using locally computed threshold based on a statistical model for frame-to-frame differences. For the extraction of representative frames (key frames) an approach is presented which distributes a given number of key frames over the sequence depending on content changes in a temporal segment of the sequence. A multimedia database is introduced, able to automatically store all bibliographic information about a recorded video as well as a visual representation of the content without any manual intervention from the user.