Scene change detection based on audio and video content analysis

Recently, the multimedia database management has been intensively studied due to the rapid growth of multimedia information and the increasing demand of its access over the Internet. Because a prohibitive amount of labor is involved in manual indexing, video indexing based on automatic content analysis has been thoroughly studied as an attractive alternative. As a result, video content analysis, which consists of both visual content analysis and audio content analysis, has attracted many research interests in recent years. This research trend, in turn, further brings the areas which are primarily built upon the content analysis module such as the video abstraction, video browsing and video retrieval, to be actively developed. In this chapter, a comprehensive survey of all these research topics will be performed.

A scene change detection method is presented in this paper, which analyzes both auditory and visual information sources and accounts for their inter-relations and coincidence to semantically identify video scenes. Audio analysis focuses on the segmentation of the audio source into three types of semantic primitives, i.e. silence, speech and music. Further processing on speech segments aims at locating speaker change instants. Video analysis attempts to segment the video source into shots, without the segmentation being affected by camera pans, zoom-ins/outs or significantly high object motion. Results from single source segmentation are in some cases suboptimal. Audio-visual interaction achieves to either enhance single source findings or extract high level semantic information. The aim of this paper is to identify semantically meaningful video scenes by exploiting the temporal correlations of both sources based on the observation that semantic changes are characterized by significant changes in both information sources. Experimentation has been carried on a real TV serial sequence composed of many different scenes with plenty of commercials appearing in-between. The results are proven to be rather promising.KeywordsSpeech SegmentAdaptive ThresholdingShot BoundaryFalse Alarm RatioShot Boundary DetectionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Developments in multimedia technologies have paved way for the storage of huge collections of video documents on computer systems. It is essential to design tools for content-based access to the documents, so as to allow an efficient exploitation of these collections. Content based analysis provides a flexible and powerfulway to access video data when compared with the other traditional video analysis techniques. The area of content based video indexing and retrieval (CBVIR), focusing on automating the indexing, retrieval and management of video, has attracted extensive research in the last decade. CBVIR is a lively area of research with enduring acknowledgments from several domains. Herein a vital assessment of contemporary researches associated with the content-based indexing and retrieval of visual information. In this paper, we present an extensive review of significant researches on CBVIR. Concise description of content based video analysis along with the techniques associated with the content based video indexing and retrieval is presented.

Content-based video retrieval is a proper solution to handle the video data. But because of their huge volumes and high dimensionality, finding a proper way to organize them for efficient search and retrieval becomes a challenging and important problem. In this paper, we present a framework for content-based video retrieval using the Shot Cluster Tree, while the latter organizes the content of shots in the tree structure. The framework can supply users with two types of queries: query-by-content and query-by-example. We also put up a novel and efficient clustering method for generating the Shot Cluster Tree structure, where shots which are visual similar and time adjacent are grouped into shot groups using the method of Sliding Shot Window firstly and then shot groups are clustered into shot clusters with the simple agglomerative hierarchical clustering method. In addition to constructing the structure of content, the Shot Cluster Tree also provides better ways to video summary and video annotation, which facilitate the video retrieval. An experimental system has been built up. Experiments verify the effectiveness of the proposed approach.

Text Segmentation of text from degraded document images is a very complex task due to high mutation between the document background and foreground region. Automatic text extraction is one of the basic feature required for content-based video indexing, automated indexing, automated annotation, structuring and retrieval tasks. Text detection from videos demands conversion of entire video into smaller framesets. Further the framesets are binarized to ease the extraction procedure. This in turn is followed by application of detection procedure on the static frames generated from the video. Text detection can lead to extraction of both superficial and embedded text. Embedded text will be the focus of this research paper because a part of the information depicted in the superficial text is already present in the embedded region. The cycle would start from conversion of dynamic video into static frames, followed by application of filters for noise removal, use of basic morphological operation like dilation and erosion, creation of bounding boxes around the textual content and finally removal of the non text region in such a manner that only the textual region in enhanced. The enhanced textual region is retained while the non textual content is eliminated.

Merits and defects of note-based video retrieval and content-based video retrieval are investigated in this paper, and complementarity of two aspects each other are also discussed. A new theory of semantic processing in content-based video retrieval is explored. An object-oriented representation modeling of the video semantics, based on the multi-layers, is presented. And on this basis, a semantic extraction frame based on the multi-semantics is constructed.

Video shot boundary detection plays an every important role in video processing. It is the first step toward video content analysis and content-based video retrieval. We develop a novel approach for video shot boundary detection based on supervised learning. Our method consists in first extracting video frame feature using a supervised kernel non-locality preserving projections, then video frames are split into abrupt transitions, gradual transitions or normal frames using two cascaded Localized-SVM classifiers. Experimental results show the effectiveness of our method.

With the increase of multimedia devices on internet, enormous number of videos are being added as part of digital content. There is a huge challenge in retrieval of these videos as mostly the videos are kept in unstructured form. Intended users try to retrieve video content as per the relevancy and need. Shot boundary detection is a significant and critical approach in the domain of digital video processing. It is the foremost critical job of content-based video retrieval and indexing. In this paper, a novel approach for shot boundary detection algorithm based on Differential evolution algorithm (DE) with SVM classifier is proposed. In this method we first calculate the curves difference of U-component histograms as the feature of difference between video frames. In the next step, Slide-Window Mean Filter to filter difference curves and SVM Classifier applying DE to detect and classify the shot transitions. The well-known TRECVID 2005, 2006, and 2007 datasets are used to test the performance of our proposed approach. The result shows the superior performance of our proposed approach, and this method can achieve high recall, precision rate, accuracy, and better computation time.

This work addresses the development of a unified approach to content-based indexing and retrieval of digital videos from television archives. The proposed approach has been designed to deal with arbitrary television genres, making it suitable for various applications. To achieve this goal, the main steps of a content-based video retrieval system are addressed in this work, namely: video segmentation, key-frame extraction, content-based video indexing and the video retrieval operation itself. Video segmentation is addressed as a typical TV broadcast structuring problem, which consists in automatically determining the boundaries of each broadcasted program (like movies, news, among others) and inter-program (for instance, commercials). Specifically, to segment the videos, Electronic Program Guide (EPG) metadata is combined with the detection of two special cues, namely, audio cuts (silence) and dark monochrome frames. On the other hand, a color histogram-based approach performs key-frame extraction. Video indexing and retrieval are accomplished by using hashing and k-d tree methods, while visual signatures containing color, shape and texture information are estimated for the key-frames, by using image and frequency domain techniques. Experimental results with the dataset of a multimedia information system especially developed for managing television broadcast archives demonstrate that our approach works efficiently, retrieving videos in 0.16 seconds on average and achieving recall, precision and F1 measure values, as high as 0.76, 0.97 and 0.86 respectively.

Chapter five - Content—Based Video Browsing and Retrieval

This paper proposes a method for content-based sports video retrieval using camera work information. Since particular camera work for a typical scene exists in sports videos, camera work transition becomes an effective cue for retrieving a sports scene based on its content. The proposed method extracts a series of camera parameters from both a user-specified scene of a retrieval key and a video stream of a retrieval target, and detects scenes having content similar to that of the key from the target by applying continuous DP matching. The method was evaluated using a video stream of a baseball game. Recall-Precision curves make its effectiveness clear.

Video is a rich information source containing both audio and visual information along with motion information embedded in it. Applications such as e-learning, live TV, video on demand, traffic monitoring, etc. need an efficient video retrieval strategy. Content-based video retrieval and superpixel segmentation are two diverse application areas of computer vision. In this work, we are presenting an algorithm for content-based video retrieval with help of Integration of Curvelet transform and Simple Linear Iterative Clustering (ICTSLIC) algorithm. Proposed algorithm consists of two steps: off line processing and online processing. In offline processing, keyframes of the database videos are extracted by employing features: Pearson Correlation Coefficient (PCC) and color moments (CM) and on the extracted keyframes superpixel generation algorithm ICTSLIC is applied. The superpixels generated by applying ICTSLIC on keyframes are used to represent database videos. On other side, in online processing, ICTSLIC superpixel segmentation is applied on query frame and the superpixels generated by segmentation are used to represent query frame. Then videos similar to query frame are retrieved through matching done by calculation of Euclidean distance between superpixels of query frame and database keyframes. Results of the proposed method are irrespective of query frame features such as camera motion, object’s pose, orientation and motion due to the incorporation of ICTSLIC superpixels as base feature for matching and retrieval purpose. The proposed method is tested on the dataset comprising of different categories of video clips such as animations, serials, personal interviews, news, movies and songs which is publicly available. For evaluation, the proposed method randomly picks frames from database videos, instead of selecting keyframes as query frames. Experiments were conducted on the developed dataset and the performance is assessed with different parameters Precision, Recall, Jaccard Index, Accuracy and Specificity. The experimental results shown that the proposed method is performing better than the other state-of-art methods.

The three packages libACA, pyACA, and ACA-Code provide reference implementations for basic approaches and algorithms for the analysis of musical audio signals in three different languages: C++, Python, and Matlab. All three packages cover the same algorithms, such as extraction of low level audio features, fundamental frequency estimation, as well as simple approaches to chord recognition, musical key detection, and onset detection. In addition, implementations of more generic algorithms useful in audio content analysis such as dynamic time warping and the Viterbi algorithm are provided. The three packages thus provide a practical cross-language and cross-platform reference to students and engineers implementing audio analysis algorithms and enable implementation-focused learning of algorithms for audio content analysis and music information retrieval.

The work presented here introduces a real-time automatic scene classifier within content-based video retrieval. In our envisioned approach end users like documentalists, not image processing experts, build classifiers interactively, by simply indicating positive examples of a scene. Classification consists of a two-stage procedure. First, small image fragments called patches are classified. Second, frequency vectors of these patch classifications are fed into a second classifier for global scene classification (e.g., city, portraits, or countryside). The first stage classifiers can be seen as a set of highly specialized, learned feature detectors, as an alternative to letting an image processing expert determine features a priori. The end user or domain expert thus builds a visual alphabet that can be used to describe the image in features that are relevant for the task at hand.We present results for experiments on a variety of patch and image classes. The scene classifier approach has been successfully applied to other domains of video content analysis, such as content-based video retrieval in television archives, automated sewer inspection, and porn filtering.

An on-line audio classification and segmentation system is presented in this research, where audio recordings are classified and segmented into speech, music, several types of environmental sounds and silence based on audio content analysis. This is the first step of our continuing work towards a general content-based audio classification and retrieval system. The extracted audio features include temporal curves of the energy function,the average zero- crossing rate, the fundamental frequency of audio signals, as well as statistical and morphological features of these curves. The classification result is achieved through a threshold-based heuristic procedure. The audio database that we have built, details of feature extraction, classification and segmentation procedures, and experimental results are described. It is shown that, with the proposed new system, audio recordings can be automatically segmented and classified into basic types in real time with an accuracy of over 90 percent. Outlines of further classification of audio into finer types and a query-by-example audio retrieval system on top of the coarse classification are also introduced.