Video caption detection and extraction using temporal information

Content-based retrieval in video databases has become an important task with the availability of large quantities of data in both public and proprietary archives. Most of video systems are based on feature classification, but problems appear because of “semantic gap” between high-level human concepts and the machine-readable low-level visual features. In this paper we adopt a relevance feedback approach (RF) to bridge the semantic gap by progressively collecting feedback from the user, which allows the machine to discover the semantic meanings of objects or events. Experimental tests conducted on more than 91 hours of video footage show an improvement of up to 90% in retrieval accuracy, compared to classic classification-based retrieval.

Efficient content-based search and retrieval of video databases has become an important industrial and consumer application due to rapid proliferation of compressed digital video data on the Internet and corporate Intranets, and due to the launch of high definition television (HDTV) broadcasts in 1998. We have developed the first HDTV video content management system that automatically analyzes motion occurring in MPEG-2 encoded videos within the compressed domain itself. Our system produces descriptors characterizing the global motion in videos for content retrieval and repurposing applications in a digital television studio. These motion-direction and magnitude based video labels can be directly, incorporated as annotation indexes into a database for querying, or used to construct higher level event descriptions of videos. Results from our ongoing experiments with tens of thousands of frames obtained from several MPEG-1,2 video streams of various genres demonstrate the good performance of our annotation system in terms of motion identification accuracy and computational efficiency.

According to the edge and gradient based algorithm for text information extraction of video image under the multimedia environment, the line structure of the video image is acted as a feature of text information to obtain the change law of line structure in video images. However, pseudo area, large noise and poor adaptability etc., appear during extraction step. Thus, a text information extraction algorithm of video images in multimedia environment is proposed. The shot of multimedia video images is segmented, and video images with text information are detected. The text of segmented video images is located, and the text information is extracted using linear extraction method. The maximum between-class variance method is adopted to optimize extracted text information, so as to achieve text information extraction of video images in multimedia environment. The experimental results show that the method considers the region of extracted text and situation of text noise in multimedia environment, and saves the extraction time compared to general extraction methods, and improves the extraction accuracy and adaptability, which can effectively advance the performance of text extraction.

It is substantiated that important in this direction is the issue of ensuring security and the necessary quality of information service provision, including the timely transfer of various types of information with sufficient quality and reliability. It is established that video information is of special importance here. However, the required bit rate level of video stream transmission is possible only in conditions of low quality of video segments. Therefore, a significant imbalance between the information load and the bandwidth of information communication networks contributes. This becomes a causal factor in the loss of the level of credibility and adequacy of video information. So, it is shown that the current issue of research concerns the improvement of the quality of providing video information services using the mobile segment on the basis of eliminating the imbalance between the significant intensity of video information flows and the limited productivity of information communications. The article proposes to use an approach based on the processing of a sequence of video segments in temporal and intra-frame spaces using the detection and elimination of new structural-statistical and structural and contextual regularities in the spectral basis. Here, to build new approaches to coding video segments, including informative video segments, it is proposed to use the theoretical foundations of non-equilibrium positional coding in spectral space.

In this paper, we describe a fully automatic video retrieval prototype system that uses an image or a video sequence of an interested identity as probe. The system is based on face vision techniques including face detection and tracking, face alignment and recognition. Given a film or TV sitcom, first face trajectories are extracted in video by head tracking that decompose the video into segments corresponding to certain identity, then frames containing faces of higher quality are selected and normalized according to face alignment results, and finally different segments are associated by face recognition. Experiments are carried out on news video, feature length film video and TV sitcom to show its effectiveness. Potential usage of our system includes intelligent DVD/VCD browsing, video database retrieval, meeting record browsing, etc.

Concept classification for consumer videos is a challenging task considering the co-occurrence of a variety objects and arbitrary motions in video segments. In this paper, we present a novel video concept classification framework with random-sampling-based spatialtemporal features. Short-term random-sampled point tracks are obtained within video segments. The spatial-temporal features are extracted from these tracks. Concept codebooks are constructed using Multiple Instance Learning upon the spatial-temporal features. The SVM classifiers are trained over codebook-based histograms for an online concept detection. We performed experiments on a video database taken from YouTube. The experimental results demonstrate that the consumer videos can be efficiently assigned concept labels by our approach.

This paper presents an analysis process of lecture video retrieval with automated video indexing and video search for lecture video databases. The video retrieval extracts the relevant metadata from the two main parts of lecture videos, namely the visual screen and audio tracks. From the visual screen, we firstly detect the slide transition and extract each unique slide frame with its temporal transition considered as the video segment. The textual metadata from slide frames is extracted and then recognized using video OCR technique. Based on OCR results, the corresponding text in video information is saved. Secondly, the speech-to-text analysis is carried out from the audio signal. Sphinx speech recognition models are used for recognition of speech-to-text conversion process. In the proposed work, the combined text information from the visual slide frame and audio signal were used to retrieve the video from the lecture video database. The performance of this combined video retrieval system shows a significant improvement in performance when compared with an individual system built using audio and text in video systems respectively.

We consider the problem of referring segmentation in images and videos with natural language. Given an input image (or video) and a referring expression, the goal is to segment the entity referred by the expression in the image or video. In this paper, we propose a cross-modal self-attention (CMSA) module to utilize fine details of individual words and the input image or video, which effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the visual input. We further propose a gated multi-level fusion (GMLF) module to selectively integrate self-attentive cross-modal features corresponding to different levels of visual features. This module controls the feature fusion of information flow of features at different levels with high-level and low-level semantic information related to different attentive words. Besides, we introduce cross-frame self-attention (CFSA) module to effectively integrate temporal information in consecutive frames which extends our method in the case of referring segmentation in videos. Experiments on benchmark datasets of four referring image datasets and two actor and action video segmentation datasets consistently demonstrate that our proposed approach outperforms existing state-of-the-art methods.

Chapter 21 - Optical flow-based representation for video action detection

Extraction of textual information is a very fundamental step in the domain of information retrieval, searching, editing, documenting, archiving, or recording the appearance of text. Appearance of text means text information is embedded in the image. Textual information can be obtained from various forms like scanners, magazines, newspapers, etc. In this survey paper, an elaborative review on text extraction is given, which includes different classifiers, feature extraction and segmentation techniques.

Novelty detection in social video has drawn much attention of researchers and is applied to many tasks in real-world applications, such as e-commerce and e-learning. Existing methods cannot address this issue effectively, since most of them do not consider the quality of videos or the long-term information of online social videos. In this paper, we propose a general framework, Long-term Information REconstruction-based Model (LIREM), which cleans the video feature information and captures both short-term and long-term spatial-temporal information of video segments to detect novelty online. We first design a novel outlier detection method for feature cleaning to improve the learning performance. Then, an LSTM-Decoder model is constructed and applied to the cleaned video segments for predicting the reconstruction error of video features. Our experiments are conducted on three real datasets, and the experimental results demonstrate the performance of our model outperforms other novelty detection models.KeywordsFeature cleaningNovelty detectionOnline social video

In this paper, a model of structure semantics of image is proposed. We describe the detailed process of mathematical inference for the whole system. The model applies the knowledge of topology, neural networks, and psychology. The system first builds a structure space T using the structure parameters extracted from an image or video segment, then T is mapped to a structure information space IT by a transformation function set phi which is based on psychometric functions. The model introduces a new regulation based on psychology and neural networks for semantic analysis of image or video databases, and we call it psychological response neural network

Rapid increase in video databases has forced the industry to have efficient and effective frameworks for video retrieval and indexing. Video segmentation into scenes is widely used for video summarization, partitioning, indexing and retrieval. In this paper, we propose a framework for scene detection mainly based on entropy and Speeded Up Robust Features (SURF) features. First, we detect the fade and abrupt boundaries based on frame entropy analysis and SURF features matching. Fade boundaries are smart indication of scenes beginning or ending in many videos and dramas, and are detected by frame entropy analysis. Before abrupt boundary detection, unnecessary frames which are obviously not abrupt boundaries, such as blank screens, high intensity influenced images, sliding credits, are removed. Candidate boundaries are detected to make SURF features efficient for abrupt boundary detection, and SURF features between candidate boundaries and their adjacent frames are used to detect the abrupt boundaries. Second, key frames are extracted from abrupt shots. We evaluate our key frame extraction with other famous algorithms and show the effectiveness of the key frames. Finally, scene boundaries are detected using sliding window of size K over the key frames in temporal order. In experimental evaluation on the TRECVID-2007 shot boundary test set, the algorithm for shot boundary achieves substantial improvements over state-of-the-art methods with the precision of 99% and the recall of 97.8%. Experimental results for video segmentation into scenes are also promising, compared to famous state-of-the-art techniques.

Scene text information extraction plays an important role in many computer vision applications. Most features in existing text extraction algorithms are only applicable to one text extraction stage (text detection or recognition), which significantly weakens the consistency in an end-to-end system, especially for the complex Chinese texts. To tackle this challenging problem, we propose a novel text structure feature extractor based on a text structure component detector (TSCD) layer and residual network for Chinese texts. Inspired by the three-layer Chinese text cognition model of a human, we combine the TSCD layer and the residual network to extract features suitable for both text extraction stages. The specialized modeling for Chinese characters in the TSCD layer simulates the key structure component cognition layer in the psychological model. And the residual mechanism in the residual network simulates the key bidirectional connection among the layers in the psychological model. Through the organic combination of the TSCD layer and the residual network, the extracted features are applicable to both text detection and recognition, as humans do. In evaluation, both text detection and recognition models based on our proposed text structure feature extractor achieve great improvements over baseline CNN models. And an end-to-end Chinese text information extraction system is experimentally designed and evaluated, showing the advantage of the proposed feature extractor as a unified feature extractor.

In multimedia-based e-learning systems, the accessibility and searchability of most lecture video content is still insufficient due to the unscripted and spontaneous speech of the speakers. Thus, it is very desirable to enable people to navigate and access specific topics within lecture videos by performing an automatic topic-wise video segmentation. This problem becomes even more challenging when the quality of such lecture videos is not sufficiently high. To this end, we first present the ATLAS system that has two main novelties: (i) a SVM hmm model is proposed to learn temporal transition cues and (ii) a fusion scheme is suggested to combine transition cues extracted from heterogeneous information of lecture videos. Subsequently, considering that contextual information is very useful in determining knowledge structures, we present the TRACE system to automatically perform such a segmentation based on a linguistic approach using Wikipedia texts. TRACE has two main contributions: (i) the extraction of a novel linguistic-based Wikipedia feature to segment lecture videos efficiently, and (ii) the investigation of the late fusion of video segmentation results derived from state-of-the-art algorithms. Specifically for the late fusion, we combine confidence scores produced by the models constructed from visual, transcriptional, and Wikipedia features. According to our experiments on lecture videos from VideoLectures.NET and NPTEL, the proposed algorithms in the ATLAS and TRACE systems segment knowledge structures more accurately compared to existing state-of-the-art algorithms.