Video retrieval using speech and text in video

In multimedia-based e-learning systems, the accessibility and searchability of most lecture video content is still insufficient due to the unscripted and spontaneous speech of the speakers. Thus, it is very desirable to enable people to navigate and access specific topics within lecture videos by performing an automatic topic-wise video segmentation. This problem becomes even more challenging when the quality of such lecture videos is not sufficiently high. To this end, we first present the ATLAS system that has two main novelties: (i) a SVM hmm model is proposed to learn temporal transition cues and (ii) a fusion scheme is suggested to combine transition cues extracted from heterogeneous information of lecture videos. Subsequently, considering that contextual information is very useful in determining knowledge structures, we present the TRACE system to automatically perform such a segmentation based on a linguistic approach using Wikipedia texts. TRACE has two main contributions: (i) the extraction of a novel linguistic-based Wikipedia feature to segment lecture videos efficiently, and (ii) the investigation of the late fusion of video segmentation results derived from state-of-the-art algorithms. Specifically for the late fusion, we combine confidence scores produced by the models constructed from visual, transcriptional, and Wikipedia features. According to our experiments on lecture videos from VideoLectures.NET and NPTEL, the proposed algorithms in the ATLAS and TRACE systems segment knowledge structures more accurately compared to existing state-of-the-art algorithms.

Lecture videos are rich with textual information and to be able to understand the text is quite useful for larger video understanding/analysis applications. Though text recognition from images have been an active research area in computer vision, text in lecture videos has mostly been overlooked. In this paper, text extraction from lecture videos are focused. For text extraction from different types of lecture videos such as slides, whiteboard lecture videos, paper lecture videos, etc. The text extraction, the text regions are segmented in video frames and extracted using recurrent neural network based OCR. And finally, the extracted text is converted into audio for ease of convenience. The designed algorithm is tested on different videos from different lectures. The experimental results show that the proposed methodology is quite efficient over existing work.

In this paper we present our experiments on text-based video indexing and retrieval. Due to expected OCR errors and the lack of semantic breadth in video text, we proposed two solutions: 1) expanding the semantics of the query word, and 2) using Glimpse to perform approximate matching instead of exact matching. The results we achieved showed that semantic expansion and Glimpse can play important roles in video retrieval based on text.

Digitalized video has become an established storage and exchange medium due to the fast development in recording technology. E-lecturing has become added professional popular. The number of lecture video data on the web is growing quickly. Therefore, an additional skillful method for video retrieval within huge lecture video archives is immediately desired. This paper comes close to regular video indexing and video stalk in large lecture videos collection. We apply Automatic video segmentation and key-frame detection to present an image instruction for the video content navigation. In this process, video will be retrieved based on content based video search for the input of speech and video text content. Histogram of gradient and Support Vector Machine is used for the feature extraction from the video and to get the accurate result respectively. This will collect required features from lecture video database as labeled with speech recognition and also retrieve lecture video from database as better than existing algorithm. Hence the proposed system will be more efficient for retrieving the videos and also improves the recognition rate.

Video text is very important semantic information, which brings precise and meaningful clues for video indexing and retrieval. However, most previous approaches did video text extraction and recognition separately, while the main difficulty of extraction and recognition with complex background wasn’t handled very well. In this paper, these difficulty is investigated by combining text extraction and recognition together as well as using OCR feedback information. The following features are highlighted in our approach: (i) an efficient character image segmentation method is proposed in consideration of most prior knowledge. (ii) text extraction are implemented both on text-row and segmented single character images, since text-row based extraction maintains the color consistency of characters and backgrounds while single character has simpler background. After that, the best binary image is chosen for recognition with OCR feedback. (iii) The K-means algorithm is used for extraction which ensures that the best extraction result is involved, which is the binary image with clear classification of text strokes and background. Finally, extensive experiments and empirical evaluations on several video text images are conducted to demonstrate the satisfying performance of the proposed approach.

The system uses image and video segmentation to improve accuracy and loss metrics. The semantic gap between CBIR content and visual qualities is addressed in the study. Hybrid feature extraction strategies may help content-based retrieval systems close this gap. This study presents a hybrid feature extraction approach to Content-Based Image and Video Retrieval (CBIVR). The study examines video segmentation as well as retrieval to improve video searches and give meaningful abstractions. The study uses keyframes to automatically extract important frames from video diaries. An effective segmentation technique removes the backdrop from input data from the HMDB along with CIFAR-10 datasets. Feature selection-based optimization reduces input variables, improving model performance and computing effort. Using rank-listed numerical properties of phenomena, a hybrid feature vector conditions and machine learning model create informed estimates. For better training data analysis, the study uses error-learning ResNets, an artificial neural network having hundreds of layers that have feed-forward connections. Phase retrieval utilizing diffracted intensity distribution recreates the sample's object plane phase shift. The loss function evaluates the author's machine learning system. Retraining the system with updated authority with pre-trained models in post-processing optimizes video retrieval. The finding opens the door to more sophisticated multimedia material retrieval applications in numerous sectors. The system obtains higher accuracy and loss metrics by utilizing image and video segmentation. Impressive performance is shown by the picture segmentation model in training, with a loss of 0.1466 and an accuracy of 94%, while competitive results are maintained on the test set, with a loss of 0.67 and an accuracy of 82.46%. The model achieves a video loss of 0.39 and a video accuracy of 90% while being trained for video segmentation. These encouraging findings demonstrate the promise of the hybrid approach in improving content-based retrieval systems, opening the door for further investigation into cutting-edge segmentation algorithms, novel training datasets, and cutting-edge deep learning architectures to revolutionize multimedia content retrieval in a wide range of contexts. DOI: https://doi.org/10.17762/ijisae.v12i4.6832

In multimedia-based e-learning systems, there are strong needs for segmenting lecture videos into topic units in order to organize the videos for browsing and to provide search capability. Automatic segmentation is highly desired because of the high cost of manual segmentation. While a lot of research has been conducted on topic segmentation of transcribed spoken text, most attempts rely on domain-specific cues and formal presentation format, and require extensive training; none of these features exist in lecture videos with unscripted and spontaneous speech. In addition, lecture videos usually have few scene changes, which imply that the visual information that most video segmentation methods rely on is not available. Furthermore, even when there are scene changes, they do not match with the topic transitions. In this paper, we make use of the transcribed speech text extracted from the audio track of video to segment lecture videos into topics. We review related research and propose a segmentation approach. Our approach utilizes features such as noun phrases and combines multiple content-based and discourse-based features. Our preliminary results show that the noun phrases are salient features and the combination of multiple features is promising to improve segmentation accuracy.

Online video-based learning has been increasingly used in educational settings. When viewing video content, students may not always maintain a high degree of attention and may neglect or ignore important content critical for effective learning. This study thus presents an attention-based video lecture review mechanism (AVLRM) that can generate video segments for review based on students' sustained attention status, as determined using brainwave signal detection technology. A quasi-experiment nonequivalent control group design was utilized to divide 55 participants from two classes of an elementary school in New Taipei City, Taiwan, into two groups. One class was randomly assigned to the experimental group, and used video lectures with the AVLRM support for learning. The other class was assigned to the control group, and used video lectures with autonomous review for learning. Analytical results indicate that students in the experimental group exhibited significantly better review effectiveness than did the control group, and this difference was especially marked for students who had a low attention level, were field-dependent, or were female. The findings show that AVLRM based on brainwave signal detection technology can precisely identify video segments that are more useful for effective review than those picked by student themselves.

We have designed a Molecular Biology massive open online course (MOOC) for a global audience. Among the learning aids offered are two types of short video segments: lecture videos (delivered unscripted by a professor) and deep dives (fully scripted and animated). While the engaged learners overwhelmingly watched the lecture video segments through to completion, some watched only a portion of each deep dive. As the deep dives take pains to follow evidence-based best practices and are more labor-intensive to make, further study of this difference in viewer retention would inform future course development decisions. Notably, course organization, length of video, lack of on-screen narrator, and identity of narrator show no correlation with this trend. Interestingly, learners who complete lecture videos but not deep dives have slightly higher overall course grades on average. Thus, our model is that learners with a higher degree of knowledge about the subject matter may feel that they do not need to complete the deep dive videos, while they feel the lecture videos are valuable. Future research will test this model.

Scene text in video contains semantic information and thus can contribute significantly to video retrieval and understanding. However, most methods detect scene text in image or single frame. Videos differ from images in temporal redundancy. In this paper we present a novel approach to detecting video scene text based on the video temporal redundancy. Video scene texts in consecutive frames have arbitrary motion due to camera or object movement. Therefore, first, we perform the motion detection in 30 consecutive frames to synthesize motion image. Second we implement video scene text detection in single frame to retrieve candidate text regions. Finally, the synthesized motion image is used to filter out candidate text regions and only keep the candidate text regions which have motion occurrence as final scene text. Experimental results show that our algorithm performs well for detecting scene text with various color, font-size and text alignment.

The 3D-AVC standard aims at improving coding efficiency by applying new techniques for utilizing intra, inter and view predictions. 3D video scenes are rendered with existing texture video and additional depth map. The depth map comes at the expense of increased computational complexity of the encoding process. For real-time applications, reducing the complexity of 3D-AVC is very important. In this paper, we present a fast intra mode decision algorithm to reduce the complexity burden in the 3D video system. The proposed algorithm uses similarity between texture video and depth map. The best intra prediction mode of the depth map is similar to that of the corresponding texture video. The early decision algorithm can be made on the intra prediction of depth map coding by using the coded intra mode of texture video. Adaptive threshold for early termination is also proposed. Experimental results show that the proposed algorithm saves the encoding time on average 29.7% without any significant loss in terms of the bit rate or PSNR value.

Motivated by the growing use of multimedia services and the explosion of multimedia collections, efficient retrieval from large-scale multimedia data has become very important in multimedia content analysis and management. In this paper, a novel ranking algorithm is proposed for video retrieval. First, video content is represented by the global and local features and second, multiple correspondence analysis (MCA) is applied to capture the correlation between video content and semantic concepts. Next, video segments are scored by considering the features with high correlations and the transaction weights converted from correlations. Finally, a user interface is implemented in a video retrieval system that allows the user to enter his/her interested concept, searches videos based on the target concept, ranks the retrieved video segments using the proposed ranking algorithm, and then displays the top-ranked video segments to the user. Experimental results on 30 concepts from the TRECVID high-level feature extraction task have demonstrated that the presented video retrieval system assisted by the proposed ranking algorithm is able to retrieve more video segments belonging to the target concepts and to display more relevant results to the users.

Object based video retrieval with local region tracking

PurposeThe purpose of this paper is to propose an indexing and teaching focus mining system for lecture videos recorded in an unconstrained environment.Design/methodology/approachBy applying the proposed algorithms in this paper, the slide structure can be reconstructed by extracting slide images from the video. Instead of applying traditional shot‐change detection methods for general videos, a new edge‐based shot‐change detection algorithm is designed specifically for lecture videos. Besides, light influence and occlusions in the lecture video can be removed to obtain more accurate results. Moreover, the teaching focus can be extracted according to instructors' behavior based on the analyses of visual and audio information extracted from the lecture video.FindingsExperiment results show the feasibility of the proposed method, that is, the slide shots can be correctly detected even if the illumination conditions are variant or the slides are obstructed by the instructor or students, and the teaching focus can be extracted to provide learners with an efficient way to study.Research limitations/implicationsThis paper provides only technical experiments, but lacks complete educational study. In the future, more subjective tests will be designed to examine the educational effects on students.Practical implicationsThis paper proposes a practical indexing and teaching focus mining system for lecture videos which can help students learn.Originality/valueThe proposed algorithms for indexing and teaching focus mining are derived and applied in lecture videos in this paper.

Study of context influence on classifiers trained under different video-document representations