Automatic video data structuring through shot partitioning and key-frame computing

Key frame extraction is an important manner of video summarization. It can be used to interpret video content quickly. Existing approaches first partition the entire video into video clips by shot boundary detection, and then, extract key frames by frame clustering. However, in most team-sport videos, a video clip usually includes many events, and it is difficult to extract the key frames related to all of these events accurately, because different events of a game shot can have features of similar appearance. As is well known, most events in team-sport videos are attack and defense conversions, which are related to global translation. Therefore, by using fine-grained partition based on the global motion, a shot could be further partitioned into more video clips, from which more key frames could be extracted and they are related to the events. In this study, global horizontal motion is introduced to further partition video clips into fine-grained video clips. Furthermore, global motion statistics are utilized to extract candidate key frames. Finally, the representative key frames are extracted based on the spatial–temporal consistence and hierarchical clustering, and the redundant frames are removed. A dataset called SportKF is built, which includes 25 videos of 197,878 frames in 112 min and 764 key frames from four types of sports (basketball, football, American football and field hockey). The experimental results demonstrate that the proposed scheme achieves state-of-the-art performance by introducing global motion statistics.

Based on compressed video segment detection, the decompression step can be omitted, features can be directly extracted from the original video data stream, and the detection rate can be accelerated. This paper analyzes the characteristics of video data, video segmentation and key frame selection. Based on the analysis of the key technologies of video retrieval technology, the compressed video stream extraction method based on video segmentation and key frame is introduced. A method of extracting key frames from MPEG compressed video based on DC coefficients and motion vectors is proposed. Experiments show that the proposed method can reduce the computational burden, and can better represent the video content.

Many multimedia applications, such as multimedia data management systems and communication systems, require efficient representation of multimedia content. Thus semantic interpretation of video content has been a popular research area. Currently, most content-based video representation involves the segmentation of video based on key frames which are generated using scene change detection techniques as well as camera/object motion. Then, video features can be extracted from key frames. However most of such research performs off-line video processing in which the whole video scope is known as a priori which allows multiple scans of the stored video files during video processing. In comparison, relatively not much research has been done in the area of on-line video processing, which is crucial in video communication applications such as on-line collaboration, news broadcasts and so on. Our research investigates on-line real-time scene change detection of multicast video over the Internet. Our on-line processing system are designed to meet the requirements of real-time video multicasting over the Internet and to utilize the successful video parsing techniques available today. The proposed algorithms extract key frames from video bitstreams sent through the MBone network, and the extracted key frames are multicasted as annotations or metadata over a separate channel to assist in content filtering such as those anticipated to be in use by on-line filtering proxies in the Internet. The performance of the proposed algorithms are demonstrated and discussed in this paper.

Rapid increase in video databases has forced the industry to have efficient and effective frameworks for video retrieval and indexing. Video segmentation into scenes is widely used for video summarization, partitioning, indexing and retrieval. In this paper, we propose a framework for scene detection mainly based on entropy and Speeded Up Robust Features (SURF) features. First, we detect the fade and abrupt boundaries based on frame entropy analysis and SURF features matching. Fade boundaries are smart indication of scenes beginning or ending in many videos and dramas, and are detected by frame entropy analysis. Before abrupt boundary detection, unnecessary frames which are obviously not abrupt boundaries, such as blank screens, high intensity influenced images, sliding credits, are removed. Candidate boundaries are detected to make SURF features efficient for abrupt boundary detection, and SURF features between candidate boundaries and their adjacent frames are used to detect the abrupt boundaries. Second, key frames are extracted from abrupt shots. We evaluate our key frame extraction with other famous algorithms and show the effectiveness of the key frames. Finally, scene boundaries are detected using sliding window of size K over the key frames in temporal order. In experimental evaluation on the TRECVID-2007 shot boundary test set, the algorithm for shot boundary achieves substantial improvements over state-of-the-art methods with the precision of 99% and the recall of 97.8%. Experimental results for video segmentation into scenes are also promising, compared to famous state-of-the-art techniques.

In a multimedia framework, digital image sequences (videos) are by far the most demanding as far as storage, search, browsing and retrieval requirements are concerned. In order to reduce the computational burden associated to video browsing and retrieval, a video sequence is usually decomposed into several scenes (shots) and each of them is characterized by means of some key frames. The proper selection of these key frames, i.e. the most representative frames in the scene, is of paramount importance for computational efficiency. In this contribution a novel key frame extraction technique based on the wavelet analysis is presented. Experimental results show the capability of the proposed algorithm to select key frames properly summarizing the shot.

Key frame extraction is one of the critical techniques in computer vision fields such as video search, video identification and video forgeries detection. The extracted key frames should be sufficient key frames that preserve main actions in a video with compact representation. The objective of this work is to improve our previous action key frames extraction algorithm (AKF) by adapting a threshold which selects action key frames as final key frames. The threshold adaptation was achieved by using the mean value, the standard deviation, and the L1-norm instead of the comparison of user summaries evaluation method to obtain a fully automatic video summarisation algorithm, and by eliminating the conditions in selecting the final key frames to reduce the complexity of the algorithm. We have validated our proposed Improved AKF on complex colour video shots instead of the simple grey level video shots. The Improved AKF algorithm was able to extract a compact number of action key frames by preventing redundant key frames, reduce processing complexity, and preserve sufficient information about the main actions in a video shot. We then evaluated the Improved AKF algorithm with the-state-of-the-art algorithms in terms of compression ratio using Paul videos and Shih-Tang dataset. The evaluation results showed that the Improved AKF algorithm achieved better compression ratio and retained sufficient information in the extracted action key frames under different testing video shots. Therefore, the improved AKF algorithm is a suitable technique for applications in computer vision fields such as passive object-based video authentication systems

In this paper, we present a fast method of key frame selection and an accurate feature point matching method for 3D reconstruction through the control of quadrotor. The quadrotor is controlled to fly at a fixed altitude and the gimbal camera has always been a downward direction. Therefore, the additional information of the quadrotor could be easily added to the 3D reconstruction process. As a result, we find a fast method to select key frames and an optimized matching method through the geometric constraints of the flight path and direction. The method is mainly to improve the efficiency of key frame selection and to get more accurate matching points. The advantage of our approach is to combine the two processes between video capture and video processing by adding flight control to the 3D scene reconstruction method. Therefore, it can use less time to complete the entire reconstruction task. The proposed approach is tested by quadrotor platform. Experiment results show that our method can greatly reduce the key frame selection time and get more matching points at the same accuracy.

Key frames are the most representative images of a video. They are used in different areas in video processing, such as indexing, retrieval and summarization. In this paper we propose a novel approach for key frames extraction based on local feature description. This approach will be used to summarize the salient visual content of videos. First, we start by generating a set of candidate keyframes. Then we detect interest points for all these candidate frames. After that we will compute repeatability between them and stock the repeatability values in a matrix. Finally we will model repeatability table by an oriented graph and the selection of keframe is inspired from shortest path algorithm A*. Realized experiments on challenging videos show the efficiency of the proposed method: it demonstrates that it is able to prevent the redundancy of the extracted key frames and maintain minimum requirements in terms of memory space.

Nowadays, a strong need exists for the efficient organization of an increasing amount of home video content. To create an efficient system for the management of home video content, it is required to categorize home video content in a semantic way. So far, a significant amount of research has already been dedicated to semantic video categorization. However, conventional categorization approaches often rely on unnecessary concepts and complicated algorithms that are not suited in the context of home video categorization. To overcome the aforementioned problem, this paper proposes a novel home video categorization method that adopts semantic home photo categorization. To use home photo categorization in the context of home video, we segment video content into shots and extract key frames that represent each shot. To extract the semantics from key frames, we divide each key frame into ten local regions and extract lowlevel features. Based on the low level features extracted for each local region, we can predict the semantics of a particular key frame. To verify the usefulness of the proposed home video categorization method, experiments were performed with home video sequences, labeled by concepts part of the MPEG-7 VCE2 dataset. To verify the usefulness of the proposed home video categorization method, experiments were performed with 70 home video sequences. For the home video sequences used, the proposed system produced a recall of 77% and an accuracy of 78%.

최근 멀티미디어 정보의 양이 매우 빠른 속도로 증가함에 따라 비디오 데이터에 대한 다양한 검색은 더욱 중요한 의미를 가지게 되었다. 비디오 데이터를 효율적으로 처리하기 위해서는 비디오 데이터가 가지고 있는 내용에 대한 정보를 데이터베이스에 저장하고 사용자들의 다양한 질의를 처리할 수 있는 의미기반 검색 기법이 요구된다. 기존의 내용기반 비디오 검색 시스템들은 주석기반 검색 또는 특징기반 검색과 같은 단일 방식으로만 검색을 하므로 검색 효율이 낮을 뿐 아니라 완전한 자동 처리가 되지 않아 시스템 관리자나 주석자의 많은 노력을 요구한다. 본 논문에서는 주석기반 검색과 특징기반 검색을 이용하여 대용량의 비디오 데이터에 대한 사용자의 다양한 의미검색을 지원하는 에이전트 기반에서의 자동화되고 통합된 비디오 의미기반 검색 시스템을 제안한다. 사용자의 기본적인 질의와 질의에 의해 추출된 키 프레임의 이미지를 선택함으로써 에이전트는 추출된 키 프레임의 주석에 대한 의미를 더욱 구체화시킨다. 또한, 사용자에 의해 선택된 키 프레임은 질의 이미지가 되어 제안하는 특징기반 검색기법을 통해 가장 유사한 키 프레임을 검색한다. 따라서 의미기반 검색을 통해 비디오 데이터의 검색의 효율을 높일 수 있도록 시스템을 설계한다. According to the rapid increase of multimedia data quantity recently, various means of video data search has been desired. In order to process video data effectively, it is required that the content information of video data is loaded in database and semantic-based retrieval method can be available for various query of users. Currently existent contents-based video retrieval systems search by single method such as annotation-based or feature-based retrieval, and show low search efficiency and requires many efforts of system administrator or annotator form less perfect automatic processing. In this paper, we propose semantic-based video retrieval system which support semantic retrieval of various users by feature-based retrieval and annotation-based retrieval of massive video data. By user's fundamental query and selection of image for key frame that extracted from query, the agent gives the detail shape for annotation of extracted key frame. Also, key frame selected by user become query image and searches the most similar key frame through feature based retrieval method that propose. Therefore, we design the system that can heighten retrieval efficiency of video data through semantic-based retrieval.

<p>A SIFT features is an effective approach for video copy detection. To detect and describe local features in images scale-invariant feature transform is used. The purpose of video copy detection is to decide whether a video segment is a copy of video from train video database or not. SIFT image features provide a set of features of such as object scaling and rotation. We first use dualthreshold method to segment the videos into segments with homogeneous content which helps to extract key frames from each segment. SIFT features are extracted from the key frames of the segments. SIFT features are very flexible to the effects of "noise" in the image. SIFT Applications include object recognition, robotic mapping and 3D modeling, navigation, image stitching, video tracking, gesture recognition, , individual identification of wildlife and match moving.</p> <p><a href="https://journalnx.com/journal-article/20150098">https://journalnx.com/journal-article/20150098</a></p>

Compressive sensing (CS) theory proves that signals can be recovered from far fewer random measurements than that suggested by Nyquist-Shannon theorem. This advantage of CS is extremely useful in some data processing applications, especially in video processing. Considering that nonlocal patch-based CS methodology has achieved an impressive performance in image CS field, it motivates us to generalize the idea into the video CS filed. In this paper, by taking account of the spatiotemporal information, we propose a novel strategy to reconstruct the non-key frames of a video sequence by introducing the nonlocal similar patch group from some key frames. Experimental results illustrate that our proposed nonlocal patch-based video CS recovery algorithm can extremely exploit the information of the key frames, and achieve a better performance compared with some existing popular video CS recovery algorithms. It is noted that our idea to construct a spatiotemporal comprehensive patch group may be combined with any nonlocal patch-based CS method for video reconstruction, which leads to various of nonlocal patch-based compressive sensing video recovery algorithms.

Video segmentation research has emerged over the last decade for biomedical image and video processing, especially in biological organism tracking. However, due to the difficulties in generating the video segmentation ground truth, the general lack of segmentation datasets with annotated ground-truth severely limits the evaluation of segmentation algorithms. This paper proposes an efficient and scalable crowdsourced approach to generate video segmentation ground-truth to facilitate database generation for general biological organism segmentation and tracking algorithm evaluation. To illustrate the proposed approach, an annotated zebrafish larvae video segmentation dataset has been generated and made freely available online. To enable the evaluation of algorithms against a ground-truth, a set of segmentation evaluation metrics are also presented. The segmentation performance of five leading segmentation algorithms is then evaluated by the metrics on the generated zebrafish video segmentation dataset.

We present an algorithm for audio-guided segmentation of the Kinect videos of Adavus in Bharatanatyam dance. Adavus are basic choreographic units of a dance sequence in Bharatanatyam. An Adavu is accompanied by percussion instruments (Tatta Palahai (wooden stick)—Tatta Kozhi (wooden block), Mridangam, Nagaswaram, Flute, Violin, or Veena) and vocal music. It is a combination of events that are either postures or small movements synchronized with rhythmic pattern of beats or Taals. We segment the videos of Adavus according to the percussion beats to determine the events for recognition of Adavus later. We use Blind Source Separation to isolate the instrumental sound from the vocal. Beats are tracked by onset detection to determine the instants in the video where the dancer assumes key postures. We also build a visualizer for test. From over 13000 input frames of 15 Adavus, 74 of the 131 key frames actually present get detected. Every detected key frame is correct. Hence, the system has 100 % precision, but only about 56 % recall.

Multi-person behavior event recognition has become an increasingly challenging research field in human–computer interaction. With the rapid development of deep learning and computer vision, it plays an important role in the inference and analysis of real sports events, that is, given the video frequency of sports events, when letting it analyze and judge the behavior trend of athletes, often faced with the limitations of large-scale data sets and hardware, it takes a lot of time, and the accuracy of the results is not high. Therefore, we propose a deep clustering learning network for motion recognition under the self-attention mechanism, which can efficiently solve the accuracy and efficiency problems of sports event analysis and judgment. This method can not only solve the problem of gradient disappearance and explosion in the recurrent neural network (RNN), but also capture the internal correlation between multiple people on the sports field for identification, etc., by using the long and short-term memory network (LSTM), and combine the motion coding information in the key frames with the deep embedded clustering (DEC) to better analyze and judge the complex behavior change types of athletes. In addition, by using the self-attention mechanism, we can not only analyze the whole process of the sports video macroscopically, but also focus on the specific attributes of the movement, extract the key posture features of the athletes, further enhance the features, effectively reduce the amount of parameters in the calculation process of self-attention, reduce the computational complexity, and maintain the ability to capture details. The accuracy and efficiency of reasoning and judgment are improved. Through verification on large video datasets of mainstream sports, we achieved high accuracy and improved the efficiency of inference and prediction. It is proved that the method is effective and feasible in the analysis and reasoning of sports videos.