Study on video key frame extraction in different scenes based on optical flow

The key frame extraction algorithm based on the indigenous disturbance variation difference video

Aiming at the problem of video key frame extraction, a density peak clustering algorithm is proposed, which uses the HSV histogram to transform high-dimensional abstract video image data into quantifiable low-dimensional data, and reduces the computational complexity while capturing image features. On this basis, the density peak clustering algorithm is used to cluster these low-dimensional data and find the cluster centers. Combining the clustering results, the final key frames are obtained. A large number of key frame extraction experiments for different types of videos show that the algorithm can extract different number of key frames by combining video content, overcome the shortcoming of traditional key frame extraction algorithm which can only extract a fixed number of key frames, and the extracted key frames can represent the main content of video accurately.

Unmanned aerial vehicles (UAVs) have become an important part of today’s world. Micro Aerial Vehicles (MAVs) are small-sized UAVs that are suited for closed spaces, indoor or covert applications. Due to their small size and limited energy and load they can bear, there are several challenges in intelligent control of these vehicles. Also, due to their limitations, using energy-consuming sensors and complex control strategies which need high computational power, is not an option. To overcome MAV’s limitations, researchers are working on visual navigation systems. In this paper, we present an efficient strategy for the guidance and navigation of a 30gr MAV in indoor environments using a mixture of visual and inertial sensors. By taking inspiration from nature and flying insects, we combined optical flow and stereopsis methods to detect stationary obstacles and avoid collisions. Our control system uses two lateral cameras to avoid collision with lateral walls. To detect frontal walls and obstacles, instead of using optical flow methods, we use two frontal cameras with the same viewing direction and calculate the discrepancy between their images to detect walls and obstacles. For guidance, we designed a guidance system that functions well with our obstacle detection system and only needs destination and MAV’s current coordinates. By implementing our control system on a simulated coaxial helicopter, we were able to record more than 150 hours of collision-free flight and successful guidance to appointed destinations. Our control system is best suited for small sized UAVs in indoor environments and only needs inexpensive sensors and microcontrollers to implement

This study adopted an improved fuzzy clustering and optical flow method for the multiscale identification and forecasting of a cloud system based on the cloud images from a 10.8-micron infrared channel of the Advanced Geosynchronous Radiation Imager. First, we used the locally constrained fuzzy c-means (FCM) clustering method to identify typhoon-dominant cloud systems. Second, we coupled the background field-constrained optical flow method with the semi-Lagrangian scheme to forecast typhoon-dominant cloud systems. The experimental results for Typhoon Maria showed that the improved FCM method was able to effectively identify changes in the cloud system while retaining its edge information through the effective removal of the offset field. The identified dominant cloud system was consistent with the precipitation field of the Global Precipitation Measurement mission. We optimized the semi-Lagrangian nonlinear extrapolation of the optical flow field by introducing background field information, thus improving the forecast accuracy of the optical flow field. Based on the assessment indicators of structural similarity, normalized mutual information, peak signal-to-noise ratio, relative standard deviation, and root mean square error, the forecast results demonstrated that the forecast effect of the background field-constrained optical flow method was better than that of the standard optical flow method.

Keyframe extraction is a very important part of the video retrieval steps. However, current keyframe extraction techniques have problems such as underrepresentation, large redundancy, poor adaptability and large computation. So, we propose an accelerated alternative to the basic task with comparable performance, and by rephrasing the task as fragment generation and prompting. We use the key frame extraction method combined with the SAM-based feature extraction model of lightweight-SAM, without additional training, to obtain an interactive video key frame extraction method for focused preference objects. In this model, we used the Yolov8-seg model to replace the VIT part in Segment-Anything Model (SAM), while adding the tracking module XMem in the keyframe extraction step, which can effectively reduce the computational amount while also ensuring the accuracy of the keyframes. Using our approach, we achieve comparable performance to the SAM approach with a 50x higher runtime speed. Recall at 50% precision (R50) is high and has significantly fewer parameters (only 70M). The ratio of key frame extraction can reach 90.3% under the premise of an average fidelity of 69.4%. We compare the performance of these methods on UCF-101, COCO, BSDS500 and other datasets, and this interactive design performs impressively in video object tracking and segmentation, and performs well in keyframe extraction for selected objects.

As the fast development of network and multimedia technology, there are mass of videos on the Internet. The problem of how to browse the large volume of video data is urgent to be solved. Key frame extraction is a very important technology in online video retrieval and video abstraction. Efficient key frame extraction can boost the development of video browsing technology. In this paper, we first review some commonly-used key frame extraction technology, and then propose a improved key frame extraction algorithm, which measures the similarity of two adjacent frames' contents in terms of the information of frame difference and extracts key frames after optimizing the frame difference. The experiment results show that the proposed algorithm can extract key information from online video efficiently, and has a good description in the aspect of motion of the camera lens.

Gas–Liquid two-phase flows are a common flow in industrial production processes. Since these flows inherently consist of discrete phases, it is challenging to accurately measure the flow parameters. In this context, a novel approach is proposed that combines the pyramidal Lucas-Kanade (L–K) optical flow method with the Split Comparison (SC) model measurement method. In the proposed approach, videos of gas–liquid two-phase flows are captured using a camera, and optical flow data are acquired from the flow videos using the pyramid L–K optical flow detection method. To address the issue of data clutter in optical flow extraction, a dynamic median value screening method is introduced to optimize the corner point for optical flow calculations. Machine learning algorithms are employed for the prediction model, yielding high flow prediction accuracy in experimental tests. Results demonstrate that the gradient boosted regression (GBR) model is the most effective among the five preset models, and the optimized SC model significantly improves measurement accuracy compared to the GBR model, achieving an R2 value of 0.97, RMSE of 0.74 m3/h, MAE of 0.52 m3/h, and MAPE of 8.0%. This method offers a new approach for monitoring flows in industrial production processes such as oil and gas.

Digital subtraction angiography is the gold standard vascular imaging and it is used for all endovascular treatment of intracranial anerysms. Optical flow imaging has been described as a potential method to evaluate cerebral hemodynamics through DSA. In this study, we aimed to compare the flow patterns measured during angiography, by using an optical flow method, with those measured by using computational fluid dynamics in intracranial aneurysms. A consecutive series of 21 patients harboring unruptured saccular intracranial aneurysms who underwent diagnostic angiography before treatment was considered. High-frame-rate digital subtraction angiography was performed to obtain an intra-aneurysmal velocity field by following the cardiac-modulated contrast wave through the vascular structures by using optical flow principles. Additionally, computational fluid dynamics modeling was performed for every case by using patient-specific inlet-boundary conditions measured with the optical flow method from both DSA and 3D rotational angiography datasets. Three independent observers compared qualitatively both the inflow direction and the apparent recirculation in regular DSA, optical flow images, and computational fluid dynamics flow patterns for each patient; κ statistics were estimated. We included 21 patients. In 14 of these 21, the flow patterns were conclusive and matching between the optical flow images and computational fluid dynamics within the same projection view (κ = .91). However, in only 8 of these 14 patients the optical flow images were conclusive and matching regular DSA images (observer κ = 0.87). In 7 of the 21 patients, the flow patterns in the optical flow images were inconclusive, possibly due to improper projection angles. The DSA-based optical flow technique was considered qualitatively consistent with computational fluid dynamics outcomes in evaluating intra-aneurysmal inflow direction and apparent recirculation. Moreover, the optical flow technique may provide the premises for new solutions for improving the visibility of flow patterns when contrast motion in DSA is not apparent. This technique is a diagnostic method to evaluate intra-aneurysmal flow patterns and could be used in the future for validation and patient evaluation.

A gesture motion detection algorithm based on optical flow method is elaborated in this paper as significant way of human-computer interaction. The skin color segmentation algorithm based on ellipse skin color model in YCbCr color space and Pyramid Lucas-Kanade optical flow method constitute the basic modules of gesture motion detection. The skin color areas are selected in sequence by skin color segmentation algorithm which produce the optical flow vectors within the Pyramid Lucas-Kanade optical flow module. Optical flow vectors of the adjacent two frames after isolation are used as parameters for judging direction. When two results after judgment are consistent, the direction of gesture movement is determined. The algorithm proposed in this paper, testified from the experiments, is of high accuracy and less time-consuming which can be applied in real-time broadly.

In the current era, most of the digital information is in the form of multimedia with a giant share of videos. Videos do have audio and visual content where the visual content has number of frames put in a sequence. Most of the consecutive frames do have very little discriminative contents. In video summarization process, several frames containing similar information do need to get processed. This leads to redundant slow processing speed and complexity, time consumption. Video summarization using key frames can ease the speed up of video processing. In this paper, novel key frames extraction method is proposed with Kekere's Proportionate Error (KPE) codebook generation techniques of vector quantization with ten different codebook sizes and two color spaces (RGB and KLUV). Experimentation done with help of the test bed of videos has shown that higher codebook sizes of KPE have given better completeness in key frame extraction for video summarization. The LUV color space with Euclidean Distance with 512 codebook size gives best performance. In square chord Distance, Mean Square Error and Euclidean Distance LUV color space gives better completeness than RGB color space for proposed KPE based video Key frame Extraction

The article is devoted to the study of the optical flow method for detecting changes in the shooting angle of city surveillance cameras. The study was conducted using data obtained from surveillance cameras in Kazan. The study was conducted under conditions of a static position of the cameras, as well as under conditions of their vertical and horizontal shift by 10 to 50 degrees. For each of the conditions, the corresponding data samples were used. To study the optical flow method, four algorithms for its implementation were used. Two algorithms were used for a sparse flow (Lucas-Kanade and Pyramidal Lucas-Kanade), and two more were used for a dense flow (Dense Inverse Search and FarneBack). To evaluate their effectiveness, and, consequently, the optical flow method, the following metrics were calculated: flow value, execution time in seconds, average shift value, proportion of non-zero vectors, shift consistency. Based on the obtained results, several important conclusions were made. With a static position of the camera, the average shift value for dense flows was significantly higher than for sparse flows. This increases the chance of obtaining false positive results, especially at night. With controlled camera displacement, the average shift value for the Lucas-Kanade and FarneBack algorithms was almost zero, which indicates their inability to reliably detect displacement. The Dense Inverse Search algorithm turned out to be the most sensitive to displacements, and Pyramidal Lucas-Kanade detected displacements, but with less accuracy. Dense flow algorithms are inferior in speed to sparse flow algorithms. Optical flow algorithms do not respond to text elements of the image. Thus, the study solved the problem. The study allowed us to evaluate the effectiveness of the selected method and algorithms in terms of the accuracy of calculating the average shift and resistance to dynamic objects. The optical flow method and its algorithmic implementation Pyramidal Lucas-Kanade are best suited for detecting the displacement of CCTV cameras in an urban environment. Minimization of false positives when the camera is stationary, combined with reliable detection of shifts, makes it an optimal choice, and immunity to text elements in the image increases its effectiveness.

The principle of an optical flow method is formulated for neutron radiography (NR) flow diagnostics to determine high-resolution velocity fields of two-phase flows. This method is based on solving an optical flow equation derived from the continuous equation for NR image analysis as an inverse problem using a variational method. The mathematical definition and physical meaning of the optical flow in NR images of a two-phase flow are clarified. As an example, the optical flow method is used to extract complex velocity fields from a time sequence of dynamic NR images acquired in an air–water two-phase flow in a flat bubbler.

The purpose of video key frame extraction is to use as few video frames as possible to represent as much video content as possible, reduce redundant video frames, and reduce the amount of computation, so as to facilitate quick browsing, content summarization, indexing, and retrieval of videos. In this paper, a method of dance motion recognition and video key frame extraction based on multifeature fusion is designed to learn the complicated and changeable dancer motion recognition. Firstly, multiple features are fused, and then the similarity is measured. Then, the video sequences are clustered by the clustering algorithm according to the scene. Finally, the key frames are extracted according to the minimum amount of motion. Through the quantitative analysis and research of the simulation results of different models, it can be seen that the model proposed in this paper can show high performance and stability. The breakthrough of video clip retrieval technology is bound to effectively promote the inheritance and development of dance, which is of great theoretical significance and practical value.

An initial assessment method that can classify as well as categorize the severity of paralysis into one of six levels according to the House-Brackmann (HB) system based on facial landmarks motion using an Optical Flow (OF) algorithm is proposed. The desired landmarks were obtained from the video recordings of 5 normal and 3 Bell's Palsy subjects and tracked using the Kanade-Lucas-Tomasi (KLT) method. A new scoring system based on the motion analysis using area measurement is proposed. This scoring system uses the individual scores from the facial exercises and grades the paralysis based on the HB system. The proposed method has obtained promising results and may play a pivotal role towards improved rehabilitation programs for patients.

A key frame extraction algorithm based on sub-shot segmentation and entropy computing is proposed. Firstly the reconstructed matrix of each frame in shots is computed through sub-space projection. Secondly sub-shot segmentation is processed according to the difference of singular value between frame and its reconstructed matrix, as well as difference of histogram feature between frame and mean value of its previous N frames. Finally two adjacent frames with the largest cross-entropy in sub-shot are obtained, in which the frame with the larger image entropy is selected as a key frame. The experimental results demonstrate that key frames extracted by the algorithm can describe shot contents efficiently.