Key Frame Estimation in Video Using Randomness Measure of Feature Point Pattern

Emotion recognition from macroexpression and microexpression has been widely used in applications such as human–computer interaction, learning status evaluation, and mental disorder diagnosis. However, due to the complexity of human macroexpressions, recognizing macroexpressions with high accuracy is a challenging task. Moreover, the short duration and low movement intensity of microexpressions make its recognition more difficult. For MM-FER (macro and microfacial expression recognition), the key information can be more efficiently expressed by a graph. In this article, a novel framework based on graph neural network named SSGNN (spatial and spectral domain features based on a graph neural network) is designed to extract spatial and spectral domain features from facial images for MM-FER, which can efficiently recognize both macroexpressions and microexpressions under the same model. SSGNN consists of two parts, SPAGNN and SPEGNN, which are used to extract spectral and spatial domain features, respectively. Experiments proved that jointly using the spectral and spatial information extracted by SSGNN can largely improve the performance of MM-FER when the training sample is limited. First, the influences of different neighbors and samples to the model performance was analyzed. Then, the contribution of SPAGNN and SPEGNN were evaluated. It was discovered that fusing the result of SPAGNN and SPEGNN at decision level further improved the performance of MM-FER. Experiment proved that SSGNN can recognize microexpression acquired by various sensors with higher accuracy under different image resolutions and image formats than the compared state-of-the-art methods in most cases. A cross-dataset experiment demonstrated the generalization ability of SSGNN.

Recognizing Human action in video is an very active research topic. There are a growing variety of human action datasets with different video length, different practitioners. Make human action recognition becomes a very difficult topic. A majority researchers solve the problem by extracting key frames from the videos. Most paper use feature Clustering methods to extract key frames in videos. On one hand, the large variety of visual content in videos make handcraft feature isn't effective enough, since there are no fixed descriptors can describe all video cases. On the other hand, traditional clustering algorithms are easily influenced by the choice of initial clustering centers. An Unsupervised feature learning and clustering method for key frame extraction is proposed in this paper, which can be used for human action recognition. Stacked auto-encoder(SAE) is trained using videos from 10 different human actions, SAE is used as a feature extractor to learn features representing human actions. Affinity Propagation Clustering algorithm is used to select key frames from video sequences. We use a variety of videos to do the experiments. Experiments demonstrate that our method can be effectively summarizing video shots considering different human actions.

Nowadays, a strong need exists for the efficient organization of an increasing amount of home video content. To create an efficient system for the management of home video content, it is required to categorize home video content in a semantic way. So far, a significant amount of research has already been dedicated to semantic video categorization. However, conventional categorization approaches often rely on unnecessary concepts and complicated algorithms that are not suited in the context of home video categorization. To overcome the aforementioned problem, this paper proposes a novel home video categorization method that adopts semantic home photo categorization. To use home photo categorization in the context of home video, we segment video content into shots and extract key frames that represent each shot. To extract the semantics from key frames, we divide each key frame into ten local regions and extract lowlevel features. Based on the low level features extracted for each local region, we can predict the semantics of a particular key frame. To verify the usefulness of the proposed home video categorization method, experiments were performed with home video sequences, labeled by concepts part of the MPEG-7 VCE2 dataset. To verify the usefulness of the proposed home video categorization method, experiments were performed with 70 home video sequences. For the home video sequences used, the proposed system produced a recall of 77% and an accuracy of 78%.

In order to solve the problem of action recognition in short video and capture the key information of video, this paper first proposes a KGAF-means method for key frame extraction. The KGAF-means method is based on the clustering principle and combines the K-means algorithm with the artificial fish swarm algorithm to realize the key frame sequence extraction. Based on the extracted key frame sequence, the RGB image and the optical flow image are separately extracted by the improved dual-stream variable convolution network. Then, using the cascading method, the image feature vector and the optical flow feature vector are fused to obtain the fused feature vector for action recognition. The selected data set is the Charades data set. The experimental results show that the mAP value of the method is 22.9 on the public dataset Charades. And the results show that the proposed method has better robustness than other network models and improves the short video action recognition effect.

For the existing traditional method to extract key frames from the video sequence with gradual change of content, there are problems of having too much redundancy or too much loss in the extraction results and other issues, a new key frames extracting method for adaptively extracting the video key frames has proposed in the paper. The experimental results have shown that the proposed method can effectively extract video key frames, especially for the video with gradual change of content, which can express more succinctly and comprehensively. The experiments on a large number of videos containing various scenes have shown that the key frames extracted by the proposed algorithm can more fully express the main content of video. When dealing with content gradient video, the effect is better. The content adaptively determines the number of key frames based on video.

We present an automatic and efficient method to register and stitch thousands of video frames into a large panoramic mosaic. Our method preserves the robustness and accuracy of image stitchers that match all pairs of images while utilizing the ordering information provided by video. We reduce the cost of searching for matches between video frames by adaptively identifying key frames based on the amount of image-to-image overlap. Key frames are matched to all other key frames, but intermediate video frames are only matched to temporally neighboring key frames and intermediate frames. Image orientations can be estimated from this sparse set of matches in time quadratic to cubic in the number of key frames but only linear in the number of intermediate frames. Additionally, the matches between pairs of images are compressed by replacing measurements within small windows in the image with a single representative measurement. We show that this approach substantially reduces the time required to estimate the image orientations with minimal loss of accuracy. Finally, we demonstrate both the efficiency and quality of our results by registering several long video sequences

Synthetic aperture radar (SAR) image target recognition has important application values in security reconnaissance and disaster monitoring. However, due to speckle noise and target orientation sensitivity in SAR images, traditional spatial domain recognition methods face challenges in accuracy and robustness. To effectively address these challenges, we propose a dual-branch spatial-frequency domain fusion recognition method with cross-attention, achieving deep fusion of spatial and frequency domain features. In the spatial domain, we propose an enhanced multi-scale feature extraction module (EMFE), which adopts a multi-branch parallel structure to effectively enhance the network’s multi-scale feature representation capability. Combining frequency domain guided attention, the model focuses on key regional features in the spatial domain. In the frequency domain, we design a hybrid frequency domain transformation module (HFDT) that extracts real and imaginary features through Fourier transform to capture the global structure of the image. Meanwhile, we introduce a spatially guided frequency domain attention to enhance the discriminative capability of frequency domain features. Finally, we propose a cross-domain feature fusion (CDFF) module, which achieves bidirectional interaction and optimal fusion of spatial-frequency domain features through cross attention and adaptive feature fusion. Experimental results demonstrate that our method achieves significantly superior recognition accuracy compared to existing methods on the MSTAR dataset.

Pan-sharpening is a panchromatic (PAN)-guided super-resolution (SR) process, focused on enhancing spatial resolution of low-resolution multi-spectral (LRMS). Existing SR-based pan-sharpening methods face two critical limitations: (1) lack of unified framework adaptable to both pan-sharpening and single image SR (SISR), and (2) inefficiencies in handling low-frequency feature redundancy in both spatial and frequency domain. Based on these, we model features in both spatial and frequency domain and propose a SR-based detail injection network (SDINet) for pan-sharpening. SDINet designs a global-local spatial extraction block (GSEB) for multi-scale feature extraction in spatial domain, accompanying a Sobel-based gated fusion block (SGFB) to non-linearly suppress low-frequency redundancy. Fourier frequency domain-based spatial detail injection block (FDIB) helps to extract supplement spatial information and to fluently convert SDINet for SISR task through input modification alone. Full-resolution and reduced-resolution experiments demonstrate the advantages of SDINet in both pan-sharpening and SISR tasks.

Key frame extraction is an important manner of video summarization. It can be used to interpret video content quickly. Existing approaches first partition the entire video into video clips by shot boundary detection, and then, extract key frames by frame clustering. However, in most team-sport videos, a video clip usually includes many events, and it is difficult to extract the key frames related to all of these events accurately, because different events of a game shot can have features of similar appearance. As is well known, most events in team-sport videos are attack and defense conversions, which are related to global translation. Therefore, by using fine-grained partition based on the global motion, a shot could be further partitioned into more video clips, from which more key frames could be extracted and they are related to the events. In this study, global horizontal motion is introduced to further partition video clips into fine-grained video clips. Furthermore, global motion statistics are utilized to extract candidate key frames. Finally, the representative key frames are extracted based on the spatial–temporal consistence and hierarchical clustering, and the redundant frames are removed. A dataset called SportKF is built, which includes 25 videos of 197,878 frames in 112 min and 764 key frames from four types of sports (basketball, football, American football and field hockey). The experimental results demonstrate that the proposed scheme achieves state-of-the-art performance by introducing global motion statistics.

To address the issues of insufficient restoration of texture details in deblurred images and inadequate learning of frequency domain features, an image deblurring algorithm based on frequency domain feature enhancement and convolutional neural networks is proposed. In this architecture, firstly, a Fourier residual module with a parallel structure is constructed to achieve collaborative learning and modeling of spatial and frequency domain features, aiming to improve frequency domain feature learning capability and the restoration effect of the texture details; secondly, a gated controlled feed-forward unit acts on the Fourier residual module to further enhance the nonlinear expression ability of the algorithm; thirdly, a supervised attention module is improved and added to the decoder to promote more effective capture of key features for image reconstruction; finally, the weighted sum of spatial domain Charbonnier loss function and frequency domain loss function is defined as a novel total loss function. In addition, to verify the performance of our proposed algorithm, we conducted experiments on the GOPRO and HIDE datasets. Through experiments on the GOPRO, we obtained an SSIM and an LPIPS of 0.961 and 0.0278, respectively. With regard to the experiments on the HIDE datasets, we obtained an SSIM and an LPIPS of 0.941 and 0.0286, respectively. As for parameter count and running time, their values were 1.197 and 9.15 × 106, respectively, obtained by the experiments on the GOPRO. In all algorithms, the values of our proposed algorithm are optimal. However, the PSNR of our proposed algorithm is very close to that of the latest comparison algorithm and is suboptimal. In a word, experimental results have demonstrated that our proposed algorithm effectively removes blur while better preserving the details and edges of the image. Therefore, it has more practical value and prospects in computer vision tasks.

Key frame selection, which aims to choose a small number of frames that well represent the entire video contents, is one of the important research issues in video content analysis. Many existing methods developed so far use some prior information about the contents of video frames based on high-level semantic features, and thus they are applicable only in particular problems. Without any prior information, approaches based on low-level index features such as clusters, correlation, and consecutive frame difference are useful. Key frames found by these approaches are expected to be as much different from each other as possible. Therefore, accurately evaluating the difference among candidate key frames is a key challenge. However, a standard similarity metric such as the squared error and the cross correlation tends to provide redundant key frames. To cope with this problem, we propose to use quadratic mutual information (QMI) as an alternative similarity measure, which allows us to capture higher-order correlation for more than two variables simultaneously. Through experiments, we demonstrate the usefulness of the proposed QMI-based key frame selection method that uses an accurate and computationally efficient least-squares QMI estimator.

3D video, which consists of a sequence of mesh models, can reproduce dynamic scenes containing 3D information. To summarize 3D video, a key frame extraction method is developed using rate-distortion (R-D) trade-off. For this purpose, an effective feature vector is extracted for each frame. Shot detection is performed using the feature vectors as a preprocessing followed by key frame extraction. Simple but reasonable definitions of rate and distortion are presented. Based on an assumption of linearity, an R-D curve is generated in each shot, where the locations of the key frames are optimized. Finally, R-D trade-off can be achieved by optimizing a cost function using a Lagrange multiplier, where the number of key frames is optimized in each shot. Therefore, our system will automatically determine the best locations and the number of key frames in the sense of R-D trade-off. Our experimental results show the extracted key frames are compact and faithful to the original 3D video.

This paper presents a semi-automatic algorithm for video object segmentation. Our algorithm assumes the use of multiple key video frames in which a semantic object of interest is defined in advance with human assistance. For video frames between every two key frames, the specified video object is tracked and segmented automatically using Learning Vector Quantization (LVQ). Each pixel of a video frame is represented by a 5-dimensional feature vector integrating spatial and color information. We introduce a parameter K to adjust the balance of spatial and color information. Experimental results demonstrate that the algorithm can segment the video object consistently with less than 2% average error when the object is moving at a moderate speed.

In this work, the idea of key frames extraction from single shots in video sequences is presented. The method is implemented by an efficient two-step algorithm, which is classified neither to clustering nor to temporal variations based techniques. In the first step, an MST (minimal spanning tree) graph is constructed, where each node is associated to a single frame of the shot. In the second step, extracts key frames based on the principle of their maximum spread, are extracted. The number of the selected key frames is controlled by an adaptively defined threshold, while the validity of the results is evaluated by the fidelity measure.

We present a key frames selection algorithm based on three iso-content principles (iso-content distance, iso-content error and iso-content distortion), so that the selected key frames are equidistant in video content according to the used principle. Two automatic approaches for defining the most appropriate number of key frames are proposed by exploiting supervised and unsupervised content criteria. Experimental results and the comparisons with existing methods from literature on large dataset of real-life video sequences illustrate the high performance of the proposed schemata.