Multimodal Video Summarization Using Machine Learning: A Comprehensive Benchmark of Feature Selection and Classifier Performance

We present a framework for selecting best audio features for audiovisual analysis and synthesis of dance figures. Dance figures are performed synchronously with the musical rhythm. They can be analyzed through the audio spectra using spectral and rhythmic musical features. In the proposed audio feature evaluation system, dance figures are manually labeled over the video stream. The music segments, which correspond to labeled dance figures, are used to train hidden Markov model (HMM) structures to learn temporal spectrum patterns for the dance figures. The dance figure recognition performances of the HMM models for various spectral feature sets are evaluated. Audio features, which are maximizing dance figure recognition performances, are selected as the best audio features for the analyzed audiovisual dance recordings. In our evaluations, mel-scale cepstral coefficients (MFCC) with their first and second derivatives, spectral centroid, spectral flux and spectral roll-off are used as candidate audio features. Selection of the best audio features can be used towards analysis and synthesis of audio-driven body animation.

The majority of color histogram equalization methods do not yield uniform histogram in gray scale. After converting a color histogram equalized image into gray scale, the contrast of the converted image is worse than that of an 1-D gray scale histogram equalized image. We propose a novel 3-D color histogram equalization method that produces uniform distribution in gray scale histogram by defining a new cumulative probability density function in 3-D color space. Test results with natural and synthetic images are presented to compare and analyze various color histogram equalization algorithms based upon 3-D color histograms. We also present theoretical analysis for nonideal performance of existing methods.

The combination of point, line, shape and color elements combined to create a physical imitation of an object is called an image. The arrangement of the box elements in the image forms pixels or matrices. each image experiences degradation or loss of quality called noise. The effect of gaussian noise is the number of colored dots that are equal to the percentage of noise. This study raises the topic of improving the quality of digital images using median filter techniques to reduce noise. In this study using color image data (Red Green Blue) as test data and then converted into grayscale images to determine the gray degree of the image. The grayscale image is stored in the database. Then noise is generated by using random numbers. Noise in the form of impulse can be positive or negative in the form of adding pixel values to the original image, or it can reduce the value of the original image. The noise type used is salt & pepper. Gray degrees 0-255 spread. Can be calculated through image histograms. To reduce noise the median filter technique is used. Image histogram as a measure of the spread of numbers from the median filter. The result is a median filter can reduce noise salt and pepper by using a matrix kernel.

In this paper, a novel vehicle detection algorithm is proposed that utilizes the color histogram of the image. The color histogram is used to search the image for regions with shadow, block symmetry, and block non-homogeneity, thereby detecting the vehicle region. First, an integral histogram of the input image is computed to decrease the amount of required computation time for the block color histograms. Then, shadow detection is performed and the block symmetry and block non-homogeneity are checked in a cascade manner to detect the vehicle in the image. Finally, the proposed scheme is applied to both still images taken in a parking lot and an on-road video sequence to demonstrate its effectiveness.

In this paper, we propose a novel video summarization technique for automatic affect analysis of participants of an online event. At first, face verification neural network is used to cluster facial regions that correspond to each participant. Next, emotional features are extracted from each face by using EfficientNet model obtained in the previous paper of the author. The features of several consecutive frames are combined into a single descriptor that is used to classify emotions. In addition, audio features are extracted using wav2vec, and an ensemble of audio and video classifiers predicts emotions for each face. Finally, dependence of these emotions on time is visualized in special color charts. In the experimental study with the AFEW dataset it was demonstrated that the proposed approach makes it possible to obtain the best-known validation accuracy 67.88%. The models were optimized using OpenVINO and gained reasonable performance even if Nvidia GPUs are unavailable. The models and source code are publicly available at https://github.com/amirabdrahimov/multimodal-emotion-recognition.

In the present study, image histogram features of both single and multi channel images were used to detect and classify the common household/office smoke and other nano-scale airborne particles. To capture the Rayleigh scattered light from the particles, a white LED which covers most of the wavelengths of visible range was used as the light source. By considering the selective colour scattering behaviour of Rayleigh scattering, the intensity variation of red, green and blue components of a scattered light by the test particles are investigated using the image histograms. Particles were classified using feature vectors constructed using the running mean and mode of the histogram maximum value index (MVI) over a window of a fixed number of frames. Two classifiers: multiple discriminant analysis (MDA) and K-nearest neighbour algorithm (for comparison) were used to classify the feature vectors. Results showed that all the test particles can be detected and classified using a white LED, which is a significant step up over detecting smoke with traditional infrared LED light detectors. Even though, the feature vectors extracted with the proposed method from both colour and single channel image histograms classified all the monotype test particles with 100% accuracy with the both classifiers, the fastest classification was obtained with MVI of single channel images classified with MDA.

Analysing user physiological responses for affective video summarisation

Video summarization focuses on creating concise versions of lengthy videos by preserving essential content. This study reveals a multimodal machine learning strategy that integrates visual and auditory information by using pretrained Gated Recurrent Neural Network architectures referred to as GARNN, combining Gated Recurrent Units (GRUs) and AlexNet, to extract audio, image, and visual features. The keyframe detection is improved by removing redundant frames and applying motion-compensated feature reduction, followed by optional PCA-based dimensionality reduction. An adversarial encoder-based Long Short-Term Memory (AE-LSTM) classifier is employed for temporal modeling by achieving high accuracy in summarization. The results got evaluated by using sensitivity, F-scores, and positive predictive values, and the method attained an average F-score of 0.985. A gated AlexNet is introduced in a multimodal GARNN-AE-LSTM framework, where motion-compensated PCA-based reduction eliminates redundancy, GRUs record temporal progression, and gating fine-tunes spatial feature selection, all of which contribute to a more accurate and efficient video summarization system. The improved F1 score demonstrates the model's effectiveness in generating accuracy in video summaries by creating a meaningful video. This approach highlights the potential of multimodal feature extraction and advanced deep learning techniques for robust video analysis and compression.

It is a problem that the results are unstable when local method is used to estimate optical flow at smooth color points. A method was presented to estimate stability through the condition number of local coefficients matrix. And the color results of global smooth were used to replace the unstable results of local color method. The combined model was provided with the advantages of high precision of local model and density of global model. Finally, the fusion optical flow result was filtered by median filter. Experiments show that the method produces density optical flow field and precision of the result can be improved.

An Improved Audio Based Key Frame Detection Model for Movie Summarization

This paper presents a new algorithm for enforcing temporal coherence on depth estimation from multi-view videos of dynamic scenes as well as the first substantial quantitative evaluation of the improvement in depth estimation accuracy due to temporal coherence. The proposed algorithm is generally applicable and practical since it bypasses explicit scene flow estimation, which has a very large state space, and relies only on optical flow which is used to impose soft constraints on depth estimation for the next frame. As a result, our algorithm is applicable to scenes with large depth and motion ranges. The output is a sequence of depth maps that can be used for novel view synthesis among other applications. While it is intuitive that enforcing temporal coherence should improve the accuracy of depth estimation, this improvement has never been assessed quantitatively due to the lack of data with ground truth. To overcome this limitation we use the image prediction error as the criterion and show that the benefits of temporal coherence are significant on a diverse set of multi-view video sequences.

According to guidelines for the evaluation of church bell chimes, the timbre of a well-sounding bell is characterized by a soft, full, bright and clear tone with a salient fundamental pitch in a low-noise sound spectrum, where a luminous, clearly pronounced striking tone can be heard with as little percussive noise as possible. In a pilot experiment, 11 bell experts and 26 laypersons evaluated the ringing of 40 loudness-matched bell recordings in terms of tonal quality, pleasantness, salience of the fundamental pitch, clarity and softness of the attack. Via signal analysis, the bell sounds were analyzed in regards to 127 audio features. Correlation and regression analyses were used to determine relationships between the bells' timbre ratings and the calculated audio features. Timbral changes resulting from an optimization of the bell clapper turned out to be particularly noticeable in respect to timbral brightness, the pronouncedness of the tonal components in the spectrum (harmonic energy) and the noise components (spectral flux above 1000 Hz). Specifically, by modifying the clapper, subjective pleasantness of the bells' timbre can be improved. With increasing clapper weight and decreasing ringing angle, the fundamental pitch of bells becomes more pronounced, with stronger harmonic components and less background noise.

The video object mechanism is based on the appearance representation of the object. Tracking objects in a video has applications in video surveillance and other some applications. In previous method does not handle the case of full occlusion. The case of full occlusion can be handled by setting an LSK similarity threshold, which stops tracking when the object is lost. The objects can be tracked based on the compression domain and pixel domain. The spatial information of the object that is being tracked is retrieved using Local Steering Kernels. The Color features are extracted using color histograms. The features similarities between a candidate object ROI and the object ROI in the previous frame and the last stored object instance in the object model are measured. For extracting the foreground extended Kalman filter is used. This increases the tracking efficiency of the algorithm. Index Terms: Color histograms, localsteeringkernels, and visualobjecttracking.

Video Shot Boundary Detection (SBD) is the fundamental process towards video summarization and retrieval. A fast and efficient SBD algorithm is necessary for real-time video processing applications. Extensive work has focused on accurate shot boundary detection at the expense of demanding computational costs. In this paper, we propose a fast SBD approach that reduces the computation pixel-wise and frame-wise while still giving satisfactory accuracy. The proposed approach substantially speeds up the computation through reducing both detection region and scope. Color histogram and mutual information are used together to measure the difference between frames. Corner distribution of frames is utilized to exclude most of false boundaries. We conduct extensive experiments to evaluate the proposed approach, and the results show that our approach can not only speed up SBD, but also detect shot boundaries with high accuracy in both Cut (CUT) and Gradual Transition (GT) boundaries.

Motion tracking of moving objects from a sequence of images is an active research area devoted to the applications of video surveillance. The motion estimation is performed by means of optical flow. In this paper, a fractional order variational model is presented for motion estimation. In particular, the proposed model generalizes the existing variational models from integer order to fractional order. More specific, the fractional order derivative describes discontinuous information about texture and edges, whereas integer order unable to do so, and therefore, a more suitable in estimating the optical flow. The Grunwald-Letnikov derivative is used as a discretization scheme to discretize the complex fractional order partial differential equations corresponding to the Euler-Lagrange equations. The resulting system of equations is solved by using the conjugate gradient method. Experimental results on various datasets verify the validity of the proposed model.