A generic audio classification and segmentation approach for multimedia indexing and retrieval

Content-based audio classification and segmentation is a basis for further audio/video analysis. In this paper, we present our work on audio segmentation and classification which employs support vector machines (SVMs). Five audio classes are considered in this paper: silence, music, background sound, pure speech, and non- pure speech which includes speech over music and speech over noise. A sound stream is segmented by classifying each sub-segment into one of these five classes. We have evaluated the performance of SVM on different audio type-pairs classification with testing unit of different- length and compared the performance of SVM, K-Nearest Neighbor (KNN), and Gaussian Mixture Model (GMM). We also evaluated the effectiveness of some new proposed features. Experiments on a database composed of about 4- hour audio data show that the proposed classifier is very efficient on audio classification and segmentation. It also shows the accuracy of the SVM-based method is much better than the method based on KNN and GMM.

An on-line audio classification and segmentation system is presented in this research, where audio recordings are classified and segmented into speech, music, several types of environmental sounds and silence based on audio content analysis. This is the first step of our continuing work towards a general content-based audio classification and retrieval system. The extracted audio features include temporal curves of the energy function,the average zero- crossing rate, the fundamental frequency of audio signals, as well as statistical and morphological features of these curves. The classification result is achieved through a threshold-based heuristic procedure. The audio database that we have built, details of feature extraction, classification and segmentation procedures, and experimental results are described. It is shown that, with the proposed new system, audio recordings can be automatically segmented and classified into basic types in real time with an accuracy of over 90 percent. Outlines of further classification of audio into finer types and a query-by-example audio retrieval system on top of the coarse classification are also introduced.

This paper proposes a security-monitoring instrument that can detect and classify the location and nature of different sounds in a room. The instrument is reliable and robust even in the presence of reverberation and in low signal to noise ratio conditions. This paper proposes a new algorithm for classifying first an audio segment as speech or nonspeech then classifies the nonspeech audio segment into its own audio type. The algorithm divides an audio segment into frames, estimates the presence of pitch in each frame, and calculates a pitch ratio parameter. This parameter is then used to classify the audio segment. The threshold used in calculating this parameter is adapted to accommodate different environments. Nonspeech audio segment has further classification using time delayed neural network to be classified into it is own type. The performance of the proposed algorithm is evaluated for different signal-to-noise ratios using a library of audio segments. The library includes speech segments and nonspeech segments such as windows breaking and footsteps. Using 0.4 second segments it is shown that the proposed algorithm can achieve an average correct decision for 94.5% of the reverberant library and 95.1% of the nonreverberant library

In a cloud setting where audio is intercepted from multiple nodes, it is of interest of identify the type of audio present at each node. Each node can be seen as an IOT (Internet of Things) device. Audio type can be speech by a particular speaker, different kinds of music or background noises, whose monitoring is useful for various IOT applications. A supervised audio classification system using sparse representation over a cloud network is presented. In this system, dictionaries are learnt from different audio streams on multiple nodes in the training stage. Audio classification is done separately on different nodes using distributed sparse representation, avoiding any centralized processing. Both training and testing is done in a distributed manner, and the final estimate of the audio type is arrived by consensus among the nodes. Given an audio segment at any node, distributed sparse coding is used to classify the segment into one of the audio classes using the different dictionary models learnt at different nodes.

Identifying different audio segments in videos is the first step for many important tasks such as event detection and speech transcription. Approaches using Mel-Frequency Cepstral coefficients (MFCCs) with Gaussian mixture models (GMMs) and hidden Markov models (HMMs) perform reasonably well in stationary conditions but do not scale to a broad range of environmental conditions. This paper focuses on the audio segmentation in broadcast soccer videos into audio classes such as Silence, Speech Only, Speech Over Crowd, Crowd Only and Excited, with an alternative feature set which is simplistic as well as robust to changes in the environment conditions. Support Vector Machines (SVMs), Neural Networks and Random Forest are used for the classification. The accuracy achieved with SVMs, Neural Networks and Random Forest are 83.80%, 86.07%, and 88.35% respectively. The proposed features and Random Forest classifier are found to achieve better accuracy compared to the other classifiers.

Introduction to Audio Analysis: A MATLAB Approach

The present paper focuses on adaptive audio detection, segmentation and classification techniques in audio broadcasting content, dedicated mainly to voice data. The suggested framework addresses a real case scenario encountered in media services and especially radio streams, aiming to fulfill diverse (semi-) automated indexing/annotation and management necessities. In this context, aggregated radio content is collected, featuring small input datasets, which are utilized for adaptive classification experiments, without searching, at this point, for a generic pattern recognition solution. Hierarchical and hybrid taxonomies are proposed, firstly to discriminate voice data in radio streams and thereafter to detect single speaker voices, and when this is the case, the experiments proceed into a final layer of gender classification. It is worth mentioning that stand-alone and combined supervised and clustering techniques are tested along with multivariate window tuning, towards the extraction of meaningful results based on overall and partial performance rates. Furthermore, the current work via data augmentation mechanisms contributes to the formulation of a dynamic Generic Audio Classification Repository to be subjected, in the future, to adaptive multilabel experimentation with more sophisticated techniques, such as deep architectures.

Detecting digital audio forgeries is a significant research focus in the field of audio forensics. In this article, the authors focus on a special form of digital audio forgery—copy-move—and propose a fast and effective method to detect doctored audios. First, the article segments the input audio data into syllables by voice activity detection and syllable detection. Second, the authors select the points in the frequency domain as feature by applying discrete Fourier transform (DFT) to each audio segment. Furthermore, this article sorts every segment according to the features and gets a sorted list of audio segments. In the end, the article merely compares one segment with some adjacent segments in the sorted list so that the time complexity is decreased. After comparisons with other state of the art methods, the results show that the proposed method can identify the authentication of the input audio and locate the forged position fast and effectively.

This paper addresses the problem of highlight sound effects detection in audio stream, which is very useful in fields of video summarization and highlight extraction. Unlike researches on audio segmentation and classification, in this domain, it just locates those highlight sound effects in audio stream. An extensible framework is proposed and in current system three sound effects are considered: laughter, applause and cheer, which are tied up with highlight events in entertainments, sports, meetings and home videos. HMMs are used to model these sound effects and a log-likelihood scores based method is used to make final decision. A sound effect attention model is also proposed to extend general audio attention model for highlight extraction and video summarization. Evaluations on a 2-hours audio database showed very encouraging results.

Audio information classification becomes a very important task for such purposes as automatic keyword spotting and other content-based audio-visual query systems. In this paper, we describe a fast and accurate audio data classification method on the MPEG coded data domain. Firstly silent segments are detected using a robust approach for different recording conditions. Then the non-silent segments are classified into three types, music, speech, and applause using temporal density, bandwidth and center frequency of subband energy. In order to be robust for a variety of audio sources as much as possible, we use Bayes discriminant function for multivariate Gaussian distribution instead of manually adjusting a threshold for each discriminator. In the experiment, every one-second of MPEG audio data is classified and about 90% of audio and speech segments have been successfully detected. As for the detection speed, less than 20% of MPEG audio decoding processing power is required.

With the rapid evolution of e-learning technology, the multiple sources of information become more and more accessible. However, the availability of a wide range of e-learning offers makes it difficult for learners to find the right content for their training needs. In this context, our paper aims to design an e-learning AI-powered Chatbot allowing interaction with learners and suggesting the e-learning content adapted to their needs. In order to achieve these objectives, we first analysed the e-learning multimedia content to extract the maximum amount of information. Then, using Natural Language Processing (NLP) techniques, we introduced a new approach to extract keywords. After that, we suggest a new approach for multimedia indexing based on extracted keywords. Finally, the Chatbot architecture is realized based on the multimedia indexing and deployed on online messaging platforms. The suggested approach aims to have an efficient way to represent the multimedia content based on keywords. We compare our approach with approaches in literature and we deduce that the use of keywords on our approach result on a better representation and reduce time to construct multimedia indexing. The core of our Chatbot is based on this indexed multimedia content which enables it to look for the information quickly. Then our designed Chatbot reduce response time and meet the learner’s need.

<p>Addressing the complexities of generating sports summaries through machine learning, our research aims to bridge the gap in audio-based event detection, particularly in soccer games. We introduce an extended ResNet-50 deep learning approach for soccer audio, emphasizing key moments from large soccer content archives through the use of transfer learning. The proposed model accurately classifies soccer audio segments into two categories: i) events, representing crucial in-game occurrences and ii) no events, denoting less impactful moments. The model involves complete audio preprocessing, the implementation of proposed model using transfer learning and the classification of events. The model’s reliability is validated using the dataset soccer action dataset compilation (SADC), involves dataset creation by football fans. Comparative analysis with pre-trained models such as VGG19, DesNet121, and EfficientNetB7 demonstrates the superior performance of the extended ResNet-50 based approach. Results across different epochs reveal consistently high accuracy, precision, recall, and F1-score, emphasizing the proposed model's effectiveness in event detection through audio classification. The paper concludes that the proposed model offers a robust solution for detecting an event from audio of soccer sports providing valuable insights for fans, analysts, and content creators to identify interested moments from soccer game with low failure.</p>

Detection of key sounds, such as applause, laugh, music, environmental noise, etc., is one of the challenges in intelligent management of multimedia information and content understanding. In this paper, we report progress in development of a reference content-based audio classification algorithm that is based on a conventional and widely accepted approach, namely signal parameterization by MFCC followed by GMM classification. Our developed labeled audio database and the conventional classification model should serve as a reference platform for an evaluation of novel, alternative or more advanced methods in audio content analysis.

Audio genre classification is recently getting a growing interest in the field of audio processing, genres are basically labels which are created and utilized for categorizing huge amount of universal music. Because of non-stationary and discontinuous nature of the audio signal, segmentation and classification have come to be a challenging task. Each genre differs from each other in their timbral, instrumental and rhythmic content of music. This paper gives the details of the genre classification using Support Vector Machine (SVM) and their comparison in terms of accuracy with other classifiers. The proposed classifier gives an accuracy of 92 % when classifying 5 musical genres which are higher compared to other classifiers used in this experiment.

Perceived predominant melody of music is the most convenient and memorable description and can be used for content-based music retrieval. However, including human voice and multiple musical instruments playing together, it is difficult to extract a predominant contour of pitches directly from MP3 music recordings. In order to build a similarity music melody model, several audio files, whose melody is perceptually similar, are collected in our experiment. Many features are extracted and a melodic similarity model is defined through analyzing each feature and combinations of them. The melody model is evaluated based on classification results of six categories of Chinese folk music using Support Vector Machine. The experiment results show that 36-dimensional Constant Q Transform (CQT) feature can represent the melody of audio music pieces accurately. Further more, the classification results for audio data with similar melody are good enough to be used in audio music classification or segmentation and subsequently are very helpful in music information retrieval system.