Audio event detection on Google's Audio Set database: Preliminary results using different types of DNNs

Detection and recognition of loud sounds and characteristic noises can significantly increase the level of safety and ensure timely response to various emergency situations. Audio event detection is the first step in recognizing audio signals in a continuous audio input stream. This article presents a number of problems that are associated with the development of sound event detection systems, such as the deviation for each environment and each sound category, overlapping audio events, unreliable training data, etc. Both methods for detecting monophonic impulsive audio event and polyphonic sound event detection methods which are used in the state-of-the-art sound event detection systems are presented. Such systems are presented in Detection and Classification of Acoustic Scenes and Events (DCASE) challenges and workshops, which take place every year. Beside a majority of works focusing on the improving overall performance in terms of accuracy many other aspects have also been studied. Several systems presented at DCASE 2021 task 4 were considered, and based on their analysis, there was a conclusion about possible future for sound event detection systems. Also the actual directions in the development of modern audio analytics systems are presented, including the study and use of various architectures of neural networks, the use of several data augmentation techniques, such as universal sound separation, etc.

Non‐speech audio event detection (AED) could be used for low‐cost, spatially diffuse surveillance applications, e.g., monitoring of vehicle activity in a national park, or of footsteps in a hallway. Experiments have shown that non‐speech AED benefits from the dynamic inference strategies such as the hidden Markov model (HMM), but that the acoustic features useful for non‐speech events may not be the same as those useful for speech. One possible solution is a tandem HMM: an HMM whose observation vector is constructed from the output of an instantaneous discriminative classifier, e.g., a neural network. The use of tandem HMMs for non‐speech AED is hindered, however, by the relatively small size of most non‐speech‐audio training corpora. This talk will demonstrate that tandem HMMs can be trained to detect non‐speech audio events using a novel form of regularized training: Baum–Welch back‐propagation (as proposed by Bengio et al.), using the conjugate‐gradient adaptive form of the Baum–Welch auxiliary functio...

From a personalized media point of view, many users favor a flexible tool to quickly browse the affective content in a video. Such affective content may cause audiences' strong reactions or special emotional experiences, such as anger, sadness, fear, joy and love. This paper attempts to extract affective content for digital videos by analyzing the subtitle files of DVD/DivX videos and utilize audio event to assist affective content detection. Firstly, videos are segmented by dialogue script partition. Compared to traditional video shot, video segmented by scripts is not affected by camera changes and shooting angles and easy to include video segments with compact content. Secondly, emotion-related vocabularies in video script are detected to locate affective video content. Using script to directly access video content avoids complex video analysis. Thirdly, audio event detection is utilized to assist affective content detection. Compared with traditional video semantic analysis, affective content analysis puts much more emphasis on the audience's reactions and emotions. Initial experiments are carried on sitcom videos because its simple video structure provides useful domain knowledge. The experimental results demonstrate that subtitle file analysis and audio event detection provides effective and efficient clues to determine the emotional content of the videos.

We propose a new deep network for audio event recognition, called AENet. In contrast to speech, sounds coming from audio events may be produced by a wide variety of sources. Furthermore, distinguishing them often requires analyzing an extended time period due to the lack of clear sub-word units that are present in speech. In order to incorporate this long-time frequency structure of audio events, we introduce a convolutional neural network (CNN) operating on a large temporal input. In contrast to previous works this allows us to train an audio event detection system end-to-end. The combination of our network architecture and a novel data augmentation outperforms previous methods for audio event detection by 16%. Furthermore, we perform transfer learning and show that our model learnt generic audio features, similar to the way CNNs learn generic features on vision tasks. In video analysis, combining visual features and traditional audio features such as MFCC typically only leads to marginal improvements. Instead, combining visual features with our AENet features, which can be computed efficiently on a GPU, leads to significant performance improvements on action recognition and video highlight detection. In video highlight detection, our audio features improve the performance by more than 8% over visual features alone.

This article investigates a novel use of the well-known audio words representations to detect specific audio events, namely gunshots and explosions, in order to get more robustness towards soundtrack variability in Hollywood movies. An audio stream is processed as a sequence of stationary segments. Each segment is described by one or several audio words obtained by applying product quantization to standard features. Such a representation using multiple audio words constructed via product quantisation is one of the novelties described in this work. Based on this representation, Bayesian networks are used to exploit the contextual information in order to detect audio events. Experiments are performed on a comprehensive set of 15 movies, made publicly available. Results are comparable to the state of the art results obtained on the same dataset but show increased robustness to decision thresholds, however limiting the range of possible operating points in some conditions. Late fusion provides a solution to this issue.

An effective and non-invasive audio monitoring system needs to be capable of simultaneous real-time detection of multiple audio events in many different environments, and locally executable on resource constrained devices, such as, smart microphones. A major challenge in this research domain is having limited available annotated data. This paper presents a novel framework to generate robust detection models of environmental and human audio events with limited available data. The framework presents the generation of a large synthetic dataset using limited data for any audio event, a novel computationally efficient feature modeling technique, named Audio2Vec, that is robust against environmental variations, and identifies and exploits the syntactic relation between audio states represented by the features and the targeted audio events. The presented framework achieves 10.3% higher F-1 scores compared to the best baseline approaches. To demonstrate the effectiveness of the framework we implemented a real-time audio monitoring system that simultaneously detects 10 audio events on a Raspberry Pi 3B and evaluate it in real home and in-car settings, that achieve F-1 scores of 0.96 and 0.956, respectively.

Ensuring passengers’ safety is one of the daily concerns of railway operators. To do this, various image and sound processing techniques have been proposed in the scientific community. Since the beginning of the 2010s, the development of deep learning made it possible to develop these research areas in the railway field included. Thus, this article deals with the audio events detection task (screams, glass breaks, gunshots, sprays) using deep learning techniques. It describes the methodology for designing a deep learning architecture that is both suitable for audio detection and optimised for embedded railway systems. We will describe how we designed from scratch two CRNN (Convolutional Recurrent Neural Network) for the detection task. And since the creation of a large and varied training database is one of the challenges of deep learning, this article also deals with the innovative methodology used to build a database of audio events in the railway environment. Finally, we will show the very promising results obtained during the experimentation in real of the model.

This paper summarizes the contributions to semantic video search that can be derived from the audio signal. Because of space restrictions, the emphasis will be on non-linguistic cues. The paper thus covers what is generally known as audio segmentation, as well as audio event detection. Using machine learning approaches, we have built detectors for over 50 semantic audio concepts.

Orca Call Detection using CNN and Spectrograms

The paper introduces a hierarchy-aware loss function in a Deep Neural Network for an audio event detection task that has a bi-level tree structured label space. The goal is not only to improve audio event detection performance at all levels in the label hierarchy, but also to produce better audio embeddings. We exploit the label tree structure to preserve that information in the hierarchy-aware loss function. Two different loss functions are separately employed. First, a triplet loss with probabilistic multi-level batch mining is introduced. Second, a quadruplet learning method is applied, which is a special case of generalized triplet learning for bi-level label taxonomy. The training is performed in a multi-task learning framework by jointly optimizing cross entropy based loss and hierarchy-aware loss function. The proposed method is found to outperform the baseline cross entropy based models at both levels of the hierarchy. The multi-task model is also able to learn better audio representations as observed in our clustering experiments. Moreover, the model is shown to transfer well when an out-of-domain dataset is used for evaluation.

Lung Auscultation is a non-invasive process of distinguishing normal respiratory sounds from abnormal ones by analyzing the airflow along the respiratory tract. With developments in the Deep Learning (DL) techniques and wider access to anonymized medical data, automatic detection of specific sounds such as crackles and wheezes have been gaining popularity. In this paper, we propose to use two sets of diversified acoustic biomarkers extracted using Discrete Wavelet Transform (DWT) and deep encoded features from the intermediate layer of a pre-trained Audio Event Detection (AED) model trained using sounds from daily activities. First set of biomarkers highlight the time frequency localization characteristics obtained from DWT coefficients. However, the second set of deep encoded biomarkers captures a generalized reliable representation, and thus indemnifies the scarcity of training samples and the class imbalance in dataset. The model trained using these features achieves a 15.05% increase in terms of the specificity over the baseline model that uses spectrogram features. Moreover, ensemble of DWT features and deep encoded feature based models show absolute improvements of 8.32%, 6.66% and 7.40% in terms of sensitivity, specificity and ICBHI-score, respectively, and clearly outperforms the state-of-the-art with a significant margin.

Audio analytic systems are receiving an increasing interest in the scientific community, not only as stand alone systems for the automatic detection of abnormal events by the interpretation of the audio track, but also in conjunction with video analytics tools for enforcing the evidence of anomaly detection. In this paper we present an automatic recognizer of a set of abnormal audio events that works by extracting suitable features from the signals obtained by microphones installed into a surveilled area, and by classifying them using two classifiers that operate at different time resolutions. An original aspect of the proposed system is the estimation of the reliability of each response of the individual classifiers. In this way, each classifier is able to reject the samples having an overall reliability below a threshold. This approach allows our system to combine only reliable decisions, so increasing the overall performance of the method. The system has been tested on a large dataset of samples acquired from real world scenarios, the audio classes of interests are represented by gunshot, scream and glass breaking in addition to the background sounds. The preliminary results obtained encourage further research in this direction.

This paper presents new approaches to improve the detection of two key audio events in a sport game (tennis) using contextual information. When analysing a tennis match using only audio information, the sound of the ball being struck and the occurrence of a line judge's shout can be obscured by players' grunts or shouts. Furthermore, if models of these two important events are trained from labelled training-data, there is often considerable audio mis-match with the test-data, which means that detection performance can be very poor. To handle this problem, we regard the players' grunts as useful contextual information that indicates the position of the events of interest. We show how to use an unsupervised learning method to build an improved model of the ball-hit event using grunt information. We can then use high-level information to distinguish grunts from line-judge shouts. This approach gives simultaneous improvements in detection of both ball-hits and line judge shouts, and is portable between different matches, unlike approaches based on the use of manually labelled training-data.

Audio event recognition, the human-like ability to identify and relate sounds from audio, is a nascent problem in machine perception. Comparable problems such as object detection in images have reaped enormous benefits from comprehensive datasets - principally ImageNet. This paper describes the creation of Audio Set, a large-scale dataset of manually-annotated audio events that endeavors to bridge the gap in data availability between image and audio research. Using a carefully structured hierarchical ontology of 632 audio classes guided by the literature and manual curation, we collect data from human labelers to probe the presence of specific audio classes in 10 second segments of YouTube videos. Segments are proposed for labeling using searches based on metadata, context (e.g., links), and content analysis. The result is a dataset of unprecedented breadth and size that will, we hope, substantially stimulate the development of high-performance audio event recognizers.

The new generation of weather observatory satellites, namely Global Precipitation Measurement (GPM) constellation satellites, is the lead observatory of the 10 highly advanced earth orbiting weather research satellites. Indeed, GPM is the first satellite that has been designed to measure light rain and snowfall, in addition to heavy tropical rainfall. This work compares the final run of the Integrated Multi-satellitE Retrievals for GPM (IMERG) product, the post real time of TRMM and Multi-satellite Precipitation Analysis (TMPA-3B42) and the Era-Interim product from the European Centre for Medium Range Weather Forecasts (ECMWF) against the Iran Meteorological Organization (IMO) daily precipitation measured by the synoptic rain-gauges over four regions with different topography and climate conditions in Iran. Assessment is implemented for a one-year period from March 2014 to February 2015. Overall, in daily scale the results reveal that all three products lead to underestimation but IMERG performs better than other products and underestimates precipitation slightly in all four regions. Based on monthly and seasonal scale, in Guilan all products, in Bushehr and Kermanshah ERA-Interim and in Tehran IMERG and ERA-Interim tend to underestimate. The correlation coefficient between IMERG and the rain-gauge data in daily scale is far superior to that of Era-Interim and TMPA-3B42. On the basis of daily timescale of bias in comparison with the ground data, the IMERG product far outperforms ERA-Interim and 3B42 products. According to the categorical verification technique in this study, IMERG yields better results for detection of precipitation events on the basis of Probability of Detection (POD), Critical Success Index (CSI) and False Alarm Ratio (FAR) in those areas with stratiform and orographic precipitation, such as Tehran and Kermanshah, compared with other satellite/model data sets. In particular, for heavy precipitation (>15 mm/day), IMERG is superior to the other products in all study areas and could be used in future for meteorological and hydrological models, etc.