Multitask learning for acoustic scene classification with topic-based soft labels and a mutual attention mechanism

Multi-task learning (MTL) has achieved success over a wide range of problems, where the goal is to improve the performance of a primary task using a set of relevant auxiliary tasks. However, when the usefulness of the auxiliary tasks w.r.t. the primary task is not known a priori, the success of MTL models depends on the correct choice of these auxiliary tasks and also a balanced mixing ratio of these tasks during alternate training. These two problems could be resolved via manual intuition or hyper-parameter tuning over all combinatorial task choices, but this introduces inductive bias or is not scalable when the number of candidate auxiliary tasks is very large. To address these issues, we present AutoSeM, a two-stage MTL pipeline, where the first stage automatically selects the most useful auxiliary tasks via a Beta-Bernoulli multi-armed bandit with Thompson Sampling, and the second stage learns the training mixing ratio of these selected auxiliary tasks via a Gaussian Process based Bayesian optimization framework. We conduct several MTL experiments on the GLUE language understanding tasks, and show that our AutoSeM framework can successfully find relevant auxiliary tasks and automatically learn their mixing ratio, achieving significant performance boosts on several primary tasks. Finally, we present ablations for each stage of AutoSeM and analyze the learned auxiliary task choices.

Common acoustic properties that different classes share degrades the performance of acoustic scene classification systems. This results in a phenomenon where a few confusing pairs of acoustic scenes dominate a significant proportion of all misclassified audio segments. In this article, we propose adopting a knowledge distillation framework that trains deep neural networks using soft labels. Soft labels, extracted from another pre-trained deep neural network, are used to reflect the similarity between different classes that share similar acoustic properties. We also propose utilizing specialist models to provide additional soft labels. Each specialist model in this study refers to a deep neural network that concentrates on discriminating a single pair of acoustic scenes that are frequently misclassified. Self multi-head attention is explored for training specialist deep neural networks to further concentrate on target pairs of classes. The goal of this article is to train a single deep neural network that demonstrates performance equivalent to, or higher than, an ensemble of multiple models, by distilling the knowledge from several models. Diverse experiments conducted using the detection and classification of acoustic scenes and events 2019 task 1-a dataset demonstrate that the knowledge distillation framework is effective in acoustic scene classification. Specialist models successfully decrease the number of misclassified audio segments in the target classes. The final single model with the proposed method that is trained by the proposed knowledge distillation from several models, including specialists trained using an attention mechanism, shows a classification accuracy of 77.63 %, higher than an ensemble of the baseline and multiple specialists.

Most attention-based methods only concentrate along the time axis, which is insufficient for Acoustic Event Detection (AED).Meanwhile, previous methods for AED rarely considered that target events possess distinct temporal and frequential scales.In this work, we propose a Multi-Scale Time-Frequency Attention (MTFA) module for AED.MTFA gathers information at multiple resolutions to generate a time-frequency attention mask which tells the model where to focus along both time and frequency axis.With MTFA, the model could capture the characteristics of target events with different scales.We demonstrate the proposed method on Task 2 of Detection and Classification of Acoustic Scenes and Events (DCASE) 2017 Challenge.Our method achieves competitive results on both development dataset and evaluation dataset.

Speech emotion recognition (SER) systems find applications in various fields such as healthcare, education, and security and defense. A major drawback of these systems is their lack of generalization across different conditions. For example, systems that show superior performance on certain databases show poor performance when tested on other corpora. This problem can be solved by training models on large amounts of labeled data from the target domain, which is expensive and time-consuming. Another approach is to increase the generalization of the models. An effective way to achieve this goal is by regularizing the models through multitask learning (MTL), where auxiliary tasks are learned along with the primary task. These methods often require the use of labeled data which is computationally expensive to collect for emotion recognition (gender, speaker identity, age or other emotional descriptors). This study proposes the use of ladder networks for emotion recognition, which utilizes an unsupervised auxiliary task. The primary task is a regression problem to predict emotional attributes. The auxiliary task is the reconstruction of intermediate feature representations using a denoising autoencoder. This auxiliary task does not require labels so it is possible to train the framework in a semi-supervised fashion with abundant unlabeled data from the target domain. This study shows that the proposed approach creates a powerful framework for SER, achieving superior performance than fully supervised single-task learning (STL) and MTL baselines. We implement the approach with sentence-level or frame-level features, demonstrating the flexibility of our approach. Additionally, the generalization of the ladder networks is evaluated in cross-corpus settings using sentence-level features, obtaining important improvements. Compared to the STL baselines, the proposed approach achieves relative gains in concordance correlation coefficient (CCC) between 3.0% and 3.5% for within corpus evaluations, and between 16.1% and 74.1% for cross corpus evaluations, highlighting the power of the architecture.

In the field of speech emotion recognition(SER), great progress has been made in feature extraction and classifier construction. However, in the natural environment, speech emotion recognition is vulnerable to environmental noise. Moreover, cross-corpus SER is a thorny problem because the gender differences and language differences between corpus. Therefore, these problems need to be solved. Considering the noise problem, wavelet threshold denoising was proposed. After that, this paper used multi-task learning method to solve the problem of languages and gender differences. Emotion recognition is primary task, while gender recognition and language recognition are auxiliary tasks. According to the multi-task learning model, the shared LSTM is designed to extract the shared features. Three private LSTMs are used to extract the features of emotion recognition ,language recognition and gender recognition respectively. Orthogonal constraint training was used to make the private space contain only private features. Two experiments are designed in this paper. The first experiment is the comparison between different wavelet bases and different numbers of decomposition layers to select suitable wavelet base and the number of decomposition layer. The second experiment is the comparison between cross-corpus SER in other literature.

ADMET property prediction via multi-task graph learning under adaptive auxiliary task selection

SummaryDeep learning has made great achievements in the field of speech recognition. With the popularization of embedded devices such as intelligent speaker and the demand for dialect interaction scenes, it poses great challenges to far‐field speech recognition and dialect language recognition. In order to solve the dialect language recognition of embedded devices in far‐field speech recognition, we propose a deep learning neural network model with multitask learning. First, the audio is passed through the end‐to‐end noise reduction model to improve the effect of audio recognition. Then we define dialect recognition as the main task and dialect area as the auxiliary task, using the multitask learning method to improve the accuracy of dialect classification. The experimental results show that the end‐to‐end noise reduction model can improve the accuracy of audio recognition, and the best effect can be 7.54% higher than the baseline, and the accuracy of dialect language recognition can be improved by about 5% through multi task learning model.

Acoustic event detection and scene classification are major research tasks in environmental sound analysis, and many methods based on neural networks have been proposed. Conventional methods have addressed these tasks separately; however, acoustic events and scenes are closely related to each other. For example, in the acoustic scene `office', the acoustic events `mouse clicking' and `keyboard typing' are likely to occur. In this paper, we propose multitask learning for joint analysis of acoustic events and scenes, which shares the parts of the networks holding information on acoustic events and scenes in common. By integrating the two networks, we expect that information on acoustic scenes will improve the performance of acoustic event detection. Experimental results obtained using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets indicate that the proposed method improves the performance of acoustic event detection by 10.66 percentage points in terms of the F-score, compared with a conventional method based on a convolutional recurrent neural network.

This paper presents a new approach for acoustic environment classification based on the discrete Hartley transform. The approach applies a Hidden Markov Model based classifier on test data composed of audio clips, in order to determine which environment is surrounding these audio clips. The approach uses features obtained from the discrete Hartley transform, leading to a set of features that require only real arithmetic computations. This can make the technique advantageous in terms of simplicity and/or in terms of computational speed. The proposed approach performance is evaluated on benchmark datasets provided from the 2013 and 2016 Detection and Classification of Acoustic Scenes and Events (DCASE) challenges. Experiments show that the proposed method is competitive compared to other recently proposed methods, and that the use of the discrete Hartley transform improves the classification performance.

Late fusion framework for Acoustic Scene Classification using LPCC, SCMC, and log-Mel band energies with Deep Neural Networks

We consider the problem of adding a large unlabeled sample from the target domain to boost the performance of a domain adaptation algorithm when only a small set of labeled examples is available from the target domain. In particular, we consider the problem setting motivated by the tasks of splice site prediction and protein localization. For example, for splice site prediction, annotating a genome using machine learning requires a lot of labeled data, whereas for non-model organisms, there are only some labeled data and lots of unlabeled data. With domain adaptation one can leverage the large amount of data from a related model organism, along with the labeled and unlabeled data from the organism of interest to train a classifier for the latter. Our goal is to analyze the three approaches of incorporating the unlabeled data—with soft labels only (i.e., Expectation-Maximization), with hard labels only (i.e., self-training), or with both soft and hard labels—for the splice site prediction and protein localization in particular, and more broadly for a general iterative domain adaptation setting. We provide empirical results on splice site prediction and protein localization indicating that using a combination of soft and hard labels performs as good as the best of the other two approaches of integrating unlabeled data.

In this paper, we present the results of the Acoustic Event Detection (AED) and Classification (AEC) evaluations carried out in February 2006 by the three participant partners from the CHIL project. The primary evaluation task was AED of the testing portions of the isolated sound databases and seminar recordings produced in CHIL. Additionally, a secondary AEC evaluation task was designed using only the isolated sound databases. The set of meeting-room acoustic event classes and the metrics were agreed by the three partners and ELDA was in charge of the scoring task. In this paper, the various systems for the tasks of AED and AEC and their results are presented.

Data augmentation, which improves the diversity of datasets by applying image transformations, has become one of the most effective techniques in visual representation learning. Usually, the design of augmentation policies faces a diversity-difficulty trade-off. On the one hand, a simple augmentation leads to a low training set diversity, which can not improve model performance significantly. On the other hand, an excessively hard augmentation has an overlarge regularization effect which harms model performance. Recently, automatic augmentation methods have been proposed to address this issue by searching the optimal data augmentation policy from a predefined searching space. However, these methods still suffer from heavy searching overhead or complex optimization objectives. In this paper, instead of searching the optimal augmentation policy, we propose to break the diversity-difficulty trade-off from a multi-task learning perspective. By formulating model learning on the augmented images and the original images as the auxiliary task and the primary task in multi-task learning respectively, the hard augmentation does not directly influence the training of the primary branch and thus its negative influence can be alleviated. Hence, neural networks can learn valuable semantic information even with a totally random augmentation policy. Experimental results on ten datasets for four tasks demonstrate the superiority of our method over the other twelve methods. Codes have been released in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ArchipLab-LinfengZhang/data-augmentation-multi-task</uri> .

Density-aware and background-aware network for crowd counting via multi-task learning

As a classical machine learning method, multi‐task learning (MTL) has been widely applied in computer vision technology. Due to deep convolutional neural network (D‐CNN) having strong ability of feature representation, the combination of MTL and D‐CNN has attracted much attention from researchers recently. However, this kind of combination has rarely been explored in the field of vehicle analysis. The authors propose a D‐CNN enhanced with weighted multi‐attribute strategy for extensive exploration of comprehensive vehicle attributes over surveillance images. Specifically, regarding to recognising vehicle model and make/manufacturer, several related attributes as auxiliary tasks are incorporated in the training process of D‐CNN structure. The proposed strategy focuses more on the main task compared with traditional MTL methods, which has assigned different weights for the main task and auxiliary tasks rather than treating all involved tasks equally. To the extent of their knowledge, this is the first report relating to the combination of D‐CNN and weighted MTL for exploration of comprehensive vehicle attributes. The following experiments will show that the proposed approach outperforms the state‐of‐the‐art method for the vehicle recognition and improves the accuracy rate by about 10% for the analysis of other vehicle attributes on the recently public CompCars dataset.