Extracting deep bottleneck features using stacked auto-encoders

This study was motivated by the hypothesis that low-frequency cortical oscillations help the brain decode the speech signal. The intelligibility (in terms of word error rate) of natural-sounding, synthetically-generated sentences was measured using a paradigm that alters speech-energy rhythm over a range of modulation frequencies. The material comprised 96 semantically unpredictable sentences (SUS), each approximately 2 s long, generated by a high-quality text-to-speech (TTS) synthesis engine. The TTS waveform was time-compressed by a factor of 3, creating a signal with a syllable rhythm three times faster than the original and whose intelligibility is poor (<50% words correct). A waveform with an artificial rhythm was produced by segmenting the time-compressed waveform into consecutive 40-ms fragments each followed by a silent interval. The parameters varied were the length of the silent interval (0-160 ms) and whether the intervals of silence were equal (“periodic”) or not (“aperiodic”). The performance curve (word error rate as a function of mean duration of silence) was U-shaped. The lowest word error rate occurred when the silence was 80-ms long and inserted periodically. This was also the condition for which word error rate increased when the silence was inserted aperiodically. These data are consistent with a model (“TEMPO”) in which low-frequency brain rhythms influence the ability to decode the speech signal. In TEMPO, optimum intelligibility is achieved when the syllable rhythm is within the range of the high-theta frequency brain rhythms (6-12 Hz), comparable to the rate at which segments and syllables are spoken in conversational speech.

We propose a feature enhancement algorithm to improve robust automatic speech recognition (ASR). The algorithm estimates a smoothed ideal ratio mask (IRM) in the Mel frequency domain using deep neural networks and a set of time-frequency unit level features that has previously been used to estimate the ideal binary mask. The estimated IRM is used to filter out noise from a noisy Mel spectrogram before performing cepstral feature extraction for ASR. On the noisy subset of the Aurora-4 robust ASR corpus, the proposed enhancement obtains a relative improvement of over 38% in terms of word error rates using ASR models trained in clean conditions, and an improvement of over 14% when the models are trained using the multi-condition training data. In terms of instantaneous SNR estimation performance, the proposed system obtains a mean absolute error of less than 4 dB in most frequency channels.

Speech activity detection (SAD) is an essential component of most speech processing tasks and greatly influences the performance of the systems. Noise and channel distortions remain a challenge for SAD systems. In this paper, we focus on a dataset of highly degraded signals, developed under the DARPA Robust Automatic Transcription of Speech (RATS) program. On this challenging data, the best-performing systems are those based on deep neural networks (DNN) trained to predict speech/non-speech posteriors for each frame. We propose a novel two-stage approach to SAD that attempts to model phonetic information in the signal more explicitly than in current systems. In the first stage, a bottleneck DNN is trained to predict posteriors for senones. The activations at the bottleneck layer are then used as input to a second DNN, trained to predict the speech/non-speech posteriors. We test performance on two datasets, with matched and mismatched channels compared to those in the training data. On the matched channels, the proposed approach leads to gains of approximately 35% relative to our best single-stage DNN SAD system. On mismatched channels, the proposed system obtains comparable performance to our baseline, indicating more work needs to be done to improve robustness to mismatched data.

This article studies the role of architecture design, i.e. choice of the number of nodes at each hidden layer, in deep neural networks (DNNs). We give a theoretical explanation that invariance and complexity of a DNN are determined by the design of its architecture. To be more specific, for DNNs with the rectified linear activation function, we prove that the variations of gradients become the largest when the bottleneck layer, the layer with the fewest nodes, changes its activation pattern and the gap of the variations of gradients of the bottleneck layer to those of the other layers is getting larger as the differences of node counts between the bottleneck layer and the other layers become larger. That is, most large variations of the function made by a DNN are concentrated on the decision boundaries corresponding to the activation patterns of the bottleneck layer. An important implication is that invariance and complexity can be improved simultaneously, which is a key to success of DNNs for many artificial intelligence domains, by designing the highest layer as the bottleneck layer. Various experiments are carried out to validate our theoretical result and its by-products.

A key challenge in radar micro-Doppler classification is the difficulty in obtaining a large amount of training data due to costs in time and human resources. Small training datasets limit the depth of deep neural networks (DNNs), and, hence, attainable classification accuracy. In this work, a novel method for diversifying Kinect-based motion capture (MOCAP) simulations of human micro-Doppler to span a wider range of potential observations, e.g. speed, body size, and style, is proposed. By applying three transformations, a small set of MOCAP measurements is expanded to generate a large training dataset for network initialization of a 30-layer deep residual neural network. Results show that the proposed training methodology and residual DNN yield improved bottleneck feature performance and the highest overall classification accuracy among other DNN architectures, including transfer learning from the 1.5 million sample ImageNet database.

This paper compares different approaches for using deep neural networks (DNNs) trained to predict senone posteriors for the task of spoken language recognition (SLR). These approaches have recently been found to outperform various baseline systems on different datasets, but they have not yet been compared to each other or to a common baseline. Two of these approaches use the DNNs to generate feature vectors which are then processed in different ways to predict the score of each language given a test sample. The features are extracted either from a bottleneck layer in the DNN or from the output layer. In the third approach, the standard i-vector extraction procedure is modified to use the senones as classes and the DNN to predict the zeroth order statistics. We compare these three approaches and conclude that the approach based on bottleneck features followed by i-vector modeling outperform the other two approaches. We also show that score-level fusion of some of these approaches leads to gains over using a single approach for short-duration test samples. Finally, we demonstrate that fusing systems that use DNNs trained with several languages leads to improvements in performance over the best single system, and we propose an adaptation procedure for DNNs trained with languages with less available data. Overall, we show improvements between 40% and 70% relative to a state-of-the-art Gaussian mixture model (GMM) i-vector system on test durations from 3 seconds to 120 seconds on two significantly different tasks: the NIST 2009 language recognition evaluation task and the DARPA RATS language identification task.

We have recently proposed a new acoustic model based on probabilistic linear discriminant analysis (PLDA) which enjoys the flexibility of using higher dimensional acoustic features, and is more capable to capture the intra-frame feature correlations. In this paper, we investigate the use of bottleneck features obtained from a deep neural network (DNN) for the PLDA-based acoustic model. Experiments were performed on the Switchboard dataset — a large vocabulary conversational telephone speech corpus. We observe significant word error reduction by using the bottleneck features. In addition, we have also compared the PLDA-based acoustic model to three others using Gaussian mixture models (GMMs), subspace GMMs and hybrid deep neural networks (DNNs), and PLDA can achieve comparable or slightly higher recognition accuracy from our experiments.

This paper proposes an efficient training-data generation approach for a deep-neural-network (DNN) based forward model of microstrip bandpass filters (BPFs), which significantly reduces the computation time required to generate training data. Conventional forward models require a large amount of training data obtained through time-consuming, high-accuracy electromagnetic (EM) simulations. To address this problem, the proposed method utilizes two types of training data generated from coarse- and fine-mesh EM simulations. First, the first DNN (DNN1) is trained using a large set of data from coarse-mesh EM simulations. Then, the second DNN (DNN2) is connected to the DNN1 to refine its output. Both DNNs are jointly trained using a small set of data obtained from fine-mesh EM simulations, resulting in an accurate forward model. As a case study, a forward model for a third-order BPF is constructed. The results show that the proposed model maintains high prediction accuracy while reducing the training data generation time by approximately 50%, thereby validating the effectiveness of the proposed approach.

Recently, a lot of research has been conducted to bring Automatic Speech Recognition (ASR) into various areas of Air Traffic Control (ATC), such as ATC simulation and training, monitoring live operators for with the aim of safety improvements, ATC workload measurement and conducting analysis on large quantities of controller-pilot speech. Due to the high accuracy requirements of the ATC context and its unique challenges, ASR has not been widely adopted in this field. In this paper, in order take advantage of the opportunities offered by the ATC context such as standardized phraseology and small vocabulary size to reduce the Word Error Rate (WER) of ASR in ATC, we perform n-best list re-ranking using syntactic knowledge. We propose a novel feature called syntactic score which is computed using syntactic rules. We also propose a WER-Sensitive Pairwise Perceptron algorithm and use the perceptron to combine the proposed feature with the decoder's confidence score. We integrate the model into the Pocketsphinx speech recognizer and evaluate the model in terms of Word Error Rate (WER) on the well known ATCOSIM and our own ATCSC corpora. The results shows that our proposed approach reduces 1.21% and 0.21% WER on the ATCSC and ATCOSIM corpora respectively.

Natural Language Processing (NLP) is a vital field of artificial intelligence that automates the study of human language. However for Malay manuscripts (MM) written in old jawi, its exposure on such field is limited. Besides, most of the studies related to MM studies and NLP were focused on rule based or rule based machine transliteration (RBMT). Hence the objective of this study is to propose a statistical approach for old jawi to modern jawi transliteration of Malay manuscript contents using Phrase Based Statistical Machine Translation (PBSMT) as its model. In order to achieve such purpose, quality score of Word Error Rate (WER) was computed on the transliteration output. Besides, the issues formerly encountered by rule based approach such as vocals limitation and homograph, reduplication, letters error and combination of multiple words were observed in the implementation. Moreover, this paper utilized exploratory approach as its research strategy and mixed method as its research method. The data for the analysis were extracted from a MM titled Bidāyat al-Mubtadī bi-Fālillah al-Muhdī. Quality score of WER was computed for the evaluation of SMT output. Afterwards, related issues were identified and assessed. The research found that quality score of PBSMT for old jawi to modern jawi transliteration was high in terms of WER, however the issues of rule based were generally addressed by PBSMT except homograph. The research is however limited to the approach of SMT that solely focused on PBSMT as its model. Moreover, the corpus size was limited to one manuscript while SMT relies on corpus size. Nevertheless the research contributes to the wider coverage on Malay language as one of the under resource languages in NLP, in form of old and modern jawi. Besides, to the best of the researcher’s knowledge, it is also the first to apply SMT (PBSMT) approach on old jawi transliteration. Most importantly, the study is to contribute on MM’s.

Characterizing biobehavioral time series data recorded under different task conditions is a critical step in neuroscience data analysis. The inherent complexity and stochasticity present in these data pose significant modeling challenges. Deep Neural Network (DNN) models have been widely adopted for analyzing such data. Despite their success, DNNs have significant limitations: they are sensitive to noise in training data and require large training datasets. These issues limit the applicability of DNNs to neuroscience data - where sample size is often smaller and data are volatile and feature noise artifacts. The present manuscript introduces a novel framework that embeds a state-space model (SSM) in the classic DNN structure. We demonstrate that this framework, which we call SSM-DNN, can overcome the sample size and noise sensitivity issues that plague classic DNNs. For this purpose, we illustrate training and inference steps when SSM-DNN is applied to a dataset recorded during a Death Implicit Association Test (D-IAT) task. This task was designed to produce biobehavioral data that facilitate decoding of participant phenotypes (e.g., person with depression vs. psychologically healthy person). We show SSM-DNN performance reaches an accuracy of 78%, which is 20% higher than state-of-art DNN decoder models. Its area under curve (AUC) is 0.8, which reflects its high specificity and sensitivity. The SSM-DNN modeling framework is scalable to high-dimensional time-series data and it can be applied broadly to neuroscience data, providing a robust and accurate decoding performance.

Typical hands-free voice capturing relies on recording a distant speaker using an array of microphones. When speech signal is recorded in a reverberant environment, the microphones capture speech corrupted by room reverberation. Increasing the distance between the speaker and recording microphones, the power ratio of the direct and late room reverberation signals is decreased, which in turn decreases the accuracy of automatic speech recognition. This paper presents a multichannel Wiener filter which exploits geometric information about early room reflection paths in order to reduce late reverberation at the filter output. As a result of such pre-processing, the accuracy of automatic speech recognition systems is improved. The presented word error rate (WER) results for simulated rooms with different reverberation times indicate that the proposed rake multichannel Wiener filtering provides an improvement in accuracy of automatic speech recognition from distance in reverberant environments. The gain in terms of word error rate achieved in considered acoustic conditions reaches several percent.

The field of speech recognition has clearly benefited from precisely defined testing conditions and objective performance measures such as word error rate. In the development and evaluation of new methods, the question arises whether the empirically observed difference in performance is due to a genuine advantage of one system over the other, or just an effect of chance. However, many publications still do not concern themselves with the statistical significance of the results reported. We present a bootstrap method for significance analysis which is, at the same time, intuitive, precise and and easy to use. Unlike some methods, we make no (possibly ill-founded) approximations and the results are immediately interpretable in terms of word error rate.

Classification of Bisyllabic Lexical Stress Patterns Using Deep Neural Networks

Deep convolutional neural networks have achieved great success in recent years. They have been widely used in various applications such as optical and SAR image scene classification, object detection and recognition, semantic segmentation, and change detection. However, deep neural networks rely on large-scale high-quality training data, and can only guarantee good performance when the training and test data are independently sampled from the same distribution. Deep convolutional neural networks are found to be vulnerable to subtle adversarial perturbations. This adversarial vulnerability prevents the deployment of deep neural networks in security-sensitive applications such as medical, surveillance, autonomous driving and military scenarios. This paper first presents a holistic view of security issues for deep convolutional neural network-based image recognition systems. The entire information processing chain is analyzed regarding safety and security risks. In particular, poisoning attacks and evasion attacks on deep convolutional neural networks are analyzed in detail. The root causes of adversarial vulnerabilities of deep recognition models are also discussed. Then, we give a formal definition of adversarial robustness and present a comprehensive review of adversarial attacks, adversarial defense, and adversarial robustness evaluation. Rather than listing existing research, we focus on the threat models for the adversarial attack and defense arms race. We perform a detailed analysis of several representative adversarial attacks on SAR image recognition models and provide an example of adversarial robustness evaluation. Finally, several open questions are discussed regarding recent research progress from our workgroup. This paper can be further used as a reference to develop more robust deep neural network-based image recognition models in dynamic adversarial scenarios.