Deep neural network features and semi-supervised training for low resource speech recognition

Not available.

—This paper presents the main improvements brought recently to the SpeeD automatic speech recognition system. Several aspects, such as speech and text resources acquisition, noise-robust speech features and feature transforms are discussed. All the updates in our ASR system are accompanied by experimental results illustrating significant improvements: between 30% and 35% relative WER reductions for various case studies (read/spontaneous speech, noisy/clean speech). In the last part of the paper, our ASR system is also compared with Google’s ASR system and a brief analysis of the results is presented. Keywords -ASR; LVCSR; noise-robust speech recognition; PNCC; LDA. I. I NTRODUCTION Large vocabulary continuous speech recognition (LVCSR) is still an unsolved topic for many languages. The reasons for this are (i) there is a lack of acoustic and linguistic resources needed for development (it is the case of so-called under-resourced languages) and (ii) the scientific research community is not stimulated by any national or international evaluation campaigns (as opposed to languages such as English, French or Chinese). The Romanian language is affected by both the aforementioned problems. In this context, the development of speech and language resources for automatic speech recognition (ASR) is a critical issue that must be addressed to push forward the research in this direction and create LVCSR systems comparable to those available for other languages. This is one of the main goals of the Speech and Dialogue (SpeeD) research group

Multivariate Gaussian based speech compensation or mapping has been developed to reduce the mismatch between training and deployment conditions for robust speech recognition. The acoustic mapping procedure can be formulated as a feature space adaptation where a noisy input signal is transformed by a multivariate Gaussian network. We propose a novel algorithm to update the network parameters based on minimizing the Kullback-Leibler distance between the core recognizer's acoustic model and transformed features. It is designed to achieve optimal overall system performance rather than MMSE on a specific feature domain. An online stochastic gradient descent learning rule is derived. We evaluate the performance of the new algorithm using a JRTk broadcast news system on a distance-talking speech corpus and compare its performance with that of previous MMSE based approaches. The experiments show the KL based approach is more effective for a large vocabulary continuous speech recognition (LVCSR) system.

We like to conversate with other people using both sounds and visuals, as our perception of speech is bimodal. Essentially echoing the same speech structure, we manage to integrate the two modalities and often understand the message better than with the eyes closed. In this work we would like to learn more about the visual nature of speech, coined lip-reading, and to make use of it towards better automatic speech recognition systems. Recent developments in the Machine Learning area, together with the release of suitable audio-visual datasets aimed at large vocabulary continuous speech recognition, have led to a renewal of the lip-reading topic, and allow us to address the recurring question of how to better integrate visual and acoustic speech.

Deep neural networks (DNN) have recently shown impressive performance as acoustic models for large vocabulary continuous speech recognition (LVCSR) tasks. Typically, the frame shift of the output of neural networks is much shorter than the average length of the modeling units, so the posterior vectors of neighbouring frames are likely to be similar. The similarity, together with the better discrimination of neural networks than typical acoustic models, shows a possibility of removing frames of neural network outputs according to the distance of posterior vectors. Then, the computation costs of beam searching can be effectively reduced. Based on that, the paper introduces a novel variable-frame-rate decoding approach based on neural network computation that accelerates the beam searching for speech recognition with minor loss of accuracy. By computing the distances of posterior vectors and removing frames with a posterior vector similar to the previous frame, the approach can make use of redundant information between frames and do a much quicker beam searching. Experiments on LVCSR tasks show a 2.4-times speed up of decoding compared to the typical framewise decoding implementation.

Iterative estimation procedures that minimize empirical risk based on general loss functions such as the Levenshtein distance have been derived as extensions of the Extended Baum Welch algorithm. While reducing expected loss on training data is a desirable training criterion, the se algorithms can be difficult to apply. They are unlike MMI estimation in that they require an explicit listing of the hy potheses to be considered and in complex problems such lists tend to be prohibitively large. To overcome this difficulty, modeling techniques originally developed to improve search efficiency in Minimum Bayes Risk decoding can be used to transform these estimation algorithms so that exact update, risk minimization procedures can be used for complex recognition problems. Experimental results in two large vocabulary speech recognition tasks show improvements over conventionally trained MMIE models.

Lattice segmentation and minimum Bayes risk discriminative training for large vocabulary continuous speech recognition

This paper proposes a novel speaker adaptation algorithm for classifying speech based on deep neural networks (DNNs). The adaptation algorithm consists of two steps. In the first step a deep neural network is trained using raw Mel-frequency cepstral coefficient (MFCC) features to discover hidden structures in the data and employing the activations of the last hidden layers of the DNN as acoustic features. In the second step using nearest neighbor, an adaptation algorithm learns speaker similarity scores based on a small amount of adaptation data from each target speaker using the DNN-based acoustic features. Based on the speaker similarity score, classification is done using a k-nearest neighbor (k-NN) classifier. The novelty of this work is that instead of modifying and re-training the DNN for speaker adaptation, which comprises a large number of parameters and is computationally expensive, activations of the learned DNN are used to project features from MFCC to a sparse DNN space, then speaker adaptation is performed based on similarity (i.e. nearest neighbor) using k-NN algorithm. With only a small amount of adaptation data, it reduces the number of phoneme classification error in the TIMIT dataset by 23%. This work also analyzes impact of deep neural networks architecture on speaker adaptation performance.

Caregiver-infant interactions shape infants' early visual experience; however, there is limited work from low-and middle-income countries (LMIC) in characterizing the visual cognitive dynamics of these interactions. Here, we present an innovative dyadic visual cognition pipeline using machine learning methods which captures, processes, and analyses the visual dynamics of caregiver-infant interactions across cultures. We undertake two studies to examine its application in both low (rural India) and high (urban UK) resource settings. Study 1 develops and validates the pipeline to process caregiver-infant interaction data captured using head-mounted cameras and eye-trackers. We use face detection and object recognition networks and validate these tools using 12 caregiver-infant dyads (4 dyads from a 6-month-old UK cohort, 4 dyads from a 6-month-old India cohort, and 4 dyads from a 9-month-old India cohort). Results show robust and accurate face and toy detection, as well as a high percent agreement between processed and manually coded dyadic interactions. Study 2 applied the pipeline to a larger data set (25 6-month-olds from the UK, 31 6-month-olds from India, and 37 9-month-olds from India) with the aim of comparing the visual dynamics of caregiver-infant interaction across the two cultural settings. Results show remarkable correspondence between key measures of visual exploration across cultures, including longer mean look durations during infant-led joint attention episodes. In addition, we found several differences across cultures. Most notably, infants in the UK had a higher proportion of infant-led joint attention episodes consistent with a child-centered view of parenting common in western middle-class families. In summary, the pipeline we report provides an objective assessment tool to quantify the visual dynamics of caregiver-infant interaction across high- and low-resource settings.

This paper presents a deep neural network (DNN) based large vocabulary continuous speech recognition (LVCSR) system for Serbian, developed using the open-source Kaldi speech recognition toolkit. The DNNs are initialized using stacked restricted Boltzmann machines (RBMs) and trained using cross-entropy as the objective function and the standard error backpropagation procedure in order to provide posterior probability estimates for the hidden Markov model (HMM) states. Emission densities of HMM states are represented as Gaussian mixture models (GMMs). The recipes were modified based on the particularities of the Serbian language in order to achieve the optimal results. A corpus of approximately 90 hours of speech (21000 utterances) is used for the training. The performances are compared for two different sets of utterances between the baseline GMM-HMM algorithm and various DNN settings.KeywordsKaldi speech recognition toolkitContinuous speech recognitionDeep neural networksSerbian

Semi-supervised and unsupervised discriminative language model training for automatic speech recognition

An acoustic model based on hidden Markov models with deep neural networks (DNN-HMM) has recently been proposed and achieved high recognition accuracy. In this paper, we investigated an elastic spectral distortion method to artificially augment training samples to help DNN-HMMs acquire enough robustness even when there are a limited number of training samples. We investigated three distortion methods - vocal tract length distortion, speech rate distortion, and frequency-axis random distortion - and evaluated those methods with Japanese lecture recordings. In a large vocabulary continuous speech recognition task with only 10 hours of training samples, a DNN-HMM trained with the elastic spectral distortion method achieved a 10.1% relative word error reduction compared with a normally trained DNN-HMM.

Hypertension is one of the most important risk factors for cardiovascular disease. The Global Burden of Disease Study in 2010 described hypertension as the leading risk factor for global disease burden, accounting for 18% of all deaths and 7% of global disability-adjusted life years.1 Furthermore, hypertension is responsible for 45% of deaths caused by ischemic heart disease and 51% of deaths caused by stroke.2 As the leading risk for death and disability, hypertension requires a global response. Reducing uncontrolled blood pressure (BP) by 25% is one of nine United Nations targets to reduce noncommunicable diseases (NCDs) by 2025.3 To that end, hypertension was the feature of World Health Day in 2013.2 The effort to reduce uncontrolled BP is based on two distinct, but integrated, approaches. One is to lower population BP through efforts such as reducing the amount of salt consumed, and the other is to identify people at risk for vascular disease and to clinically manage their hypertension to reduce global cardiovascular risk.4 The task of clinically managing increased BP globally is daunting. In 2008, 40% of the global population older than 25 years had hypertension, representing approximately 1 billion people.5 Further, the burden of hypertension is greatest where resources are the lowest.6 For example, the African region has a hypertension prevalence rate of 46% in adults older than 25 years, compared with the Americas, which have a prevalence rate of 35%.4 In Haiti, the prevalence rate of hypertension in men and women older than 40 years is 69.1% and 67.2%, respectfully.7 Despite the high burden, awareness, and treatment, control rates are suboptimal in most developing countries, which are disproportionately impacted by hypertension. This also underlines the “know-do gap” in terms of transfer of evidence to policy and practice. Simplistically, the sequential steps to the clinical management of hypertension are: (1) the identification of people whose BP is high; (2) behavioral lifestyle counseling; (3) assessing vascular risk to identify those in whom pharmacologic interventions are cost-effective; (4) prescribing the indicated pharmacologic therapies to reduce vascular risk; and (5) titrating pharmacologic and lifestyle behavior therapy to achieve recommended risk target levels.8, 9 In most low-resource settings (LRS), the current major initial barrier is the identification of people whose BP is high.2 BP screening programs are the first step in identifying a large number of people with hypertension,10 especially in LRS where many people are unaware and do not have adequate access to a health care system. Systematic reviews of the literature have recommended BP screening.11, 12 Both the Canadian Task Force on Preventative Health Care (CTFPHC) and the US Preventive Task Force (USPSTF) have recommended screening for hypertension in adults aged 18 years or older.10, 11 These systematic reviews demonstrate substantial indirect evidence that BP screening can identify adults at increased cardiovascular risk, that diagnosis of hypertension leads to treatment, and that treatment leads to improved cardiovascular outcomes.10, 11 Hypertension control can thus be a gateway to manage other vascular and metabolic risks linked to NCDs. In a randomized controlled trial, Kaczorowski and colleagues demonstrated that a community-based BP screening program that included cardiovascular risk assessment and education sessions reduced cardiovascular morbidity and mortality.13 Additionally, there is evidence to suggest that once high BP is controlled in screened individuals, there is improvement in quality of life and work absenteeism.14-16 However, there are no standardized approaches to assist in the development of community-based BP screening programs in LRS. The systematic reviews from the CTFPHC and USPSTF found scant literature on screening programs in LRS.10, 11 While describing the Canadian landscape, Lindsay and colleagues5 demonstrated a higher prevalence of hypertension and cardiovascular disease among aboriginal populations, who also have reduced access to preventative health care, and called on practitioners to screen people who have infrequent access to health care. John and colleagues17 describes the success of a BP screening program in rural India, which uses trained primary care workers to screen older adults who do not have access to clinic-based screening. Similarly, Strogatz and colleagues demonstrate the utility of a hypertension screening program in rural North Carolina.18 Furthermore, Deepa and colleagues19 demonstrated that the “rule of halves” is still applicable in a large South Indian population, where approximately half of the individuals with hypertension are undetected, that half of those detected are not treated, and that half of those treated are not controlled. Indeed, LRS may require innovative and region-specific approaches to tackle the immense challenge of hypertension diagnosis and control.20-23 Successful approaches in LRS have been described by Kenerson with the implementation of the Haiti Hypertension Program24 and by Ordunez-Garcia with the Cuban Hypertension Program.25 These examples underline the immense need for creation and sustainability of valid BP screening programs in LRS. In order to begin addressing this gap, a World Hypertension League (WHL) committee was established on “Blood Pressure Screening Programs in Low Resource Settings” to create and disseminate comprehensive and easy-to-use resources to aid in the development and sustainability of hypertension screening programs. To that end, the WHL has developed a standard set of resources to guide the development of BP screening programs in LRS. The recommendations and resources are outlined in this manuscript and can be found at http://www.whleague.org/index.php/j-stuff/blood-pressure-assessment-train-the-trainer. In addition, supporting videos on BP screening can be found at http://www.whleague.org/index.php/j-stuff/resource-center. It is noted that the WHL BP screening committee is pilot testing these resources with the intent to refine the recommendations and resources. The “Train the Trainer” module consists of a standard set of six resources followed by a quality-improvement feedback resource evaluation form intended to provide a practical framework on how to create, sustain, and evaluate BP screening programs in LRS. The resources are intended to be used by experts. Hence, perhaps the first need is for low-resource regions of the world and national hypertension organizations to develop “Train the Trainer” workshops to increase capacity to utilize the resources. The first meeting of the Pan-African Society of Hypertension in Cameroon, December 2014, hosted such a session. Resource 1 introduces the critical components of a successful screening program. All adults older than 18 years old should be screened for hypertension.5, 6 However, if resources are scarce, most efforts should concentrate on more vulnerable individuals such as older adults. Key components of a successful BP screening program include training to accurately measure BP, use of accurate equipment, providing education on the meaning and health impact of hypertension, and adequate follow-up and access to health care for people with high BP. Other key issues described in this resource include the importance of assessing global cardiovascular risk in people with hypertension and locating the screening program in areas frequented by those at highest risk for hypertension. Resource 2 describes the importance of appropriate selection of BP devices. In LRS, a semi-automated device that has passed international standards for accuracy is recommended. A link to approved devices is provided. We describe the challenges of LRS settings, including intermittent electricity, and offer potential solutions. We recognize that many centers use manual devices; however, they are not recommended because they are rarely used appropriately and require ongoing training and assessment. The WHL is currently involved in facilitating the field testing of automated BP devices. Resource 3 and 3a describe a comprehensive algorithm on how to organize a BP screening training program. Key messages include selection of appropriate staff to oversee and conduct the screening. The program should include training on the risks of hypertension and nonpharmacologic treatment recommendations, such as the Dietary Approaches to Stop Hypertension (DASH) diet.26 BP attendants should know how to accurately select cuff size, measure BP, interpret BP readings, and refer to health care professionals when needed. BP training should be formally evaluated through a standardized form (Resource 3a). Resource 4 describes the recommended technique for measuring BP using a semi- or fully automated device in a screening program. Key components include selection of an appropriate device (semi- or fully automated when available), patient preparation, measurement technique, and recording on a data collection form. A picture illustrates the appropriate technique and patient preparation. The WHL Web site features a video illustrating the recommended BP measurement technique to aid in training and will be available in several languages (https://www.youtube.com/watch?v=egBmUw0Y0IE). Resource 5 describes the appropriate interpretation of BP readings and dissemination of this knowledge to patients. A BP reading of <140/90 mm Hg is interpreted as normal and annual BP check is recommended. A BP reading of 140 mm Hg to 179/90–109 mm Hg is considered elevated and follow-up with a health care professional within a few weeks is recommended. Shorter follow-up is recommended with BPs that are in the higher range. A BP reading >180 mm Hg to 199/110 mm Hg is considered significantly elevated and follow-up with a health care professional as soon as possible is recommended. If a patient demonstrates evidence of end-organ effects, such as chest pain, shortness of breath, or visual changes, immediate hospital referral is recommended. Resource 5 also includes a sample letter given to the patient that explains his/her BP, follow-up instruction, and lifestyle modifications to lower cardiovascular risk. Resource 6 describes a BP data collection form to be used at screening programs. It is important not only to measure BP, but also to record BP readings in a standardized manner. These data are useful in assessment of the utility of a screening program. This serves to further enhance program development and research within communities and regions. The clinical impact of undiagnosed hypertension in LRS has substantially increased on a global scale over the past few decades and is projected to increase further. NCDs are projected to be a significant barrier to economic development in LRS.4 BP screening programs provide a cost-effective and WHL-endorsed intervention to improve the rates of hypertension detection, control, and treatment, with the goal of reducing cardiovascular morbidity and mortality. WHL resources provide comprehensive and practical tools to establish and sustain a successful BP screening program. In the upcoming year there will be ongoing pilot testing of these resources in LRS. We call on national hypertension and cardiovascular organizations to make use of these resources in the development of BP screening programs. These resources will be updated regularly with ongoing feedback and field-testing results. Drs Mangat, Campbell, Mohan, Khalsa, Berbari, Jean-Charles, Kenerson, Lemogoum, Orias, Veiga, and Zhang report no specific funding in relation to this research and have no conflicts of interest to declare. Dr Niebylski is a paid contractor for the World Hypertension League but has no other conflicts to declare. Lyne Cloutier has received funding for the pilot projects from the Agence Universitaire de la Francophonie and has no conflicts of interest to declare. The World Hypertension League is grateful for the assistance of people listed here and the nations of Cameroon, Brazil, and Haiti, where pilot testing of the resources will be initiated.

This paper deals with the use of a stochastic context‐free grammar (SCFG) for large vocabulary continuous speech recognition; in particular, an SCFG with phrase‐level dependency rules is built. Unlike n‐gram models, the SCFG can describe not only local constraints but also global constraints pertaining to the sentence as a whole, thus making possible language models with great expressive power. However, the inside‐outside algorithm must be used for estimation of the SCFG parameters, which involves a great amount of calculation, proportional to the third power of the number of nonterminal symbols and of the input string length. Hence, due to problems in dealing with extensive text corpora, the SCFG has hardly been applied as a language model for very large vocabulary continuous speech recognition. The proposed phrase‐level dependency SCFG allows a significant reduction of the computational load. In experiments with the EDR corpus, the proposed method proved effective. In experiments with the Mainichi corpus, a large‐scale phrase‐level dependency SCFG was built for a very large vocabulary continuous speech recognition system. Speech recognition tests with a vocabulary of about 5000 words showed that the proposed method could not compare with the trigram model in performance; however, when it was used in combination with a trigram model, the error rate was reduced by 14% compared to the trigram model alone. © 2002 Wiley Periodicals, Inc. Syst Comp Jpn, 33(13): 48–59, 2002; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.1172

Deep Convolutional Neural Networks for Large-scale Speech Tasks