Subspace mixture model for low-resource speech recognition in cross-lingual settings

This paper concerns cross-lingual acoustic modeling in the case when there are limited target language resources. We build on an approach in which a subspace Gaussian mixture model (SGMM) is adapted to the target language by reusing the globally shared parameters estimated from out-of-language training data. In current cross-lingual systems, these parameters are fixed when training the target system, which can give rise to a mismatch between the source and target systems. We investigate a maximum a posteriori (MAP) adaptation approach to alleviate the potential mismatch. In particular, we focus on the adaptation of phonetic subspace parameters using a matrix variate Gaussian prior distribution. Experiments on the GlobalPhone corpus using the MAP adaptation approach results in word error rate reductions, compared with the cross-lingual base-line systems and systems updated using maximum likelihood, for training conditions with 1 hour and 5 hours of target language data.

Subspace Gaussian mixture based language modeling for large vocabulary continuous speech recognition

Current speech recognition systems are often based on HMMs with state-clustered Gaussian Mixture Models (GMMs) to represent the context dependent output distributions. Though highly successful, the standard form of model does not exploit any relationships between the states, they each have separate model parameters. This paper describes a general class of model where the context-dependent state parameters are a transformed version of one, or more, canonical states. A number of published models sit within this framework, including, semi-continuous HMMs, subspace GMMs and the HMM error model. A set of preliminary experiments illustrating some of this model’s properties using CMLLR transformations from the canonical state to the context dependent state are described.

In this paper, the improvements in the recently implemented Kannada speech recognition system is demonstrated in detail. The Kannada automatic speech recognition (ASR) system consists of ASR models which are created by using Kaldi, IVRS call flow and weather and agricultural commodity prices information databases. The task specific speech data used in the recently developed spoken dialogue system had high level of different background noises. The different types of noises present in collected speech data had an adverse effect on the on line and off line speech recognition performances. Therefore, to improve the speech recognition accuracy in Kannada ASR system, a noise reduction algorithm is developed which is a fusion of spectral subtraction with voice activity detection (SS-VAD) and minimum mean square error spectrum power estimator based on zero crossing (MMSE-SPZC) estimator. The noise elimination algorithm is added in the system before the feature extraction part. An alternative ASR models are created using subspace Gaussian mixture models (SGMM) and deep neural network (DNN) modeling techniques. The experimental results show that, the fusion of noise elimination technique and SGMM/DNN based modeling gives a better relative improvement of 7.68% accuracy compared to the recently developed GMM-HMM based ASR system. The least word error rate (WER) acoustic models could be used in spoken dialogue system. The developed spoken query system is tested from Karnataka farmers under uncontrolled environment.

DBMiP: A pre-training method for information propagation over deep networks

Noise adaptive training (NAT) is an effective approach to normalise the environmental distortions in the training data.This paper investigates the model-based NAT scheme using joint uncertainty decoding (JUD) for subspace Gaussian mixture models (SGMMs).A typical SGMM acoustic model has much larger number of surface Gaussian components, which makes it computationally infeasible to compensate each Gaussian explicitly.JUD tackles the problem by sharing the compensation parameters among the Gaussians and hence reduces the computational and memory demands.For noise adaptive training, JUD is reformulated into a generative model, which leads to an efficient expectation-maximisation (EM) based algorithm to update the SGMM acoustic model parameters.We evaluated the SGMMs with NAT on the Aurora 4 database, and obtained higher recognition accuracy compared to systems without adaptive training.

Formally verifying functional and security properties of a large-scale production operating system is highly desirable. However, it is challenging as such OSes are often written in multiple source languages that have no formal semantics - a prerequisite for formal reasoning. To avoid expensive formalization of the semantics of multiple high-level source languages, we present a lightweight and rigorous verification toolchain that verifies OS code at the binary level, targeting ARM machines. To reason about ARM instructions, we first translate the ARM Specification Language that describes the semantics of the ARMv8 ISA into the PVS7 theorem prover and verify the translation. We leverage the radare2 reverse engineering tool to decode ARM binaries into PVS7 and verify the translation. Our translation verification methodology is a lightweight formal validation technique that generates large-scale instruction emulation test lemmas whose proof obligations are automatically discharged. To demonstrate our verification methodology, we apply the technique on two OSes: Google's Zircon and a subset of Linux. We extract a set of 370 functions from these OSes, translate them into PVS7, and verify the correctness of the translation by automatically discharging hundreds of thousands of proof obligations and tests. This took 27.5 person-months to develop.

Automatic speech recognition depends on large amounts of transcribed speech recordings in order to estimate the parameters of the acoustic model. Recording such large speech corpora is time-consuming and expensive; as a result, sufficient quantities of data exist only for a handful of languages—there are many more languages for which little or no data exist. Given that there are acoustic similarities between speech in different languages, it may be fruitful to use data from a well-resourced source language to estimate the acoustic models for a recognizer in a poorly-resourced target language. Previous approaches to this task have often involved making assumptions about shared phonetic inventories between the languages. Unfortunately pairs of languages do not generally share a common phonetic inventory. We propose an indirect way of transferring information from a source language acoustic model to a target language acoustic model without having to make any assumptions about the phonetic inventory overlap. To do this, we employ tandem features, in which class-posteriors from a separate classifier are decorrelated and appended to conventional acoustic features. Tandem features have the advantage that the language of the speech data used to train the classifier need not be the same as the target language to be recognized. This is because the class-posteriors are not used directly, so do not have to be over any particular set of classes. We demonstrate the use of tandem features in cross-lingual settings, including training on one or several source languages. We also examine factors which may predict a priori how much relative improvement will be brought about by using such tandem features, for a given source and target pair. In addition to conventional phoneme class-posteriors, we also investigate whether articulatory features (AFs)—a multi-stream, discrete, multi-valued labeling of speech—can be used instead. This is motivated by an assumption that AFs are less language-specific than a phoneme set.

We consider the pronunciation assessment of vowels of Indian English uttered by speakers with Gujarati L1 using confidence measures obtained by automatic speech recognition. The goodness-of-pronunciation measure as captured by the acoustic likelihood scores can be effective only when the acoustic models used are appropriate for the task i.e. detecting errors in the target language (Indian English) typical of speakers of Gujarati (the source language). Thus the speech data used for acoustic model training is expected to have a prominent influence on system performance. In the absence of labeled speech databases of either the source or target language, we investigate specific combinations of acoustic models trained on available databases of American English and Hindi. It is observed that Indian English speech is better represented by Hindi speech models for vowels common to the two languages rather than by American English models. Further, adaptation with a limited amount of Indian English speech improves the system performance.

Only a handful of the world's languages are abundant with the resources that enable practical applications of speech processing technologies. One of the methods to overcome this problem is to use the resources existing in other languages to train a multilingual automatic speech recognition (ASR) model, which, intuitively, should learn some universal phonetic representations. In this work, we focus on gaining a deeper understanding of how general these representations might be, and how individual phones are getting improved in a multilingual setting. To that end, we select a phonetically diverse set of languages, and perform a series of monolingual, multilingual and crosslingual (zero-shot) experiments. The ASR is trained to recognize the International Phonetic Alphabet (IPA) token sequences. We observe significant improvements across all languages in the multilingual setting, and stark degradation in the crosslingual setting, where the model, among other errors, considers Javanese as a tone language. Notably, as little as 10 hours of the target language training data tremendously reduces ASR error rates. Our analysis uncovered that even the phones that are unique to a single language can benefit greatly from adding training data from other languages - an encouraging result for the low-resource speech community.

We have recently proposed a new acoustic model based on probabilistic linear discriminant analysis (PLDA) which enjoys the flexibility of using higher dimensional acoustic features, and is more capable to capture the intra-frame feature correlations. In this paper, we investigate the use of bottleneck features obtained from a deep neural network (DNN) for the PLDA-based acoustic model. Experiments were performed on the Switchboard dataset — a large vocabulary conversational telephone speech corpus. We observe significant word error reduction by using the bottleneck features. In addition, we have also compared the PLDA-based acoustic model to three others using Gaussian mixture models (GMMs), subspace GMMs and hybrid deep neural networks (DNNs), and PLDA can achieve comparable or slightly higher recognition accuracy from our experiments.

The present study proposes a source speech register analysis model that incorporates textual and situational components and parameters of register equivalence in the target language to identify the relations between the source language register and student interpreter’s performance, textual register mismatch between the source and target languages, and the relevance between register equivalence parameters in the target language and interpreting quality. The findings are as follows: first, source language register does have influence on interpreter’s performance, and relations exist between the two. In general, higher source language register brings better interpreting quality. Three assessment criteria and the overall quality differ across different levels of registers in the source language. Second, register mismatch between the source and target languages happens since interpreters are inclined to lower the register of interpreting output from the high-register source speech and increase the register level in the target language in the case of the low-register speech. Third, intonation and voice have the highest correlations with the interpreters’ performance. Fluency ranks the second, and lexical-semantic choices the third.

The ongoing debate on code switching (CS) in multilingual societies and Foreign Language (FL) classes is still relevant and worth considering. This linguistic phenomenon simply refers to altering from one language to another during speech and writing (Richards and Schmidt 2013). While some authorities propose that interactions between learners and instructors in the FL class be carried out only in the target language, other opinions favour a careful blend of both target and source languages. Ghana, a multilingual country in West Africa, adopted English as its official language during the colonial era. French was later introduced to promote regional integration among Ghana and her neighbours, especially since Ghana is greatly surrounded by francophone countries within the subregion (Gandolfi et al. 2021). French has since then been the second most widely taught FL in the country, with a presence in almost all stages of the Ghanaian academic trajectory. This multilingual position of Ghana makes it common to find elements of CS present in FL classrooms, where both instructors and learners occasionally resort to English for instructional conversations (Chachu 2016). This study focuses on university students’ perceptions of CS between English as the source language and French as a foreign language (FFL) during the instruction of the latter. Some frameworks for teaching and learning FFL in Ghana prefer that the language of instruction and interaction be limited to the target language only (Soku and Azanku 2015). The Common European Framework of References for Languages (CEFR) however, suggests the possible use of CS for purposes of instruction when there are favourable conditions (Council of Europe 2001). To identify the roles of CS and investigate FL students’ opinions on this, our data was gathered from classroom observations and questionnaires that had both Likert scale and open-ended questions. This was at the end of their first semester of studying university-level FFL. It was observed that CS plays a significant role in the understanding of linguistic concepts and greatly improves students’ engagement and participation in the FL class. Of the 36 respondents, 34 felt their academic needs were met when the language of instruction and interaction alternated between French and English. Twenty-nine students reported losing focus easily in the FFL class in instances where the medium of interaction was limited to French alone, while 26 reported feeling nervous whenever they were asked to only communicate in French. For metalinguistic aspects of the language, 33 students, constituting 91.7% were of the view that a careful blend of English and French could go a long way to aid their understanding and foster the learning of grammatical concepts. Generally, students attested to better retention of vocabulary in the target language when there was a good blend of both languages. It is recommended therefore that language instructors resort periodically to CS to ensure understanding and engagement from students. CS also helps learners have the opportunity to intervene with their limited linguistic structures when they are faced with linguistic challenges in the FL classroom. Click here to read the corresponding ISSOTL blog post.

The study focuses on teaching the translation of audiovisual memes: comic image/video and text combinations that tend to go viral on the Internet. The key features of a meme are a comic effect resulting from surprise, disrupted expectations, the clash between the text and the video; “viral” expansion across the Internet and other social media. Mastering the skills required to translate such audiovisual content is essential since memes are highly popular on the Internet with all age groups, there are numerous allusions references to the popular memes in the mass media, fiction, movies, and TV series. Teaching audiovisual meme translation shall consist of analyzing the meme properties, building meme comprehension and metacognitive analysis skills, and mastering specific approaches and techniques for meme representation in the source and target languages. An emphasis should be placed on developing the ability to adequately reconstruct meme ecosystems in the source language and find their analogs in the target language as well as the skills required to identify archetypical narratives in the source language and build them in the target language. It is shown that memes have some similarities to set expressions and proverbs. The meme translation curriculum shall build the ability to find and translate set expressions and proverbs in audiovisual works. It is also shown that translation teachers should have specialized training in teaching the translation of memes as a special type of audiovisual content. It can be implemented in the format of short seminars/webinars, or within the teacher further education framework.

This paper discusses English-poetry translation, Heusca by John Conford into Indonesian translated by Chairil Anwar. The purpose of this study is to determine how the translator considers aesthetic and expressive values from the source language and translate them into target language. Aesthetic values are the means of delivery thought through the use of poetic language such as diction or word choice, metaphor, imageri, and others figurative language. Meanwhile, the expressive values are the means of delivery thoughts through the author's mind or emotions such as structure, rhyme, and pronunciation. The method used in this study is a qualitative descriptive analysis of the meaning. The results of analysis show that in terms of meaning, aesthetic values, and structures, Heusca is the result of a semantic translation and faithful to source language. Translator managed to retain the meaning of source language. In other word, the translation can meet the requirements of fidelity. In terms of expressive values in particular aspects of meter and rhyme are not so faithful to source language because the source language and the target language have some differences. Basically, this is acceptable that the Indonesian language as target language has a very big difference in terms of pronunciation with English as the source language. As a result, meter and rhyme of target language may not be forced into the source language without damaging the naturalness of meaning for target language readers. It can be said that the translation can meet the transparency criteria because source language appeared in poetry comes naturally to native speakers of English.