A framework for Sudanese Arabic – English Mixed Speech Processing

In recent years, with the rise of human-computer interaction and the successful application of end-to-end models in the field of speech recognition, the construction of end-to-end speech recognition models has received extensive attention. Relying on the multi-task learning method and the connection between language identification and speech recognition, we proposed an end-to-end Transformer model, which is a multilingual speech recognition model integrating language identification. The model takes the speech recognition task as the main task and the language identification task as the auxiliary task. In this paper, the validity of the model is verified by using the datasets of 13 languages in the 2021 Oriental Language Recognition challenge (OLR). The experimental results show that the model constructed in this paper has a relative improvement of 37.46% in the speech recognition task compared with the baseline system proposed by the OLR organizer. The accuracy of language identification reaches 89.70 %. The results can get the fifth place in the 2021 OLR constraint track of speech recognition equally.

It is significant to highlight that the demand of multilingual systems in human-computer interfaces is growing and automatic language identification (LID) is becoming increasingly important for the speech community as a fundamental tool to enrich theses systems. Research in this field has been active for several years. LID becomes also a useful resource in Multilingual Security System (MSS) providing methodologies to improve the robustness and the effectiveness of these systems. The main goal of our project is the development of Multilingual LVCSR systems in the Basque context. Nowadays, our work is oriented to the use of Basque in internet. The present work describes the development of a Language Identification (LID) system oriented to robust Multilingual Speech Recognition for under-resourced languages in the Basque context.

This study focuses on code-switching (CS) in French/Algerian Arabic bilingual communities and investigates how speech technologies, such as automatic data partitioning, language identification and automatic speech recognition (ASR) can serve to analyze and classify this type of bilingual speech. A preliminary study carried out using a corpus of Maghrebian broadcast data revealed a relatively high presence of CS Alge-rian Arabic as compared to the neighboring countries Morocco and Tunisia. Therefore this study focuses on code switching produced by bilingual Algerian speakers who can be considered native speakers of both Algerian Arabic and French. A specific corpus of four hours of speech from 8 bilingual French Algerian speakers was collected. This corpus contains read speech and conversational speech in both languages and includes stretches of code-switching. We provide a linguistic description of the code-switching stretches in terms of intra-sentential and inter-sentential switches, the speech duration in each language. We report on some initial studies to locate French, Arabic and the code-switched stretches, using ASR system word posteriors for this pair of languages.

In multilingual communication through a common language among both native speakers (NS) and non-native speakers (NNS), NNS may encounter problems in comprehending the messages of NS or following conversations. Even though automated speech recognition (ASR) transcripts provide support to NNS, such transcripts may contain errors and impose the need to simultaneously listen and read. To reduce this burden, we propose adding another channel (i.e., highlighting) through which NS can help NNS by highlighting the critical parts of transcripts, thus making them more useful to NNS. In a laboratory study involving 14 triads (two NS and one NNS in each triad), participants engaged in collaborative discussions under two conditions: audio conferencing plus ASR transcripts with and without the highlighting function. NS showed various motivations to perform the extra task of highlighting. The highlighting efforts helped NS themselves focus on the discussion and enhanced their task performance while increasing the clarity and comfort perceived by NNS during communication. Having NS generating highlights can benefit both NS and NNS, but in different ways. We discuss the implications for research and design of multilingual collaborative work.

Speech recognition of individuals who are listening to a nonnative language is significantly degraded in the presence of background noise and may be influenced by proficiency, age of acquisition, language experience, and daily use of the nonnative language. The purpose of this study is to examine and compare speech recognition in noise performance across test conditions with varying signal-to-noise ratios (SNRs) as well as the presence of vocal and spatial cues in listeners who speak American English as a native language or Mandarin Chinese as a native language. Self-rated English proficiency and experience were collected for native Mandarin Chinese speakers to determine its relationship to performance on the test measures. A cross-sectional repeated measures design was used for the study. Four groups of participants were included in the study. The adult groups consisted of 25 adults who speak native English and 25 adults who speak native Mandarin Chinese with English as an additional language. The pediatric groups consisted of 16 children who speak native English and 16 children who speak native Mandarin Chinese with English as an additional language. Percent correct speech recognition in noise was assessed at three SNRs (-3, 0, +3 dB) using the adult or pediatric versions of the AzBio sentence test. The Listening in Spatialized Noise-Sentence (LiSN-S) test was used to determine the effect of providing spatial and vocal cues on the speech recognition in noise performance of the groups of participants. The data for each age group and test measure were analyzed with a repeated measures analysis of variance. Correlation analyses were performed to examine relationships between English proficiency and experience on performance across the speech recognition test conditions. Analysis of the data from the adult or pediatric AzBio sentence test identified a significant effect of native language for adults but no significant effect for children. The higher SNRs yielded better performance for all listeners. On the LiSN-S test, results for the adult and pediatric groups were similar and showed significantly better performance for the native English speakers in every test condition. The demographic and language characteristics that most affected speech recognition performance across the test measures included the length of time the person lived in the United States, the age of English acquisition, the number of minutes per day English was spoken by the participant, and the self-rated English proficiency. The findings in this study highlight the importance and benefit of higher SNRs as well as the provision of vocal and spatial cues for improving speech recognition performance in noise of adult and pediatric listeners who speak Mandarin Chinese as a native language.

Speech, languages, and background voices are overlapped in the multi-speaker environment. The speech separation process is necessary for downstream applications like language and speaker identification and ASR tasks. A combined speech separation and language recognition approach is proposed in this paper. First, the monolingual information is extracted from a multi-speaker multilingual speech scenario, then fed the separated speech to a language identification model. The speech separation model is trained in a single-channel overlapped speech data, and the language identification model is trained in Oriental language corpus. The combined effect of these two models is evaluated in the task of language identification. When the language recognition network is fed with the audio spectrograms separated by the source separation network, the Chinese and Korean language recognition results are significantly improved in terms of accuracy and recall rate than the mixed audio spectrograms.

This article discusses the current requirement for scientific research to be published in the English language and the problems arising from this requirement not only for authors whose first language is not English but also for society and for the world's scientific community. The assistance which is and should be available to authors is considered as well as a future looking towards multilingual publication.

This paper introduces a speech separation system as a front-end processing step for automatic speech recognition (ASR). It employs computational auditory scene analysis (CASA) to separate the target speech from the interference speech. Specifically, the mixed speech is preprocessed based on auditory peripheral model. Then a pitch tracking is conducted and the dominant pitch is used as a main cue to find the target speech. Next, the time frequency (TF) units are merged into many segments. These segments are then combined into streams via CASA initial grouping. A regrouping strategy is employed to refine these streams via amplitude modulate (AM) cues, which are finally organized by the speaker recognition techniques into corresponding speakers. Finally, the output streams are reconstructed to compensate the missing data in the abovementioned processing steps by a cluster based feature reconstruction. Experimental results of ASR show that at low TMR (<-6dB) the proposed method offers significantly higher recognition accuracy. Index term: speech separation, CASA, speaker recognition, pitch tracking, units grouping, reconstruction

In this paper, we extend our previous work on direct recognition of single-channel multi-talker mixed speech using permutation invariant training (PIT). We propose to adapt the PIT models with auxiliary features such as pitch and i-vector, and to exploit the gender information with multi-task learning which jointly optimizes for the speech recognition and speaker-pair prediction. We also compare CNN-BLSTMs against BLSTM-RNNs used in our previous PIT-ASR model. The experimental results on the artificially mixed two-talker AMI data indicate that our proposed model improvements can reduce word error rate (WER) by ~ 10.0% relative to our previous work for both speakers in the mixed speech. Our results also confirm that PIT can be easily combined with advanced techniques to improve the performance on multi-talker speech recognition.

The classical language identification architecture would require a collection of languages independent text and speech information for training by the system before it can identify the languages correctly. This paper also address language identification framework however with data has been downsized considerably from general language identification architecture. The system goal is to identify the type of language being spoken to the system based on a series of trained speech with sound file features and without any language text data or lexical knowledge of the spoken language. The system is also expected to be able to be deployed in mobile platform in future. This paper is specifically about measuring the performance optimisation of audio filters on a CNN model integration for the language identification system. There are several metric to gauge the performance identification system for a classification problem. Precision, recall and F1 Scores is presented for the performance evaluation with different combination of filters together with CNN model as the framework of the language identification system. The goal is not to get the best filter for noise, instead to identify the filter that is a good fit to develop language model with environmental noise for a robust language identification system. Our experiments manage to identify the best combination of filters to increase the accuracy of language identification using short speech. This resulting us to modify our pre-processing phase in the overall language identification system.

Utilising the known language of origin of a name can be useful when predicting the pronunciation of the name. When this language is not known, automatic language identification (LID) can be used to influence which language-specific grapheme-to-phoneme (G2P) predictor is triggered to produce a pronunciation for the name. We investigate the implications when both the LID system and the G2P system generate errors: what influence does this have on a resulting speech recognition system? We experiment with different approaches to LID-based dictionary creation and report on results in four South African languages: Afrikaans, English, Sesotho and isiZulu.

When pumpkin is closer to onion than to squash: The structure of the second language lexicon

With an ever‐declining number of papers written in languages other than English, the idiom of Shakespeare is, today more than ever before, the international language of science [1], [2]. In 2014, according to PubMed's language filter, 95.4% of life science publications retrieved by the search engine were written in English, in line with the trend of the last several decades [1], [2]. Today, among the 20 countries with the highest number of publications per year on PubMed, only France and China apparently publish less than 95% of their papers in English [1]. However, English is only natively spoken by around 5% of the world population and only about 20 countries use it as their primary language (http://www.ethnologue.com/statistics/size). Not surprisingly, in 2014, only 4 out of the 20 countries with highest number of life science publications have English as their primary language [3], and in the 5‐year period 2008–2012, nearly 54% of papers on PubMed were written in countries where English is not the primary idiom [1]. So, is it wise to keep English as the language of science? According to Howy Jacobs's 2012 editorial [4], the answer is yes. Jacobs argues that, because English is the language used to publish the vast majority of scientific articles today, it must be considered and protected as the language of science. His stance, he says, is not imperialistic—his native language is English—because he would willingly replace English should “a majority of the world's scientists” vote for it. …

Statistical approaches in speech technology, whether used for statistical language models, trees, hidden Markov models or neural networks, represent the driving forces for the creation of language resources (LR), e.g., text corpora, pronunciation and morphology lexicons, and speech databases. This paper presents a system architecture for the rapid construction of morphologic and phonetic lexicons, two of the most important written language resources for the development of ASR (automatic speech recognition) and TTS (text-to-speech) systems. The presented architecture is modular and is particularly suitable for the development of written language resources for inflectional languages. In this paper an implementation is presented for the Slovenian language. The integrated graphic user interface focuses on the morphological and phonetic aspects of language and allows experts to produce good performances during analysis. In multilingual TTS systems, many extensive external written language resources are used, especially in the text processing part. It is very important, therefore, that representation of these resources is time and space efficient. It is also very important that language resources for new languages can be easily incorporated into the system, without modifying the common algorithms developed for multiple languages. In this regard the use of large external language resources (e.g., morphology and phonetic lexicons) represent an important problem because of the required space and slow look-up time. This paper presents a method and its results for compiling large lexicons, using examples for compiling German phonetic and morphology lexicons (CISLEX), and Slovenian phonetic (SIflex) and morphology (SImlex) lexicons, into corresponding finite-state transducers (FSTs). The German lexicons consisted of about 300,000 words, SIflex consisted of about 60,000 and SImlex of about 600,000 words (where 40,000 words were used for representation using finite-state transducers). Representation of large lexicons using finite-state transducers is mainly motivated by considerations of space and time efficiency. A great reduction in size and optimal access time was achieved for all lexicons. The starting size for the German phonetic lexicon was 12.53 MB and 18.49 MB for the morphology lexicon. The starting size for the Slovenian phonetic lexicon was 1.8 MB and 1.4 MB for the morphology lexicon. The final size of the corresponding FSTs was 2.78 MB for the German phonetic lexicon, 6.33 MB for the German morphology lexicon, 253 KB for SIflex and 662 KB for the SImlex lexicon. The achieved look-up time is optimal, since it only depends on the length of the input word and not on the size of the lexicon. Integration of lexicons for new languages into the multilingual TTS system is easy when using such representations and does not require any changes in the algorithms used for such lexicons.

We present a test of a revised version of the Second Language Linguistic Perception (L2LP) model, a computational model of the acquisition of second language (L2) speech perception and recognition. The model draws on phonetic, phonological, and psycholinguistic constructs to explain a number of L2 learning scenarios. However, a recent computational implementation failed to validate a theoretical proposal for a learning scenario where the L2 has less phonemic categories than the native language (L1) along a given acoustic continuum. According to the L2LP, learners faced with this learning scenario must not only shift their old L1 phoneme boundaries but also reduce the number of categories employed in perception. Our proposed revision to L2LP successfully accounts for this updating in the number of perceptual categories as a process driven by the meaning of lexical items, rather than by the learners' awareness of the number and type of phonemes that are relevant in their new language, as the previous version of L2LP assumed. Results of our simulations show that meaning-driven learning correctly predicts the developmental path of L2 phoneme perception seen in empirical studies. Additionally, and to contribute to a long-standing debate in psycholinguistics, we test two versions of the model, with the stages of phonemic perception and lexical recognition being either sequential or interactive. Both versions succeed in learning to recognize minimal pairs in the new L2, but make diverging predictions on learners' resulting phonological representations. In sum, the proposed revision to the L2LP model contributes to our understanding of L2 acquisition, with implications for speech processing in general.