Exploring Solutions for Text-to-Speech Synthesis of Low-Resource Languages

Recent advances in research in the field of natural language processing have facilitated many new applications. Natural Language Processing basically involves automating the synthesis and generation of natural languages. However, most of the developments have happened only for a few dominant languages spoken widely like English, Chinese, etc. Some of the developments are also observed for dominant Indian languages like Hindi, Bengali Tamil, etc. However, research into most of the other languages spoken in the world is at a very primitive stage. This paper aims to highlight issues related to one such language for which many resources are not available in a computable format. We report our initial work on Assamese with limited resources. Assamese is the official language of Assam, India and it is also the most spoken Indian language in North East India. We report results of a few experiments carried with a very low amount of training, as resources available for Assamese are not adequate. Part of speech tagging is fundamental to any NLP application. Though some amount of research has been conducted for automatic parts of speech tagging, they have used large amount of data that are not available in public domain. We have modified the viterbi algorithm used for Hiden Markov Model and applied it for automatic parts of speech tagging for Assamese. We have also applied certain language characteristics that can be used for improving accuracy. The key is to use a training dataset as small as possible so that the dependency on training data is limited. We conclude the paper with a brief discussion on the scope of Assamese POS tagging in the future.

Multi-Language Speech-to-Text Translation (ST) plays a pivotal role in bridging linguistic barriers by converting spoken language into written text across different languages.This project aims to develop a robust ST model tailored for low-resource Indian languages, specifically targeting the Indo-Aryan and Dravidian language families.The dataset used consists of speeches from conferences and TED Talks, along with their corresponding transcriptions in English (source language) and translations in Hindi, Bengali, and Tamil (target languages).By tackling the lack of data and disparities in attention within low-resource languages, the paper strives to create an efficient ST system capable of real-world deployment.Additionally, existing resources in related languages are leveraged and word-level translation resources are explored to enhance translation accuracy.

This paper describes a technique for synthesizing speech with any desired voice. The technique is based on an HMM-based text-to-speech (TTS) system and MLLR adaptation algorithm. To generate speech of an arbitrarily given target speaker, speaker-independent speech units, i.e., average voice models, is adapted to the target speaker using MLLR framework. In addition to spectrum and pitch adaptation, we derive an algorithm for adaptation of state duration. We demonstrate that a few sentences uttered by a target speaker are sufficient to adapt not only voice characteristics but also prosodic features. Synthetic speech generated from adapted models using only four sentences is very close to that from speaker dependent models trained using a large amount of speech data.

Many investigations on speech nonlinearities have been carried out and these studies provide strong evidences to support nonlinear system modelling of speech production. The nonlinear characteristics that these studies point to are analogous to chaotic systems. This paper aims to provide evidence of chaotic nature of speech signal and use it for feature extraction to distinguish synthetic and natural speech. The feature used to extract chaos is Lyapunov Exponent (LE). The synthetic speech is found to have higher values of LE in comparison with natural speech. We propose a new feature based on LE for detection of synthetic speech. The synthetic speech used is from Hidden Markov Model (HMM)-based speech synthesis system (HTS) trained using low resource Indian language-Gujarati. This work may find its application for improving robustness of speaker verification (SV) systems against imposture attack using synthetic speech.

In this study, we propose an L2 Korean-speaking assessment task that uses elicited imitation (EI) as a tool to build an L2 Korean learner corpus. The corpus in this study differs from other Korean learner corpora as it is meant for use as a language teaching tool. Consequently, several speaking assessment tasks linked to accredited L2 Korean proficiency tests were performed to collect a large amount of speech data from learners of various nationalities. EI is used to measure language proficiency and has the advantage of being able to explore the spoken interlanguage of learners. For this study, we designed and proposed a Korean EI task and showed how it can be used, including scoring it. We also explained how it can be validated in the future and used to build an L2 Korean learner corpus.

The factors influencing people's food decisions, such as one's mood and eating environment, are important information to foster self-reflection and to develop personalized healthy diet. But, it is difficult to consistently collect them due to the heavy data capture burden. In this work, we examine how speech input supports capturing everyday food practice through a week-long data collection study (N = 11). We deployed FoodScrap, a speech-based food journaling app that allows people to capture food components, preparation methods, and food decisions. Using speech input, participants detailed their meal ingredients and elaborated their food decisions by describing the eating moments, explaining their eating strategy, and assessing their food practice. Participants recognized that speech input facilitated self-reflection, but expressed concerns around re-recording, mental load, social constraints, and privacy. We discuss how speech input can support low-burden and reflective food journaling and opportunities for effectively processing and presenting large amounts of speech data.

Practical applications of wordspotting, such as spoken message retrieval and browsing, require the ability to process large amounts of speech data at speeds many times faster than real-time. The paper presents a novel approach to this problem in which all of the stored audio material is preprocessed off-line to generate a phoneme lattice. At search time, putative word matches are found in this lattice using symmetric dynamic programming. The paper presents the details of the algorithms used and compares performance with a number of conventional approaches using a 20 keyword vocabulary on the DARPA Resource Management Task. The results show that the proposed method is very much faster yet performs acceptably compared to conventional systems which depend on keyword-specific training or prior knowledge of the test set vocabulary. >

By constraining the lexical content of input speech, text-dependent speaker verification (TD-SV) offers more reliable performance than text-independent speaker verification (TI-SV) when dealing with short utterances. Because speech with constrained lexical content is harder to collect, often TD models are fine-tuned from a TI model using a small target phrase dataset. However, sometimes the target phrase dataset is too tiny for fine-tuning, which is the main obstacle for deploying TD-SV. One solution is to fine-tune the model using medium-size multi-phrase TD data and then deploy the model on the target phrase. Although this strategy does help in some cases, the performance is still sub-optimal because the model is not optimized for the target phrase. Inspired by the recent progress in meta-learning, we propose a three-stage pipeline for adapting a TI model to a TD model for the target phrase. Firstly, a TI model is trained using a large amount of speech data. Then, we use a multi-phrase TD dataset to tune the TI model via model-agnostic meta-learning. Finally, we perform fast adaptation using a small target phrase dataset. Results show that the three-stage pipeline consistently outperforms multi-phrase and target phrase fine-tuning.

Speech emotion recognition based on formant characteristics feature extraction and phoneme type convergence

Over the past few years, Large Language Models of Code (Code LLMs) have started to have a significant impact on programming practice. Code LLMs are also emerging as building blocks for research in programming languages and software engineering. However, the quality of code produced by a Code LLM varies significantly by programming language. Code LLMs produce impressive results on high-resource programming languages that are well represented in their training data (e.g., Java, Python, or JavaScript), but struggle with low-resource languages that have limited training data available (e.g., OCaml, Racket, and several others). This paper presents an effective approach for boosting the performance of Code LLMs on low-resource languages using semi-synthetic data. Our approach, called M ulti PL-T, generates high-quality datasets for low-resource languages, which can then be used to fine-tune any pretrained Code LLM. M ulti PL-T translates training data from high-resource languages into training data for low-resource languages in the following way. 1) We use a Code LLM to synthesize unit tests for commented code from a high-resource source language, filtering out faulty tests and code with low test coverage. 2) We use a Code LLM to translate the code from the high-resource source language to a target low-resource language. This gives us a corpus of candidate training data in the target language, but many of these translations are wrong. 3) We use a lightweight compiler to compile the test cases generated in (1) from the source language to the target language, which allows us to filter our obviously wrong translations. The result is a training corpus in the target low-resource language where all items have been validated with test cases. We apply this approach to generate tens of thousands of new, validated training items for five low-resource languages: Julia, Lua, OCaml, R, and Racket, using Python as the source high-resource language. Furthermore, we use an open Code LLM (StarCoderBase) with open training data (The Stack), which allows us to decontaminate benchmarks, train models without violating licenses, and run experiments that could not otherwise be done. Using datasets generated with M ulti PL-T, we present fine-tuned versions of StarCoderBase and Code Llama for Julia, Lua, OCaml, R, and Racket that outperform other fine-tunes of these base models on the natural language to code task. We also present Racket fine-tunes for two very recent models, DeepSeek Coder and StarCoder2, to show that M ulti PL-T continues to outperform other fine-tuning approaches for low-resource languages. The M ulti PL-T approach is easy to apply to new languages, and is significantly more efficient and effective than alternatives such as training longer.

Vocal Tract Length Normalization (VTLN) is used to design vocal tract length normalized Automatic Speech Recognition (ASR) systems. It has led to improvement in the performance of ASR systems by taking into account the physiological differences among speakers. Recently, a number of speech recognition applications are being developed for Indian languages. In this paper, we use state-of-the-art method for VTLN based on maximum likelihood approach. A vowel recognition system has been developed for two low resourced Indian languages, viz., Gujarati and Marathi. Appropriate warping factors have been obtained for all speakers considered for training and testing procedures. An improvement in the performance of vowel recognition is observed as compared to state-of-the-art Mel Frequency Cepstral Coefficients (MFCC).

Synthetic speech technology, now approaching human-like naturalness due to advancements in deep learning, has turned to focusing on personality or persona design as a common practice in the industry. The present study aimed to identify speech characteristics affecting personality impressions in synthetic and natural speech. Thirty native Korean speakers participated in a personality rating experiment in which they evaluated natural Korean sentences and their synthetic counterparts in terms of the Big-Five personality model. Acoustic analyses were performed to examine voice quality and prosody, including Intonational Phrase (IP) boundary tones. The results revealed that scores of agreeableness, conscientiousness, and emotional stability increased overall when the voices contained greater aperiodicity in the harmonics (i.e., were likely breathier) and were weaker in energy. The results also demonstrated that different prosodic features affected personality perception in synthetic and natural speech; synthetic speech with a wider F0 range received higher scores on extroversion, openness, and emotional stability. In contrast, the effect of IP boundary tones was most frequently found for female natural speech, which contained a wider range of tones, including multitonals (e.g., LHL%). Our findings suggest that intonation is one of the key factors which can be adjusted to generate synthetic speech with various personalities.

When translating specific words from one language to another, it is necessary to take into consideration several different variables to guarantee correctness, precision, quality, and contextual appropriateness. Context-aware translation systems face several issues due to the complexities of natural language. These challenges include contextual disambiguation, cultural and linguistic nuances, subject matter expertise, and maintaining translation consistency. In low-resource languages such as Tamil, which suffer from limited linguistic resources, high word-sense ambiguity, and flexible word order, the translation process is further complicated. These issues are magnified in low-resource languages. To address these concerns, the proposed study will provide linguistic resources for pre-training Neural Machine Translation (NMT) systems. This will be accomplished by developing a translation system that is language-independent, document-specific, and context-based, making it suitable for languages with limited resources. The research presents a novel word vector format known as Word2Line (W2L), which minimises the time required for the process. With the use of TF-IDF, the system can recognise document-specific words in the English language, which serves as the source language, and then predict the context-based translations of those words in Tamil, the target language.

There is a growing need for a model for automatic word-level language detection due to the growing usage of multilingual text. To identify code-switched and code-mixed sentences in the Kokborok language and English at the word level, we propose an unsupervised model. Numerous studies on code-mixed text have been published, including for several Indian languages. However, to the best of our knowledge, our work represents a pioneering effort, being the first to identify languages for low-resource English-Kokborok language pairs. We have employed a method that merges data from a frequency dictionary with data from a character n-gram model. The Viterbi technique combines a character n-gram Markov model with a frequency lexicon to aid in precise language identification at the word level. The proposed method performed well, with a word-level accuracy of 93.15%, which is better than the BiLSTM-CRF model of 88.5% and the rule-based baseline of 85.3%. This demonstrates the effectiveness of combining lexicon and statistical methodology for handling low-resource languages without depending on massive annotated datasets.

We propose a multitask learning (MTL) approach to improve low-resource automatic speech recognition using deep neural networks (DNNs) without requiring additional language resources. We first demonstrate that the performance of the phone models of a single low-resource language can be improved by training its grapheme models in parallel under the MTL framework. If multiple low-resource languages are trained together, we investigate learning a set of universal phones (UPS) as an additional task again in the MTL framework to improve the performance of the phone models of all the involved languages. In both cases, the heuristic guideline is to select a task that may exploit extra information from the training data of the primary task(s). In the first method, the extra information is the phone-to-grapheme mappings, whereas in the second method, the UPS helps to implicitly map the phones of the multiple languages among each other. In a series of experiments using three low-resource South African languages in the Lwazi corpus, the proposed MTL methods obtain significant word recognition gains when compared with single-task learning (STL) of the corresponding DNNs or ROVER that combines results from several STL-trained DNNs.