Towards automatic transcription of spontaneous presentations

This paper deals with an approach to Automatic Language Identification using only prosodic modeling. The actual approach for language identification focuses mainly on phonotactics because it gives the best results. We propose here to evaluate the relevance of prosodic information for language identification with read studio recording (previous experiment [1]) and spontaneous telephone speech. For read speech, experiments were performed on the five languages of the MULTEXT database [2]. On the MULTEXT corpus, our prosodic system achieved an identification rate of 79 % on the five languages discrimination task. For spontaneous speech, experiments are made on the ten languages of the OGI Multilingual telephone speech corpus [3]. On the OGI MLTS corpus, the results are given for languages pair discrimination tasks, and are compared with results from [4]. As a conclusion, if our prosodic system achieves good performance on read speech, it might not take into account the complexity of spontaneous speech prosody. 1.

This paper describes two experiments aimed at exploring the relationship between linguistic aspects and perceived proficiency in read and spontaneous speech. 5,000 utterances of read speech by 50 non-native speakers of Korean in Experiment 1, and of 6,000 spontaneous speech utterances in Experiment 2 were scored for proficiency by native human raters and were analyzed by factors known to be related to perceived proficiency. The results show that the factors investigated in this study can be employed to predict proficiency ratings, and the predictive power of fluency and pitch and accent accuracy is strong for both read and spontaneous speech. We also observe that while proficiency ratings of read speech are mainly related to segmental accuracy, those of spontaneous speech appear to be more related to pitch and accent accuracy. Moreover, proficiency in read speech does not always equate to the proficiency in spontaneous speech, and vice versa, with Pearson’s per-speaker correlation score of 0.535.

Emotions are an integral part of social interactions, serving vital functions in the management of our interpersonal relationships. Cues to emotional meaning can be conveyed through a variety of modalities, from facial expressions and body gestures to the linguistic and non-linguistic, or prosodic aspects of speech. As automatic speech systems continue to grow in use, the ability to recognize emotional cues from these modalities is no longer solely expected in human communication, but also in human-computer interactions with the goal of developing more user-friendly and effective applications. This thesis explores how speech emotion recognition (SER) systems can be more robust to factors of language, culture, and context through the lens of emotional speech and emotional prosody from bilingual English-speaking Singaporeans of Chinese, Malay, or Indian ethnicity. Specifically, the thesis aims to examine how Singaporeans from the three ethnic groups may vary in their perception of emotions in speech, and to explore how they express emotions via speech prosody. The recognition and production processes of emotional communication in Singapore English was investigated through two emotion classes – anger and happiness. An emotion perception test was conducted using acted angry and happy speech as stimuli to understand how Singapore English speakers recognized emotions in speech. The results of the perception test revealed that respondents displayed very similar patterns in their rating of emotional speech stimuli. The study also found that respondents tended to rated anger with low dominance. At the same time, a spontaneous speech corpus known as the Singapore English Spontaneous Emotional Speech Corpus (SESESC) was also developed, featuring audio recordings of 44 Singapore English speakers from the three ethnic groups playing a co-operative video game with either a familiar partner or a confederate. More than 36,000 intervals were annotated in-context of the recordings by each annotator based on a modified emotion annotation scheme that included information regarding the emotion category and arousal levels. Intervals that were labelled as conveying joy or anger with full agreement from the annotators were selected following a series of criteria to facilitate comparable prosodic analyses given the inherent dynamicity of spontaneous speech. The analysis focused on exploring the F0 movements of the selected utterances in tandem with periodic energy. Overall, it was observed that utterances conveying happiness were associated more with level contours, while angry utterances were produced with more steep falling contours especially on the utterance-final syllable. A tendency for an F0 peak to be produced on the second syllable of the utterance was also observed in both sets of data analyzed, alluding to the interactions between emotional prosody and the intonational structure of Singapore English. Finally, an emotion classification was conducted using a subset of the corpus as test data for a fine-tuned SER model. Results showed that while utterances conveying happiness could be automatically recognized to a high percentage of 86%, there was also a high percentage of misclassification of anger. This result appears to counter findings in past SER studies, which typically reported high levels of recognition accuracy for anger. Taken together, the thesis contributes an understanding of how spontaneous speech and emotional cues on the prosodic level interact even in task-oriented conversations, with further considerations of how these patterns may vary among bilinguals in multicultural contexts, so as to enable more robust speech emotion recognition systems in the future.

The recognition of read and spontaneous speech in local vernacular: The case of Zurich German

Pronunciation variants across system configuration, language and speaking style

This paper addresses the issue of producing synthetic speech for an interpreted dialogue where the attitudinal colouring of an original utterance is to be preserved; it describes differences in speaking style between read and spontaneous speech from the viewpoint of synthesis research and discusses the design of a synthesis system incorporating labels to encode the prosodic and segmental variation simultaneously. Spontaneous speech confronts us with phenomena that were not encountered in corpora of prepared or read speech, and to account for these we have to identify increasingly higher-level factors of discourse structure and speaker involvement. The paper makes three specific claims: (a) that it is better to label the distinctive characteristics of speech through higher-level context dependencies, and to select units for synthesis from appropriate contexts, rather than attempt to predict and modify fine phonetic detail; (b) that the labelling of segmental and prosodic characteristics can be done adequately for speech synthesis using automatic techniques, leaving the human labeller free to identify higher-level discourse-related aspects of the speech; and (c) that instead of minimizing the size of the source database of speech units, we should rather be concerned to maximize its variety and to efficiently select from it the units that most closely express the characteristics of the target speech. The CHATR resynthesis toolkit performs many of these tasks.

We have constructed a new speech data corpus consisting of the utterances of 221 elderly Japanese people (average age: 79.2) with the aim of improving the accuracy of automatic speech recognition (ASR) for the elderly. ASR is a beneficial modality for people with impaired vision or limited hand movement, including the elderly. However, speech recognition systems using standard recognition models, especially acoustic models, have been unable to achieve satisfactory performance for the elderly. Thus, creating more accurate acoustic models of the speech of elderly users is essential for improving speech recognition for the elderly. Using our new corpus, which includes the speech of elderly people living in three regions of Japan, we conducted speech recognition experiments using a variety of DNN-HNN acoustic models. As training data for our acoustic models, we examined whether a standard adult Japanese speech corpus (JNAS), an elderly speech corpus (S-JNAS) or a spontaneous speech corpus (CSJ) was most suitable, and whether or not adaptation to the dialect of each region improved recognition results. We adapted each of our three acoustic models to all of our speech data, and then re-adapt them using speech from each region. Without adaptation, the best recognition results were obtained when using the S-JNAS trained acoustic models (total corpus: 21.85% Word Error Rate). However, after adaptation of our acoustic models to our entire corpus, the CSJ trained models achieved the lowest WERs (entire corpus: 17.42%). Moreover, after readaptation to each regional dialect, the CSJ trained acoustic model with adaptation to regional speech data showed tendencies of improved recognition rates. We plan to collect more utterances from all over Japan, so that our corpus can be used as a key resource for elderly speech recognition in Japanese. We also hope to achieve further improvement in recognition performance for elderly speech.

This article presents a probabilistic scheme for detecting the interruption point (IP) in spontaneous speech based on inter-syllable boundary-based prosodic features. Because of the high error rate in spontaneous speech recognition, a combined acoustic model considering both syllable and subsyllable recognition units, is firstly used to determine the inter-syllable boundaries and output the recognition confidence of the input speech. Based on the finding that IPs always occur at inter-syllable boundaries, a probability distribution of the prosodic features at the current potential IP is estimated. The Conditional Random Field (CRF) model, which employs the clustered prosodic features of the current potential IP and its preceding and succeeding inter-syllable boundaries, is employed to output the IP likelihood measure. Finally, the confidence of the recognized speech, the probability distribution of the prosodic features and the CRF-based IP likelihood measure are integrated to determine the optimal IP sequence of the input spontaneous speech. In addition, pitch reset and lengthening are also applied to improve the IP detection performance. The Mandarin Conversional Dialogue Corpus is adopted for evaluation. Experimental results show that the proposed IP detection approach obtains 10.56% and 6.5% more effective results than the hidden Markov model and the Maximum Entropy model respectively under the same experimental conditions. Besides, the IP detection error rate can be further reduced by 9.15% using pitch reset and lengthening information. The experimental results confirm that the proposed model based on inter-syllable boundary-based prosodic features can effectively detect the interruption point in spontaneous Mandarin speech.

Nowadays, automatic speech recognizers have become quite goodin recognizing well prepared fluent speech (e.g. news readings).However, the recognition of spontaneous speech is still problem-atic. Some important reasons for this are that spontaneous speechis usually less articulated and contains a lot of disfluencies. In thispaper, a new methodology for coping with disfluencies is presentedand evaluated. The basic idea is to detect disfluencies and to deter-mine the nature of these disfluencies prior to the recognition, and touse that information to control/modify the search. At present, themethodology has been elaborated for filled pauses (FP) and wordrepetitions (WR). It enables us to eliminate about one associated normal word error per disfluency without introducing a significantaugmentation of the computational load. 1. Introduction The most frequently occurring disfluencies in spontaneous speechare filled pauses (FPs). If one has a sufficiently large spontaneousspeech corpus with FP transcriptions available, one can create ded-icated acoustic FP models to supplement the traditional triphonemodel set. One can also introduce lexical FP models (pronuncia-tion variants in the lexicon). In [7], a combination of acoustic andlexical FP models yielded a reduction of the word error rate (WER)of 7.8 % relative for medical dictation (with respect to a baselinesystem with one FP model in the lexicon).Even when good FP models are available, FPs may still con-fuse the recognizer because they lead to word sequences (-grams)that are not observed in the kind of written material that is normallyused for language model (LM) development. Extending this mate-rialwithspontaneous speech transcriptionswould beanother meansof improving a spontaneous speech recognizer. However, the nec-essary resources to accomplish this for a general task are usuallynot available (and certainly not for Dutch). For a limited domainlike medical dictation, the situation is different, and experiments byPakhomov [4]showed thatthetechnique can work : the WERcouldbe reduced from 32.2 to 28.5 %.A simple way of dealing with FPs at the level of the LM is toignore the FP in the LM context of the next word. However, Stol-cke and Shriberg [8] come to the surprising conclusion that discard-ing FPs from the trigram context actually increases the perplexity.However, they looked at the speech stretches that were supplied totherecognizer and thatweredelimitedonthebasisofacoustic crite-ria. The FPs occurring at sentence boundaries often appeared in themiddle of such stretches. By only discarding sentence internal FPsthe perplexity did decrease, as expected. In [5] the speech stretchescorresponded to sentences and all FPs were sentence internal. Forthis material, the discarding strategy resulted in a 4 % decrease ofthe overall perplexity and a 30 % decrease of the perplexity of theword after an FP.The problems with the discarding of FPs in the LM are that thenotion of a sentence is not always clear in spontaneous speech, andthat even when it is, the sentence boundaries only become availableafter the recognition (with FPs) has been accomplished.Apart from FPs, there are other important disfluency types inspontaneous speech, such as the word repetition (WR). It is definedas an event consisting of a completed (not interrupted) word (W)and a repetition (R) of that word. Clearly, WRs do not require anynew acoustic or lexical models. However, just like FPs, they maydisturb the recognizer at the level of the language model.In this paper we investigate adaptive search methodologies thataim at improving the recognition of spontaneous speech in thevicinity of FPsand WRs without affecting the performance in otherregions. In section 2 we briefly outline the basic principles of thenew adaptive approach and we further elaborate and evaluate it insections 3 and 4.

Sequence labeling to detect stuttering events in read speech

A new speech corpus of super-elderly Japanese for acoustic modeling

The Acoustic Voice Quality Index (AVQI) is a widely adopted tool for assessing dysphonia, incorporating sustained vowel and continuous speech samples to enhance ecological validity. Despite its broad use, the reliability of AVQI measurements, particularly in digital health applications, remains underexplored. This study aims to review the literature on AVQI's development and validation from the perspective of internal consistency of acoustic measurements, and to assess the amount of speech required to reliably determine the AVQI of a voice recording. Two sub-studies are described. Study 1: A narrative review was conducted using Google Scholar and Scopus to identify studies mentioning “AVQI” and “reliability”. Key methodological details were extracted from studies reporting AVQI measurements, summarized, and discussed in terms of how the internal consistency of acoustic measurements was ensured across studies where AVQI had been validated or applied. Study 2: Recordings of read and spontaneous speech as well as sustained vowels produced by 54 native Swedish speakers (22 female, 32 male; age range: 46–78 years) were assessed in terms of the amount of speech required to obtain a reliable acoustic measurement of the speakers' voices. Simulations were performed using read and spontaneous speech materials of varying lengths. The variability in AVQI and its sub-measures was analyzed relative to the length (in words or in seconds) of continuous speech included. The result of study 1 shows that out of 129 identified studies, 85 reported on AVQI measurements. The review highlighted substantial variability in continuous speech lengths used across studies, ranging from 4–200 words. A recommendation of 3 s of voiced segments was often adhered to, but was found to lack sufficiently robust grounding. The simulations indicated that AVQI achieves internal consistency at speech lengths of approximately 50 words (or 20 s), which is longer than the current recommendation. Both read and spontaneous speech provided stable AVQI measurements at these speech lengths. AVQI thresholds obtained using speech lengths shorter than 50 words (20 s) may require re-evaluation. Robust standardization of continuous speech lengths is essential for the successful adoption of AVQI in digital health applications.

Many phonological processes can be affected by segmental context spanning word boundaries, which often lead to variable outcomes. This paper tests the idea that some of this variability can be explained by reference to production planning. We examine coronal stop deletion (CSD), a variable process conditioned by preceding and upcoming phonological context, in a corpus of spontaneous British English speech, as a means of investigating a number of variables associated with planning: Prosodic boundary strength, word frequency, conditional probability of the following word, and speech rate. From the perspective of production planning, (1) prosodic boundaries should affect deletion rate independently of following context; (2) given the locality of production planning, the effect of the following context should decrease at stronger prosodic boundaries; and (3) other factors affecting planning scope should modulate the effect of upcoming phonological material above and beyond the modulating effect of prosodic boundaries. We build a statistical model of CSD realization, using pause length as a quantitative proxy for boundary strength, and find support for these predictions. These findings are compatible with the hypothesis that the locality of production planning constrains variability in speech production, and have practical implications for work on CSD and other variable processes.

Discriminative keyword spotting

Long-Term Effects of Gender-Affirming Hormone Therapy from Adolescence to Adulthood on Voice.