Synthesizing Spontaneous Speech

This paper describes two experiments aimed at exploring the relationship between linguistic aspects and perceived proficiency in read and spontaneous speech. 5,000 utterances of read speech by 50 non-native speakers of Korean in Experiment 1, and of 6,000 spontaneous speech utterances in Experiment 2 were scored for proficiency by native human raters and were analyzed by factors known to be related to perceived proficiency. The results show that the factors investigated in this study can be employed to predict proficiency ratings, and the predictive power of fluency and pitch and accent accuracy is strong for both read and spontaneous speech. We also observe that while proficiency ratings of read speech are mainly related to segmental accuracy, those of spontaneous speech appear to be more related to pitch and accent accuracy. Moreover, proficiency in read speech does not always equate to the proficiency in spontaneous speech, and vice versa, with Pearson’s per-speaker correlation score of 0.535.

This paper deals with an approach to Automatic Language Identification using only prosodic modeling. The actual approach for language identification focuses mainly on phonotactics because it gives the best results. We propose here to evaluate the relevance of prosodic information for language identification with read studio recording (previous experiment [1]) and spontaneous telephone speech. For read speech, experiments were performed on the five languages of the MULTEXT database [2]. On the MULTEXT corpus, our prosodic system achieved an identification rate of 79 % on the five languages discrimination task. For spontaneous speech, experiments are made on the ten languages of the OGI Multilingual telephone speech corpus [3]. On the OGI MLTS corpus, the results are given for languages pair discrimination tasks, and are compared with results from [4]. As a conclusion, if our prosodic system achieves good performance on read speech, it might not take into account the complexity of spontaneous speech prosody. 1.

The recognition of read and spontaneous speech in local vernacular: The case of Zurich German

The aim of this paper is twofold: (i) give a large-scale description in realized word-final schwas of French lexical words for different regions (North vs. South) and different speaking styles (read vs. spontaneous speech); (ii) highlight differences in prosodic features and test these differences via automatic classification techniques. The proposed study relies on a subset of 12.5 hours of the French PFC corpus. Manually transcribed speech was segmented and labeled using automatic speech alignment and a pronunciation dictionary including optional word-final schwas for all words ending in a consonant. F0 and intensity values were extracted and averaged over segments. Our study revealed that, for both speaking styles, word-final schwas of southern French tended to keep relatively high F0 values and longer durations in comparison with northern French where F0 tends to drop on a word-final schwa. On average, spontaneous speech featured smaller F0 drops between final full vowel and subsequent word-final schwa vowel as well as longer durations. The automatic North/South classification of word-final schwas achieved better results for spontaneous speech. As for distinguishing between speaking styles, southern French obtained slightly better scores than the northern varieties.

This paper reports various investigations on recognizing spontaneous presentation speech in connection with the “Spontaneous Speech” national project started in 1999. Presentation speech uttered by 10 male speakers of approximately 4.5 hours duration has been recognized. Experimental results show that acoustic and language modeling based on an actual spontaneous speech corpus is far more effective than conventional modeling based on read speech. The recognition accuracy has a wide speaker-tospeaker variability according to the speaking rate, the number of fillers, the number of repairs, etc. It was confirmed that unsupervised speaker adaptation of acoustic models was effective to improve the recognition accuracy. The recognition accuracy for spontaneous speech is, however, still rather low, and there remains a large number of research issues.

Discriminative keyword spotting

Sequence labeling to detect stuttering events in read speech

This article presents a probabilistic scheme for detecting the interruption point (IP) in spontaneous speech based on inter-syllable boundary-based prosodic features. Because of the high error rate in spontaneous speech recognition, a combined acoustic model considering both syllable and subsyllable recognition units, is firstly used to determine the inter-syllable boundaries and output the recognition confidence of the input speech. Based on the finding that IPs always occur at inter-syllable boundaries, a probability distribution of the prosodic features at the current potential IP is estimated. The Conditional Random Field (CRF) model, which employs the clustered prosodic features of the current potential IP and its preceding and succeeding inter-syllable boundaries, is employed to output the IP likelihood measure. Finally, the confidence of the recognized speech, the probability distribution of the prosodic features and the CRF-based IP likelihood measure are integrated to determine the optimal IP sequence of the input spontaneous speech. In addition, pitch reset and lengthening are also applied to improve the IP detection performance. The Mandarin Conversional Dialogue Corpus is adopted for evaluation. Experimental results show that the proposed IP detection approach obtains 10.56% and 6.5% more effective results than the hidden Markov model and the Maximum Entropy model respectively under the same experimental conditions. Besides, the IP detection error rate can be further reduced by 9.15% using pitch reset and lengthening information. The experimental results confirm that the proposed model based on inter-syllable boundary-based prosodic features can effectively detect the interruption point in spontaneous Mandarin speech.

Emotions are an integral part of social interactions, serving vital functions in the management of our interpersonal relationships. Cues to emotional meaning can be conveyed through a variety of modalities, from facial expressions and body gestures to the linguistic and non-linguistic, or prosodic aspects of speech. As automatic speech systems continue to grow in use, the ability to recognize emotional cues from these modalities is no longer solely expected in human communication, but also in human-computer interactions with the goal of developing more user-friendly and effective applications. This thesis explores how speech emotion recognition (SER) systems can be more robust to factors of language, culture, and context through the lens of emotional speech and emotional prosody from bilingual English-speaking Singaporeans of Chinese, Malay, or Indian ethnicity. Specifically, the thesis aims to examine how Singaporeans from the three ethnic groups may vary in their perception of emotions in speech, and to explore how they express emotions via speech prosody. The recognition and production processes of emotional communication in Singapore English was investigated through two emotion classes – anger and happiness. An emotion perception test was conducted using acted angry and happy speech as stimuli to understand how Singapore English speakers recognized emotions in speech. The results of the perception test revealed that respondents displayed very similar patterns in their rating of emotional speech stimuli. The study also found that respondents tended to rated anger with low dominance. At the same time, a spontaneous speech corpus known as the Singapore English Spontaneous Emotional Speech Corpus (SESESC) was also developed, featuring audio recordings of 44 Singapore English speakers from the three ethnic groups playing a co-operative video game with either a familiar partner or a confederate. More than 36,000 intervals were annotated in-context of the recordings by each annotator based on a modified emotion annotation scheme that included information regarding the emotion category and arousal levels. Intervals that were labelled as conveying joy or anger with full agreement from the annotators were selected following a series of criteria to facilitate comparable prosodic analyses given the inherent dynamicity of spontaneous speech. The analysis focused on exploring the F0 movements of the selected utterances in tandem with periodic energy. Overall, it was observed that utterances conveying happiness were associated more with level contours, while angry utterances were produced with more steep falling contours especially on the utterance-final syllable. A tendency for an F0 peak to be produced on the second syllable of the utterance was also observed in both sets of data analyzed, alluding to the interactions between emotional prosody and the intonational structure of Singapore English. Finally, an emotion classification was conducted using a subset of the corpus as test data for a fine-tuned SER model. Results showed that while utterances conveying happiness could be automatically recognized to a high percentage of 86%, there was also a high percentage of misclassification of anger. This result appears to counter findings in past SER studies, which typically reported high levels of recognition accuracy for anger. Taken together, the thesis contributes an understanding of how spontaneous speech and emotional cues on the prosodic level interact even in task-oriented conversations, with further considerations of how these patterns may vary among bilinguals in multicultural contexts, so as to enable more robust speech emotion recognition systems in the future.

Individuals who share a dialect nonetheless differ consistently in phonetic expression. The origin of such idiolectal differences has been sought in sociolinguistically and in physiologically motivated accounts. Our study focused on characteristic differences in the habitual rate of speech production within a dialect, aiming to calibrate the contribution of neuromuscular constraint in idiolect differences. We sampled and analyzed the temporal characteristics of spontaneous and read speech produced by a set of adolescent female talkers from Brooklyn, New York. Spontaneous speech was elicited by open-ended questions, while the Farm Script was used as the text for production of read speech at comfortable and at rapid rates. Overall, the rate of rapid read speech was correlated with the rate of comfortable spontaneous speech and comfortable reading, indicating a clear contribution of neuromuscular constraints to characteristic differences in speech rate. However, some talkers exhibiting the ability to produce rapid speech at the fastest rates adopted comfortable speaking rates at the slower end of the distribution. In light of these findings, we doubt that idiolectal differences in speaking rate can be accountable solely by appeal to differences between individuals in agility of oral articulators. [Work supported by NIH DC00308.]

The Acoustic Voice Quality Index (AVQI) is a widely adopted tool for assessing dysphonia, incorporating sustained vowel and continuous speech samples to enhance ecological validity. Despite its broad use, the reliability of AVQI measurements, particularly in digital health applications, remains underexplored. This study aims to review the literature on AVQI's development and validation from the perspective of internal consistency of acoustic measurements, and to assess the amount of speech required to reliably determine the AVQI of a voice recording. Two sub-studies are described. Study 1: A narrative review was conducted using Google Scholar and Scopus to identify studies mentioning “AVQI” and “reliability”. Key methodological details were extracted from studies reporting AVQI measurements, summarized, and discussed in terms of how the internal consistency of acoustic measurements was ensured across studies where AVQI had been validated or applied. Study 2: Recordings of read and spontaneous speech as well as sustained vowels produced by 54 native Swedish speakers (22 female, 32 male; age range: 46–78 years) were assessed in terms of the amount of speech required to obtain a reliable acoustic measurement of the speakers' voices. Simulations were performed using read and spontaneous speech materials of varying lengths. The variability in AVQI and its sub-measures was analyzed relative to the length (in words or in seconds) of continuous speech included. The result of study 1 shows that out of 129 identified studies, 85 reported on AVQI measurements. The review highlighted substantial variability in continuous speech lengths used across studies, ranging from 4–200 words. A recommendation of 3 s of voiced segments was often adhered to, but was found to lack sufficiently robust grounding. The simulations indicated that AVQI achieves internal consistency at speech lengths of approximately 50 words (or 20 s), which is longer than the current recommendation. Both read and spontaneous speech provided stable AVQI measurements at these speech lengths. AVQI thresholds obtained using speech lengths shorter than 50 words (20 s) may require re-evaluation. Robust standardization of continuous speech lengths is essential for the successful adoption of AVQI in digital health applications.

Multiethnolectal ways of speaking have been emerging for 30 years in culturally and linguistically diverse neighborhoods of European cities, including Zurich (Switzerland). Among the prosodic features of Germanic multiethnolects, a so-called ‘staccato’ rhythm has been mentioned in several studies. For instance, a comparison between two groups of adolescents (12 speakers each) showed that speakers of multiethnolectal Zurich German displayed slower syllable rates and less vowel duration variability than speakers of a rather traditional dialect. This study compares syllable rate and speech rhythm metrics (nPVI-V, nPVI-C) in spontaneous and read speech of 48 Zurich German adolescents. In a regression analysis, rhythmic measures were compared with the perception of how multiethnolectal the speakers sounded (rating score). The results showed that syllable rate and nPVI-V were related to rating score independently of speaking style (read, spontaneous speech): Speakers who were perceived as more multiethnolectal had a slower syllable rate and less vowel duration variability. Such findings were not observed for nPVI-C. These results suggest that syllable rate and speech rhythm (at least, vowel duration variability) are stable phonetic features of multiethnolectal Zurich German, since the relationship between these features and the perception of multiethnolectal speech was observed in both read and spontaneous speech.

본 연구는 2대 이상 서울에서 거주 중인 30대와 60-70대 토박이 화자들을 대상으로 하여 연령(세대) 변수가 낭독 발화와 자유 발화의 음향음성학적 차이를 밝히는 데에 기여하는지를 살폈다. 또한, 낭독발화와 자유 발화의 억양 차이도 함께 분석되었다. 측정된 계량적 지표들 중 기본주파수의 표준 편차와 연속단위 변동지수, 기본주파수의 음역과 최솟값은 낭독 발화보다 자유 발화에서 연령 변수에 상관없이 유의미하였으나, 기본주파수의 최댓값과 평균값은 30대 화자 집단에 한해서만 유의미하였다. 발화 속도는 두 집단에서 모두 유의미하긴 하였으나 60대 화자 집단에서 더 큰 효과 크기를 보였고, 두 연령 집단에서 서로 다른 방향으로 변화하였다. 억양의 경우, 억양구의 수는 세대 변수와 상호 작용을 보이며 발화 형식에 따라 유의미하게 다르게 나타났으나 억양구 경계 성조는 발화 형식에 따라 큰 차이를 보이지 않았다. 이러한 결과는 세대에 따른 유창성 차이와 노년층 화자 성대의 노화 등에 기인한다고 생각된다. 전반적으로 본 연구 결과는 앞으로의 발화 형식 연구에서 사회언어학적 요인들이 좀 더 적극적으로 고려되어야 함을 시사한다.

Oral cancer speech is a disease which impacts more than half a million people worldwide every year. Analysis of oral cancer speech has so far focused on read speech. In this paper, we 1) present and 2) analyse a three-hour long spontaneous oral cancer speech dataset collected from YouTube. 3) We set baselines for an oral cancer speech detection task on this dataset. The analysis of these explainable machine learning baselines shows that sibilants and stop consonants are the most important indicators for spontaneous oral cancer speech detection.

Background and Objectives: Population rapid growth and demographic shift is leading to a rise in neurodegenerative disorders such as dementia and mild cognitive impairment (MCI). Evidence indicates that MCI is not the earliest phase of prodromal AD. Subjective Memory Decline (SMD) refers to a self-perceived decline in cognitive abilities compared to previous functioning levels in individuals with normal cognition. Language impairment represents a critical marker of neurodegenerative disorders and early memory decline in healthy older adults. Methods: This review was conducted in accordance with PRISMA Statement guidelines. The inclusion criteria of the selection process were set as follows: (1) All studies analyzed spontaneous speech samples in individuals with SMD or individuals with +αβ amyloid. (2) Studies reported language performance indicators (e.g., lexical, syntactic, semantic, phonetic, or fluency measures) derived from spontaneous speech. (3) The study population included participants with SMD based on recognized diagnostic criteria or self-reported cognitive complaints without objective cognitive impairment. (4) Studies were written in English. (5) The time frame of studies was 5 years. Results: The present work is a review of speech features—particularly from spontaneous and narrative speech—and methods that can serve as sensitive indicators of early cognitive changes due to AD pathology. Conclusions: Spontaneous speech analysis, through acoustic and temporal parameters such as silence duration, phrasal segment length, and speech segment frequency, offers a rich window into the subtle cognitive and linguistic changes that reflect early memory decline in healthy older adults. Spontaneous speech performance could be a scalable, low-cost, and non-invasive diagnostic tool in proactive cognitive health.