Interruption Point Detection of Spontaneous Speech Using Inter-Syllable Boundary-Based Prosodic Features

Recognizing emotions using natural or spontaneous speech are extremely difficult compared to doing the same for acted or elicited speeches. Speech emotion recognition for real conversation such as spontaneous speech requires linguistic information of the speech to be included in the speech emotion recognition component to achieve a high recognition rate. However, with the lack of digital speech resources of an under-resourced language, this requirement poses a problem. In this paper, speech emotion recognition of spontaneous speech in Malay language using prosodic features and Random Forest classifier is presented. We also investigate the influence of age categorized as children, young adults and middle-aged on emotion recognition. Ninety spontaneous speech sentences from 30 native speakers of Malay language are collected and classified into three emotions, which are happy, angry and sad. Results show that the spontaneous speech of middle-aged group achieved the highest accuracy rate followed by children age group and finally the young adults. While sad emotions are recognized satisfactorily across all age groups, confusions exist between happy and angry emotions.

This paper presents a progress report on a relatively difficult ASR task on a spontaneous speech corpus - Mandarin Conversational Dialogue Corpus (MCDC). A DNN-based acoustic model is constructed based on the CLDNN structure with a large dataset that comprises two spontaneous-speech corpora and one read-speech corpus. The study uses a large text dataset formed by seven corpora to train an efficient general language model (LM). Two adapted LMs specially for spontaneous speech recognition are also constructed. Experimental results showed that the best performances of 26.3% in character error rate (CER) and 32.5% in word error rate (WER) were reached on MCDC. They represented 27.9% and 22.2% of relative CER and WER reductions as compared with the performances by the previous best HMM-based method. This confirms that the proposed method is promising in tackling on Mandarin spontaneous speech recognition.

This paper describes the development of the EARS (effective, affordable, reusable speech-to-text) Chinese corpus, a telephony conversational speech database for speech processing. The EARS database is the first of its kind collected for Mandarin Chinese telephony spontaneous speech. The purpose of developing this EARS Chinese corpus is to collect Mandarin conversations between either strangers or friends, which cover a wide range of topics, over landline and cellular channels. All the speech data are annotated with standard Chinese character transcription as well as specific mark-ups for spontaneous speech. This corpus will be used for conversational and spontaneous Mandarin speech recognition tasks, under the DARPA EARS framework. This paper introduces the design, development, structure, and initial phonetic analysis of the first 50-hour collection of this corpus. Additional 300 to 500 hours of data will be collected and transcribed between 2004 and 2005.

Emotions are an integral part of social interactions, serving vital functions in the management of our interpersonal relationships. Cues to emotional meaning can be conveyed through a variety of modalities, from facial expressions and body gestures to the linguistic and non-linguistic, or prosodic aspects of speech. As automatic speech systems continue to grow in use, the ability to recognize emotional cues from these modalities is no longer solely expected in human communication, but also in human-computer interactions with the goal of developing more user-friendly and effective applications. This thesis explores how speech emotion recognition (SER) systems can be more robust to factors of language, culture, and context through the lens of emotional speech and emotional prosody from bilingual English-speaking Singaporeans of Chinese, Malay, or Indian ethnicity. Specifically, the thesis aims to examine how Singaporeans from the three ethnic groups may vary in their perception of emotions in speech, and to explore how they express emotions via speech prosody. The recognition and production processes of emotional communication in Singapore English was investigated through two emotion classes – anger and happiness. An emotion perception test was conducted using acted angry and happy speech as stimuli to understand how Singapore English speakers recognized emotions in speech. The results of the perception test revealed that respondents displayed very similar patterns in their rating of emotional speech stimuli. The study also found that respondents tended to rated anger with low dominance. At the same time, a spontaneous speech corpus known as the Singapore English Spontaneous Emotional Speech Corpus (SESESC) was also developed, featuring audio recordings of 44 Singapore English speakers from the three ethnic groups playing a co-operative video game with either a familiar partner or a confederate. More than 36,000 intervals were annotated in-context of the recordings by each annotator based on a modified emotion annotation scheme that included information regarding the emotion category and arousal levels. Intervals that were labelled as conveying joy or anger with full agreement from the annotators were selected following a series of criteria to facilitate comparable prosodic analyses given the inherent dynamicity of spontaneous speech. The analysis focused on exploring the F0 movements of the selected utterances in tandem with periodic energy. Overall, it was observed that utterances conveying happiness were associated more with level contours, while angry utterances were produced with more steep falling contours especially on the utterance-final syllable. A tendency for an F0 peak to be produced on the second syllable of the utterance was also observed in both sets of data analyzed, alluding to the interactions between emotional prosody and the intonational structure of Singapore English. Finally, an emotion classification was conducted using a subset of the corpus as test data for a fine-tuned SER model. Results showed that while utterances conveying happiness could be automatically recognized to a high percentage of 86%, there was also a high percentage of misclassification of anger. This result appears to counter findings in past SER studies, which typically reported high levels of recognition accuracy for anger. Taken together, the thesis contributes an understanding of how spontaneous speech and emotional cues on the prosodic level interact even in task-oriented conversations, with further considerations of how these patterns may vary among bilinguals in multicultural contexts, so as to enable more robust speech emotion recognition systems in the future.

The amount of legislative documents produced within the past decade has risen dramatically, making it difficult for law practitioners to consult and update legislation. Named Entity Recognition (NER) systems have the untapped potential to extract information from legal documents, which can improve information retrieval and decision-making processes. We introduce the UlyssesNER-Br, a corpus of Brazilian Legislative Documents for NER with quality baselines. The presented corpus consists of bills and legislative consultations from Brazilian Chamber of Deputies. We implemented Conditional Random Field (CRF) and Hidden Markov Model (HMM) models, and the promising F1-score of 80.8% in the analysis by categories and 81.04% in the analysis by types, was achieved with the CRF model. The entities with the best average F1-score results were “FUNDlei” and “DATA”, and the ones with the worst results were “EVENTO” and “PESSOAgrupoind”. The corpus was also evaluated using a BiLSTM-CRF and Glove architectures provided by the pioneering state-of-the-art paper, achieving F1-score of 76.89% in the analysis by categories and 59.67% in the analysis by types.

In this paper, a pipelining empirical mode decomposition is presented to reduce the computing time of the emotionalized spontaneous speaker or speech recognition processing. This is a novel approach for integrating the pipelining technique into the standard empirical mode decomposition of the Hilbert-Huang transform. In addition, there is reduced about 45% of the computing time when the emotionalized spoken signal through our segmentation and pipelining processes. Based on the designed processing of emotionalized spontaneous speaker or speech recognition, the segmented and processed voice signals are recomposed back for constructing the speech and speaker models, or to identify which existed model is the most similar one. In the final part of this paper, a comparison of the speech recognized rate between standard and pipelining empirical mode decompositions are presented, and an equivalent effect in the recognition will be found. In practice, speaker or speech recognitions in an emotionalized spontaneous speech are very difficult. The existing speech recognition methods often fail to capture inherent voiceprint features from an emotionalized speech, such as the voice with a passionate intonation. And some of the existed methods to extract the pure voiceprint from an emotionalized spoken signal are very expensive in computation and time, so that technique is impossible to use in a real-time environment like smart houses. But, this paper presents a solution to improve the emotionalized spontaneous speaker or speech recognition processing to fit the real-time request.

Nowadays, automatic speech recognizers have become quite goodin recognizing well prepared fluent speech (e.g. news readings).However, the recognition of spontaneous speech is still problem-atic. Some important reasons for this are that spontaneous speechis usually less articulated and contains a lot of disfluencies. In thispaper, a new methodology for coping with disfluencies is presentedand evaluated. The basic idea is to detect disfluencies and to deter-mine the nature of these disfluencies prior to the recognition, and touse that information to control/modify the search. At present, themethodology has been elaborated for filled pauses (FP) and wordrepetitions (WR). It enables us to eliminate about one associated normal word error per disfluency without introducing a significantaugmentation of the computational load. 1. Introduction The most frequently occurring disfluencies in spontaneous speechare filled pauses (FPs). If one has a sufficiently large spontaneousspeech corpus with FP transcriptions available, one can create ded-icated acoustic FP models to supplement the traditional triphonemodel set. One can also introduce lexical FP models (pronuncia-tion variants in the lexicon). In [7], a combination of acoustic andlexical FP models yielded a reduction of the word error rate (WER)of 7.8 % relative for medical dictation (with respect to a baselinesystem with one FP model in the lexicon).Even when good FP models are available, FPs may still con-fuse the recognizer because they lead to word sequences (-grams)that are not observed in the kind of written material that is normallyused for language model (LM) development. Extending this mate-rialwithspontaneous speech transcriptionswould beanother meansof improving a spontaneous speech recognizer. However, the nec-essary resources to accomplish this for a general task are usuallynot available (and certainly not for Dutch). For a limited domainlike medical dictation, the situation is different, and experiments byPakhomov [4]showed thatthetechnique can work : the WERcouldbe reduced from 32.2 to 28.5 %.A simple way of dealing with FPs at the level of the LM is toignore the FP in the LM context of the next word. However, Stol-cke and Shriberg [8] come to the surprising conclusion that discard-ing FPs from the trigram context actually increases the perplexity.However, they looked at the speech stretches that were supplied totherecognizer and thatweredelimitedonthebasisofacoustic crite-ria. The FPs occurring at sentence boundaries often appeared in themiddle of such stretches. By only discarding sentence internal FPsthe perplexity did decrease, as expected. In [5] the speech stretchescorresponded to sentences and all FPs were sentence internal. Forthis material, the discarding strategy resulted in a 4 % decrease ofthe overall perplexity and a 30 % decrease of the perplexity of theword after an FP.The problems with the discarding of FPs in the LM are that thenotion of a sentence is not always clear in spontaneous speech, andthat even when it is, the sentence boundaries only become availableafter the recognition (with FPs) has been accomplished.Apart from FPs, there are other important disfluency types inspontaneous speech, such as the word repetition (WR). It is definedas an event consisting of a completed (not interrupted) word (W)and a repetition (R) of that word. Clearly, WRs do not require anynew acoustic or lexical models. However, just like FPs, they maydisturb the recognizer at the level of the language model.In this paper we investigate adaptive search methodologies thataim at improving the recognition of spontaneous speech in thevicinity of FPsand WRs without affecting the performance in otherregions. In section 2 we briefly outline the basic principles of thenew adaptive approach and we further elaborate and evaluate it insections 3 and 4.

We present an image classification system based on a Conditional Random Field (CRF) model trained on simple features obtained from a small number of semantically representative image patches. The CRFs are very powerful to handle complex parts dependencies due to their approach based on the effective

In order to improve the performance of Khmer named entity recognition(NER), a NER method based on conditional random field (CRF) model fusing with Khmer entity characteristics constraints is proposed in this paper. First of all, we carried out analyses on the Khmer entity characteristics, summarized the constraint on these entity characteristics and introduced into the CRF; then solved the labeling sequence by integer linear programming integrated entity characteristic constraint and obtained a model of CRF integrated with constraints based on Khmer entity characteristics. Based on a contrastive experiment, CRF model of the constraint has a better performance than traditional CRF model when carrying out the Khmer NER.

The recognition of read and spontaneous speech in local vernacular: The case of Zurich German

Sequence labeling to detect stuttering events in read speech

The Conditional Random Fields (CRFs) only can deal with the sequence data of Markov property. And it cannot realize the relationship labeling with more fine structure between semantic roles. An approach to semantic role labeling (SRL) based on Chain Conditional Random Fields (CCRFs) Model was proposed. The long-distance dependencies between different state variants were handled effectively via labeling Hierarchical Dependencies and Brother Dependencies of syntactic dependency tree. Moreover, some new combinative features and prepositional phrase also were added though taking advantages of any features can be added in CRFs model. The experiments were implemented on CoNLL 2008 Shared Task. The results indicate the proposed method can improve precision and recall rate of the system.

Contour detection is a fundamental problem in computer vision. However, there is still a considerable disparity between detection results and actual contours. To detect object-level contours on the basis of comprehensive analysis of potential edges, we present a deep-learning-based approach with a conditional random fields (CRF) model. We obtain the initial edgemap with a VGGNet-based model, and establish a contour correlation graph (CCG) to describe the potential edges and their relationships. Then a CRF model is adopted to fulfill the optimization on the basis of the analysis of object-level contour characteristics and to predict the validity of candidate contours. The experiment results on various datasets demonstrate that the proposed method outperforms the existing methods by a noticeable margin.

Accurately recognizing human activities from sensor data recorded in a smart home setting is a challenging task. Typically, probabilistic models such as the hidden Markov model (HMM) or conditional random fields (CRF) are used to map the observed sensor data onto the hidden activity states. A weakness of these models, however, is that the type of distribution used to model state durations is fixed. Hidden semi-Markov models (HSMM) and semi-Markov conditional random fields (SMCRF) model duration explicitly, allowing state durations to be modelled accurately. In this paper we compare the recognition performance of these models on multiple fully annotated real world datasets consisting of several weeks of data. In our experiments the HSMM consistently outperforms the HMM, showing that accurate duration modelling can result in a significant increase in recognition performance. SMCRFs only slightly outperform CRFs, showing that CRFs are more robust in dealing with violations of the modelling assumptions. The datasets used in our experiments are made available to the community to allow further experimentation.

Domain-specific knowledge is often recorded by experts in the form of unstructured text. For example, in the medical domain, clinical notes from electronic health records contain a wealth of information. Similar practices are found in other domains. The challenge we discuss in this paper is how to identify and extract part names from technicians repair notes, a noisy unstructured text data source from General Motors’ archives of solved vehicle repair problems, with the goal to develop a robust and dynamic reasoning system to be used as a repair adviser by service technicians. In the present work, we discuss two approaches to this problem. We present an algorithm for ontology-guided entity disambiguation that uses existing knowledge sources, such as domain-specific taxonomies and other structured data. We illustrate its use in the automotive domain, using GM parts ontology and the unit structure of repair manuals text to build context models, which are then used to disambiguate mentions of part-related entities in the text. We also describe extraction of part names with a small amount of annotated data using hidden Markov models (HMM) with shrinkage, achieving an f-score of approximately 80%. Next, we used linear-chain conditional random fields (CRF) in order to model observation dependencies present in the repair notes. Using CRF did not lead to improved performance, but a slight improvement over the HMM results was obtained by using a weighted combination of the HMM and CRF models.