Apply pipelining empirical mode decomposition to accelerate an emotionalized speech processing

Nowadays, automatic speech recognizers have become quite goodin recognizing well prepared fluent speech (e.g. news readings).However, the recognition of spontaneous speech is still problem-atic. Some important reasons for this are that spontaneous speechis usually less articulated and contains a lot of disfluencies. In thispaper, a new methodology for coping with disfluencies is presentedand evaluated. The basic idea is to detect disfluencies and to deter-mine the nature of these disfluencies prior to the recognition, and touse that information to control/modify the search. At present, themethodology has been elaborated for filled pauses (FP) and wordrepetitions (WR). It enables us to eliminate about one associated normal word error per disfluency without introducing a significantaugmentation of the computational load. 1. Introduction The most frequently occurring disfluencies in spontaneous speechare filled pauses (FPs). If one has a sufficiently large spontaneousspeech corpus with FP transcriptions available, one can create ded-icated acoustic FP models to supplement the traditional triphonemodel set. One can also introduce lexical FP models (pronuncia-tion variants in the lexicon). In [7], a combination of acoustic andlexical FP models yielded a reduction of the word error rate (WER)of 7.8 % relative for medical dictation (with respect to a baselinesystem with one FP model in the lexicon).Even when good FP models are available, FPs may still con-fuse the recognizer because they lead to word sequences (-grams)that are not observed in the kind of written material that is normallyused for language model (LM) development. Extending this mate-rialwithspontaneous speech transcriptionswould beanother meansof improving a spontaneous speech recognizer. However, the nec-essary resources to accomplish this for a general task are usuallynot available (and certainly not for Dutch). For a limited domainlike medical dictation, the situation is different, and experiments byPakhomov [4]showed thatthetechnique can work : the WERcouldbe reduced from 32.2 to 28.5 %.A simple way of dealing with FPs at the level of the LM is toignore the FP in the LM context of the next word. However, Stol-cke and Shriberg [8] come to the surprising conclusion that discard-ing FPs from the trigram context actually increases the perplexity.However, they looked at the speech stretches that were supplied totherecognizer and thatweredelimitedonthebasisofacoustic crite-ria. The FPs occurring at sentence boundaries often appeared in themiddle of such stretches. By only discarding sentence internal FPsthe perplexity did decrease, as expected. In [5] the speech stretchescorresponded to sentences and all FPs were sentence internal. Forthis material, the discarding strategy resulted in a 4 % decrease ofthe overall perplexity and a 30 % decrease of the perplexity of theword after an FP.The problems with the discarding of FPs in the LM are that thenotion of a sentence is not always clear in spontaneous speech, andthat even when it is, the sentence boundaries only become availableafter the recognition (with FPs) has been accomplished.Apart from FPs, there are other important disfluency types inspontaneous speech, such as the word repetition (WR). It is definedas an event consisting of a completed (not interrupted) word (W)and a repetition (R) of that word. Clearly, WRs do not require anynew acoustic or lexical models. However, just like FPs, they maydisturb the recognizer at the level of the language model.In this paper we investigate adaptive search methodologies thataim at improving the recognition of spontaneous speech in thevicinity of FPsand WRs without affecting the performance in otherregions. In section 2 we briefly outline the basic principles of thenew adaptive approach and we further elaborate and evaluate it insections 3 and 4.

This article presents a probabilistic scheme for detecting the interruption point (IP) in spontaneous speech based on inter-syllable boundary-based prosodic features. Because of the high error rate in spontaneous speech recognition, a combined acoustic model considering both syllable and subsyllable recognition units, is firstly used to determine the inter-syllable boundaries and output the recognition confidence of the input speech. Based on the finding that IPs always occur at inter-syllable boundaries, a probability distribution of the prosodic features at the current potential IP is estimated. The Conditional Random Field (CRF) model, which employs the clustered prosodic features of the current potential IP and its preceding and succeeding inter-syllable boundaries, is employed to output the IP likelihood measure. Finally, the confidence of the recognized speech, the probability distribution of the prosodic features and the CRF-based IP likelihood measure are integrated to determine the optimal IP sequence of the input spontaneous speech. In addition, pitch reset and lengthening are also applied to improve the IP detection performance. The Mandarin Conversional Dialogue Corpus is adopted for evaluation. Experimental results show that the proposed IP detection approach obtains 10.56% and 6.5% more effective results than the hidden Markov model and the Maximum Entropy model respectively under the same experimental conditions. Besides, the IP detection error rate can be further reduced by 9.15% using pitch reset and lengthening information. The experimental results confirm that the proposed model based on inter-syllable boundary-based prosodic features can effectively detect the interruption point in spontaneous Mandarin speech.

This paper describes the development of the EARS (effective, affordable, reusable speech-to-text) Chinese corpus, a telephony conversational speech database for speech processing. The EARS database is the first of its kind collected for Mandarin Chinese telephony spontaneous speech. The purpose of developing this EARS Chinese corpus is to collect Mandarin conversations between either strangers or friends, which cover a wide range of topics, over landline and cellular channels. All the speech data are annotated with standard Chinese character transcription as well as specific mark-ups for spontaneous speech. This corpus will be used for conversational and spontaneous Mandarin speech recognition tasks, under the DARPA EARS framework. This paper introduces the design, development, structure, and initial phonetic analysis of the first 50-hour collection of this corpus. Additional 300 to 500 hours of data will be collected and transcribed between 2004 and 2005.

The recognition of read and spontaneous speech in local vernacular: The case of Zurich German

A statistical coarticulatory model is presented for spontaneous speech recognition, where knowledge of the dynamic, target-directed behavior in the vocal tract resonance is incorporated into the model design, training, and in likelihood computation. The principal advantage of the new model over the conventional HMM is the use of a compact, internal structure that parsimoniously represents long-span context dependence in the observable domain of speech acoustics without using additional, context-dependent model parameters. The new model is formulated mathematically as a constrained, nonstationary, and nonlinear dynamic system, for which a version of the generalized EM algorithm is developed and implemented for automatically learning the compact set of model parameters. A series of experiments for speech recognition and model synthesis using spontaneous speech data from the Switchboard corpus are reported. The promise of the new model is demonstrated by showing its consistently superior performance over a state-of-the-art benchmark HMM system under controlled experimental conditions. Experiments on model synthesis and analysis shed insight into the mechanism underlying such superiority in terms of the target-directed behavior and of the long-span context-dependence property, both inherent in the designed structure of the new dynamic model of speech.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

Speech and speaker recognition technology has made very significant progress in the past 50 years. The progress can be summarized by the following changes: (1) from template matching to corpus-base statistical modeling, e.g., HMM and n-grams, (2) from filter bank/spectral resonance to Cepstral features (Cepstrum + DCepstrum + DDCepstrum), (3) from heuristic time-normalization to DTW/DP matching, (4) from gdistanceh-based to likelihood-based methods, (5) from maximum likelihood to discriminative approach, e.g., MCE/GPD and MMI, (6) from isolated word to continuous speech recognition, (7) from small vocabulary to large vocabulary recognition, (8) from context-independent units to context-dependent units for recognition, (9) from clean speech to noisy/telephone speech recognition, (10) from single speaker to speaker-independent/adaptive recognition, (11) from monologue to dialogue/conversation recognition, (12) from read speech to spontaneous speech recognition, (13) from recognition to understanding, (14) from single-modality (audio signal only) to multi-modal (audio/visual) speech recognition, (15) from hardware recognizer to software recognizer, and (16) from no commercial application to many practical commercial applications. Most of these advances have taken place in both the fields of speech recognition and speaker recognition. The majority of technological changes have been directed toward the purpose of increasing robustness of recognition, including many other additional important techniques not noted above.

Recognizing emotions using natural or spontaneous speech are extremely difficult compared to doing the same for acted or elicited speeches. Speech emotion recognition for real conversation such as spontaneous speech requires linguistic information of the speech to be included in the speech emotion recognition component to achieve a high recognition rate. However, with the lack of digital speech resources of an under-resourced language, this requirement poses a problem. In this paper, speech emotion recognition of spontaneous speech in Malay language using prosodic features and Random Forest classifier is presented. We also investigate the influence of age categorized as children, young adults and middle-aged on emotion recognition. Ninety spontaneous speech sentences from 30 native speakers of Malay language are collected and classified into three emotions, which are happy, angry and sad. Results show that the spontaneous speech of middle-aged group achieved the highest accuracy rate followed by children age group and finally the young adults. While sad emotions are recognized satisfactorily across all age groups, confusions exist between happy and angry emotions.

This paper presents a progress report on a relatively difficult ASR task on a spontaneous speech corpus - Mandarin Conversational Dialogue Corpus (MCDC). A DNN-based acoustic model is constructed based on the CLDNN structure with a large dataset that comprises two spontaneous-speech corpora and one read-speech corpus. The study uses a large text dataset formed by seven corpora to train an efficient general language model (LM). Two adapted LMs specially for spontaneous speech recognition are also constructed. Experimental results showed that the best performances of 26.3% in character error rate (CER) and 32.5% in word error rate (WER) were reached on MCDC. They represented 27.9% and 22.2% of relative CER and WER reductions as compared with the performances by the previous best HMM-based method. This confirms that the proposed method is promising in tackling on Mandarin spontaneous speech recognition.

This paper concerns effective speaker adaptation methods to solve the over-training problem in speaker verification, which frequently occurs when modeling a speaker with sparse training data. While various speaker adaptations have already been applied to speech recognition, these methods have not yet been formally considered in speaker verification. This paper proposes speaker adaptation methods using a combination of maximum a posteriori (MAP) and maximum likelihood linear regression (MLLR) adaptations, which are successfully used in speech recognition, and applies to speaker verification. Our aim is to remedy the small training data problem by investigating effective speaker adaptations for speaker modeling. Experimental results show that the speaker verification system using a weighted MAP and MLLR adaptation outperforms that of the conventional speaker models without adaptation by a factor of up to 5 times. From these results, we show that the speaker adaptation method achieves significantly better performance even when only small training data is available for speaker verification.

In this work, we deal with advanced context-dependent automatic speech recognition (ASR) of Czech spontaneous talk using hidden Markov models (HMM). Context-dependent units (e.g. triphones, diphones) in ASR systems provide significant improvement against simple non-context-dependent units. However, for spontaneous speech recognition we had to overcome some very challenging tasks. For one, the number of syllables compared to the size of spontaneous speech corpus makes the usage of context-dependent units very difficult. The main part of this article shows problems and procedures to effectively build and use a syllable-based ASR with the LASER (ASR system developed at Department of Computer Science and Engineering, Faculty of Applied Sciences). The procedures are usable with virtual any modern ASR.

Masked Language Models (MLM) are self-supervised neural networks trained to fill in the blanks in a given sentence with masked tokens. Despite the tremendous success of MLMs for various text based tasks, they are not robust for spoken language understanding, especially for spontaneous conversational speech recognition noise. In this work we introduce Warped Language Models (WLM) in which input sentences at training time go through the same modifications as in MLM, plus two additional modifications, namely inserting and dropping random tokens. These two modifications extend and contract the sentence in addition to the modifications in MLMs, hence the word "warped" in the name. The insertion and drop modification of the input text during training of WLM resemble the types of noise due to Automatic Speech Recognition (ASR) errors, and as a result WLMs are likely to be more robust to ASR noise. Through computational results we show that natural language understanding systems built on top of WLMs perform better compared to those built based on MLMs, especially in the presence of ASR errors.

Human-computer interaction for virtual-real fusion

The process of speaker recognition in a noisy and distant environment is a difficult task as it faces numerous challenges before clean speaker speech signal reaching the microphone. While developing a deep neural network for robust functioning in extreme conditions, the selection of a perfectly compatible optimizer, loss function, and dropout is necessary. This paper presents a comparative study of the optimization process in the neural network, how loss function effectively unites in seeking the optimizer. It emphasizes on the selection of the number of input nodes, hidden layers, and time consumed by each set of selections. This study elaborates the accuracy obtained at different combinations of parameters for robust deep neural network structure. This paper is classified under speaker and speech recognition process into improving accuracy. Experiment results shows that Adam optimizer with 150 epochs offers around 95% accuracy for speaker classification under the noisy condition at different SNR values.

For communication, speech is one of the natural forms. A person’s voice contains various parameters that convey information such as emotions, gender, attitude, health, and identity. Determination of these parameters will help to further develop the technology into a reliable and consistent means of identification using speaker recognition. Speaker recognition technologies have wide application areas especially in authentication; surveillance and forensic speaker recognition. In addition, speaker recognition refers to the automated method of identifying or confirming the identity of an individual based on his/her voice. Speech recognition strips out the personal differences to detect the words. Speaker recognition typically disregards the language and meaning to detect the physical person behind the speech. Speech recognition is language-dependent, while Speaker recognition is independent of language. In essence, voice biometrics provides speaker recognition rather than speech recognition. The most accepted form of identification for a human is his/her speech signal. Principally the speaker recognition is the computing task of validating a user’s claimed identity 108using characteristics extracted from their voice. The speaker recognition process based on a speech signal is treated as one of the most exciting technologies of human recognition. For Speaker identification activities we mainly emphasize the physical features of signal. Speakers could be categorized as speaker identification and speaker verification. In speaker identification, the obtained features are compared with all the speaker’s features which are stored in a voice model database and in speaker verification the obtained features are only compared with the stored features of the speaker he/she claimed to be. In this chapter the general principles of speaker recognition, methodology, and applications are discussed.