Combining Several ASR Outputs in a Graph-Based SLU System

End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting

Spoken language understanding (SLU) in dialog systems is generally performed using a natural language understanding (NLU) model based on the hypotheses produced by an automatic speech recognition (ASR) system. However, when new spoken dialog applications are built from scratch in real user environments that often have sub-optimal audio characteristics, ASR performance can suffer due to factors such as the paucity of training data or a mismatch between the training and test data. To address this issue, this paper proposes an ASR-free, end-to-end (E2E) modeling approach to SLU for a cloud-based, modular spoken dialog system (SDS). We evaluate the effectiveness of our approach on crowdsourced data collected from non-native English speakers interacting with a conversational language learning application. Experimental results show that our approach is particularly promising in situations with low ASR accuracy. It can further improve the performance of a sophisticated CNN-based SLU system with more accurate ASR hypotheses by fusing the scores from E2E system, i.e., the overall accuracy of SLU is improved from 85.6% to 86.5%.

Spoken Language Understanding (SLU) is a part of spoken dialogue system (SDS) which represents speech data into semantic form required to understand user's intention. The human-computer interaction system is most commonly placed in noisy environment. This introduces error in Automatic Speech Recognizer (ASR). SLU takes into account ASR errors. Thus robustness of SLU plays important role in efficient working of human computer interaction system. The SLU consist of semantic decoder and belief tracker. Recent line of research models semantic decoder as a classification task or sequence to sequence learning. The SLU models incorporate learning models for classification. The learning model requires lot of labelled data for training. The process of obtaining labelled data is expensive and time consuming. Thus it is desirable that SLU must be able to maintain its accuracy even if sparse training data is available. The objective of this research is to evaluate robustness in terms of accuracy and data sparsity of SLU incorporating Relevance Vector Classifier (RVC) and Stochastic Gradient Descent (SGD) classifier. The framework of SLU incorporates semantic tuple classifier (STC). The STC performs dialogue act classification and slot filling by using classifiers. The classifiers that were incorporated in semantic decoder are Support Vector Machine (SVM), SGD and RVC. The dataset used is DSTC II. The SVM classifier showed maximum robustness and better accuracy as compared to other classifiers. For SVM, classifier the Inter Cross Entropy (ICE) and accuracy were 1.0 and 0.91, respectively. For RVC, ICE and accuracy were 4.055 and 0.52.

We present a comprehensive study on building and adapting RNN transducer (RNN-T) models for spoken language understanding (SLU). These end-to-end (E2E) models are constructed in three practical settings: a case where verbatim transcripts are available, a constrained case where the only available annotations are SLU labels and their values, and a more restrictive case where transcripts are available but not corresponding audio. We show how RNN-T SLU models can be developed starting from pre-trained automatic speech recognition (ASR) systems, followed by an SLU adaptation step. In settings where real audio data is not available, artificially synthesized speech is used to successfully adapt various SLU models. When evaluated on two SLU data sets, the ATIS corpus and a customer call center data set, the proposed models closely track the performance of other E2E models and achieve state-of-the-art results.

Spoken language understanding (SLU) topic has seen a lot of progress these last three years, with the emergence of end-to-end neural approaches. Spoken language understanding refers to natural language processing tasks related to semantic extraction from speech signal, like named entity recognition from speech or slot filling task in a context of human-machine dialogue. Classically, SLU tasks were processed through a cascade approach that consists in applying, firstly, an automatic speech recognition process, followed by a natural language processing module applied to the automatic transcriptions. These three last years, end-to-end neural approaches, based on deep neural networks, have been proposed in order to directly extract the semantics from speech signal, by using a single neural model. More recent works on self-supervised training with unlabeled data open new perspectives in term of performance for automatic speech recognition and natural language processing. In this paper, we present a brief overview of the recent advances on the French MEDIA benchmark dataset for SLU, with or without the use of additional data. We also present our last results that significantly outperform the current state-of-the-art with a Concept Error Rate (CER) of 11.2%, instead of 13.6% for the last state-of-the-art system presented this year.

Decoding speaker's intent is a crucial part of spoken language understanding (SLU). The presence of noise or errors in the text transcriptions, in real life scenarios make the task more challenging. In this paper, we address the spoken language intent detection under noisy conditions imposed by automatic speech recognition (ASR) systems. We propose to employ confusion2vec word feature representation to compensate for the errors made by ASR and to increase the robustness of the SLU system. The confusion2vec, motivated from human speech production and perception, models acoustic relationships between words in addition to the semantic and syntactic relations of words in human language. We hypothesize that ASR often makes errors relating to acoustically similar words, and the confusion2vec with inherent model of acoustic relationships between words is able to compensate for the errors. We demonstrate through experiments on the ATIS benchmark dataset, the robustness of the proposed model to achieve state-of-the-art results under noisy ASR conditions. Our system reduces classification error rate (CER) by 20.84% and improves robustness by 37.48% (lower CER degradation) relative to the previous state-of-the-art going from clean to noisy transcripts. Improvements are also demonstrated when training the intent detection models on noisy transcripts.

This paper describes an effective and robust approach based on finite state word network for Mandarin spoken language understanding (SLU) in specific domain. A kind of syntax for grammar representation is defined to efficiently specify the utterances which may be spoken in a task. Moreover, arbitrary semantic meaning can be added into grammars conveniently. Then, the grammars are complied into a finite state word network, which contains both literal and semantic information defined by the grammars. A robust parser is implemented based on 3-dimensional dynamic programming. Given a transcription from an automatic speech recognition (ASR) system, the parser searches for the best path in the word network that matches the recognition text most closely. The semantic meaning of the transcription can then be extracted from the best path. Experimental results demonstrate the good performance and robustness of the proposed approach on a Mandarin SLU task.

In the process of spoken language understanding (SLU), the first step is to label semantic concept (or class). Since there are variations induced by automatic speech recognition (ASR) or missing field of keywords in limited domain, extracting the main semantic concepts will be the key problem in SLU. In this paper, we utilize some effective features based on conditional random field (CRF) to segment Chinese named entities, which include people's name and other names in domain. The goal of this paper is to acquire the key semantic concept regardless of whether it is a complete or fragment concept. Experiments show that the whole of performance of named entities recognition arrives at 96.67%.

Spoken Language Understanding (SLU) systems typically consist of a set of machine learning models that operate in conjunction to produce an SLU hypothesis. The generated hypothesis is then sent to downstream components for further action. However, it is desirable to discard an incorrect hypothesis before sending it downstream. In this work, we present two designs for SLU hypothesis rejection modules: (i) scheme R1 that performs rejection on domain specific SLU hypothesis and, (ii) scheme R2 that performs rejection on hypothesis generated from the overall SLU system. Hypothesis rejection modules in both schemes reject/accept a hypothesis based on features drawn from the utterance directed to the SLU system, the associated SLU hypothesis and SLU confidence score. Our experiments suggest that both the schemes yield similar results (scheme R1: 2.5% FRR @ 4.5% FAR, scheme R2: 2.5% FRR @ 4.6% FAR), with the best performing systems using all the available features. We argue that while either of the rejection schemes can be chosen over the other, they carry some inherent differences which need to be considered while making this choice. Additionally, we incorporate ASR features in the rejection module (obtaining an 1.9% FRR @ 3.8% FAR) and analyze the improvements.

Voice search is the technology that enables users to access information using spoken queries. Automatic speech recognizer (ASR) is one of the key modules for voice search systems. However, the high error rate of the state-of-the-art large vocabulary continuous speech recognition (LVCSR) is the bottleneck for most voice search systems. In this paper, we first build a baseline system using language model (LM) with domain-specific information. To improve our system, we propose a forward-backward LVCSR system combination method to decrease the search errors in speech recognition. This also helps to improve the spoken language understanding (SLU) performance. Experiment results show that our proposed method improves the performance of speech recognition by 5.7% relative CER reduction and increases the F1-measure of SLU by 1.5% absolute on our test set.KeywordsVoice searchLVCSRSLUforward-backwardsystem combination

Expanding Internet connectivity has had tremendous influence on people’s everyday life, since they do everything on their phones and laptops [1]. Several items have been developed by various researchers in order to improve the lives of people, notably the elderly and disabled, while remaining technologically advanced. Voice-command-enabled technologies, such as SIRI and Google voice commands, are the most useful. These systems are based on the Speech recognition module, which is one of the most important module that can make human-machine communication easier. Automatic Speech Recognition (ASR) has made significant progress toward human-like performance by employing a data-driven method [2]. In this paper, we created an Arabic voice command dataset which include 10 commands spoken by 10 speaker and repeated 10 times. The obtained dataset, despite its size, was evaluated on four speech processing tasks and achieved an accuracy of 95.9% in ASR, and a macro F1score of 99.67%, 100%, 100%, and 97.98%, in speaker identification, gender recognition, accent recognition, and spoken language understanding, respectively.

In the chapter we describe procedures for Croatian speech recognition which are used in a limited domain spoken dialog system for Croatian speech. The dialog system would provide information about weather in different regions of Croatia for different time periods (Žibert et al., 2003). The spoken dialog system includes modules for automatic speech recognition (ASR), spoken language understanding and text-to-speech synthesis. In this work ASR module based on data-driven statistical and rule-based knowledge approach is discussed.

Spoken language understanding (SLU), which converts user requests in natural language to machine-interpretable expressions, is becoming an essential task. The lack of training data is an important problem, especially for new system tasks, because existing SLU systems are based on statistical approaches. In this paper, we proposed to use two sources of the "wisdom of crowds," crowdsourcing and knowledge community website, for improving the SLU system. We firstly collected paraphrasing variations for new system tasks through crowdsourcing as seed data, and then augmented them using similar questions from a knowledge community website. We investigated the effects of the proposed data augmentation method in SLU task, even with small seed data. In particular, the proposed architecture augmented more than 120,000 samples to improve SLU accuracies.

Human-computer interaction (HCI) is significantly impacted by delayed responses from a spoken dialogue system. Hence, end-to-end (e2e) spoken language understanding (SLU) solutions have recently been proposed to decrease latency. Such approaches allow for the extraction of semantic information directly from the speech signal, thus bypassing the need for a transcript from an automatic speech recognition (ASR) system. In this paper, we propose a compact e2e SLU architecture for streaming scenarios, where chunks of the speech signal are processed continuously to predict intent and slot values. Our model is based on a 3D convolutional neural network (3D-CNN) and a unidirectional long short-term memory (LSTM). We compare the performance of two alignment-free losses: the connectionist temporal classification (CTC) method and its adapted version, namely connectionist temporal localization (CTL). The latter performs not only the classification but also localization of sequential audio events. The proposed solution is evaluated on the Fluent Speech Command dataset and results show our model ability to process incoming speech signal, reaching accuracy as high as 98.97 % for CTC and 98.78 % for CTL on single-label classification, and as high as 95.69 % for CTC and 95.28 % for CTL on two-label prediction.

This paper describes an alarm detection system dedicated to process automatically customer feedbacks in call-centers. Previous studies presented a strategy that consists in the robust detection of subjective opinions about a particular topic in a spoken message. In the present study, we focus on the alarm detection problem in a customer spoken feedback application. We want to characterize each customer's survey with a degree of emergency. All the messages considered as urgent need a quick answer from the call-center service in order to satisfy the customer. The strategy proposed is based on a classification scheme that takes into account all the features that can characterize a survey: answers to the closed questions, topics and opinions detected in the open question spoken message, confidence scores from the Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) modules. A field trial realized among real customers has shown that despite the ASR robustness issues, our system efficiently ranks the most urgent messages and brings a finer analysis on the surveys than the one provided by processing the closed questions alone.