End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource setting

We present a comprehensive study on building and adapting RNN transducer (RNN-T) models for spoken language understanding (SLU). These end-to-end (E2E) models are constructed in three practical settings: a case where verbatim transcripts are available, a constrained case where the only available annotations are SLU labels and their values, and a more restrictive case where transcripts are available but not corresponding audio. We show how RNN-T SLU models can be developed starting from pre-trained automatic speech recognition (ASR) systems, followed by an SLU adaptation step. In settings where real audio data is not available, artificially synthesized speech is used to successfully adapt various SLU models. When evaluated on two SLU data sets, the ATIS corpus and a customer call center data set, the proposed models closely track the performance of other E2E models and achieve state-of-the-art results.

Spoken language understanding (SLU) in dialog systems is generally performed using a natural language understanding (NLU) model based on the hypotheses produced by an automatic speech recognition (ASR) system. However, when new spoken dialog applications are built from scratch in real user environments that often have sub-optimal audio characteristics, ASR performance can suffer due to factors such as the paucity of training data or a mismatch between the training and test data. To address this issue, this paper proposes an ASR-free, end-to-end (E2E) modeling approach to SLU for a cloud-based, modular spoken dialog system (SDS). We evaluate the effectiveness of our approach on crowdsourced data collected from non-native English speakers interacting with a conversational language learning application. Experimental results show that our approach is particularly promising in situations with low ASR accuracy. It can further improve the performance of a sophisticated CNN-based SLU system with more accurate ASR hypotheses by fusing the scores from E2E system, i.e., the overall accuracy of SLU is improved from 85.6% to 86.5%.

In this paper, we present an approach to Spoken Language Understanding (SLU) where we perform a combination of multiple hypotheses from several Automatic Speech Recognizers (ASRs) in order to reduce the impact of recognition errors in the SLU module. This combination is performed using a Grammatical Inference algorithm that provides a generalization of the input sentences by means of a weighted graph of words. We have also developed a specific SLU algorithm that is able to process these graphs of words according to a stochastic semantic modelling.The results show that the combinations of several hypotheses from the ASR module outperform the results obtained by taking just the 1-best transcription.

Spoken language understanding (SLU) topic has seen a lot of progress these last three years, with the emergence of end-to-end neural approaches. Spoken language understanding refers to natural language processing tasks related to semantic extraction from speech signal, like named entity recognition from speech or slot filling task in a context of human-machine dialogue. Classically, SLU tasks were processed through a cascade approach that consists in applying, firstly, an automatic speech recognition process, followed by a natural language processing module applied to the automatic transcriptions. These three last years, end-to-end neural approaches, based on deep neural networks, have been proposed in order to directly extract the semantics from speech signal, by using a single neural model. More recent works on self-supervised training with unlabeled data open new perspectives in term of performance for automatic speech recognition and natural language processing. In this paper, we present a brief overview of the recent advances on the French MEDIA benchmark dataset for SLU, with or without the use of additional data. We also present our last results that significantly outperform the current state-of-the-art with a Concept Error Rate (CER) of 11.2%, instead of 13.6% for the last state-of-the-art system presented this year.

Expanding Internet connectivity has had tremendous influence on people’s everyday life, since they do everything on their phones and laptops [1]. Several items have been developed by various researchers in order to improve the lives of people, notably the elderly and disabled, while remaining technologically advanced. Voice-command-enabled technologies, such as SIRI and Google voice commands, are the most useful. These systems are based on the Speech recognition module, which is one of the most important module that can make human-machine communication easier. Automatic Speech Recognition (ASR) has made significant progress toward human-like performance by employing a data-driven method [2]. In this paper, we created an Arabic voice command dataset which include 10 commands spoken by 10 speaker and repeated 10 times. The obtained dataset, despite its size, was evaluated on four speech processing tasks and achieved an accuracy of 95.9% in ASR, and a macro F1score of 99.67%, 100%, 100%, and 97.98%, in speaker identification, gender recognition, accent recognition, and spoken language understanding, respectively.

Spoken Language Understanding (SLU) is a part of spoken dialogue system (SDS) which represents speech data into semantic form required to understand user's intention. The human-computer interaction system is most commonly placed in noisy environment. This introduces error in Automatic Speech Recognizer (ASR). SLU takes into account ASR errors. Thus robustness of SLU plays important role in efficient working of human computer interaction system. The SLU consist of semantic decoder and belief tracker. Recent line of research models semantic decoder as a classification task or sequence to sequence learning. The SLU models incorporate learning models for classification. The learning model requires lot of labelled data for training. The process of obtaining labelled data is expensive and time consuming. Thus it is desirable that SLU must be able to maintain its accuracy even if sparse training data is available. The objective of this research is to evaluate robustness in terms of accuracy and data sparsity of SLU incorporating Relevance Vector Classifier (RVC) and Stochastic Gradient Descent (SGD) classifier. The framework of SLU incorporates semantic tuple classifier (STC). The STC performs dialogue act classification and slot filling by using classifiers. The classifiers that were incorporated in semantic decoder are Support Vector Machine (SVM), SGD and RVC. The dataset used is DSTC II. The SVM classifier showed maximum robustness and better accuracy as compared to other classifiers. For SVM, classifier the Inter Cross Entropy (ICE) and accuracy were 1.0 and 0.91, respectively. For RVC, ICE and accuracy were 4.055 and 0.52.

The comprehension of spoken language has been characterized by a number of “local” theories that have focused on specific aspects of the task: models of word recognition, models of selective attention, accounts of thematic role assignment at the sentence level, and so forth. The ease of language understanding (ELU) model (Rönnberg et al., 2013) stands as one of the few attempts to offer a fully encompassing framework for language understanding. In this paper we discuss interactions between perceptual, linguistic, and cognitive factors in spoken language understanding. Central to our presentation is an examination of aspects of the ELU model that apply especially to spoken language comprehension in adult aging, where speed of processing, working memory capacity, and hearing acuity are often compromised. We discuss, in relation to the ELU model, conceptions of working memory and its capacity limitations, the use of linguistic context to aid in speech recognition and the importance of inhibitory control, and language comprehension at the sentence level. Throughout this paper we offer a constructive look at the ELU model; where it is strong and where there are gaps to be filled.

Spoken Language Understanding (SLU) aims to extract the semantics frame of user queries, which is a core component in a task-oriented dialog system. With the burst of deep neural networks and the evolution of pre-trained language models, the research of SLU has obtained significant breakthroughs. However, there remains a lack of a comprehensive survey summarizing existing approaches and recent trends, which motivated the work presented in this article. In this paper, we survey recent advances and new frontiers in SLU. Specifically, we give a thorough review of this research field, covering different aspects including (1) new taxonomy: we provide a new perspective for SLU filed, including single model vs. joint model, implicit joint modeling vs. explicit joint modeling in joint model, non pre-trained paradigm vs. pretrained paradigm; (2) new frontiers: some emerging areas in complex SLU as well as the corresponding challenges; (3) abundant open-source resources: to help the community, we have collected, organized the related papers, baseline projects and leaderboard on a public website where SLU researchers could directly access to the recent progress. We hope that this survey can shed a light on future research in SLU field.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

This paper describes an effective and robust approach based on finite state word network for Mandarin spoken language understanding (SLU) in specific domain. A kind of syntax for grammar representation is defined to efficiently specify the utterances which may be spoken in a task. Moreover, arbitrary semantic meaning can be added into grammars conveniently. Then, the grammars are complied into a finite state word network, which contains both literal and semantic information defined by the grammars. A robust parser is implemented based on 3-dimensional dynamic programming. Given a transcription from an automatic speech recognition (ASR) system, the parser searches for the best path in the word network that matches the recognition text most closely. The semantic meaning of the transcription can then be extracted from the best path. Experimental results demonstrate the good performance and robustness of the proposed approach on a Mandarin SLU task.

In the process of spoken language understanding (SLU), the first step is to label semantic concept (or class). Since there are variations induced by automatic speech recognition (ASR) or missing field of keywords in limited domain, extracting the main semantic concepts will be the key problem in SLU. In this paper, we utilize some effective features based on conditional random field (CRF) to segment Chinese named entities, which include people's name and other names in domain. The goal of this paper is to acquire the key semantic concept regardless of whether it is a complete or fragment concept. Experiments show that the whole of performance of named entities recognition arrives at 96.67%.

Spoken language understanding (SLU), which is a core component of the task-oriented dialogue system, has made substantial progress in the research of single-turn dialogue. However, the performance in multi-turn dialogue is still not satisfactory in the sense that the existing multi-turn SLU methods have low portability and compatibility for other single-turn SLU models. Further, existing multi-turn SLU methods do not exploit the historical predicted results when predicting the current utterance, which wastes helpful information. To gap those shortcomings, in this paper, we propose a novel Result-based Portable Framework for SLU (RPFSLU). RPFSLU allows most existing single-turn SLU models to obtain the contextual information from multi-turn dialogues and takes full advantage of predicted results in the dialogue history during the current prediction. Experimental results on the public dataset KVRET have shown that all SLU models in baselines acquire enhancement by RPFSLU on multi-turn SLU tasks.

The ATIS (Air Travel Information Service) corpus will be soon celebrating its 30th birthday. Designed originally to benchmark spoken language systems, it still represents the most well-known corpus for benchmarking Spoken Language Understanding (SLU) systems. In 2010, in a paper titled What is left to be understood in ATIS? [1], Tur et al. discussed the relevance of this corpus after more than 10 years of research on statistical models for performing SLU tasks. Nowadays, in the Deep Neural Network (DNN) era, ATIS is still used as the main benchmark corpus for evaluating all kinds of DNN models, leading to further improvements, although rather limited, in SLU accuracy compared to previous state-of-the-art models. We propose in this paper to investigate these results obtained on ATIS from a qualitative point of view rather than just a quantitative point of view and answer the two following questions: what kind of qualitative improvement brought DNN models to SLU on the ATIS corpus? Is there anything left, from a qualitative point of view, in the remaining 5% of errors made by current state-of-the-art models?

Decoding speaker's intent is a crucial part of spoken language understanding (SLU). The presence of noise or errors in the text transcriptions, in real life scenarios make the task more challenging. In this paper, we address the spoken language intent detection under noisy conditions imposed by automatic speech recognition (ASR) systems. We propose to employ confusion2vec word feature representation to compensate for the errors made by ASR and to increase the robustness of the SLU system. The confusion2vec, motivated from human speech production and perception, models acoustic relationships between words in addition to the semantic and syntactic relations of words in human language. We hypothesize that ASR often makes errors relating to acoustically similar words, and the confusion2vec with inherent model of acoustic relationships between words is able to compensate for the errors. We demonstrate through experiments on the ATIS benchmark dataset, the robustness of the proposed model to achieve state-of-the-art results under noisy ASR conditions. Our system reduces classification error rate (CER) by 20.84% and improves robustness by 37.48% (lower CER degradation) relative to the previous state-of-the-art going from clean to noisy transcripts. Improvements are also demonstrated when training the intent detection models on noisy transcripts.

In the chapter we describe procedures for Croatian speech recognition which are used in a limited domain spoken dialog system for Croatian speech. The dialog system would provide information about weather in different regions of Croatia for different time periods (Žibert et al., 2003). The spoken dialog system includes modules for automatic speech recognition (ASR), spoken language understanding and text-to-speech synthesis. In this work ASR module based on data-driven statistical and rule-based knowledge approach is discussed.