HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management

Pre-trained encoder-decoder models are widely applied in Task-Oriented Dialog (TOD) systems on the session level, mainly focusing on modeling the dialog semantic information. Dialogs imply structural information indicating the interaction among user utterances, belief states, database search results, system acts and responses, which is also crucial for TOD systems. In addition, for the system acts, additional pre-training and datasets are considered to improve their accuracies, undoubtedly introducing a burden. Therefore, a novel end-to-end TOD system named Winnie is proposed in this paper to improve the TOD performance. First, to make full use of the intrinsic structural information, supervised contrastive learning is adopted to narrow the gap in the representation space between text representations of the same category and enlarge the overall continuous representation margin between text representations of different categories in dialog context. Then, a system act classification task is introduced for policy optimization during fine-tuning. Empirical results show that Winnie substantially improves the performance of the TOD system. By introducing the supervised contrastive and system act classification losses, Winnie achieves state-of-the-art results on benchmark datasets, including MultiWOZ2.2, In-Car, and Camrest676. Their end-to-end combined scores are improved by 3.2, 1.9, and 1.1 points, respectively.

Large language models (LLMs) like ChatGPT have demonstrated the ability to perform a variety of natural language processing (NLP) tasks. However, it's unclear whether ChatGPT can serve as a task-oriented dialogue system. In this paper, we evaluate the impact of ChatGPT on task-oriented dialogue (TOD) systems and perform a comprehensive analysis to learn its benefits and challenges. We find that ChatGPT performs well on relatively simple dialogue understanding tasks such as intent detection and slot filling, but fails to understand complex multi-turn conversations and interact with Knowledge Base(KB) in dialogue state tracking and response generation. Future LLM-based TOD work should pay more attention to (1) incorporating domain knowledge (2) understanding complex instructions (3) modeling long-term memory (4) interacting with external knowledge bases.

Task-Oriented Dialogue (TOD) systems are designed to carry out specific tasks by tracking dialogue states and generating appropriate responses to help users achieve defined goals. Recently, end-to-end dialogue models pre-trained based on large datasets have shown promising performance in the conversational system. However, they share the same parameters to train tasks of the dialogue system (NLU, DST, NLG), so debugging each task is challenging. Also, they require a lot of effort to fine-tune large parameters to create a task-oriented chatbot, making it difficult for non-experts to handle. Therefore, we intend to train relatively lightweight and fast models compared to PLM. In this paper, we propose an End-to-end TOD system with Task-Optimized Adapters which learn independently per task, adding only small number of parameters after fixed layers of pre-trained network. We also enhance the performance of the DST and NLG modules through reinforcement learning, overcoming the learning curve that has lacked at the adapter learning and enabling the natural and consistent response generation that is appropriate for the goal. Our method is a model-agnostic approach and does not require prompt-tuning as only input data without a prompt. As results of the experiment, our method shows competitive performance on the MultiWOZ benchmark compared to the existing end-to-end models. In particular, we attain state-of-the-art performance on the DST task of 2.2 dataset.

Task-oriented dialogue (TOD) systems typically retrieve domain-specific knowledge from external knowledge bases to generate accurate responses. Most existing works focus on understanding unimodal textual context and generating text-only responses. This reliance on text-modality information overlooks the richness of real-world multi-modal data, thereby limiting the effectiveness of TOD systems. To address this problem, we propose a unified multi-modal context-knowledge model named MIND. MIND employs a text-image alignment strategy to achieve cross-modal understanding between textual and visual modalities. In addition, a multi-modal TOD dataset named MuCoK has been specifically designed for dialogue objectives in real-world multi-modal scenarios. To the best of our knowledge, MIND is the first TOD model capable of understanding multi-modal dialogue contexts, enabling multi-modal knowledge retrieval, and generating coherent multi-modal responses. Experiment results on two benchmarks demonstrate that our MIND achieves state-of-the-art performance across all evaluation metrics, surpassing even the multi-modal large language models.

Natural language understanding is a critical module in task-oriented dialogue systems. Recently, state-of-the-art approaches use deep learning methods and transformers to improve the performance of dialogue systems. In this work, we propose a natural language understanding model with a specific-shopping named entity recognizer using a joint learning-based BERT transformer for task-oriented dialogue systems in the Persian Language. Since there is no published available dataset for Persian online shopping dialogue systems, to tackle the lack of data, we propose two methods for generating training data: fully-simulated and semi-simulated method. We created a simulated dataset with a hybrid of rule-based and template-based generation methods and a semi-simulated dataset where the language generation part is done by a human to increase the quality of the dataset. Our experiments with the natural language understanding module show that a combination of the datasets can improve results. These dataset generation methods can apply in other domains for low-resource languages in task-oriented dialogue systems too to solve the cold start problem of datasets.

Even as progress in speech technologies and task and dialog modeling has allowed the development of advanced spoken dialog systems, the low-level interaction behavior of those systems often remains rigid and inefficient. Based on an analysis of human-human and human-computer turn-taking in naturally occurring task-oriented dialogs, we define a set of features that can be automatically extracted and show that they can be used to inform efficient end-of-turn detection. We then frame turn-taking as decision making under uncertainty and describe the Finite-State Turn-Taking Machine (FSTTM), a decision-theoretic model that combines data-driven machine learning methods and a cost structure derived from Conversation Analysis to control the turn-taking behavior of dialog systems. Evaluation results on CMU Let's Go, a publicly deployed bus information system, confirm that the FSTTM significantly improves the responsiveness of the system compared to a standard threshold-based approach, as well as previous data-driven methods.

Most prior work on task-oriented dialogue systems are restricted to limited coverage of domain APIs. However, users oftentimes have requests that are out of the scope of these APIs. This work focuses on responding to these beyond-API-coverage user turns by incorporating external, unstructured knowledge sources. Our approach works in a pipelined manner with knowledge-seeking turn detection, knowledge selection, and response generation in sequence. We introduce novel data augmentation methods for the first two steps and demonstrate that the use of information extracted from dialogue context improves the knowledge selection and end-to-end performances. Through experiments, we achieve state-of-the-art performance for both automatic and human evaluation metrics on the DSTC9 Track 1 benchmark dataset, validating the effectiveness of our contributions.

Building task-oriented dialogue systems has become a topic of interest in the research community and industry. The task-oriented dialogue system is a closed-domain dialogue system that can perform specific tasks for users. The natural language understanding module of a task-oriented dialogue system is crucial because it is related to a task-oriented dialogue system that provides correctional services for users. The natural language understanding module of a task-oriented dialogue system performs two tasks: intent detection and slot filling. The intent detection task can be regarded as a text classification task; a classification model is trained to predict the intention of the user from the user’s input information. The slot filling task can be regarded as a sequence analysis task; a sequence analysis model is trained to predict the details of the user’s intention. In this paper, we proposed a novel model based on a transformer encoder for intent detection and slot filling. It follows the encoder-decoder structure, including a vanilla Transformer encoder, a bidirectional LSTM encoder, a linear classification decoder for intent detection, and a conditional random field decoder for slot filling. The experimental results on two public datasets show that our proposed model outperforms the existing methods based on the Transformer and can be combined with BERT to achieve better intent detection and slot filling results.

Recent handcrafts on dialog manager in task-oriented dialog systems (TODS) offer great promises on handling conversations. However, most tend to be shortsighted in handling advancing conversations. Modelling the future direction on conversations is crucial for TODS that can be scaled across multi-domain. This paper proposes a novel architectural model for the dialog manager, (MAS_DM). In this model, the dialog manager is a MAS. The architecture consists of multiple intelligent interacting agents, namely, state agent, master agent, and dialog agents. Each agent performs a set of tasks to achieve the overall goal of advancing the conversation within a topic. In this paper, the particular component of the Dialogue Manager, and Strategy selection has been discussed in detail. The notion of learning is essential, since it is intended to provide a means to which the agents will adapt to their environment. We show how to combine MAS and RL to enable agents learn a topic of interest and support an advancing conversation on the same. This will enable the realization of advancing conversations between a human and the TODS on a given topic.

Last few decades have witnessed substantial breakthroughs on several areas of speech and language understanding research, specifically for building human to machine conversational dialog systems. Dialog systems, also known as interactive conversational agents, virtual agents or sometimes chatbots, are useful in a wide range of applications ranging from technical support services to language learning tools and entertainment. Recent success in deep neural networks has spurred the research in building data-driven dialog models. In this chapter, we present state-of-the-art neural network architectures and details on each of the components of building a successful dialog system using deep learning. Task-oriented dialog systems would be the focus of this chapter, and later different networks are provided for building open-ended non-task-oriented dialog systems. Furthermore, to facilitate research in this area, we have a survey of publicly available datasets and software tools suitable for data-driven learning of dialog systems. Finally, appropriate choice of evaluation metrics are discussed for the learning objective.

Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system's capabilities via strict user goals, namely "user familiarity" bias.This data bias deepens when it combines with datadriven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations.Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios.In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (opengoal).Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues.Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation.From this, we discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities.We discuss its characteristics and toxicity while showing recent large language models can also suffer from this behavior. Work done as a research intern at NAVER AI Lab. Work done while at NAVER AI Lab. High User Familiarity with the System Low User Familiarity with the System(I should provide price range, food type, area (north, east, south, west), and booking info) Hi! Please book a fancy

End-to-end task-oriented dialogue systems rely heavily on an understanding of the dialog history. This often faces the challenge of inferring which dialog history information is critical to generating responses. In this paper, we address this challenge by leveraging a dialog history manager component that dynamically focuses on dialog history memory. It performs multiple add and forget operations by fusing an enhanced entity representation of dialog history and Knowledge Base (KB) information as queries, remembering entities relevant to responses and filtering out unimportant information. Experimental results on an open task-oriented dialogue dataset show that our model outperforms the baseline system in terms of effectiveness and produces contextually consistent responses.

Automated task-oriented dialog based system, generally stated as Chatbot, is widely used nowadays by service-oriented platforms such as banking, mobile service providers and travel management firms. The most imperative part of the task-oriented dialog system is to distinguish the intent of the queries asked. If the system erroneously identifies the intent of the query, then the given answer is either incorrect or not related to the query asked. This raises the risk of deteriorating the reliability of the entire system and the organization. Such kind of systems struggle when a user asks queries that contain words for which training classes are not available. These classes may be termed as unseen classes. Our aim is to find the unseen classes in an automated task-oriented dialog system. This paper focuses on open-world learning. Specifically, we propose a deep learning-based Intent Identification framework, OWI, to identify unseen classes for an automated dialog-based system. The OWI framework is based on convolutional neural network with a 1-vs-rest output layer to identify the unseen classes. The proposed model is evaluated on various performance matrices. In addition, we compare OWI with an existing state-of-the-art model. The experimental results show that the OWI outperforms the existing model with respect to identifying unseen classes.

An indispensable component in task-oriented dialogue systems is the dialogue state tracker, which keeps track of users' intentions in the course of conversation. The typical approach towards this goal is to fill in multiple pre-defined slots that are essential to complete the task. Although various dialogue state tracking methods have been proposed in recent years, most of them predict the value of each slot separately and fail to consider the correlations among slots. In this paper, we propose a slot self-attention mechanism that can learn the slot correlations automatically. Specifically, a slot-token attention is first utilized to obtain slot-specific features from the dialogue context. Then a stacked slot self-attention is applied on these features to learn the correlations among slots. We conduct comprehensive experiments on two multi-domain task-oriented dialogue datasets, including MultiWOZ 2.0 and MultiWOZ 2.1. The experimental results demonstrate that our approach achieves state-of-the-art performance on both datasets, verifying the necessity and effectiveness of taking slot correlations into consideration.

Dialogue systems are a popular natural language processing (NLP) task as it is promising in real-life applications. It is also a complicated task since many NLP tasks deserving study are involved. As a result, a multitude of novel works on this task are carried out, and most of them are deep learning based due to their outstanding performance. In this survey, we mainly focus on the deep learning based dialogue systems. We comprehensively review state-of-the-art research outcomes in dialogue systems and analyze them from two angles: model type and system type. Specifically, from the angle of model type, we discuss the principles, characteristics, and applications of different models that are widely used in dialogue systems. This will help researchers acquaint these models and see how they are applied in state-of-the-art frameworks, which is rather helpful when designing a new dialogue system. From the angle of system type, we discuss task-oriented and open-domain dialogue systems as two streams of research, providing insight into the hot topics related. Furthermore, we comprehensively review the evaluation methods and datasets for dialogue systems to pave the way for future research. Finally, some possible research trends are identified based on the recent research outcomes. To the best of our knowledge, this survey is the most comprehensive and up-to-date one at present for deep learning based dialogue systems, extensively covering the popular techniques. We speculate that this work is a good starting point for academics who are new to the dialogue systems or those who want to quickly grasp up-to-date techniques in this area.