Multi-turn dialogue-oriented pretrained question generation model

Numerical expression evaluation from sentences using Natural Language processing is a well-known problem domain. Evaluation of noise-free mathematical expressions has been done previously to assess the ability of the recurrent neural network to solve numerical expressions. However, the evaluation of numerical expressions when present in noisy format, i.e., operator and operands are represented “in-word” and mixed numerical format, are not well nourished till now. This paper presents a comparative study to analyze the ability of sequence-to-sequence neural network based Encoder-Decoder models to calculate the numerical output of the noisy expressions. We have treated the problem as a sequence-to-sequence based classification problem instead of a regression problem as considered earlier, to deal with the noisy data. Performance of three different Encoder-Decoder models namely Unidirectional, Bidirectional, and Attention based Encoder-Decoder LSTM models are used to evaluate the noisy expressions. It has been experimentally observed that Bidirectional and Attention based Encoder-Decoder LSTM models perform better than Unidirectional EncoderDecoder model and the sequence-to-sequence model.

In recent years, Question Generation (QG) has gained significant attention as a research topic, particularly in the context of its potential to support automatic reading comprehension assessment preparation. However, current QG models are mostly trained on factoid-type datasets, which tend to produce questions that are too simple for assessing advanced abilities. One promising alternative is to train QG models on exam-type datasets, which contain questions that require content reasoning. Unfortunately, there is a shortage of such training data compared to factoid-type questions. To address this issue and improve the quality of QG for generating advanced questions, we propose the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Handover QG</i> framework. This framework involves the joint training of exam-type QG and factoid-type QG, and controls the question generation process by interleavingly using the exam-type QG decoder and the factoid-type QG decoder. Furthermore, we employ reinforcement learning to enhance QG performance. Our experimental evaluation shows that our model significantly outperforms the compared baselines, with a BLEU-4 score increase from 5.31 to 6.48. Human evaluation also confirms that the questions generated by our model are answerable and appropriately difficult. Overall, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Handover QG</i> framework offers a promising solution for improving QG performance in generating advanced questions for reading comprehension assessment.

The standard evaluation metric of automatic speech recognition (ASR) is the word error rate (WER), which measures the dissimilarity between recognized word sequences and their ground truth. Many training algorithms designed to reduce sequence-level errors such as WER have been proposed for hidden Markov model (HMM)-based ASR, e.g., state-level minimum Bayes risk (sMBR). However, these approaches cannot be used directly for encoder-decoder model based end-to-end ASR, because the encoder-decoder model employs very different mechanisms from HMM-based approaches. In this paper, we propose a new method for optimizing the encoder-decoder model based on a sequence-level evaluation metric. Since the WER is not directly differentiable, we adopt a policy gradient objective function to train the encoder-decoder model, which enables us to minimize the expected WER of the model predictions. This training method employs the scoring of multiple hypotheses as in the decoding stage while usual cross entropy training uses only the ground truth. Therefore, we can expect it to improve the decoding results of the encoder-decoder model. We perform experiments using the Tedlium corpus to demonstrate the potential of our proposed method for improving the recognition performance of the encoder-decoder model.

The question generation task can automatically generate large-scale questions to provide training data for reading comprehension tasks and QA systems, which are crucial for low-resource languages such as Tibetan. At present, due to the emergence of large-scale datasets and pre-trained language models in the Chinese and English domains, the task of question generation in the Chinese and English domains has been well developed, while the research on question generation in the Tibetan is still in its infancy. The main reason is the lack of datasets and the relatively backward development of various models in Tibetan. To solve the above questions, this paper constructs a Tibetan pre-trained language model TiBERT to provide a basis for the development of various downstream tasks, to expand the datasets of Tibetan machine reading comprehension, this paper proposes a Tibetan question generation model named TQGR. The model consists of two parts, the question generation and question quality assessment. The question generation adopts the classic seq2seq architecture to generate questions, and the question quality assessment improves the quality of generated questions by evaluating the fluency reward score, word repetition rate reward score and interrogative words classification auxiliary task. Finally, the experimental results show that our model has higher performance than baseline models, and ablation experiments demonstrate the effectiveness of the three mechanism.

The potential of automated question generation is role play in the multi-fields and multi-applications such as question and answering systems, examination systems, and information retrieval. Before learning the question generated, one should understand how to classify questions. This research aims to generate possible questions considering the possible question categories from question classification based on Natural Language Processing. In this research, we compared the results on Logistic Regression, Support Vector Machine, and Multinomial Naï ve Bayes, which were traditional classification models. The deep learning techniques were Convolutional Neural Networks, Bidirectional Long Short-Term Memory, combined CNN and BiLSTM models, and BERT models. The experimental results show that the preprocessing phase using Natural Language Processing could enhance question classification. The classification of the sentence to question classification attained an average micro $F_{1} -$ score of 91.40% when applied BERT model by pre-trained WangchanBERTa on simple sentences. In contrast, the satisfying score with an average micro $F_{1} -$ score of 82.07% (from 80.37% on original input) when applied to add all POS tags unigram + bigram TF-IDF by using the SVM model. The experimental results when the CNN model with GloVe on adding focusing POS tags is a satisfactory score with an average micro $F_{1} -$ score of 79.79%.

There are numerous publications of Question Generation (QG) in English but few in Thai. More than a million question-answer pairs are available in the English language, compared with only around 12,000 question-answer pairs in the Thai language. This paper presents a method to improve automatic Thai answer-agnostic QG from a dataset of insufficient size. Our evaluation showed that a QG model which was trained by the pre-trained model MT5 from a Thai dataset achieved a BLEU-1 score of 56.19. We proposed a method to generate synthetic data and an additional mechanism by using a single pre-trained model. Our best model outperformed the previous model by achieving a BLEU-1 score of 59.03. The results from the human evaluation in fluency score was 4.40, the relevance score 4.65, and the answer-ability score 4.7 out of 5.0.

Coding according to the International Classification of Diseases (ICD)-10 and its clinical modifications (CM) is inherently complex and expensive. Natural Language Processing (NLP) assists by simplifying the analysis of unstructured data from electronic health records, thereby facilitating diagnosis coding. This study investigates the suitability of transformer models for ICD-10 classification, considering both encoder and encoder-decoder architectures. The analysis is performed on clinical discharge summaries from the Medical Information Mart for Intensive Care (MIMIC)-IV dataset, which contains an extensive collection of electronic health records. Pre-trained models such as BioBERT, ClinicalBERT, ClinicalLongformer, and ClinicalBigBird are adapted for the coding task, incorporating specific preprocessing techniques to enhance performance. The findings indicate that increasing context length improves accuracy, and that the difference in accuracy between encoder and encoder-decoder models is negligible.

Automatic question generation from text or paragraph is a great challenging task which attracts broad attention in natural language processing. Because of the verbose texts and fragile ranking methods, the quality of top generated questions is poor. In this paper, we present a novel framework Automatic Chinese Question Generation (ACQG) to generate questions from text or paragraph. In ACQG, we use an adopted TextRank to extract key sentences and a template-based method to construct questions from key sentences. Then a multi-feature neural network model is built for ranking to obtain the top questions. The automatic evaluation result reveals that the proposed framework outperforms the state-of-the-art systems in terms of perplexity. In human evaluation, questions generated by ACQG rate a higher score.

We study how to improve the performance of Question Answering over Knowledge Base (KBQA) by utilizing the factoid Question Generation (QG) in this paper. The task of question generation (QG) is to generate a corresponding natural language question given the input answer, while question answering (QA) is a reverse task to find a proper answer given the question. For the KBQA task, the answer could be regarded as a fact containing a predicate and two entities from the knowledge base. Training an effective KBQA system needs a lot of labeled data which are hard to acquire. And a trained KBQA system still performs poor when answering the questions corresponding with unseen predicates in the training process. To solve these challenges, we propose a unified framework to combine the QG and QA with the help of knowledge base and text corpus. The models of QA and QG are first trained jointly on the gold dataset, then the QA model is fine tuned by utilizing a supplemental dataset constructed by the QG model with the help of text evidence. We conduct experiments on two datasets SimpleQuestions and WebQSP with the Freebase knowledge base. Empirical results show that our framework improves the performance of KBQA and performs comparably with or even better than the state-of-the-arts.

Question generation in open-domain dialogue systems is a challenging but less-explored task. It aims to enhance the interactivity and persistence of human-machine interactions. Previous work mainly focuses on question generation in the setting of single-turn dialogues, or investigates it as a data augmentation method for machine comprehension. We propose a Context-augmented Neural Question Generation (CNQG) model that leverages the conversational context to generate questions for promoting interactivity and persistence of multi-turn dialogues. More specifically, we formulate the task of question generation as a two-stage process. First, we employ an encoder-decoder framework to predict a question pattern, which denotes a set of representative interrogatives, and identify the potential topics from the conversational context by employing point-wise mutual information. Then, we generate the question by decoding the concatenation of the current dialogue utterance, the pattern, and the topics with an attention mechanism. To the best of our knowledge, ours is the first work on question generation in multi-turn open-domain dialogue systems. Our experimental results on two publicly available multi-turn conversation datasets show that CNQG outperforms the state-of-the-art baselines in terms of BLEU-1, BLEU-2, Distinct-1 and Distinct-2. In addition, we find that CNQG allows one to efficiently distill useful features from long contexts, and maintain robust effectiveness even for short contexts.

This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec.

The advent of the Internet resulted in availability of e-content in huge quantities; this e-content includes information in different domains. This information is being exploited for various reasons and applications. As the formal evaluation and assessment techniques are undergoing transformations, there is also a need for a voluminous Question Bank (QB). The manual creation of questions is a time-consuming and expensive task as it requires domain experts and there is a growing trend towards question bank based question paper generation for assessment. Hence the problem of automatic question generation has attracted attention of the Natural Language Processing (NLP) research community; this problem can be handled by NLP. Automatic generation of high-quality questions is the long-term goal of the Question Generation (QG) research community. The automatic question generation aims to generate all possible questions from the input text or document. This survey article presents the issues, challenges and approaches as well as reviews the reported research work in the context of automatic question generation for different languages at national and international level.

Endowing dialogue systems with personas is essential to deliver more human-like conversations. However, this problem is still far from well explored due to the difficulties of both embodying personalities in natural languages and the persona sparsity issue observed in most dialogue corpora. This paper proposes a pre-training based personalized dialogue model that can generate coherent responses using persona-sparse dialogue data. In this method, a pre-trained language model is used to initialize an encoder and decoder, and personal attribute embeddings are devised to model richer dialogue contexts by encoding speakers' personas together with dialogue histories. Further, to incorporate the target persona in the decoding process and to balance its contribution, an attention routing structure is devised in the decoder to merge features extracted from the target persona and dialogue contexts using dynamically predicted weights. Our model can utilize persona-sparse dialogues in a unified manner during the training process, and can also control the amount of persona-related features to exhibit during the inference process. Both automatic and manual evaluation demonstrates that the proposed model outperforms state-of-the-art methods for generating more coherent and persona consistent responses with persona-sparse data.

Pre-Trained Language Models and Their Applications

A survey on textual entailment based question answering