Deep Neural Networks for Natural Language Understanding: A Multi-Task Learning Approach

With the notable success of pretrained language models, the pretraining-fine-tuning paradigm has become a dominant solution for natural language understanding (NLU) tasks. Typically, the training instances of a target NLU task are introduced in a completely random order and treated equally at the fine-tuning stage. However, these instances can vary greatly in difficulty, and similar to human learning procedures, language models can benefit from an easy-to-difficult curriculum. Based on this concept, we propose a curriculum learning (CL) framework. Our framework consists of two stages, Review and Arrange, targeting the two main challenges in curriculum learning, i.e., how to define the difficulty of instances and how to arrange a curriculum based on the difficulty, respectively. In the first stage, we devise a cross-review (CR) method to train several teacher models first and then review the training set in a crossed manner to distinguish easy instances from difficult instances. In the second stage, two sampling algorithms, a coarse-grained arrangement (CGA) and a fine-grained arrangement (FGA), are proposed to arrange a curriculum for language models in which the learning materials start from the easiest instances, and more difficult instances are gradually added into the training procedure. Compared to previous heuristic CL methods, our framework can avoid the errors caused by a gap in difficulty between humans and machines and has strong generalization ability. We conduct comprehensive experiments, and the results show that our curriculum learning framework, without any manual model architecture design or use of external data, obtains significant and universal performance improvements on a wide range of NLU tasks in different languages.

Procedural knowledge, or how-to knowledge, is the knowledge acquired from natural language understanding of instructions in procedural text. Procedural knowledge bases containing textual descriptions of tasks in procedures have witnessed explosive growth recently. This has facilitated a significant body of work in various natural language understanding tasks. A rich source of procedural text is in the form of recipes describing food preparation procedures. The ready availability of online recipes has enabled progress in food computing, which refers to computing tasks related to recipes, such as food perception, recipe image recognition and calorie estimation, and food-oriented retrieval of recipes. However, past work on food computing has not covered the procedural knowledge inherent in recipes and the natural language understanding tasks required to uncover that knowledge. We seek to address this by presenting an overview of recent work in natural language understanding tasks in food computing and describing how this contributes to how-to knowledge and future applications.

Detecting biases in natural language understanding (NLU) for African American Vernacular English (AAVE) is crucial to developing inclusive natural language processing (NLP) systems.To address dialectinduced performance discrepancies, we introduce AAVENUE (AAVE Natural Language Understanding Evaluation), a benchmark for evaluating large language model (LLM) performance on NLU tasks in AAVE and Standard American English (SAE).AAVENUE builds upon and extends existing benchmarks like VALUE, replacing deterministic syntactic and morphological transformations with a more flexible methodology leveraging LLMbased translation with few-shot prompting, improving performance across several evaluation metrics when translating key tasks from the GLUE and SuperGLUE benchmarks.We compare AAVENUE and VALUE translations using five popular LLMs and a comprehensive set of metrics including fluency, BARTScore, quality, coherence, and understandability.Additionally, the fluency of AAVENUE is validated by annotations from AAVE speakers.Our evaluations reveal that LLMs consistently perform better on SAE tasks than AAVEtranslated versions, underscoring inherent biases and highlighting the need for more inclusive NLP models.We have open-sourced our source code on GitHub and created a website to showcase our work at https://aavenue.live.

Detecting user intents from utterances is the basis of natural language understanding (NLU) task. To understand the meaning of utterances, some work focuses on fully representing utterances via semantic parsing in which annotation cost is labor-intentsive. While some researchers simply view this as intent classification or frequently asked questions (FAQs) retrieval, they do not leverage the shared utterances among different intents. We propose a simple and novel multi-point semantic representation framework with relatively low annotation cost to leverage the fine-grained factor information, decomposing queries into four factors, i.e., topic, predicate, object/condition, query type. Besides, we propose a compositional intent bi-attention model under multi-task learning with three kinds of attention mechanisms among queries, labels and factors, which jointly combines coarse-grained intent and fine-grained factor information. Extensive experiments show that our framework and model significantly outperform several state-of-the-art approaches with an improvement of 1.35%-2.47% in terms of accuracy.

A method for creating a vision-and-language (V&L) model is to extend a language model through structural modifications and V&L pre-training. Such an extension aims to make a V&L model inherit the capability of natural language understanding (NLU) from the original language model. To see how well this is achieved, we propose to evaluate V&L models using an NLU benchmark (GLUE). We compare five V&L models, including single-stream and dual-stream models, trained with the same pre-training. Dual-stream models, with their higher modality independence achieved by approximately doubling the number of parameters, are expected to preserve the NLU capability better. Our main finding is that the dual-stream scores are not much different than the single-stream scores, contrary to expectation. Further analysis shows that pre-training causes the performance drop in NLU tasks with few exceptions. These results suggest that adopting a single-stream structure and devising the pre-training could be an effective method for improving the maintenance of language knowledge in V&L extensions.

In recent years, multitask learning has turned out to be of great success in various applications. Though single task model training has promised great results throughout these years, it ignores valuable information which might help in estimating better learning parameters. By virtue of learning-related tasks, multitask learning has been able to generalize the models even better. In this paper, we try to enhance the feature mapping of multitask deep learning models by sharing features among related tasks and inductive transfer learning. We have explored the learning of task relationship among various tasks for acquiring more benefits from multitask learning. The proposed enhanced model is compared with two state-of-the-art multitask deep learning models, namely Hyperface and FaceNet. The results show better performance of the proposed model in predicting age, gender and ethnicity on the UTK face dataset.KeywordsNeural networksMultitask deep learningDeep learningRecurrent neural networksConvolution neural networks

Natural Language Understanding (NLU) and Natural Language Generation (NLG) are the general methods that support machine understanding of text content. They play a very important role in the text information processing system including recommendation and question and answer systems. There are many researches in the field of NLU such as Bag of words, N-Gram, and neural network language model. These models have achieved a good performance in NLU and NLG tasks. However, since they require lots of training data, it is difficult to obtain rich data in practical applications. Thus, pretraining becomes important. This paper proposes a semisupervised way to deal with math word problem (MWP) tasks using unsupervised pretraining and supervised tuning methods, which are based on the Unified pretrained Language Model (UniLM). The proposed model requires fewer training data than traditional models since it uses model parameters of tasks that have been learned before to initialize the model parameters of new tasks. In this way, old knowledge helps new models successfully perform new tasks from old experiences instead of from scratch. Moreover, in order to help the decoder make accurate predictions, we combine the advantages of AR and AE language models to support one-way, sequence-to-sequence, and two-way predictions. Experiments, carried out on MWP tasks with 20,000+ mathematical questions, show that the improved model outperforms the traditional models with a maximum accuracy of 79.57%. The impact of different experiment parameters is also studied in the paper and we found that a wrong arithmetic order leads to incorrect solution expression generation.

Paraphrase identification is a crucial task in natural language understanding, especially in cross-language information retrieval. Nowadays, Multi-Task Deep Neural Network (MT-DNN) has become a state-of-the-art method that brings outstanding results in paraphrase identification [1]. In this paper, our proposed method based on MT-DNN [2] to detect similarities between English and Vietnamese sentences, is proposed. We changed the shared layers of the original MT-DNN from original the BERT [3] to other pre-trained multi-language models such as M-BERT [3] or XLM-R [4] so that our model could work on cross-language (in our case, English and Vietnamese) information retrieval. We also added some tasks as improvements to gain better results. As a result, we gained 2.3% and 2.5% increase in evaluated accuracy and F1. The proposed method was also implemented on other language pairs such as English – German and English – French. With those implementations, we got a 1.0%/0.7% improvement for English – German and a 0.7%/0.5% increase for English – French.

Abstract. Large language models (LLMs) have revolutionized the field of natural language processing (NLP), demonstrating remarkable capabilities in understanding, generating, and manipulating human language. This comprehensive review explores the development, applications, optimizations, and challenges of LLMs. This paper begin by tracing the evolution of these models and their foundational architectures, such as the Transformer, GPT, and BERT. We then delve into the applications of LLMs in natural language understanding tasks, including sentiment analysis, named entity recognition, question answering, and text summarization, highlighting real-world use cases. Next, we examine the role of LLMs in natural language generation, covering areas such as content creation, language translation, personalized recommendations, and automated responses. We further discuss LLM applications in other NLP tasks like text style transfer, text correction, and language model pre-training. Subsequently, we explore techniques for optimizing and improving LLMs, including model compression, explainability, robustness, and security. Finally, we address the challenges posed by the significant computational requirements, sample inefficiency, and ethical considerations surrounding LLMs. We conclude by discussing potential future research directions, such as efficient architectures, few-shot learning, bias mitigation, and privacy-preserving techniques, which will shape the ongoing development and responsible deployment of LLMs in NLP.

Case or agr. Just as abstract Case and agr are cover terms for syntactic features with often idiosyncratic morphological realization, thematic roles seem most useful simply as syntactic designations for positions that enter into argument and adjunct relations. This is a good book, not just for its critique of theta theory, but also for the positive contribution of further development of the Decompositional Theory. Although the book is a revised dissertation, it doesn't read like one: the style is engaging and philosophical, and the exemplification is detailed and clear. And while it does not cover every thematic-role-based proposal ever made, the book is nevertheless to be recommended to anyone who has puzzled over the content of thematic roles. [Edwin Battistella, University of Alabama in Birmingham] Psychology of language: An introduction to sentence and discourse processes. By Murray Singer. Hillsdale, NJ: Lawrence Erlbaum, 1990. Pages xi, 308. $24.95. Recent years have seen the emergence of an essentially interdisciplinary, cognitive-science approach to natural language processing, with some of the most fruitful work focusing on discourse structure and processing. Singer's book is intended as an introductory graduate textbook, though it is also suitable as a desk reference for the linguist, cognitive psychologist, or computer scientist with an interest in natural language processing. An introductory graduate textbook must accomplish several purposes: (i) familiarize students with key theoretical concepts; (ii) refer them to the fundamental literature in which those concepts are discussed; (iii) provide them with exemplars that teach them how to apply the concepts insightfully to the data; and (iv) inculcate a sense of relevance that enables students to read professional articles and evaluate their theoretical significance. S's book is probably above average by these standards. Key concepts are clearly if succinctly explained, fundamental articles are cited in each of the relevant disciplines, and examples are frequent and detailed enough to satisfy the most exacting linguist. The book's weaknesses are those endemic to any introductory text: it emphasizes consensus, spending more time summarizing established results than exploring open issues and controversies, and the passages which explain key concepts are often terse, so that the instructor may sometimes wish to assign source articles as background reading. Overall, the book appears to be a valuable introductory textbook for a course in psycholinguistics or discourse processing. It is divided into ten chapters, as follows: Ch. 1, 'Introduction'; Ch. 2, 'Language and meaning: Representing and remembering discourse'; Ch. 3, 'Syntax and parsing processes'; Ch. 4, The role of knowledge in language comprehension'; Ch. 5, 'Understanding coherent discourse'; Ch. 6, Theme'; Ch. 7, 'Inference processes'; Ch. 8, 'Understanding stories'; Ch. 9, 'Question answering and sentence verification'; and Ch. 10, 'Natural language understanding by computers—and people'. One of the noteworthy aspects of the book is its thorough integration of psycholinguistic data with the concepts of discourse analysis, including coherence, the givennew contract, thematic structure, and scripts. [Paul Deane, University of Central Florida.] An essay on grammar-parser relations. By Jan van de Koot. Dordrecht: Foris, 1990. Pp. xii, 152. Paper $24.50. In this monograph, K investigates the possible relations between grammars and parsers. Any theory of the mind that postulates separate 240 LANGUAGE, VOLUME 68, NUMBER 1 (1992) levels of representations raises the question of what relations are possible, justified and beneficial among the several levels. Two such theories are Marr's theory of vision and Chomskyan generative grammar, with their separation of competence from performance. The book under review belongs to the research line of computer modelling of performance started by Parsifal (Mitchell Marcus, A theory of syntactic recognition for natural language, Cambridge, MA: MIT Press, 1980) in the Government-Binding framework. The monograph starts with a critical introduction to the relevant issues. Through extensive quotations from Marr's work (David Marr, Vision, San Francisco, CA: Freeman, 1982) K presents the methodological justification for multileveled theories of cognition. He then proceeds to explore, in the more limited domain of the language faculty, the necessary features of a psychologically real theory of grammar. The arguments that led some scholars to consider transformational grammar not psychologically real are presented and rebutted by the introduction of a more articulated framework for evaluation, which takes into account the separateness of competence grammar, runtime grammar, parsing technique, and evaluation metrics. Although nothing new is added to the debate here, the main issues of a discussion that spans a decade are reviewed clearly and concisely. Ch. 2 is an overview of several proposals in the framework of deterministic parsing. Most notably, the first such parser, Parsifal, is presented in detail. Arguing against some recent criticism of Parsifal's use of look-ahead, K claims that the use of look-ahead to resolve local ambiguities is empirically correct if coupled with the assumption that lexical-semantic information can be used during syntactic processing, when it is needed for disambiguation. K also reviews the formal properties of Parsifal, drawing on other works in the literature to argue that determinism provides a functional explanation of WH-islands at most, and that it is neutral with respect to grammatical formalism. Ch. 3 explores the relation between the Barriers theory (Noam Chomsky, Barriers. Cambridge, MA: MIT Press, 1986) and deterministic parsing. The parsing platform is Parsifal, with alterations and augmentations that link it more tightly to the grammar and make it more accurate empirically. An algorithm for chain formation is proposed here. K discusses its integration in the existing parser and its application to some linguistic facts, such as violations of the Subject and Adjunct Conditions and parasitic gaps. Some interesting ideas are put forth in this chapter, although more extensive exemplification and, if not a real implementation, at least lower-level specifications of the algorithms would have substantiated the proposal more convincingly. The book ends with a number of theoretical investigations of the possible contribution of the theory of performance to the theory of competence. In general, the reader interested in these issues will find in K's book an accurate and critical review of the main issues in the tradition of deterministic parsing, in itself a needed and useful contribution. Because the book is not intended as a review, however, more space should have been devoted to the author's new proposal; interesting ideas, such as those related to the algorithms for chain formation, are not given enough space and detailed description. The book would have also benefited from more careful editing and typesetting. [Paola Merlo. University ofMaryland.] Politically speaking: The pragmatic analysis of political language. By John Wilson. (Language in Society series.) Basil Blackwell, 1990. Pp. 203. Cloth $45.95, paper $16.95. Politically speaking offers an interesting and frequently amusing look at political speech from the perspective of linguistic pragmatics. Unlike Robin Lakoffs Talking power (New York: Basic Books, 1990), it does not investigate political power or even politics in a generic sense; and, unlike Frederick Bowers's Linguistic aspects oflegislative expression (Vancouver: University of British Columbia Press, 1990). it does not examine statutory language. Rather, Wilson explores politicians' utterances in speeches and interviews. Examples from other cultures are used, but mostly the study deals with recent British and American political figures. A notable feature of the book, W's quotations—from Margaret Thatcher's 'We are a grandmother' to Ronald Reagan's '[in England] a criminal with a gun, even if he was arrested for burglary, was tried for first degree murder and hanged if found

Large language models (LLMs), consisting of billions and trillions of parameters, have demonstrated exceptional ability in natural language understanding (NLU) and natural language generation (NLG) tasks. Increases in their numbers of parameters and model sizes have resulted in better performance and accuracy. However, models with such enormous numbers of parameters incur significant computational costs and resources, making them challenging to fine tune and adapt to a specific downstream task. Several parameter-efficient fine-tuning (PEFT) techniques have been proposed to address this issue. This study demonstrates the improvement obtained over the base LLaMA3-8B model using two prominent PEFT techniques: LoRA and QLoRA. We use the sequence classification task of sentiment analysis to conduct the experiments. Additionally, we analyze the effects of hyperparameter adjustments (r and α) on the model’s performance. We examine the tradeoff between efficiency and memory savings obtained using the quantized LoRA (QLoRA) technique. We also investigate and compare the performance changes of LoRA and QLoRA techniques obtained after adapting to attention layers (query, key, value, and project) to all the linear layers during fine tuning. We report the findings of our work along with limitations and future directions.

It is by now widely accepted that a number of tasks in natural language understanding (NLU) require the storage of and reasoning with a vast amount of background (commonsense) knowledge. While several efforts have been made to build such ontologies, a consensus on a scientific methodology for ontological design is yet to emerge. In this paper we suggest an approach to building a commonsense ontology for language understanding using language itself as a design guide. The idea is rooted in Frege's conception of compositional semantics and is related to the idea of type inferences in strongly-typed, polymorphic programming languages. The method proposed seems to (i) resolve the problem of multiple inheritance; (ii) suggest an explanation for polysemy and metaphor; and (iii) provide a step towards establishing a systematic approach to ontological design.

Document-level relation extraction with multi-layer heterogeneous graph attention network

Coreference Resolution (CR) is a crucial yet challenging task in natural language understanding, often constrained by task-specific architectures and encoder-based language models that demand extensive training and lack adaptability. This study introduces the first multilingual CR methodology which leverages decoder-only LLMs to handle both overt and zero mentions. The article explores how to model the CR task for LLMs via five different instruction sets using a controlled inference method. The approach is evaluated across three LLMs: Llama 3.1, Gemma 2, and Mistral 0.3. The results indicate that LLMs, when instruction-tuned with a suitable instruction set, can surpass state-of-the-art task-specific architectures. Specifically, our best model, a fully fine-tuned Llama 3.1 for multilingual CR, outperforms the leading multilingual CR model (i.e., Corpipe 24 single stage variant) by 2 percentage points on average across all languages in the CorefUD v1.2 dataset collection.

Text matching is a basic and important task in natural language understanding, this paper proposes a new model BBMC for the problem of insufficient feature extraction ability of existing text matching models, which integrates BiLSTM and multi-scale CNN on the basis of BERT. First, the word embedding representation of the text is obtained by the BERT, and then the semantic features of the text are further extracted by the double-layer BiLSTM, followed by the multi-scale CNN model, the key local features are extracted, and finally the linear and SoftMax function are used to classify. Experimental results on the LCQMC dataset show that the BBMC has been improved to a certain extent compared with other methods, and the accuracy on the test set can be best achieved 88.01%.