On Randomized Classification Layers and Their Implications in Natural Language Generation

Among studies on natural language generation (NLG), computational creativity, and human-computer interaction; there is a vision of witnessing these tools collaborating with humans in generating and authoring creative content. This study aims to systematically review published studies discussing creative writing and story generation during the period of 2016-2021. This work seeks to identify the primary research methods used in NLG and creative writing studies, to locate how these studies are distributed geographically, and finally, to classify the subfields or common keywords primarily used in NLG involving creative writing. The findings suggest that experiment studies and problem-solving were the most common research methods in NLG and creative writing. Major identified themes in the reviewed articles include story generation, language models, and co-creativity, along with some gaps in foreign language translation and humour generation studies. The majority of the studies suggest that NLG tasks had a positive impact on creative writing. Common tasks related to NLG and creative writing are typically using keywords such as story generation, co-creativity, co-writing, user interface and writing tools. In future work, we aim to explore more GPT-3 capabilities in creative writing, in addition to creative writing applications in foreign language translation tasks.

Natural Language Understanding (NLU) and Natural Language Generation (NLG) are the general methods that support machine understanding of text content. They play a very important role in the text information processing system including recommendation and question and answer systems. There are many researches in the field of NLU such as Bag of words, N-Gram, and neural network language model. These models have achieved a good performance in NLU and NLG tasks. However, since they require lots of training data, it is difficult to obtain rich data in practical applications. Thus, pretraining becomes important. This paper proposes a semisupervised way to deal with math word problem (MWP) tasks using unsupervised pretraining and supervised tuning methods, which are based on the Unified pretrained Language Model (UniLM). The proposed model requires fewer training data than traditional models since it uses model parameters of tasks that have been learned before to initialize the model parameters of new tasks. In this way, old knowledge helps new models successfully perform new tasks from old experiences instead of from scratch. Moreover, in order to help the decoder make accurate predictions, we combine the advantages of AR and AE language models to support one-way, sequence-to-sequence, and two-way predictions. Experiments, carried out on MWP tasks with 20,000+ mathematical questions, show that the improved model outperforms the traditional models with a maximum accuracy of 79.57%. The impact of different experiment parameters is also studied in the paper and we found that a wrong arithmetic order leads to incorrect solution expression generation.

Transformer-based language models are able to generate fluent text and be efficiently adapted across various natural language generation tasks. However, language models that are pretrained on large unlabeled web text corpora have been shown to suffer from degenerating toxic content and social bias behaviors, consequently hindering their safe deployment. Various detoxification methods were proposed to mitigate the language model's toxicity; however, these methods struggled to detoxify language models when conditioned on prompts that contain specific social identities related to gender, race, or religion. In this study, we propose Reinforce-Detoxify; A reinforcement learning-based method for mitigating toxicity in language models. We address the challenge of safety in language models and propose a new reward model that is able to detect toxic content and mitigate unintended bias towards social identities in toxicity prediction. The experiments demonstrate that the Reinforce-Detoxify method for language model detoxification outperforms existing detoxification approaches in automatic evaluation metrics, indicating the ability of our approach in language model detoxification and less prone to unintended bias toward social identities in generated content.

This paper addresses the problem of automatic generation of natural language descriptions for ontology-described artifacts. The original motivation for the work is the challenge of providing textual narratives of automatically generated scientific workflows (e.g., paragraphs that scientists can include in their publications). The paper presents two systems which generate descriptions of sets of atoms derived from a collection of ontologies. The first system, called nlgPhylogeny, demonstrates the feasibility of the task in the Phylotastic project, providing evolutionary biologists with narrative for automatically generated analysis workflows. nlgPhylogeny utilizes the fact that the Grammatical Framework (GF) is suitable for the natural language generation (NLG) task; the paper shows how elements of the ontologies in Phylotastic, such as web services and information artifacts, can be encoded in GF for the NLG task. The second system, called \(\mathtt{\small nlgOntology}^{A}\), is a generalization of nlgPhylogeny. It eliminates the requirement that a GF needs to be defined and proposes the use of annotated ontologies for NLG. Given a set of annotated ontologies, \(\mathtt{\small nlgOntology}^{A}\) generates a GF suitable for the creation of natural language descriptions of sets of atoms derived from these ontologies. The paper describes the algorithms used in the development of nlgPhylogeny and \(\mathtt{\small nlgOntology}^{A}\) and discusses potential applications of these systems.

Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives and desires different properties of generated text. The complexity makes automatic evaluation of NLG particularly challenging. Previous work has typically focused on a single task and developed individual evaluation metrics based on specific intuitions. In this paper, we propose a unifying perspective based on the nature of information change in NLG tasks, including compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog). Information alignment between input, context, and output text plays a common central role in characterizing the generation. With automatic alignment prediction models, we develop a family of interpretable metrics that are suitable for evaluating key aspects of different NLG tasks, often without need of gold reference data. Experiments show the uniformly designed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in each of diverse tasks, including text summarization, style transfer, and knowledge-grounded dialog.

Information extraction (IE) aims to automatically identify and extract information about specific interests from raw texts. Despite the abundance of solutions based on fine-tuning pretrained language models, IE in the context of few-shot and zero-shot scenarios remains highly challenging due to the scarcity of training data. Large language models (LLMs), on the other hand, can generalize well to unseen tasks with few-shot demonstrations or even zero-shot instructions and have demonstrated impressive ability for a wide range of natural language understanding or generation tasks. Nevertheless, it is unclear, whether such effectiveness can be replicated in the task of IE, where the target tasks involve specialized schema and quite abstractive entity or relation concepts. In this paper, we first examine the validity of LLMs in executing IE tasks with an established prompting strategy and further propose multiple types of augmented prompting methods, including the structured fundamental prompt (SFP), the structured interactive reasoning prompt (SIRP), and the voting-enabled structured interactive reasoning prompt (VESIRP). The experimental results demonstrate that while directly promotes inferior performance, the proposed augmented prompt methods significantly improve the extraction accuracy, achieving comparable or even better performance (e.g., zero-shot FewNERD, FewNERD-INTRA) than state-of-the-art methods that require large-scale training samples. This study represents a systematic exploration of employing instruction-following LLM for the task of IE. It not only establishes a performance benchmark for this novel paradigm but, more importantly, validates a practical technical pathway through the proposed prompt enhancement method, offering a viable solution for efficient IE in low-resource settings.

With the rapid development of large language models (LLMs) in natural language processing and generation tasks, researchers are continuously validating their application potential in the field of medical artificial intelligence (AI). Generative Pre-trained Transformer 5 (GPT-5), Pathways Language Model 2 (PaLM-2), Large Language Model Meta AI 3 (Llama-3), Gemini-2.5, and various medical vertical models have demonstrated excellent performance in many medical and health-related tasks. However, LLMs still face key bottlenecks in real-world medical scenarios, including “illusions” slow knowledge updates, and a lack of interpretability. These problems severely restrict the safe deployment of large language models in high-risk medical scenarios and have become a major obstacle to the transition of medical artificial intelligence from research to clinical application. Retrieval-augmented generation (RAG) technology has therefore become an important solution for improving the credibility of medical LLMs. RAG significantly reduces the risk of information errors by retrieving authoritative medical knowledge before generation while maintaining knowledge updates without retraining. A review of research progress on RAG in the medical field is of great theoretical value and practical significance for promoting the design, evaluation, and standardized application of trustworthy medical AI. This review aims to systematically summarize the research progress of RAG in medical scenarios, including the technical frameworks, typical applications, and development trends of three types of methods—naive RAG, advanced RAG, and modular RAG—and further discuss their significance in medical reliability, health equity, and personalized medicine. The naive RAG architecture, through a standard “index-retrieval-generation” process, transforms external knowledge bases into a structured vector space that can be queried by an LLM, achieving positive results in electronic medical record (EMR) summary, preoperative assessment, and medical question answering (QA). However, naive RAG still has limitations in retrieval accuracy and cross-modal processing. To address these shortcomings, advanced RAG methods have been devised that significantly improve the model’s decision-making capabilities by improving retrieval strategies, enhancing inference chains, and introducing self-reflection mechanisms. Furthermore, modular RAG design offers composable, multi-module systems to support multi-source knowledge integration and complex task decomposition. In terms of application value, RAG’s contributions to the healthcare field are primarily reflected in these aspects. On the one hand, it significantly improves the reliability of medical AI by introducing traceable knowledge to reduce illusions. On the other hand, it helps promote healthcare equity by driving a shift toward a patient-centered healthcare model through localized knowledge bases and multilingual support. However, the deployment of RAG in medical scenarios still faces many challenges, such as data privacy risks. Future research should focus on improving self-supervised reflection capabilities and developing cross-modal knowledge fusion technologies. As research develops, RAG is expected to become a core foundational capability of future intelligent healthcare systems, further promoting the development of safe and efficient intelligent healthcare.

The task of natural language generation by computer requires a method for the selection of lexical items that refer to concepts. A generation system cannot assume that every concept will have an appropriate lexical entry associated with it. Similarly, a generation system must avoid redundant text and must strive for the most specific descriptions possible. This paper presents an algorithm for lexical selection that takes advantage of a conceptual network in which we represent the relationships between concepts (such as which concepts subsume each other and what differentiates concepts at different levels of generality). When the algorithm chooses a lexical realization for a concept, it first checks to see if the concept to be expressed is associated with a lexical entry. If so, that entry is used. If the concept does not have the appropriate lexical associations, the algorithm generates a phrase with a more general head term and restrictive modifiers. It does this by using the conceptual network to compute a semantically appropriate head term, and then modifying the request for generation by adding restrictions that differentiate the concept associated with the head term from the concept initially requested. Similarly, the conceptual network is used to eliminate redundant modifiers by computing the information contained in a lexical item that is chosen as a head term. Here the algorithm modifies the generation request to avoid restating the extra information. This algorithm is being implemented within Penman, a computerized system for the generation of English text from statements in a first-order predicate calculus language.

We introduce the discussion mechanism into the multiagent communicating encoder-decoder architecture for Natural Language Generation (NLG) tasks and prove that by applying the discussion mechanism, the communication between agents becomes more effective. Generally speaking, an encoder-decoder architecture predicts target-sequence word by word in several time steps. At each time step of prediction, agents with the discussion mechanism predict the target word after several discussion steps. In the first step of discussion, agents make their choice independently and express their decision to other agents. In the next discussion step, agents collect other agents’ decision to update their own decisions, then express the updated decisions to others again. After several iterations, the agents make their final decision based on a well-communicated situation. The benefit of the discussion mechanism is that multiple encoders can be designed as different structures to fit the specified input or to fetch different representations of inputs.We train and evaluate the discussion mechanism on Table to Text Generation, Text Summarization and Image Caption tasks, respectively. Our empirical results demonstrate that the proposed multi-agent discussion mechanism is helpful for maximizing the utility of the communication between agents.

Large language models (LLMs), consisting of billions and trillions of parameters, have demonstrated exceptional ability in natural language understanding (NLU) and natural language generation (NLG) tasks. Increases in their numbers of parameters and model sizes have resulted in better performance and accuracy. However, models with such enormous numbers of parameters incur significant computational costs and resources, making them challenging to fine tune and adapt to a specific downstream task. Several parameter-efficient fine-tuning (PEFT) techniques have been proposed to address this issue. This study demonstrates the improvement obtained over the base LLaMA3-8B model using two prominent PEFT techniques: LoRA and QLoRA. We use the sequence classification task of sentiment analysis to conduct the experiments. Additionally, we analyze the effects of hyperparameter adjustments (r and α) on the model’s performance. We examine the tradeoff between efficiency and memory savings obtained using the quantized LoRA (QLoRA) technique. We also investigate and compare the performance changes of LoRA and QLoRA techniques obtained after adapting to attention layers (query, key, value, and project) to all the linear layers during fine tuning. We report the findings of our work along with limitations and future directions.

Despite the success of text-to-text pre-trained models in various natural language generation (NLG) tasks, the generation performance is largely restricted by the number of labeled data in downstream tasks, particularly in data-to-text generation tasks. Existing works mostly utilize abundant unlabeled structured data to conduct unsupervised pre-training for task adaption, which fail to model the complex relationship between source structured data and target texts. Thus, we introduce self-training as a better few-shot learner than task-adaptive pre-training, which explicitly captures this relationship via pseudo-labeled data generated by the pre-trained model. To alleviate the side-effect of low-quality pseudo-labeled data during self-training, we propose a novel method called Curriculum-Based Self-Training (CBST) to effectively leverage unlabeled data in a rearranged order determined by the difficulty of text generation. Experimental results show that our method can outperform fine-tuning and task-adaptive pre-training methods, and achieve state-of-the-art performance in the few-shot setting of data-to-text generation.

Early Exiting is one of the most popular methods to achieve efficient inference. Current early exiting methods adopt the (weighted) sum of the cross entropy loss of all internal classifiers as the objective function during training, imposing all these classifiers to predict all instances correctly. However, during inference, as long as one internal classifier predicts an instance correctly, it can accelerate without losing accuracy. Thus, there is a notable gap between training and inference. We propose ConsistentEE, an early exiting method that is consistent in training and inference. ConsistentEE formulates the early exiting process as a reinforcement learning problem. A policy network is added to decide whether an instance should exit or continue. The training objective of ConsistentEE only requires each instance to be predicted correctly by one internal classifier. Additionally, we introduce the concept "Memorized Layer" to measure the hardness of an instance. We incorporate the memorized layer into reward function design, which allows "easy'' instances to focus more on acceleration while ``hard'' instances to focus more on accuracy. Experimental results show that our method outperforms other baselines on various natural language understanding and generation tasks using PLMs and LLMs as backbones respectively.

Large Language Models (LLMs) have shown impressive capabilities in natural language understanding and generation tasks. However, their reliance on text-only input limits their ability to handle tasks that require multimodal reasoning. To overcome this, Multimodal Large Language Models (MLLMs) have been introduced, enabling inputs such as images, text, video and audio. While MLLMs address some limitations, they often suffer from hallucinations because of over-reliance on internal knowledge and face high computational costs. Traditional vector-based multimodal RAG systems attempt to mitigate these issues by retrieving supporting information, but often suffer from cross-modal misalignment, where independently retrieved text and image content cannot align meaningfully. Motivated by the structured retrieval capabilities of text-based knowledge graph RAG, this paper proposes VisGraphRAG to address the challenge by modelling structured relationships between images and text within a unified MMKG. This structure enables more accurate retrieval and better alignment across modalities, resulting in more relevant and complete responses. The experimental results show that VisGraphRAG significantly outperforms the vector database-based baseline RAG, achieving a higher answer accuracy of 0.7629 compared to 0.6743. Besides accuracy, VisGraphRAG also shows superior performance in key RAGAS metrics such as multimodal relevance (0.8802 vs 0.7912), showing its stronger ability to retrieve relevance information across modalities. These results underscore the effectiveness of the proposed Multimodal Knowledge Graph (MMKG) methods in enhancing cross-modal alignment and supporting more accurate, context-aware generation in complex multimodal tasks.

Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple low-resource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised cross-lingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results.We experimented with few-shot training (with 1000 supervised data points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analyses to demonstrate the robustness of ZmBART.

Evaluating Natural Language Generation (NLG) systems is a challenging task. Firstly, the metric should ensure that the generated hypothesis reflects the reference's semantics. Secondly, it should consider the grammatical quality of the generated sentence. Thirdly, it should be robust enough to handle various surface forms of the generated sentence. Thus, an effective evaluation metric has to be multifaceted. In this paper, we propose an automatic evaluation metric incorporating several core aspects of natural language understanding (language competence, syntactic and semantic variation). Our proposed metric, RoMe, is trained on language features such as semantic similarity combined with tree edit distance and grammatical acceptability, using a self-supervised neural network to assess the overall quality of the generated sentence. Moreover, we perform an extensive robustness analysis of the state-of-the-art methods and RoMe. Empirical results suggest that RoMe has a stronger correlation to human judgment over state-of-the-art metrics in evaluating system-generated sentences across several NLG tasks.