Graphine: A Dataset for Graph-aware Terminology Definition Generation

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

Text compression is of considerable theoretical and practical interest. It is, for example, becoming increasingly important for satisfying the requirements of fitting a large database onto a single CD-ROM. Many of the compression techniques discussed in the literature are model based. We here propose the notion of a formal grammar as a flexible model of text generation that encompasses most of the models offered before as well as, in principle, extending the possibility of compression to a much more general class of languages. Assuming a general model of text generation, a derivation is given of the well known Shannon entropy formula, making possible a theory of information based upon text representation rather than on communication. The ideas are shown to apply to a number of commonly used text models. Finally, we focus on a Markov model of text generation, suggest an information theoretic measure of similarity between two probability distributions, and develop a clustering algorithm based on this measure. This algorithm allows us to cluster Markov states, and thereby base our compression algorithm on a smaller number of probability distributions than would otherwise have been required. A number of theoretical consequences of this approach to compression are explored, and a detailed example is given.

Data-to-text generation task aims at generating text from structured data. In this work, we focus on a relatively new and challenging equation-to-text generation task – generating math word problems from equations and propose a novel equation-to-problem text generation model. Our model first utilizes a template-aware equation encoder and a Variational AutoEncoder (VAE) model to bridge the gap between abstract math tokens and text. We then introduce a topic selector and a topic controller to prevent topic drifting problems. To avoid the commonsense violation issues, we design a pre-training stage together with a commonsense enforcement mechanism. We construct a dataset to evaluate our model through both automatic metrics and human evaluation. Experiments show that our model significantly outperforms baseline models. Further analysis shows our model is effective in tackling topic drifting and commonsense violation problems.

Network representation learning is receiving increasing attention from scholars. Among them, methods based on graph neural networks have become particularly popular. However, most existing methods currently only focus on networks with a single type of relations. In the real world, networks contain a wealth of diverse information, with multiple types of relationships between nodes. In this paper, we propose a graph neural network-based multiplex network representation learning model (GNMRL). We model nodes within each layer of the multiplex network by aggregating neighbor information. Additionally, since nodes are influenced by different network layers, we integrate node interaction information across network layers. We conducted systematic experiments on four datasets, and the results show that GNMRL outperforms other comparison methods in both link prediction and node classification tasks.

Humans benefit from previous experiences when taking actions. Similarly, related examples from the training data also provide exemplary information for neural dialogue models when responding to a given input message. However, effectively fusing such exemplary information into dialogue generation is non-trivial: useful exemplars are required to be not only literally-similar, but also topic-related with the given context. Noisy exemplars impair the neural dialogue models understanding the conversation topics and even corrupt the response generation. To address the issues, we propose an exemplar guided neural dialogue generation model where exemplar responses are retrieved in terms of both the text similarity and the topic proximity through a two-stage exemplar retrieval model. In the first stage, a small subset of conversations is retrieved from a training set given a dialogue context. These candidate exemplars are then finely ranked regarding the topical proximity to choose the best-matched exemplar response. To further induce the neural dialogue generation model consulting the exemplar response and the conversation topics more faithfully, we introduce a multi-source sampling mechanism to provide the dialogue model with both local exemplary semantics and global topical guidance during decoding. Empirical evaluations on a large-scale conversation dataset show that the proposed approach significantly outperforms the state-of-the-art in terms of both the quantitative metrics and human evaluations.

Computer assisted writing system

Recent text generation models are easy to generate relevant and fluent text for the given text, while lack of causal reasoning ability when we change some parts of the given text. Counterfactual story rewriting is a recently proposed task to test the causal reasoning ability for text generation models, which requires a model to predict the corresponding story ending when the condition is modified to a counterfactual one. Previous works have shown that the traditional sequence-to-sequence model cannot well handle this problem, as it often captures some spurious correlations between the original and counterfactual endings, instead of the causal relations between conditions and endings. To address this issue, we propose a sketch-and-customize generation model guided by the causality implicated in the conditions and endings. In the sketch stage, a skeleton is extracted by removing words which are conflict to the counterfactual condition, from the original ending. In the customize stage, a generation model is used to fill proper words in the skeleton under the guidance of the counterfactual condition. In this way, the obtained counterfactual ending is both relevant to the original ending and consistent with the counterfactual condition. Experimental results show that the proposed model generates much better endings, as compared with the traditional sequence-to-sequence model.

Teachers need to provide numerous examples sentences for students to translate in the process of teaching English, but the number of sentences given by teachers to practice subjectively is not enough. Therefore, the study constructs a text generation model using an improved convolutional neural network semantic segmentation method, where the corpus utterances are keyword extracted and new shorter utterances are generated based on the keywords for language learners to practice translation. The research first uses the textRank algorithm to extract semantic keywords to obtain a dataset, and then uses CNN to construct an encoder to achieve semantic encoding of the keyword dataset. However, during the research process, it was found that traditional CNN models are relatively sensitive to the location of input data. Therefore, the research introduces the idea of Decomposition Machine (FM) to improve the encoder. In order to control text generation, research has introduced a weighted additive attention mechanism in the decoding process to associate the meaning of the generated text sequence with the meaning of the keyword set. Based on this, a text generation model for generating a translation related corpus in English teaching is constructed. This results in a text generation model that can be used to generate a translation-linked corpus for English language teaching. The average BLEU value of model 1 is 0.924, Inform value is 98.40, the Success value is 97.64, and the Combine value is 101.24, which can achieve high-quality text generation by the keyword lexical meaning and provide technical guarantee for the establishment of the corpus in educating in English.

Human use language not just to convey information but also to express their inner feelings and mental states. In this work, we adapt the state-of-the-art language generation models to generate affective (emotional) text. We posit a model capable of generating affect-driven and topic focused sentences without losing grammatical correctness as the affect intensity increases. We propose to incorporate emotion as prior for the probabilistic state-of-the-art text generation model such as GPT-2. The model gives a user the flexibility to control the category and intensity of emotion as well as the topic of the generated text. Previous attempts at modelling fine-grained emotions fall out on grammatical correctness at extreme intensities, but our model is resilient to this and delivers robust results at all intensities. We conduct automated evaluations and human studies to test the performance of our model, and provide a detailed comparison of the results with other models. In all evaluations, our model outperforms existing affective text generation models.

Human use language not just to convey information but also to express their inner feelings and mental states. In this work, we adapt the state-of-the-art language generation models to generate affective (emotional) text. We posit a model capable of generating affect-driven and topic focused sentences without losing grammatical correctness as the affect intensity increases. We propose to incorporate emotion as prior for the probabilistic state-of-the-art text generation model such as GPT-2. The model gives a user the flexibility to control the category and intensity of emotion as well as the topic of the generated text. Previous attempts at modelling fine-grained emotions fall out on grammatical correctness at extreme intensities, but our model is resilient to this and delivers robust results at all intensities. We conduct automated evaluations and human studies to test the performance of our model, and provide a detailed comparison of the results with other models. In all evaluations, our model outperforms existing affective text generation models.

BackgroundThe current research on the application of generative artificial intelligence in has shown rapid growth, and most of the existing research focuses on general nursing scenarios. And is still in the exploratory stage in the specialty field and lacks a systematic compendium of its application in the field of cardiovascular specialty nursing.ObjectiveA review of studies related to the application of generative artificial intelligence in the field of specialized cardiovascular nursing informing relevant research and practice.MethodsThis review was conducted according to the JBI guidelines for scoping reviews, using the PRISMA-ScR reporting tool.The databases searched included PubMed, Embase, Web of Science, APA PsycNetInfo (Ovid), the Cochrane Library, MEDLINE (Ovid), CINAHL, SinoMed, CNKI, WanFang databases, peer-reviewed studies written in English and Chinese from the time of database construction until March 29, 2025. Eligible studies were screened by title and abstract, and full-text screening was performed by two independent evaluators.ResultsNineteen studies were included in this review, and the main application tools were 9 text generation models, 3 multimodal generation models, 4 temporal prediction models, and 3 chatbots. The application scenarios mainly include clinical decision support, patient health management, and nursing education and counseling. Generative artificial intelligence has outstanding effects in reducing nursing workload and precise intervention, but there are limitations in the field of health education and counseling, such as logical disconnection, poor information quality, and lack of humanistic care.ConclusionsGenerative artificial intelligence provides technical support for the intelligent transformation of cardiovascular specialty care, especially effective in the areas of clinical decision support and patient health management. There is an urgent need to solve the existing problems to promote its in-depth application in this field, and it is suggested that future research focuses on the construction of specialized multimodal models.

Commonsense knowledge is crucial for artificial intelligence systems to understand natural language. Previous commonsense knowledge acquisition approaches typically rely on human annotations (for example, ATOMIC) or text generation models (for example, COMET.) Human annotation could provide high-quality commonsense knowledge, yet its high cost often results in relatively small scale and low coverage. On the other hand, generation models have the potential to automatically generate more knowledge. Nonetheless, machine learning models often fit the training data well and thus struggle to generate high-quality novel knowledge. To address the limitations of previous approaches, in this paper, we propose an alternative commonsense knowledge acquisition framework DISCOS (from DIScourse to COmmonSense), which automatically populates expensive complex commonsense knowledge to more affordable linguistic knowledge resources. Experiments demonstrate that we can successfully convert discourse knowledge about eventualities from ASER, a large-scale discourse knowledge graph, into if-then commonsense knowledge defined in ATOMIC without any additional annotation effort. Further study suggests that DISCOS significantly outperforms previous supervised approaches in terms of novelty and diversity with comparable quality. In total, we can acquire 3.4M ATOMIC-like inferential commonsense knowledge by populating ATOMIC on the core part of ASER. Codes and data are available at https://github.com/HKUST-KnowComp/DISCOS-commonsense.

Numerical simulation of underground CO2 storage requires accurate, fast and field-scale calculations of saturation and reservoir pressures, yet high-fidelity physics-based simulators are computationally expensive to do a rapid analysis. Recent advances in deep learning, especially Graph Neural Networks (GNNs), offer promising alternatives by approximating simulation outputs while dramatically reducing computational costs by minimizing the total reliance on traditional simulators. However, training GNN models on large-scale, spatiotemporal datasets remains a significant bottleneck when constrained to a single GPU. To address the computational bottleneck of training GNN on large-scale and high-resolution datasets, we propose to use multi-GPU scalable training of GNNs. The U-Net Enhanced Graph Neural Network (U-GCN) specifically was adopted to predict the spatiotemporal evolution of gas saturation and pressure buildup in CO2 storage in saline aquifers. The UGCN model was trained across multiple GPUs, allowing for concurrent gradient computations and parameter updates on separate GPU processes. The model operates natively on regular reservoir meshes to learn spatiotemporal evolution of pressure and saturation from thousands of simulation cases, and the training pipeline couples process-per-GPU data parallelism with a node-local caching strategy that deamplifies the source data file (in Hierarchical Data Format HDF5 format) Input-Output (I/O) by performing a single deserialization per node and broadcasting preprocessed graphs to sibling ranks. This eliminates multi-node startup contention and renders runs compute bound. Scaling experiments from 1 to 16 GPUs demonstrate substantial wall-clock reductions while preserving validation behavior. With per-GPU batch size 100, total training time decreased from 955.46 s (1 GPU) to 145.17 s (16 GPUs), a 6.58× speedup (≈41% parallel efficiency). With per-GPU batch size 8, time decreased from 5519.00 s to 1047.17 s, a 5.27× speedup (≈33% efficiency). The speedups are reported per epoch. Cross-node overheads relative to ideal doubling were ∼11–35%, more pronounced at small per-GPU batches where collective latency dominates. Training and validation losses rose modestly as GPU count increased under weak scaling—consistent with the larger global batch and fewer optimizer updates per epoch—yet validation tracked training across all settings, indicating stable generalization and no data leakage. These effects are mitigated by standard large-batch remedies (e.g., linear learning-rate scaling with warmup, longer schedules, optional gradient accumulation). By integrating domain-aware batching, multi-GPU data parallelism, mixed precision, and node-local caching with a GNN tailored to structured meshes, the proposed approach delivers a reproducible, high-throughput storage surrogate training regime that scales to multi-node clusters without compromising model validity. The resulting surrogates enable rapid scenario screening and outer-loop workflows (optimization and UQ) at orders-of-magnitude lower training cost compared to conventional GNN training on single a GPU baseline.

Dialogue systems help various real applications interact with humans in an intelligent natural way. In dialogue systems, the task of dialogue generation aims to generate utterances given previous utterances as contexts. Among various spectrums of dialogue generation approaches, end-to-end neural generation models have received an increase of attention. These end-to-end neural generation models are capable of generating natural-sounding sentences with a unified neural encoder-decoder network structure. The end-to-end structure sequentially encodes each word in an input context and generates the response word-by-word deterministically during decoding. However, lack of variation and limited ability in capturing long-term dependencies between utterances still challenge existing approaches. In this paper, we propose a novel hierarchical variational memory network (HVMN), by adding the hierarchical structure and the variational memory network into a neural encoder-decoder network. By emulating human-to-human dialogues, our proposed method can capture both the high-level abstract variations and long-term memories during dialogue tracking, which enables the random access of relevant dialogue histories. Extensive experiments conducted on three large real-world datasets verify a significant improvement of our proposed model against state-of-the-art baselines for dialogue generation.

Neural models are widely applied to headline generation. Template-based methods are a promising direction to overcome the shortcomings of the neural headline generation (NHG) model in generating duplicate or extra words. Previous works often retrieve relevant headlines from the training data and adopt them as the soft template to guide the NHG model. However, these works had two drawbacks: reliance on additional retrieval tools, and uncertainty regarding semantic consistency between the retrieved headline and the source article. The NHG model uncertainty can be utilized to generate hypotheses. The hypotheses generated based on a well-trained NHG model not only contain salient information but also exhibit diversity, making them suitable as soft templates. In this study, we use a basic NHG model to generate multiple diverse hypotheses as candidate templates. Then, we propose a novel Multiple-Hypotheses-based NHG (MH-NHG) model. Experiments on English headline generation tasks demonstrate that it outperforms several baseline systems and achieves a comparable performance with the state-of-the-art system. This indicates that MH-NHG can generate more accurate headlines guided by multiple hypotheses.