Compression, information theory, and grammars: a unified approach

this paper proposed the modification of the classic Shannon's formula of information entropy by utilizing a concept that we discovered on April 28, 2016 namely Harmony in Gradation - the Formula for Everything. In more detail, the current formula for calculating information entropy contains the weaknesses because it only contains we call the Harmony, whereas every correct formula that involves the union or random variable which has more than two elements refer to the Harmony in Gradation concept should contain two conflicting elements, namely the Harmony and the Gradation. Then, there are two steps for preparing the modification of Shannon's entropy formula, which the first is making a formula that has at once the Harmony and the Gradation but has the opposite understanding of the Shannon's entropy formula, and we call it the Cavity channel formula, then the second is compiles the entropy modified formula. Furthermore, in this paper, it is proven that by using the formula proposed in this paper, the calculation of entropy will be more accurate compared to the use of the Shannon's formula and will be more in line with the reality of the measurement results. Finally, there are three weaknesses of Shannon's formula which are simply proven here, first is an error in the maximum entropy value, the second is an inaccurate error represents harmony between symbols, and third is not being able to see significant gradation changes in the appearance of symbols.

Precisely defining the terminology is the first step in scientific communication. Developing neural text generation models for definition generation can circumvent the labor-intensity curation, further accelerating scientific discovery. Unfortunately, the lack of large-scale terminology definition dataset hinders the process toward definition generation. In this paper, we present a large-scale terminology definition dataset Graphine covering 2,010,648 terminology definition pairs, spanning 227 biomedical subdisciplines. Terminologies in each subdiscipline further form a directed acyclic graph, opening up new avenues for developing graph-aware text generation models. We then proposed a novel graph-aware definition generation model Graphex that integrates transformer with graph neural network. Our model outperforms existing text generation models by exploiting the graph structure of terminologies. We further demonstrated how Graphine can be used to evaluate pretrained language models, compare graph representation learning methods and predict sentence granularity. We envision Graphine to be a unique resource for definition generation and many other NLP tasks in biomedicine.

Computer assisted writing system

Recent text generation models are easy to generate relevant and fluent text for the given text, while lack of causal reasoning ability when we change some parts of the given text. Counterfactual story rewriting is a recently proposed task to test the causal reasoning ability for text generation models, which requires a model to predict the corresponding story ending when the condition is modified to a counterfactual one. Previous works have shown that the traditional sequence-to-sequence model cannot well handle this problem, as it often captures some spurious correlations between the original and counterfactual endings, instead of the causal relations between conditions and endings. To address this issue, we propose a sketch-and-customize generation model guided by the causality implicated in the conditions and endings. In the sketch stage, a skeleton is extracted by removing words which are conflict to the counterfactual condition, from the original ending. In the customize stage, a generation model is used to fill proper words in the skeleton under the guidance of the counterfactual condition. In this way, the obtained counterfactual ending is both relevant to the original ending and consistent with the counterfactual condition. Experimental results show that the proposed model generates much better endings, as compared with the traditional sequence-to-sequence model.

Human use language not just to convey information but also to express their inner feelings and mental states. In this work, we adapt the state-of-the-art language generation models to generate affective (emotional) text. We posit a model capable of generating affect-driven and topic focused sentences without losing grammatical correctness as the affect intensity increases. We propose to incorporate emotion as prior for the probabilistic state-of-the-art text generation model such as GPT-2. The model gives a user the flexibility to control the category and intensity of emotion as well as the topic of the generated text. Previous attempts at modelling fine-grained emotions fall out on grammatical correctness at extreme intensities, but our model is resilient to this and delivers robust results at all intensities. We conduct automated evaluations and human studies to test the performance of our model, and provide a detailed comparison of the results with other models. In all evaluations, our model outperforms existing affective text generation models.

Human use language not just to convey information but also to express their inner feelings and mental states. In this work, we adapt the state-of-the-art language generation models to generate affective (emotional) text. We posit a model capable of generating affect-driven and topic focused sentences without losing grammatical correctness as the affect intensity increases. We propose to incorporate emotion as prior for the probabilistic state-of-the-art text generation model such as GPT-2. The model gives a user the flexibility to control the category and intensity of emotion as well as the topic of the generated text. Previous attempts at modelling fine-grained emotions fall out on grammatical correctness at extreme intensities, but our model is resilient to this and delivers robust results at all intensities. We conduct automated evaluations and human studies to test the performance of our model, and provide a detailed comparison of the results with other models. In all evaluations, our model outperforms existing affective text generation models.

BackgroundThe current research on the application of generative artificial intelligence in has shown rapid growth, and most of the existing research focuses on general nursing scenarios. And is still in the exploratory stage in the specialty field and lacks a systematic compendium of its application in the field of cardiovascular specialty nursing.ObjectiveA review of studies related to the application of generative artificial intelligence in the field of specialized cardiovascular nursing informing relevant research and practice.MethodsThis review was conducted according to the JBI guidelines for scoping reviews, using the PRISMA-ScR reporting tool.The databases searched included PubMed, Embase, Web of Science, APA PsycNetInfo (Ovid), the Cochrane Library, MEDLINE (Ovid), CINAHL, SinoMed, CNKI, WanFang databases, peer-reviewed studies written in English and Chinese from the time of database construction until March 29, 2025. Eligible studies were screened by title and abstract, and full-text screening was performed by two independent evaluators.ResultsNineteen studies were included in this review, and the main application tools were 9 text generation models, 3 multimodal generation models, 4 temporal prediction models, and 3 chatbots. The application scenarios mainly include clinical decision support, patient health management, and nursing education and counseling. Generative artificial intelligence has outstanding effects in reducing nursing workload and precise intervention, but there are limitations in the field of health education and counseling, such as logical disconnection, poor information quality, and lack of humanistic care.ConclusionsGenerative artificial intelligence provides technical support for the intelligent transformation of cardiovascular specialty care, especially effective in the areas of clinical decision support and patient health management. There is an urgent need to solve the existing problems to promote its in-depth application in this field, and it is suggested that future research focuses on the construction of specialized multimodal models.

Commonsense knowledge is crucial for artificial intelligence systems to understand natural language. Previous commonsense knowledge acquisition approaches typically rely on human annotations (for example, ATOMIC) or text generation models (for example, COMET.) Human annotation could provide high-quality commonsense knowledge, yet its high cost often results in relatively small scale and low coverage. On the other hand, generation models have the potential to automatically generate more knowledge. Nonetheless, machine learning models often fit the training data well and thus struggle to generate high-quality novel knowledge. To address the limitations of previous approaches, in this paper, we propose an alternative commonsense knowledge acquisition framework DISCOS (from DIScourse to COmmonSense), which automatically populates expensive complex commonsense knowledge to more affordable linguistic knowledge resources. Experiments demonstrate that we can successfully convert discourse knowledge about eventualities from ASER, a large-scale discourse knowledge graph, into if-then commonsense knowledge defined in ATOMIC without any additional annotation effort. Further study suggests that DISCOS significantly outperforms previous supervised approaches in terms of novelty and diversity with comparable quality. In total, we can acquire 3.4M ATOMIC-like inferential commonsense knowledge by populating ATOMIC on the core part of ASER. Codes and data are available at https://github.com/HKUST-KnowComp/DISCOS-commonsense.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

Teachers need to provide numerous examples sentences for students to translate in the process of teaching English, but the number of sentences given by teachers to practice subjectively is not enough. Therefore, the study constructs a text generation model using an improved convolutional neural network semantic segmentation method, where the corpus utterances are keyword extracted and new shorter utterances are generated based on the keywords for language learners to practice translation. The research first uses the textRank algorithm to extract semantic keywords to obtain a dataset, and then uses CNN to construct an encoder to achieve semantic encoding of the keyword dataset. However, during the research process, it was found that traditional CNN models are relatively sensitive to the location of input data. Therefore, the research introduces the idea of Decomposition Machine (FM) to improve the encoder. In order to control text generation, research has introduced a weighted additive attention mechanism in the decoding process to associate the meaning of the generated text sequence with the meaning of the keyword set. Based on this, a text generation model for generating a translation related corpus in English teaching is constructed. This results in a text generation model that can be used to generate a translation-linked corpus for English language teaching. The average BLEU value of model 1 is 0.924, Inform value is 98.40, the Success value is 97.64, and the Combine value is 101.24, which can achieve high-quality text generation by the keyword lexical meaning and provide technical guarantee for the establishment of the corpus in educating in English.

AI-based content generation has become increasingly important across various industries, en-abling users to create textual and visual content efficiently using artificial intelligence. This project presents the design and development of a unified AI-powered content generation platform implemented without relying on external APIs. The proposed system utilizes LLaMA-based large language models for text generation and diffusion models for image generation, deployed locally to ensure greater control, data privacy, and reduced dependency on third-party services. The system allows users to generate articles, blogs, sum-maries, and AI-generated images through a single web-based interface. User inputs are processed by the in-tegrated language and image generation models, and the generated outputs are delivered in real time. The system architecture is designed to efficiently manage model inference, content generation workflows, and user interactions. This project demonstrates the feasibility of building a standalone AI content generation system using open-source models and highlights its applicability in domains such as education, digital content creation, and media automation.

We present FaseehGPT, a specialized transformer-based language model designed for high-quality Arabic text generation in resource-constrained environments. Unlike existing Arabic language models that primarily focus on understanding tasks, FaseehGPT is optimized for generative applications while maintaining computational efficiency suitable for deployment on consumer-grade hardware. The model employs a decoder-only transformer architecture with 70.7 million parameters, trained on a carefully curated corpus of 8.7 million Arabic texts spanning colloquial tweets, formal news articles, and classical literature. Our approach leverages the morphological richness of Arabic through strategic tokenization using a pre-trained Arabic BERT tokenizer, enabling effective handling of the language’s complex derivational and inflectional patterns. Extensive evaluation demonstrates FaseehGPT’s capability to generate coherent, contextually appropriate text across multiple Arabic varieties and registers. The model achieves competitive performance while requiring significantly fewer computational resources than comparable systems, with training completed on a single NVIDIA T4 GPU. We provide comprehensive technical details, reproducible training procedures, and make the complete model and codebase publicly available to advance Arabic NLP research. Evaluation metrics show consistent improvement across training epochs, with final perplexity scores indicating strong language modeling performance comparable to larger models in the Arabic domain. https://huggingface.co/alphatechlogics/FaseehGPT

Data-to-text generation task aims at generating text from structured data. In this work, we focus on a relatively new and challenging equation-to-text generation task – generating math word problems from equations and propose a novel equation-to-problem text generation model. Our model first utilizes a template-aware equation encoder and a Variational AutoEncoder (VAE) model to bridge the gap between abstract math tokens and text. We then introduce a topic selector and a topic controller to prevent topic drifting problems. To avoid the commonsense violation issues, we design a pre-training stage together with a commonsense enforcement mechanism. We construct a dataset to evaluate our model through both automatic metrics and human evaluation. Experiments show that our model significantly outperforms baseline models. Further analysis shows our model is effective in tackling topic drifting and commonsense violation problems.

This paper focuses on simulating text dialogues in which impressions between speakers improve during speed dating. This simulation involves selecting an utterance from multiple candidates generated by a text generation model that replicates a specific speaker's utterances, aiming to improve the impression from their dialogue partner. Accurately selecting an utterance that improves the impression is crucial for the simulation. We believe that whether an utterance improves the speaker's impression from their partner may depend on the personalities of both parties. However, recent methods for utterance selection do not consider the impression per utterance or the personalities. To address this issue, we propose a method that predicts whether an utterance improves the speaker's impression from their partner, considering the personalities. The evaluation results showed that personalities are useful in predicting impression changes per utterance. Furthermore, we conducted a human evaluation of simulated dialogues using our method. The results showed that it could simulate dialogues more favorably received than those selected without considering personalities.

Federated Learning Transformers for Personalized Text Generation in Privacy-Sensitive User Settings