Clinical short text classification method based on ALBERT and GAT

Modelling short text is challenging due to the small number of word co-occurrence and insufficient semantic information that affects downstream Natural Language Processing (NLP) tasks, for example, text classification. Gathering information from external sources is expensive and may increase noise. For efficient short text classification without depending on external knowledge sources, we propose Expressive Short text Classification (EStC). EStC consists of a novel document context-aware semantically enriched topic model called the Short text Topic Model (StTM) that captures words, topics and documents semantics in a joint learning framework. In StTM, the probability of predicting a context word involves the topic distribution of word embeddings and the document vector as the global context, which obtains by weighted averaging of word embeddings on the fly simultaneously with the topic distribution of words without requiring an additional inference method for the document embedding. EStC represents documents in an expressive (number of topics × number of word embedding features) embedding space and uses a linear support vector machine (SVM) classifier for their classification. Experimental results demonstrate that EStC outperforms many state-of-the-art language models in short text classification using several publicly available short text data sets.

Short text understanding is a key task and popular issue in current natural language processing. Because the content of short texts is characterized by sparsity and semantic limitation, the traditional search methods that analyze only the semantics of literal text for short text understanding and similarity matching have certain restrictions. In this paper, we propose a combined method based on knowledge-based conceptualization and a transformer encoder. Specifically, for each term in a short text, we obtain its concepts and enrich the short text information from a knowledge base based on cooccurrence terms and concepts, construct a convolutional neural network (CNN) to capture local context information, and introduce the subnetwork structure based on a transformer embedding encoder. Then, we embed these concepts into a low-dimensional vector space to obtain more attention from these concepts based on a transformer. Finally, the concept space and transformer encoder space construct the understanding models. An experiment shows that the method in this paper can effectively capture more semantics of short texts and can be applied to a variety of applications, such as short text information retrieval and short text classification.

Long Short-Term Memory (LSTM) networks are unique to exercise data in its memory cell with long-term memory as Natural Language Processing (NLP) tasks have inklings of intensive time and computational power due to their complex structures like magnitude language model Transformer required to pre-train and learn billions of data performing different NLP tasks. In this paper, a dynamic chaotic model is proposed for the objective of transforming neurons states in network with neural dynamic characteristics by restructuring LSTM as Chaotic Neural Oscillatory-Long-Short Term Memory (CNO-LSTM), where neurons in LSTM memory cells are weighed in substitutes by oscillatory neurons to speed up computational training of language model and improve text classification accuracy for real-world applications. From the implementation perspective, five popular datasets of general text classification including binary, multi classification and multi-label classification are used to compare with mainstream baseline models on NLP tasks. Results showed that the performance of CNO-LSTM, a simplified model structure and oscillatory neurons state in exercising different types of text classification tasks are above baseline models in terms of evaluation index such as Accuracy, Precision, Recall and F1. The main contributions are time reduction and improved accuracy. It achieved approximately 46.76% of the highest reduction training time and 2.55% accuracy compared with vanilla LSTM model. Further, it achieved approximately 35.86% in time reduction compared with attention model without oscillatory indicating that the model restructure has reduced GPU dependency to improve training accuracy.

The graph attention network (GAT) [1] has started to become a mainstream neural network architecture since 2018, yielding remarkable performance gains in various natural language processing (NLP) tasks. Although GAT has reached the state-of-the-art (SOTA) performance as a recent success in relation prediction in knowledge graph, the current model is still limited by the following two aspects: (1) the existing model only considers the neighbors from the inbound-direction of the given entity, but ignores the rich neighborhood information from outbound-directions; (2) the existing model only uses the $k$ -th hop output to learn the multi-hop embeddings, which leads to the loss of a large amount of early-stage embedding information (e.g., one-hop) at the graph attention step. In this study, we propose a novel bidirectional graph attention network (BiGAT) to learn the hierarchical neighbor propagation. In our proposed BiGAT, an inbound-directional GAT and an outbound-directional GAT are introduced to capture sufficient neighborhood information before propagating the bidirectional neighborhood information to learn the multi-hop feature embeddings in a hierarchical manner. Experiments conducted on the four publicly available datasets show that BiGAT achieves the competitive results in comparison to other SOTA methods.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

Incorporating context-relevant concepts into convolutional neural networks for short text classification

Text extraction is an important part of natural language processing (NLP) tasks. Most NLP tasks like text classification, machine translation, text-to-speech, text-based language identification, text summarization, and named-entity recognition involve the use of textual data. Such data is limited for low-resourced languages making it difficult to experiment advanced NLP techniques on these languages. This paper presents a Python-based toolkit for text analysis and text extraction from different types of images, documents, and audio files. The toolkit is built as a library that has functions that can be imported and utilized for text extraction.

Towards a Novel Weakly Supervised Joint Approach of Named Entity Recognition and Normalization for Noisy Text

Text classification is a crucial task in natural language processing. Due to the characteristics of text structure, achieving the best result remains an ongoing challenge. In this paper, we propose an ensemble model which outperforms the state-of-the-art. We first utilize rule-based n-gram approach to extend corpus. Then two different features, global dependencies of word and local semantic feature, are extracted by gated recurrent unit and global average pooling model respectively. In order to take advantage of the complementarity of the global and local features, a decision-level fusion is applied to fuse those different kinds of features. We evaluate the quality of our model on various public datasets, including sentiment analysis, ontology classification and text categorization. Experimental results show that our model can effectively learn representations for language modeling, and achieves the best accuracy of text categorization.

The dominant paradigm in Natural Language Processing (NLP) has traditionally relied on sequential modelling, often overlooking the rich, non-linear structural dependencies inherent in linguistic data. Graph Neural Networks (GNNs) offer a powerful alternative by modelling text as structured graphs, yet a systematic evaluation of their comparative efficacy across tasks of varying structural complexity remains under-explored. This paper presents a comprehensive comparative study of three prominent GNN architectures- Graph Convolutional Networks (GCN), Graph Attention Networks (GAT), and Graph SAGE- applied to three distinct NLP tasks: Text Classification, Semantic Role Labelling (SRL), and Abstract Meaning Representation (AMR) Parsing. We construct task-specific graph topologies, utilizing heterogeneous corpus graphs for document classification and syntactic dependency trees for SRL and AMR. Our empirical results, derived from benchmarks on the 20 Newsgroups, CoNLL-2009, and AMR 2.0 datasets, reveal a clear correlation between architectural properties and task complexity. While GCNs provide a robust and efficient baseline for global text classification (Accuracy: 96.6%), they struggle with the fine-grained structural modelling required for semantic tasks. Conversely, GATs demonstrate superior performance on syntax-heavy tasks, achieving a Labelled F1 score of 85.8% on SRL and a Smatch score of 74.5 on AMR parsing, significantly outperforming the GCN baseline (82.5% and 71.2%, respectively). These findings suggest that the anisotropic aggregation capability of attention mechanisms is critical for capturing the nuanced, long-range dependencies in natural language, establishing GAT as the preferred architecture for structure-dependent NLP applications.

In this paper, we present the preliminary results on the analysis of deep learning terms used for natural language processing (NLP) tasks. We propose a statistical analysis of papers published from 2012 to 2015 in the main ACL conferences. Our aim is investigating which DL term, and consequently which DL method, is mostly used for each specific NLP task, since its introduction in the field. In order to do this, our first contribution is the development of two terminological lists, referring respectively to DL methods for text analysis and NLP tasks. The list of deep learning terms contains 41 terms and acronyms, as well as a NLP term list contains 145 terms and acronyms. From our corpus, the frequencies of various terms have been extracted with respect to the ACL conference and the publication year. After the preliminary data analysis, we decided to restrict the extraction process to abstract texts. We applied multivariate techniques called correspondence analysis in order to visualize and evaluate the joint behavior of our variables.

In this study, we explore the potential of machine learning (ML) models after different text representation methods on the balanced IMDB dataset, which is widely regarded as a gold standard in sentiment analysis, one of the Natural Language processing (NLP) tasks. On the open source IMDB movie reviews dataset, we first undertake data cleaning and text representation with data preprocessing steps. Then, we apply sentiment classification using different ML models. In order to evaluate the models, we used precision (P), recall (R), F1-score (F1), and area under curve (AUC), as well as receiver operating characteristic (ROC). It is worth noting that text feature extraction with Bidirectional Encoder Representations from Transformers (BERT) provided the highest performance in all models, with the SVM model offering particularly promising results. In this model, we observed the following results: ACC 0.9033, F1 0.9308, R 0.9015, R 0.9015, P 0.9072, AUC 0.9638, and ROC 0.96. These findings suggest that NLP techniques and, in particular, machine learning models that employ BERT may offer high levels of accuracy and reliability in text classification problems. It would be beneficial for future studies to validate these findings using BERT on different NLP tasks. This would help to evaluate the effectiveness and applicability of the models in practice.

Work orders, equipment information, technical records and best practices documents contain within them a wealth of insights related to Equipment Maintenance which can be unlocked with Natural Language Processing tasks like classification, clustering, named entity recognition or part of speech tagging. But obtaining large enough labelled data sets in Equipment Maintenance domain manually is prohibitive and very expensive. This lack of labeled Equipment Maintenance data can be overcome with Large Language Models (LLMs) such as GPT-3, BERT that are pretrained transformer networks, considered state-of-the-art when it comes to Natural Language Processing (NLP) tasks. However, the vocabulary understood by these LLMs are mostly from English Language and need to be fine-tuned to understand industry and organization specific vocabulary and acronyms. This paper explores the potential of a domain specific LLM model for oil and gas industries. Data that are of good quality and that provide a comprehensive overview of industry are collected. This corpus of text contains documents like work orders, equipment data and technical documents. A custom tokenizer is trained on this data to identify domain specific terminology. A comparative study is done with other off-the-shelf tokenizers: BERT and RoBERTa, to compare the effectiveness of the tokenization. With millions of work orders and equipment documents, training pipelines had to parallelized so that training can occur on multiple GPUs. A comprehensive study of multiple training methods is done in this paper. Model and tokenizer developed were packaged and archived to be consumed in machine learning pipelines to specific use-cases across the organization. For an organization adopting digital transformation, the availability of an organization specific LLM is an enabler to extract insights from millions of documents containing free text. The applicability of such models spans across multiple disciplines like Maintenance, Reliability, Safety etc. and streamlines the development of highly accurate and robust text analytics.

Towards a Novel Weakly Supervised Joint Approach of Named Entity Recognition and Normalization for Noisy Text

Many tasks in natural language processing like offensive, toxic, or emotional text classification are subjective by nature. Humans tend to perceive textual content in their own individual way. Existing methods commonly rely on the agreed output values, the same for all consumers. Here, we propose personalized solutions to subjective tasks. Our four new deep learning models take into account not only the content but also the specificity of a given human. The models represent different approaches to learning the representation and processing data about text readers. The experiments were carried out on four datasets: Wikipedia discussion texts labelled with attack, aggression, and toxicity, as well as opinions annotated with ten numerical emotional categories. Emotional data was considered as multivariate regression (multitask), whereas Wikipedia data as independent classifications. All our models based on human biases and their representations significantly improve the prediction quality in subjective tasks evaluated from the individual’s perspective.