Explicit Interaction Model towards Text Classification

Long Short-Term Memory (LSTM) networks are unique to exercise data in its memory cell with long-term memory as Natural Language Processing (NLP) tasks have inklings of intensive time and computational power due to their complex structures like magnitude language model Transformer required to pre-train and learn billions of data performing different NLP tasks. In this paper, a dynamic chaotic model is proposed for the objective of transforming neurons states in network with neural dynamic characteristics by restructuring LSTM as Chaotic Neural Oscillatory-Long-Short Term Memory (CNO-LSTM), where neurons in LSTM memory cells are weighed in substitutes by oscillatory neurons to speed up computational training of language model and improve text classification accuracy for real-world applications. From the implementation perspective, five popular datasets of general text classification including binary, multi classification and multi-label classification are used to compare with mainstream baseline models on NLP tasks. Results showed that the performance of CNO-LSTM, a simplified model structure and oscillatory neurons state in exercising different types of text classification tasks are above baseline models in terms of evaluation index such as Accuracy, Precision, Recall and F1. The main contributions are time reduction and improved accuracy. It achieved approximately 46.76% of the highest reduction training time and 2.55% accuracy compared with vanilla LSTM model. Further, it achieved approximately 35.86% in time reduction compared with attention model without oscillatory indicating that the model restructure has reduced GPU dependency to improve training accuracy.

Text FCG: Fusing Contextual Information via Graph Learning for text classification

Text extraction is an important part of natural language processing (NLP) tasks. Most NLP tasks like text classification, machine translation, text-to-speech, text-based language identification, text summarization, and named-entity recognition involve the use of textual data. Such data is limited for low-resourced languages making it difficult to experiment advanced NLP techniques on these languages. This paper presents a Python-based toolkit for text analysis and text extraction from different types of images, documents, and audio files. The toolkit is built as a library that has functions that can be imported and utilized for text extraction.

Machine learning methods, including Deep learnings are popular for data processing. Deep learning methods have shown great promise in applied in natural language processing (NLP) tasks. Text classification is a supervised machine learning task that involves labelled documents to train classifier. Previous works involving machine learning and deep learning methods for text classification have been tested with relatively small- sized data instances. In this paper, we compared the performance of the machine leaning and deep learning algorithms in text classification task. This paper also studied, explored and compared the scalability of these methods with respect to bigger data instances. We used support vector machines (SVM), Logistic regression, Random forest and Naive Bayes Machine leaning algorithms and convolutional neural network (CNN) deep learning method. The task involved a multi-class classification problem involving six (6) classes consisting of six thousand (6,000) data instances with an average of 20 sentences in each data instance. The CNN deep leaning algorithm outperformed all the machine learning algorithms, achieving an accuracy of over 85%. This is because the filter weights are leaning are updated in backward propagation in each epoch, hence, this result in better result compared to the traditional methods.

Creating deep neural networks for text classification tasks using grammar genetic programming

Text classification is one of the basic tasks of natural language processing. In recent years, deep learning has been widely used in text classification tasks. The representative one is the convolutional neural network. The convolutional neural network(CNN) is limited by the size of the local window and can only extract local features of the text. For long texts like news, CNN cannot learn the longterm dependence of the long text. Another model of deep learning recurrent neural networks based on long short-term memory (LSTM) can learn the long-term dependence of text. Therefore, in the work of this paper, combining the advantages of CNN and LSTM, a LSTM_CNN Hybrid model is constructed for Chinese news text classification tasks. We first use LSTM to learn the longterm dependence of text, then we design a shallow convolution structure to further extract the semantic features of the text, and finally use the max-pooling operation to filter to obtain important features for classification. The model we proposed has achieved good results on the News dataset.

Text classification is a fundamental task in natural language processing (NLP). Deep-learning-based text classification methods usually have two stages: training and inference. However, the training dataset is only used in the training stage. To make full use of the training dataset in the inference stage in order to improve model performance, we propose a k-nearest neighbors retrieval augmented method (KRA) for deep-learning-based text classification models. KRA works by first constructing a storage system that stores the embeddings of the training samples during the training stage. During the inference stage, the model retrieves the top k-nearest neighbors of the testing text from the storage. Then, we use text augmentation methods to expand the retrieved neighbors, including traditional augmentation methods and a large language model (LLM)-based method. Next, the method weights the augmented neighbors based on their distances from the target text and incorporates their labels into the inference of the final results accordingly. We evaluate our KRA method on six benchmark datasets using four commonly used deep learning models: CNN, LSTM, BERT, and RoBERTa. The results demonstrate that KRA significantly improves the classification performance of these models, with an average accuracy improvement of 0.3% for BERT and up to 0.4% for RoBERTa. These improvements highlight the effectiveness and generalizability of KRA across different models and datasets, making it a valuable enhancement for a wide range of text classification tasks.

Incorporating context-relevant concepts into convolutional neural networks for short text classification

Text classification is a fundamental task in natural language processing (NLP). Context semantics can greatly improve the accuracy of text classification tasks. Although there are some popular methods in obtaining semantics, current context semantic analysis techniques, due to limited accuracy, are still a great bottleneck for text classification. This paper introduces a novel model, the densely connected Bidirectional LSTM with Max-pooling of CNN network (Dense-BiLSTM-MP), which greatly enhances the context of semantic information. In this model, a densely connected bidirectional long short-term memory (BiLSTM) model, as well as multiple max-pooling layers of convolutional network, are applied to obtain an increasingly enhanced assessment of context, and extract the key features, respectively. Experiments were conducted on four public datasets: YELP, 20NewsGroup, THUNews and AG. The experimental results show that the proposed model outperforms state of the art methods on several datasets. Furthermore, discussions on the Dense-BiLSTM-MP model’s performance in short texts and long texts were given, respectively.

The advances in deep learning have led to great achievements in many Natural Language Processing (NLP) tasks. With the nature of language, i.e., sequential data, most NLP tasks can be framed into the sequence learning framework, such as text generation. As one of the most important foundations for modern NLP techniques, autoregressive generation models have achieved dominant performance in a great deal of NLP tasks. Therefore, this thesis emphasizes improving the autoregressive generation model for different NLP tasks. While many tasks can naturally fit into the sequence learning framework, some of them, e.g., building discourse parsing tree, require sophisticated designs to fit into neural models. Therefore, this thesis firstly emphasizes a novel unified framework for discourse parsing, which builds a discourse tree in a top-down depth-first manner, and it frames the task as an autoregressive generation task with the goal of each step being the prediction of the node position given a piece of text. The proposed approach is proven effective with extensive empirical experiments. In addition, I extend the above framework by proposing a hierarchical decoder, which leverages the information from parents and siblings of the nodes that are currently processed. The proposed decoder utilizes the nature of the tree structure and further improves the experiment performance on both discourse parsing and dependency parsing tasks. On the other hand, the de facto strategies, i.e., cross entropy loss and teacher forcing, for training the autoregressive generation models have been shown problematic in certain aspects. For example, cross entropy loss, which is one of the widely leveraged training objective functions, often leads to text degeneration in text generation, and teacher forcing suffers from the exposure bias problem, where there exists a mismatch between the training and testing setup. For text degeneration, I introduce a class of diminishing attentions, which enforces the submodularity of the coverage calculated by cross attention in the sequence-to-sequence model. The proposed diminishing attentions achieve notable improvement on several neural text generation tasks, including text summarization, machine translation, and image paragraph generation. Further, I propose a novel training objective, ScaleGrad, to replace cross entropy, which significantly reduces the degeneration problem in different text generation tasks. In fact, ScaleGrad can be extended to problems beyond text degeneration. It provides wide flexibility to inject different inductive biases into the text generation model by directly modifying the gradient information in the output layer. Next, for the exposure bias problem, this thesis introduces a novel type of scheduled sampling based on training accuracy, which requires only minimal hyper-parameter tuning compared to existing scheduled sampling methods. Additionally, a novel imitation loss is proposed to further enforce the model’s generative behavior to match the teacher-forced behavior. Moreover, this thesis demonstrates that reducing exposure bias can improve the robustness of language models against repetition and toxic errors.

Many tasks in natural language processing like offensive, toxic, or emotional text classification are subjective by nature. Humans tend to perceive textual content in their own individual way. Existing methods commonly rely on the agreed output values, the same for all consumers. Here, we propose personalized solutions to subjective tasks. Our four new deep learning models take into account not only the content but also the specificity of a given human. The models represent different approaches to learning the representation and processing data about text readers. The experiments were carried out on four datasets: Wikipedia discussion texts labelled with attack, aggression, and toxicity, as well as opinions annotated with ten numerical emotional categories. Emotional data was considered as multivariate regression (multitask), whereas Wikipedia data as independent classifications. All our models based on human biases and their representations significantly improve the prediction quality in subjective tasks evaluated from the individual’s perspective.

Tsetlin Machine (TM) is an interpretable pattern recognition algorithm based on propositional logic, which has demonstrated competitive performance in many Natural Language Processing (NLP) tasks, including sentiment analysis, text classification, and Word Sense Disambiguation. To obtain human-level interpretability, legacy TM employs Boolean input features such as bag-of-words (BOW). However, the BOW representation makes it difficult to use any pre-trained information, for instance, word2vec and GloVe word representations. This restriction has constrained the performance of TM compared to deep neural networks (DNNs) in NLP. To reduce the performance gap, in this paper, we propose a novel way of using pre-trained word representations for TM. The approach significantly enhances the performance and interpretability of TM. We achieve this by extracting semantically related words from pre-trained word representations as input features to the TM. Our experiments show that the accuracy of the proposed approach is significantly higher than the previous BOW-based TM, reaching the level of DNN-based models.

Text classification is one of the most popular tasks in natural language processing, especially in the context of sentiment classification. Insufficient training data poses a significant challenge in many text classification studies. This research focuses on optimizing classification performance using the Passive Aggressive (PA) algorithm, leveraging limited training data. It compares conventional text representation methods like TF-IDF with modern approaches employing word embeddings such as FastText and BERT. The primary dataset encompasses sentiment issues related to Kaesang Pangarep's appointment as the chairman of PSI, gathered through Twitter crawling, and classified into positive, negative, and neutral sentiment labels. Two versions of the training data, each containing only 300 balanced tweets for positive, negative, and neutral classes, were used. The data was split 80% for training and 20% for validation in the search for an optimal model. External data with different issues and pre-existing sentiment labels was used to augment the training data. Experimental results demonstrated that the BERT language model, used as input features for the Passive Aggressive method with hyperparameter tuning, outperformed TF-IDF features. Evaluation on the test data revealed that BERT features with Passive Aggressive achieved an F1-score of 0.52, surpassing the conventional TF-IDF representation with an F1-score of 0.42. The utilization of the BERT language model significantly contributed to improving text classification performance in the field of natural language processing, particularly for the Passive Aggressive method.

The goal of text classification is to assign labels to text units accurately, which is a basic task in natural language processing. This technology has shown great value in many practical application scenarios, covering spam detection, emotional tendency analysis, user intent recognition, and many other aspects. In recent years, because of the excellent performance of graph convolutional neural networks (GCNs) in processing non-European spatial data, it has become a research hotspot in text classification and has been widely adopted. This paper first introduces the background knowledge and working principle of graph convolutional neural networks. Then the text classification method based on GCNs is described. The last thing discussed is the limitations of graph convolutional neural networks, as well as the difficulties and future development directions in this field. This paper serves as a guide for researchers and practitioners in relevant fields to fully comprehend the latest advancements in text classification methods using graph convolutional neural networks, and by revealing its limitations and challenges, it also provides guidance for future research and development work, intending to promote technical progress and application expansion in this field.

Text classification is the most fundamental and essential task in natural language processing. The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, raising the need for a comprehensive and updated survey. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021, focusing on models from traditional models to deep learning. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification. We then discuss each of these categories in detail, dealing with both the technical developments and benchmark datasets that support tests of predictions. A comprehensive comparison between different techniques, as well as identifying the pros and cons of various evaluation metrics are also provided in this survey. Finally, we conclude by summarizing key implications, future research directions, and the challenges facing the research area.