INFLUENCE OF LEARNING RATE PARAMETER IN DIFFERENT TRANSFORMER MODELS IN TEXT CLASSIFICATION TASK

Text classification is a task of assigning a set of text documents into predefined classes based on the classifier that learns from training samples; labelled or unlabeled. Binary text classifiers provide a way to separate related documents from a large dataset. However, the existing binary text classifiers are not grounded in reality due to the issue of overfitting. They try to find a clear boundary between relevant and irrelevant objects rather than understand the decision boundary. Normally, the decision boundary cannot be described as a clear boundary because of the numerous uncertainties in text documents. This paper attempts to address this issue by proposing an effective model based on sliding window technique (SW) and Support Vector Machine (SVM) to deal with the uncertain boundary and to improve the effectiveness of binary text classification. This model aims to set the decision boundary by dividing the training documents into three distinct regions (positive, boundary, and negative regions) to ensure the certainty of extracted knowledge to describe relevant information. The model then organizes training samples for the learning task to build a multiple SVMs based classifier. The experimental results using the standard dataset Reuters Corpus Volume 1 (RCV1) and TREC topics for text classification, show that the proposed model significantly outperforms six state-of-the-art baseline models in binary text classification.

Large-scale pre-trained language models such as BERT have brought much better performance to text classification. However, their large sizes can lead to sometimes prohibitively slow fine-tuning and inference. To alleviate this, various compression methods have been proposed; however, most of these methods solely consider reducing inference time, often ignoring significant increases in training time, and thus are even more resource consuming. In this article, we focus on lottery ticket extraction for the BERT architecture. Inspired by observations that representations at lower layers are often more useful for text classification, we propose that we can identify the winning ticket of BERT for binary text classification through adaptive truncation, i.e., a process that drops the top-k layers of the pre-trained model based on simple, fast computations. In this way, the cost for compressing and fine-tuning, as well as inference, can be vastly reduced. We present experiments on eight mainstream binary text classification datasets covering different input styles (i.e., single-text and text-pair), as well as different typical tasks (e.g., sentiment analysis, acceptability judgement, textual entailment, semantic similarity analysis and natural language inference). Compared with some strong baselines, our method saved 78.1% time and 31.7% memory on average, and up to 86.7 and 48% in extreme cases, respectively. We also saw good performance, often outperforming the original language model.

Document classification has already been widely studied. In fact, some studies compared feature selection techniques or feature space transformation whereas some others compared the performance of different algorithms. Recently, following the rising interest towards the Support Vector Machine, various studies showed that the SVM outperforms other classification algorithms. So should we just not bother about other classification algorithms and opt always for SVM?We have decided to investigate this issue and compared SVM to kNN and naive Bayes on binary classification tasks. An important issue is to compare optimized versions of these algorithms, which is what we have done. Our results show all the classifiers achieved comparable performance on most problems. One surprising result is that SVM was not a clear winner, despite quite good overall performance. If a suitable preprocessing is used with kNN, this algorithm continues to achieve very good results and scales up well with the number of documents, which is not the case for SVM. As for naive Bayes, it also achieved good performance.

How to use advanced deep learning technology to build an effective and powerful text classification model, extract text semantic attributes and achieve good classification results on large-scale test datasets has always been the main problem and severe challenge of text classification tasks. The purpose of this paper is to study the application optimization of NLP system in text semantics and text classification based on deep learning technology. A WC-GCN classification algorithm combining GCN and Attention mechanism is proposed. The related tasks of text classification are discussed, including three typical text classification tasks, including sentiment analysis, topic analysis, and news classification. The algorithm model proposed in this paper is verified from the experimental point of view. Compared with other deep learning-based text classification methods, the proposed algorithm model performs better on the public datasets of three text classification tasks. The news text also achieved a high accuracy rate.

Even if recent Transformer-based architectures, such as BERT, achieved impressive results in semantic processing tasks, their fine-tuning stage still requires large scale training resources. Usually, Data Augmentation (DA) techniques can help to deal with low resource settings. In Text Classification tasks, the objective of DA is the generation of well-formed sentences that i) represent the desired task category and ii) are novel with respect to existing sentences. In this paper, we propose a neural approach to automatically learn to generate new examples using a pre-trained sequence-to-sequence model. We first learn a task-oriented similarity function that we use to pair similar examples. Then, we use these example pairs to train a model to generate examples. Experiments in low resource settings show that augmenting the training material with the proposed strategy systematically improves the results on text classification and natural language inference tasks by up to 10% accuracy, outperforming existing DA approaches.

Knowledge enhanced pre-trained language models (K-PLMs) are shown to be effective for many public tasks in the literature, but few of them have been successfully applied in practice. To address this problem, we propose K-AID, a systematic approach that includes a low-cost knowledge acquisition process for acquiring domain knowledge, an effective knowledge infusion module for improving model performance, and a knowledge distillation component for reducing the model size and deploying K-PLMs on resource-restricted devices (e.g., CPU) for real-world application. Importantly, instead of capturing entity knowledge like the majority of existing K-PLMs, our approach captures relational knowledge, which contributes to better improving sentence-level text classification and text matching tasks that play a key role in question answering (QA). We conducted a set of experiments on five text classification tasks and three text matching tasks from three domains, namely E-commerce, Government, and Film&TV, and performed online A/B tests in E-commerce. Experimental results show that our approach is able to achieve substantial improvement on sentence-level question answering tasks and bring beneficial business value in industrial settings.

Abbreviations are unavoidable yet critical parts of the medical text. Using abbreviations, especially in clinical patient notes, can save time and space, protect sensitive information, and help avoid repetitions. However, most abbreviations might have multiple senses, and the lack of a standardized mapping system makes disambiguating abbreviations a difficult and time-consuming task. The main objective of this study is to examine the feasibility of sequence labeling methods for medical abbreviation disambiguation. Specifically, we explore the capability of sequence labeling methods to deal with multiple unique abbreviations in a single text. We use two public datasets to compare and contrast the performance of several transformer models pre-trained on different scientific and medical corpora. Our proposed sequence labeling approach outperforms the more commonly used text classification models for the abbreviation disambiguation task. In particular, the SciBERT model shows a strong performance for both sequence labeling and text classification tasks over the two considered datasets. Furthermore, we find that abbreviation disambiguation performance for the text classification models becomes comparable to that of sequence labeling only when postprocessing is applied to their predictions, which involves filtering possible labels for an abbreviation based on the training data.

Class imbalance is a common and critical challenge in text classification tasks, where the underrepresentation of certain classes often impairs the ability of classifiers to learn minority class patterns effectively. According to the “garbage in, garbage out” principle, even high-performing models may fail when trained on skewed distributions. To address this issue, this study investigates the impact of oversampling techniques, specifically the Synthetic Minority Over-sampling Technique (SMOTE) and thirty of its variants, on two benchmark text classification datasets: TREC and Emotions. Each dataset was vectorized using the MiniLMv2 transformer model to obtain semantically rich representations, and classification was performed using six machine learning algorithms. The balanced and imbalanced scenarios were compared in terms of F1-Score and Balanced Accuracy. This work constitutes, to the best of our knowledge, the first large-scale, systematic benchmarking of SMOTE-based oversampling methods in the context of transformer-embedded text classification. Furthermore, statistical significance of the observed performance differences was validated using the Friedman test. The results provide practical insights into the selection of oversampling techniques tailored to dataset characteristics and classifier sensitivity, supporting more robust and fair learning in imbalanced natural language processing tasks.

Text classification is one of the fundamental tasks in natural language processing. Recently, deep neural networks have achieved promising performance in the text classification task compared to shallow models. Despite of the significance of deep models, they ignore the fine-grained (matching signals between words and classes) classification clues since their classifications mainly rely on the text-level representations. To address this problem, we introduce the interaction mechanism to incorporate word-level matching signals into the text classification task. In particular, we design a novel framework, EXplicit interAction Model (dubbed as EXAM), equipped with the interaction mechanism. We justified the proposed approach on several benchmark datasets including both multilabel and multi-class text classification tasks. Extensive experimental results demonstrate the superiority of the proposed method. As a byproduct, we have released the codes and parameter settings to facilitate other researches.

With the fast-paced prevalence of smartphones, binary short text classification (STC) is becoming a basic and challenging issue, and relevant STC algorithms can be successfully used in spam filtering for short message service (SMS), wechat, microblogging, and so on. In this manuscript, we address the structural feature of SMS documents and propose a structural learning framework, which decomposes the complex binary STC problem according to the SMS document structure, and predicts the final category by combining several sub-predictions. Supported by our index of string-frequence, we also implement some STC domain classifiers. The experimental results show that the performance of two previous STC algorithms can be upgraded by the structural learning framework, and our STC domain classifiers can achieve the state-of-the-art performance on the task of Chinese SMS spam filtering within the structural learning framework.

The paper considers two linguistic models, analyzed the possibility of their use for the text data classification as well as their associations in the integrated texts presentation. A cognitive approach for the text classification issues is presented. An algorithm to identify the words basic level using WordNet is considered. A model for text classification based on the pre-trained word embeddings is presented. The model consists of three layers: embedding layer Long-Short Term Memory (LSTM) layer, and softmax layer. The model was trained and evaluated on the 20 Newsgroups dataset. The classification quality was assessed by F- measure, precision and recall. The obtained results analysis is carried out. Both described models show good results, low scores for some texts are explained. The advantages and limitations of the linguistic models are shown. In future works the authors are going to combine proposed models and modify them. Thus, for model based on word embedding there are pretty vast opportunities for extension: from experimenting with different word embeddings and various distance metrics to more complicated architecture of layers and even promising state of the art artificial neural network models, activation functions and their modifications. In addition, there is research area of proper ensemble strategy selection.

Text classification is a fundamental task in many applications such as topic labeling, sentiment analysis, and spam detection. The text syntactic relationship and word sequence are important and useful for text classification. How to model and incorporate them to improve performance is one key challenge. Inspired by human behavior in understanding text. In this paper, we combine the syntactic relationship, sequence structure, and semantics for text representation, and propose an attention-enhanced capsule network-based text classification model. Specifically, we use graph convolutional neural networks to encode syntactic dependency trees, build multi-head attention to encode dependencies relationship in text sequence, merge with semantic information by capsule network at last. Extensive experiments on five datasets demonstrate that our approach can effectively improve the performance of text classification compared with state-of-the-art methods. The result also shows capsule network, graph convolutional neural network, and multi-headed attention has integration effects on text classification tasks.

Due to the continuous growth and advancement of technology, an enormous volume of text data is generated daily across various sources including social media platforms, websites, search engines, healthcare records, and news articles. Extracting meaningful patterns from text data, such as viewpoints, related theories, journal distribution, facts, and the development of online news text, is a challenging task due to the varying lengths of the texts. One issue arises from the length of the text data itself, and another challenge lies in extracting valuable features, especially in news articles. In the deep learning models, the convolutional neural networks (CNNs) are capable of capturing local features in text data, but unable to capture the structural information or semantic relationships between words. Consequently, a sole CNN network often yields poor performance in text classification tasks, whereas the Gated Recurrent Unit (GRU) is adept at effectively extracting semantic information and understanding the global structural relationships present in textual data. This paper presents a solution to the problem by introducing a new text classification that integrates the strengths of CNN and GRU. The proposed hybrid models incorporate word vectorization and word dispersion in parallel. Initially, the model trains word vectors using the Word2vec model and then leverages the GRU model to capture semantic information from text sentences. Subsequently, the CNN method is employed to capture crucial semantic features, leading to classification using the SoftMax layer. Experimental findings demonstrated that the proposed hybrid GRU_CNN model outperformed and achieved accuracy 97.73% as compared to individual CNN, LSTM, and GRU models in terms of classification effectiveness and accuracy.

Depression is currently one of the most prevalent mental disorders and its incidence has been rising significantly in Malaysia amid the Covid-19 pandemic. While previous studies have demonstrated the potential of artificial intelligence technology in analysing social media texts to detect signs of depression, most of these studies have focused on English textual content. Considering that Mandarin is the second most widely spoken language worldwide, it is worthwhile to explore depression detection techniques specifically tailored for Mandarin textual content. This research aims to examine the effectiveness of the BERT model in text classification, particularly for detecting depression in Mandarin. The study proposes the utilization of the BERT model to analyse social media posts related to depression. The model is trained using the WU3D dataset, which comprises a collection of over 2 million text data sourced from Sina Weibo, a prominent Chinese social media platform. Given the dataset's inherent imbalance, text augmentation techniques were employed to assess whether they contribute to improved model performance. The findings suggest that the BERT model trained on the original dataset outperformed the model trained on the augmented dataset. This implies that the BERT model is well-equipped to handle imbalanced datasets effectively. Furthermore, it is speculated that the augmented dataset did not introduce novel information or knowledge during the model training process. Notably, the highest-performing model achieved an impressive accuracy rate of 88% on the testing dataset.

Short text classification plays an important role in NLP and its applications span a wide range of activities such as sentiment analysis, spam detection. Recently, attention mechanism is widely used in text classification task. Inspired by this, a text classification model based on multi-attention network(MAN) is proposed in this study, which perform well in extracting information related to text category. In our model, we combine the textual information based on multi-attention mechanism, which enables model to focus on global information of the sentence. We tested effectiveness of our model using several standard text classification datasets. Experiment told that our model achieved state-of-the-art results on all datasets.