Exploring the Potential of Transformers in Natural Language Processing -A Study on Text Classification

Natural Language Processing (NLP) is a science that integrates computer knowledge, mathematical knowledge and linguistic knowledge, and text classification and recognition is considered to be an important research field and direction of natural language processing. This paper mainly studies the realization of text classification model through the processing method of text data in natural language processing and the theoretical knowledge and technical means of machine learning. We summarize the existing text classification algorithms, analyze their applicable scenarios, and optimize on the basis of these algorithm models. The paper proposes a Chinese text classification algorithm based on weight preprocessing. Algorithm based on the weight preprocessing link, so that the optimized classifier model can improve the accuracy of the existing text classifier. In this paper, the English Newsgroups corpus is used for experimental verification. The experimental results show that the classification accuracy and accuracy of the improved algorithm are better than those of the traditional text classification algorithm, thereby improving the accuracy of English text classification.

Natural Language Processing (NLP) involves the use of algorithms and models and various computational techniques to analyze, process, and generate natural language data, including speech and text. NLP helps computers interact with humans in a more natural way, which has become increasingly important as more humancomputer interactions take place. NLP allows machines to process and analyze voluminous unstructured data, including social media posts, newspaper articles, reviews from customers, emails, and others. It helps organizations extract insights, automate tasks, and improve decision-making by enabling machines to understand and generate human-like language. A linguistic background is essential for understanding NLP. Linguistic theories and models help in developing NLU systems, as NLP specialists need to understand the structure and rules of language. NLU systems are organized into different components, including language modelling, parsing, and semantic analysis. NLU systems may be assessed through the use of metrics that includes measures like precision and recall, as well as indicators that convey meaningful information that include F1 score and others. Semantics and knowledge representation are central to NLU, as they involve understanding the meaning of words and sentences and representing this information in a way that machines can use. Approaches to knowledge representation include semantic networks, ontologies, and vector embeddings. Language modelling is an essential step in NLP that sees usage in applications like speech recognition, text generation, and text completion and also in areas such as machine translation. Ambiguity Resolution remains a major challenge in NLP, as language is often ambiguous and context-dependent. Some common applications of NLP include sentiment analysis, chatbots, virtual assistants, machine translation, speech recognition, text classification, text summarization, and information extraction. In this chapter, we show the applicability of a popular unsupervised learning technique, viz., clustering through K-Means. The efficiency provided by the K-Means algorithm can be improved through the use of an optimization loop. The prospects for NLP are promising, with an increasing demand for AI-powered language technologies in various industries, including healthcare, finance, and e-commerce. There is also a growing need for ethical and responsible AI systems that are transparent and accountable.

Big data analytics (BDA) is an advanced analytic technique used with very large and diverse sets of data from different sources. Natural language processing (NLP) is a technology that interfaces with different fields such as computer science, linguistics, and human-computer interactions. Over the past few years, there is a growing number of firms, which are using different BDA and NLP applications in their businesses. Only a few of the research have investigated different dimensions of NLP and BDA and their impacts on the overall organizational performance. There is a growing interest among researchers and practitioners in understanding the consequences for firms that adopt BDA and NLP applications. In this context, the aim of this article is to determine the factors for the usage of BDA and NLP applications in business. With the help of dynamic capability view theory and existing literature, a theoretical model was developed conceptually. Later, the model was validated using structural equation modeling approach considering 1287 samples from 23 firms, primarily based in Asia and Europe, which use NLP and BDA applications. The article finds that NLP and BDA applications help the firms to improve their operational efficiency, which in turn improves the overall firm performance.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

The dominant paradigm in Natural Language Processing (NLP) has traditionally relied on sequential modelling, often overlooking the rich, non-linear structural dependencies inherent in linguistic data. Graph Neural Networks (GNNs) offer a powerful alternative by modelling text as structured graphs, yet a systematic evaluation of their comparative efficacy across tasks of varying structural complexity remains under-explored. This paper presents a comprehensive comparative study of three prominent GNN architectures- Graph Convolutional Networks (GCN), Graph Attention Networks (GAT), and Graph SAGE- applied to three distinct NLP tasks: Text Classification, Semantic Role Labelling (SRL), and Abstract Meaning Representation (AMR) Parsing. We construct task-specific graph topologies, utilizing heterogeneous corpus graphs for document classification and syntactic dependency trees for SRL and AMR. Our empirical results, derived from benchmarks on the 20 Newsgroups, CoNLL-2009, and AMR 2.0 datasets, reveal a clear correlation between architectural properties and task complexity. While GCNs provide a robust and efficient baseline for global text classification (Accuracy: 96.6%), they struggle with the fine-grained structural modelling required for semantic tasks. Conversely, GATs demonstrate superior performance on syntax-heavy tasks, achieving a Labelled F1 score of 85.8% on SRL and a Smatch score of 74.5 on AMR parsing, significantly outperforming the GCN baseline (82.5% and 71.2%, respectively). These findings suggest that the anisotropic aggregation capability of attention mechanisms is critical for capturing the nuanced, long-range dependencies in natural language, establishing GAT as the preferred architecture for structure-dependent NLP applications.

BackgroundMachine learning systems are part of the field of artificial intelligence that automatically learn models from data to make better decisions. Natural language processing (NLP), by using corpora and learning approaches, provides good performance in statistical tasks, such as text classification or sentiment mining.ObjectiveThe primary aim of this systematic review was to summarize and characterize, in methodological and technical terms, studies that used machine learning and NLP techniques for mental health. The secondary aim was to consider the potential use of these methods in mental health clinical practiceMethodsThis systematic review follows the PRISMA (Preferred Reporting Items for Systematic Review and Meta-analysis) guidelines and is registered with PROSPERO (Prospective Register of Systematic Reviews; number CRD42019107376). The search was conducted using 4 medical databases (PubMed, Scopus, ScienceDirect, and PsycINFO) with the following keywords: machine learning, data mining, psychiatry, mental health, and mental disorder. The exclusion criteria were as follows: languages other than English, anonymization process, case studies, conference papers, and reviews. No limitations on publication dates were imposed.ResultsA total of 327 articles were identified, of which 269 (82.3%) were excluded and 58 (17.7%) were included in the review. The results were organized through a qualitative perspective. Although studies had heterogeneous topics and methods, some themes emerged. Population studies could be grouped into 3 categories: patients included in medical databases, patients who came to the emergency room, and social media users. The main objectives were to extract symptoms, classify severity of illness, compare therapy effectiveness, provide psychopathological clues, and challenge the current nosography. Medical records and social media were the 2 major data sources. With regard to the methods used, preprocessing used the standard methods of NLP and unique identifier extraction dedicated to medical texts. Efficient classifiers were preferred rather than transparent functioning classifiers. Python was the most frequently used platform.ConclusionsMachine learning and NLP models have been highly topical issues in medicine in recent years and may be considered a new paradigm in medical research. However, these processes tend to confirm clinical hypotheses rather than developing entirely new information, and only one major category of the population (ie, social media users) is an imprecise cohort. Moreover, some language-specific features can improve the performance of NLP methods, and their extension to other languages should be more closely investigated. However, machine learning and NLP techniques provide useful information from unexplored data (ie, patients’ daily habits that are usually inaccessible to care providers). Before considering It as an additional tool of mental health care, ethical issues remain and should be discussed in a timely manner. Machine learning and NLP methods may offer multiple perspectives in mental health research but should also be considered as tools to support clinical practice.

Text classification is one of the important applications of Natural Language Processing (NLP) in which text is classified into pre-defined categories or classes. Several state-of-the-arts have already been conducted on text classification in different Indian languages, however, limited researches on text classification performed in Odia. Moreover, the news headlines classification is a text classification technique which is performed in our study mainly focusing on Odia language. The Convolutional Neural Network (CNN) based Deep Learning technique is used to classify the Odia news headlines dataset available at Kaggle repository. The dataset consisting of approx. 15,200 Odia news headlines is split into training and testing sets of 80:20 ratio and is categorized into different classes, such as sports, business, and entertainment by using CNN. The efficacy and efficiency of the proposed CNN-based Odia news headlines classification is evaluated in contrary to other existing machine learning and deep learning models, such as Neural Networks, LSTM (Long Short-Term Memory), and Bi-LSTM (Bidirectional Long Short-Term Memory) with respect to different performance metrics like precision, accuracy, recall, F1-score, training time, and AUC-ROC (Area Under the Receiver Operating Characteristic Curve). The simulation results revealed that the proposed CNN model outperformed with respect to other models.

Analytics of machine learning-based algorithms for text classification

Text classification is a fundamental task in natural language processing (NLP). Deep-learning-based text classification methods usually have two stages: training and inference. However, the training dataset is only used in the training stage. To make full use of the training dataset in the inference stage in order to improve model performance, we propose a k-nearest neighbors retrieval augmented method (KRA) for deep-learning-based text classification models. KRA works by first constructing a storage system that stores the embeddings of the training samples during the training stage. During the inference stage, the model retrieves the top k-nearest neighbors of the testing text from the storage. Then, we use text augmentation methods to expand the retrieved neighbors, including traditional augmentation methods and a large language model (LLM)-based method. Next, the method weights the augmented neighbors based on their distances from the target text and incorporates their labels into the inference of the final results accordingly. We evaluate our KRA method on six benchmark datasets using four commonly used deep learning models: CNN, LSTM, BERT, and RoBERTa. The results demonstrate that KRA significantly improves the classification performance of these models, with an average accuracy improvement of 0.3% for BERT and up to 0.4% for RoBERTa. These improvements highlight the effectiveness and generalizability of KRA across different models and datasets, making it a valuable enhancement for a wide range of text classification tasks.

Hierarchical and lateral multiple timescales gated recurrent units with pre-trained encoder for long text classification

Text classification is a very important problem in Nature Language Processing (NLP). The text classification based on shallow machine-learning models takes too much time and energy to extract features of data, but only obtains poor performance. Recently, deep learning methods are widely used in text classification and result in good performance. In this paper, we propose a Convolutional Neural Network (CNN) with multi-size convolution and multi-type pooling for text classification. In our method, we adopt CNNs to extract features of the texts and then select the important information of these features through multi-type pooling. Experiments show that the CNN with multi-convolution and multi-type pooling (CNN-MCMP) obtains better performance on text classification compared with both the shallow machine-learning models and other CNN architectures.

Incorporating context-relevant concepts into convolutional neural networks for short text classification

As one of the core tasks of natural language processing technology, text classification methods general-ly face the problems of insufficient global semantic capture and limited feature focusing ability when processing long texts or complex semantics. To address this issue, a deep learning model that integrates improved convolutional neural networks, unbalanced bidirectional long short-term memory networks, and multi-head attention mechanisms is proposed. Utilizing an improved bidirectional long short-term memory network to capture global semantic information, while dynamically focusing on key features through a multi head attention mechanism to enhance the model's adaptability to classification tasks. The performance of the model is validated through experiments on AG News (short text) and IMDb (long text) datasets. The results show that in short text classification, the proposed method has an accu-racy rate of 96% and a classification error rate of only 1.46%. In the task of long text classification, the method proposed in the study has a product under the curve of 0.98. In adversarial attack testing, the accuracy rates of adversarial samples generated by different methods are 92.85% and 90.63%, respec-tively, with the lowest robustness degradation rates of 3.72% and 5.49%, respectively. In cross domain generalization testing, it shows the least classification errors and superior cross domain adaptability. These results validate the high performance, robustness, and wide applicability of the method. The re-search indicates that this approach can validly improve the performance of text classification and pro-vide new solutions for natural language processing related tasks in long text and multi-category scenar-ios.

Text classification is a key task of the Natural Language Processing (NLP) field that aims at assigning predefined categories to textual documents. Performing text classification requires features that effectively represent the content and the meaning of textual documents. Selecting a suitable method for term weighting is of central importance and can improve the quality of the classification method. In this paper, we propose to a new text classification solution to perform Category-based Feature Augmentation (CFA) on the document representation. First, a term-category feature matrix is derived from a modified version of the supervised Term-Frequency Inverse-Category-Frequency (TF-ICF) weighting model. This is done by embedding the TF-ICF matrix in a one-layer feed-forward neural network. The latter is trained using the gradient descent algorithm allowing to iteratively update the term-category matrix until reaching convergence. The model produces category-based feature vector representations that are used to augment the document representations and perform the classification task. Experimental results on four benchmark datasets show that our lean model approach improves text classification accuracy and is significantly more efficient compared with its deep model alternatives.

Creating deep neural networks for text classification tasks using grammar genetic programming