The Automatic Text Classification Method Based on BERT and Feature Union

With the development of Internet technology and the transformation of news media to informatization, news images, texts, and other information have exploded. The news image and text classification can effectively solve the disorder problem of news information. The early news image text classification is to establish artificial classifiers according to specific classification rules, but the classification error rate is high and the classification speed is slow. Later, machine learning technology replaced manual classification of news image texts. Although the classification efficiency has been greatly improved, the classification time is still a bit long. The bidirectional encoder representations from transformers (BERT) model uses transformer and encoder to pretrain news image text to improve classification efficiency. By comparing the differences between machine learning and BERT models in news image text classification, the experiments showed that the average precision, recall, and F1 values of the news image text classification algorithm using the BERT model were 96.6%, 95.7%, and 96.1%, respectively. All three evaluation criteria were about 5% more than the classification algorithm of the machine learning model. The classification speed of the news image text classification algorithm using the BERT model was 1.8 times that of the news image text classification algorithm based on support vector machine. Therefore, the news image text classification algorithm using the BERT model can improve the classification accuracy and efficiency of news image text.

Bidirectional Encoder Representations from Transformers (BERT) has gained increasing attention from researchers and practitioners as it has proven to be an invaluable technique in natural languages processing. This is mainly due to its unique features, including its ability to predict words conditioned on both the left and the right context, and its ability to be pretrained using the plain text corpus that is enormously available on the web. As BERT gained more interest, more BERT models were introduced to support different languages, including Arabic. The current state of knowledge and practice in applying BERT models to Arabic text classification is limited. In an attempt to begin remedying this gap, this review synthesizes the different Arabic BERT models that have been applied to text classification. It investigates the differences between them and compares their performance. It also examines how effective they are compared to the original English BERT models. It concludes by offering insight into aspects that need further improvements and future work.

The article examines the use of neural networks in telemedicine for patient self-diagnosis, focusing on custom LSTM (Long Short-Term Memory) models and pre-trained BERT (Bidirectional Encoder Representations from Transformers) models. Telemedicine has become essential in modern healthcare, particularly due to the rise in remote consultations and diagnostics during global health crises. The study reviews research in this area, highlighting contributions from international and Ukrainian experts. It explores the effectiveness of neural networks in interpreting patient symptom descriptions and generating accurate diagnoses. The custom LSTM model processes text-based symptom descriptions but faces challenges with limited datasets, impacting its diagnostic accuracy. On the other hand, the BERT model leverages transfer learning, achieving over 92% accuracy in predictions, thanks to its ability to understand complex language inputs. An enhanced LSTM model is also presented, incorporating Dropout and Batch Normalization layers, which help increase the robustness and accuracy of the predictions while managing the model's complexity. The article also discusses the technical challenges of training these neural networks, including the variability of symptom descriptions and the need for large annotated medical datasets. Proposed solutions include domain-specific enhancements and data augmentation to improve the models' performance. In comparing the three models, the study identifies that while the custom LSTM model is more flexible and simple, it struggles with generalization. The BERT model, by contrast, excels in understanding and accuracy but requires more computational resources. The enhanced LSTM model offers a balance between complexity and performance. In conclusion, neural networks offer promising advances in telemedicine by providing accurate, AI-assisted self-diagnosis. Although challenges remain, such as data requirements and computing power, models like BERT are key to improving healthcare accessibility and accuracy in telemedicine applications. Among the main advantages of using artificial intelligence for the healthcare system are in-depth diagnostic analysis, individual treatment regimens, and optimization of medical processes. However, modern technologies in the medical field are not yet able to completely replace experienced specialists.

Individuals with health anomalies often share their experiences on social media sites, such as Twitter, which yields an abundance of data on a global scale. Nowadays, social media data constitutes a leading source to build drug monitoring and surveillance systems. However, a proper assessment of such data requires discarding mentions which do not express drug-related personal health experiences. We automate this process by introducing a novel deep learning model. The model includes character-level and word-level embeddings, embedding-level attention, convolu- tional neural networks (CNN), bidirectional gated recurrent units (BiGRU), and context-aware attentions. An embedding for a word is produced by integrating both word-level and character-level embeddings using an embedding-level attention mechanism, which selects the salient features from both embeddings without expanding dimensionality. The resultant embedding is further analyzed by three CNN layers independently, where each extracts unique n-grams. BiGRUs followed by attention layers further process the outputs from each CNN layer. Besides, the resultant embedding is also encoded by a BiGRU with attention. Our model is developed to cope with the intricate attributes inherent to tweets such as vernacular texts, descriptive medical phrases, frequently misspelt words, abbreviations, short messages, and others. All these four outputs are summed and sent to a softmax classifier. We built a dataset by incorporating tweets from two benchmark datasets designed for the same objective to evaluate the performance. Our model performs substantially better than existing models, including several customized Bidirectional Encoder Representations from Transformers (BERT) models with an F1-score of 0.772.

BackgroundDeaths related to physical trauma impose a heavy burden on society, and the Abbreviated Injury Scale (AIS) is an important tool for injury research. AIS covers injuries to various parts of the human body and scores them based on the severity of the injury. In practical applications, the complex AIS coding rules require experts to encode by consulting patient medical records, which inevitably increases the difficulty, time, and cost of evaluation of patient and also puts higher demands on the workload of information collection and processing. In some cases, the sheer number of patients or the inability to access detailed medical records necessary for coding further complicates independent AIS codes.ObjectiveThis study aims to use advanced deep learning techniques to predict AIS codes based on easily accessible diagnostic information of patients to improve the accuracy of trauma assessment.MethodsWe used a dataset of patients with trauma (n=26,810) collected by the Chongqing Daping Hospital between October 2013 and June 2024. We mainly selected the patient’s diagnostic information, injury description, cause of injury, injury region, injury types, and present illness history as the key feature inputs. We used a robust optimization Bidirectional Encoder Representations from Transformers (BERT) pretraining method to embed these features and constructed a prediction model based on BERT. This model aims to predict AIS codes and comprehensively evaluate its performance through a 5-fold cross-validation. We compared the BERT model with previous research results and current mainstream machine learning methods to verify its advantages in prediction tasks. In addition, we also conducted external validation of the model using 244 external data points from the Chongqing Emergency Center.ResultsThe BERT model proposed in this paper performs significantly better than the comparison model on independent test datasets with an accuracy of 0.8971, which surpassed the previous study by 10 % points. In addition, the area under the curve (AUC value of the BERT model is 0.9970, and the F1-score is 0.8434. In the external dataset, the accuracy, AUC, and F1-score results of the model are 0.7131, 0.8586, and 0.6801, respectively. These results indicate that our model has high generalization ability and prediction accuracy.ConclusionsThe BERT model we proposed is mainly based on diagnostic information to predict AIS codes, and its prediction accuracy is superior to previous investigations and current mainstream machine learning methods. It has a high generalization ability in external datasets.

Event Extraction (EE) is a pivotal task for NLP, where important events in the narrative text need to be detected and recognized. We present an alternative method for extracting events from Hans Christian Andersen's fairy tales, utilizing Few-Shot Learning with BERT (Bidirectional Encoder Representations from Transformers) and RNN (Recurrent Neural Network) in this paper. We selected Andersen's fairy tales because they are characterized by rich narratives and symbolic language, which also often prevents automatic event extraction. To reduce reliance on labeled samples, we utilize the Few-Shot Learning method, which enables the model to learn from a small number of labeled event examples trivially. The BERT model is used to generate deep representations by modeling the context between words and sentences. RNN is essential to capture the sequence of events in the story, which determines the structure of the narrative. The findings demonstrate that the proposed framework significantly improves event extraction, with high values of evaluation metrics such as in accuracy, precision, recall, and F1-score. The proposed method is also effective in extracting non-explicit events while keeping the narrative context. Despite the challenges posed by metaphorical language and subjective events, this work demonstrates that Few-Shot Learning, BERT, and RNNs offer a promising solution to the task of event extraction from complex narratives.

Sentence Boundary Disambiguation (SBD) is crucial for building datasets for tasks such as machine translation, syntactic analysis, and semantic analysis. Currently, most automatic sentence segmentation in Tibetan adopts the methods of rule-based and statistical learning, as well as the combination of the two, which have high requirements on the corpus and the linguistic foundation of the researchers and are more costly to annotate manually. In this study, we explore Tibetan SBD using deep learning technology. Initially, we analyze Tibetan characteristics and various subword techniques, selecting Byte Pair Encoding (BPE) and Sentencepiece (SP) for text segmentation and training the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. Secondly, we studied the Tibetan SBD based on different BERT pre-trained language models, which mainly learns the ambiguity of the shad (“།”) in different positions in modern Tibetan texts and determines through the model whether the shad (“།”) in the texts has the function of segmenting sentences. Meanwhile, this study introduces four models, BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, based on the BERT model for performance comparison. Finally, to verify the performance of the pre-trained language models on the SBD task, this study conducts SBD experiments on both the publicly available Tibetan pre-trained language model TiBERT and the multilingual pre-trained language model (Multi-BERT). The experimental results show that the F1 score of the BERT (BPE) model trained in this study reaches 95.32% on 465,669 Tibetan sentences, nearly five percentage points higher than BERT (SP) and Multi-BERT. The SBD method based on pre-trained language models in this study lays the foundation for establishing datasets for the later tasks of Tibetan pre-training, summary extraction, and machine translation.

It is essential to understand how environmental factors and cultural backgrounds affect emotional responses to build Human-Computer Interaction (HCI) technologies. This study uses the BERT (Bidirectional Encoder Representations from Transformers) model to investigate the emotional landscape of interactions on the X platform across various cultural contexts. The aims of the study are as follows: (1) Gather and prepare X platform data from COVID-19 sources in India, Pakistan, Malaysia, the US, the UK, Australia, and Canada. (2) Use a BERT model to qualitatively analyze tweet sentiment. (3) Assess the accuracy of the model and look for sentiment trends in tweets from nations throughout the pandemic. The BERT model successfully classified sentiment, as seen by its 80.29% accuracy rate on test data. Sentiment research showed that positive sentiment was far more prevalent in the US, Canada, and Australia, indicating that these countries were better able to adjust to the COVID-19 situation. Stability was seen in the balanced sentiment distributions displayed by Pakistan, India, and the United Kingdom. Despite having fewer data points, Pakistan and Malaysia continued to have largely positive attitudes. This study provides the basis for a comparative analysis of emotional responses by taking contextual and cultural aspects into account.

Twitter has emerged as a critical platform for capturing public sentiment, offering a valuable source for sentiment analysis. This study presents a comparative evaluation of two BERT (Bidirectional Encoder Representations from Transformers) models—baseline and fine-tuned—targeted at analyzing Indonesian-language tweets. Employing the CRISP-DM framework, the methodology encompasses automated data crawling, comprehensive text pre-processing (including case folding, cleaning, tokenisation, normalisation, and data augmentation), and model development using the IndoBERT-base-p1 architecture. The experimental results reveal that the fine-tuned BERT model achieves significantly improved performance over the non-optimized model, with accuracy, precision, recall, and F1-score values reaching 91%, 0.91, 0.90, and 0.91, respectively. These findings indicate the fine-tuned model's superior ability to capture linguistic subtleties and contextual sentiment features within informal social media text. Furthermore, the model is deployed in a web-based application for real-time sentiment classification, demonstrating its practical applicability. This study underscores the effectiveness of Fine-tuning in enhancing BERT-based sentiment analysis for low-resource languages. It highlights its potential for informing decisionmaking in digital communication, marketing, and policy research contexts.

Investor attitude is an important factor in driving the behavior of the stock market, usually dwarfing conventional financial markets. Although previous study has investigated the mental side of investing, this has been concentrated mostly on the developed world and has had no real-time behavioral incorporation. This study fulfills this requirement by investigating the roles played by investor mentality, media reporting, and macroeconomic triggers in determining stock market volatility during bear markets with a special emphasis on Kathmandu. The goal is to measure the effect of sentiment on market performance based on both structured survey data and unstructured social media and news text. A hybrid methodological design is used, combining statistical modeling (Chi-Square test) and a machine learning-based sentiment classification approach utilizing a fine-tuned BERT (Bidirectional Encoder Representations from Transformers) model. Social media and financial news information are both drawn from a 2022–2025 Kaggle Stock Market Sentiment dataset, and NVivo provides thematic analysis of investor interviews. The following BERT model, coded through Python's Hugging Face Transformers package, resulted in a 9.2% improvement on accuracy compared to standard sentiment analysis models such as logistic regression and SVM, providing richer contextual knowledge of investor emotion. Empirical findings indicate a statistically significant relationship between sentiment and investment behavior (χ² = 23.250, p = 0.006), validating market change behavior. This study not only emphasizes sentiment as an important predictive variable but also proposes a scalable, real-time predictive framework for risk-sensitive investment strategy. The application of this model can improve the early identification of volatility and guide sentiment-based trading systems for investors, analysts, and policymakers in developing markets.

Large language models (LLMs) have emerged as a milestone in the realm of machine learning, offering groundbreaking capabilities in understanding human language with remarkable coherence and versatility. Their application spans a multitude of domains, ranging from text classification to information extraction. Such models are first pre-trained on vast data corpora to learn general language patterns and then are fine-tuned on specific downstream tasks. Despite their impressive capabilities, LLMs encounter several limitations, including sub-optimal performance in understanding the domain-specific language of drug labels, limited performance in analyzing class-imbalanced data, and high computational complexity of training. Human prescription drug labeling contains a summary of the essential scientific information needed for the safe and effective use of the drug and includes critical information about drug products, such as pharmacokinetics and adverse events. Automatic information extraction from drug labels can facilitate processes such as finding the adverse reaction of the drugs or finding the interaction of one drug with another drug. Although LLMs are an interesting solution to perform this task, because of their training on generic data, they may not possess the depth of knowledge or the ability required to understand the language in such a specialized field. To address this issue, this work presents the developed PharmBERT, which is a BERT model specifically pretrained on the drug labels, publicly available online for the NLP community. We demonstrate that our model outperforms models such as vanilla BERT, ClinicalBERT, and BioBERT in multiple NLP tasks in the drug label domain. Another shortcoming of LLMs lies in their handling of class-imbalanced data, leading to a subpar performance in identifying and accurately responding to minority classes. This class-imbalance can skew the model's learning process, favoring majority classes, and thus diminishing its effectiveness in recognizing and appropriately addressing less represented categories. To cope with these challenges, we present a simple modification of standard fine-tuning. Specifically, we propose a two-stage fine-tuning. In Stage 1, we fine-tune the final layer of the pretrained model with class-balanced augmented data, generated using ChatGPT. As a large generative language model, ChatGPT is capable of generating novel yet contextually similar responses to a given text sample, which makes it an excellent candidate for data augmentation. In Stage 2, we perform the standard fine-tuning. Our modification has several benefits: (1) it leverages pretrained representations by only fine-tuning a small portion of the model parameters while keeping the rest untouched; (2) it allows the model to learn an initial representation of the specific task; and importantly (3) it protects the learning of tail classes from being at a disadvantage during the model updating. The experimental results show that the proposed two-stage fine-tuning outperforms vanilla fine-tuning and state-of-the-art methods on the above datasets. The third issue of LLMs is the high computational complexity of training. Due to the large number of parameters in many state-of-the-art LLMs, such as Bidirectional Encoder Representations from Transformers (BERT), the process of fine-tuning exhibits a high level of intricacy. One attractive solution to this issue is parameter-efficient fine-tuning, which means only fine-tuning a small portion of the model and not changing the rest. However, the question that which portion of the BERT model is the most important one for fine-tuning remains unanswered. Here, we first analyze different components in the BERT model to see which one undergoes the most change after fine-tuning. We show that output LayerNorm changes more than other components when finetuned for different General Language Understanding Evaluation (GLUE) tasks. Then we show that only fine-tuning this component reaches a comparable, or in some cases better, performance to full fine-tuning and other parameter-efficient fine-tuning methods. Moreover, we use Fisher information to find out which parameters in LayerNorm are the most important ones and show that with a small performance degradation, many NLP tasks in the GLUE benchmark can be solved by fine-tuning only a small, selected portion of LayerNorm. In summary, LLMs hold immense potential to redefine and enhance various fields, including drug-label analysis, but they also suffer from limitations like sub-optimal performance in understanding domain-specific texts of drug labels, not learning minority classes in the class-imbalanced data, and high computational complexity. In this thesis, for each one of these shortcomings, first, the problem is analyzed and then a solution is provided.

BackgroundThe accumulation of vast electronic medical records (EMRs) through medical informatization creates significant research value, particularly in obstetrics. Diagnostic standardization across different health care institutions and regions is vital for medical data analysis. Large language models (LLMs) have been extensively used for various medical tasks. Prompt engineering is key to use LLMs effectively.ObjectiveThis study aims to evaluate and compare the performance of LLMs with various prompt engineering techniques on the task of standardizing obstetric diagnostic terminology using real-world obstetric data.MethodsThe paper describes a 4-step approach used for mapping diagnoses in electronic medical records to the International Classification of Diseases, 10th revision, observation domain. First, similarity measures were used for mapping the diagnoses. Second, candidate mapping terms were collected based on similarity scores above a threshold, to be used as the training data set. For generating optimal mapping terms, we used two LLMs (ChatGLM2 and Qwen-14B-Chat [QWEN]) for zero-shot learning in step 3. Finally, a performance comparison was conducted by using 3 pretrained bidirectional encoder representations from transformers (BERTs), including BERT, whole word masking BERT, and momentum contrastive learning with BERT (MC-BERT), for unsupervised optimal mapping term generation in the fourth step.ResultsLLMs and BERT demonstrated comparable performance at their respective optimal levels. LLMs showed clear advantages in terms of performance and efficiency in unsupervised settings. Interestingly, the performance of the LLMs varied significantly across different prompt engineering setups. For instance, when applying the self-consistency approach in QWEN, the F1-score improved by 5%, with precision increasing by 7.9%, outperforming the zero-shot method. Likewise, ChatGLM2 delivered similar rates of accurately generated responses. During the analysis, the BERT series served as a comparative model with comparable results. Among the 3 models, MC-BERT demonstrated the highest level of performance. However, the differences among the versions of BERT in this study were relatively insignificant.ConclusionsAfter applying LLMs to standardize diagnoses and designing 4 different prompts, we compared the results to those generated by the BERT model. Our findings indicate that QWEN prompts largely outperformed the other prompts, with precision comparable to that of the BERT model. These results demonstrate the potential of unsupervised approaches in improving the efficiency of aligning diagnostic terms in daily research and uncovering hidden information values in patient data.

With the increasing number of web texts, the classification of web texts has become an important task. In this paper, the text word vector representation method is first analyzed, and bidirectional encoder representations from transformers (BERT) are selected to extract the word vector. The bidirectional gated recurrent unit (BiGRU), convolutional neural network (CNN), and attention mechanism are combined to obtain the context and local features of the text, respectively. Experiments were carried out using the THUCNews dataset. The results showed that in the comparison between word-to-vector (Word2vec), Glove, and BERT, the BERT obtained the best classification result. In the classification of different types of text, the average accuracy and F1 value of the BERT-BGCA method reached 0.9521 and 0.9436, respectively, which were superior to other deep learning methods such as TextCNN. The results suggest that the BERT-BGCA method is effective in classifying web texts and can be applied in practice.

Semantic matching is one of the critical technologies for intelligent customer service. Since Bidirectional Encoder Representations from Transformers (BERT) is proposed, fine-tuning on a large-scale pre-training language model becomes a general method to implement text semantic matching. However, in practical application, the accuracy of the BERT model is limited by the quantity of pre-training corpus and proper nouns in the target domain. An enhancement method for knowledge based on domain dictionary to mask input is proposed to solve the problem. Firstly, for modul input, we use keyword matching to recognize and mask the word in domain. Secondly, using self-supervised learning to inject knowledge of the target domain into the BERT model. Thirdly, we fine-tune the BERT model with public datasets LCQMC and BQboost. Finally, we test the model’s performance with a financial company’s user data. The experimental results show that after using our method and BQboost, accuracy increases by 12.12% on average in practical applications.

The rising complexity of cyberattacks poses significant challenges to traditional intrusion detection system (IDS). Approaches that rely mainly on feature engineering and rule matching often fail to address new and complex attack patterns effectively. To address these challenges, this paper proposes a network IDS based on bidirectional encoder representations from transformers (BERT), which aims to leverage recent advances in the field of natural language processing (NLP) to enhance the performance of IDS. The core of our approach lies in converting network traffic data into textual representations that can be processed by applying advanced NLP techniques. Specifically, we first convert various types of information in network traffic into a natural-language-like sequence representation, which allows us to utilize a pre-trained BERT model for feature extraction. The BERT model, with its bidirectional encoder structure, captures complex contextual information in sequences, producing high-quality feature representations. After feature extraction is completed, we further perform classification using a classifier based on a bidirectional gated recurrent unit (GRU) network, which is able to efficiently process sequence data and reduce computational complexity while capturing temporal dependencies in the sequences. By training the classifier on the basis of the extracted features, we are able to achieve an accurate distinction between normal and abnormal traffic. To validate the effectiveness of our approach, we conducted experiments on several public datasets, and the experimental results show that the BERT-based network IDS significantly outperforms traditional machine learning (ML) and deep learning (DL) methods in terms of detection performance. The BERT model enhances detection accuracy and effectively captures complex attack patterns, compared to traditional methods. Our research not only demonstrates the great potential of NLP technology in the field of network security but also provides a new research direction for cross-domain applications. By treating network traffic as a “language,” our approach resolves issues that traditional methods struggle with while being scalable and adaptable to various network environments and attack patterns.