Optimizing BERT Models with Fine-Tuning for Indonesian Twitter Sentiment Analysis

With the development of Internet technology and the transformation of news media to informatization, news images, texts, and other information have exploded. The news image and text classification can effectively solve the disorder problem of news information. The early news image text classification is to establish artificial classifiers according to specific classification rules, but the classification error rate is high and the classification speed is slow. Later, machine learning technology replaced manual classification of news image texts. Although the classification efficiency has been greatly improved, the classification time is still a bit long. The bidirectional encoder representations from transformers (BERT) model uses transformer and encoder to pretrain news image text to improve classification efficiency. By comparing the differences between machine learning and BERT models in news image text classification, the experiments showed that the average precision, recall, and F1 values of the news image text classification algorithm using the BERT model were 96.6%, 95.7%, and 96.1%, respectively. All three evaluation criteria were about 5% more than the classification algorithm of the machine learning model. The classification speed of the news image text classification algorithm using the BERT model was 1.8 times that of the news image text classification algorithm based on support vector machine. Therefore, the news image text classification algorithm using the BERT model can improve the classification accuracy and efficiency of news image text.

The article examines the use of neural networks in telemedicine for patient self-diagnosis, focusing on custom LSTM (Long Short-Term Memory) models and pre-trained BERT (Bidirectional Encoder Representations from Transformers) models. Telemedicine has become essential in modern healthcare, particularly due to the rise in remote consultations and diagnostics during global health crises. The study reviews research in this area, highlighting contributions from international and Ukrainian experts. It explores the effectiveness of neural networks in interpreting patient symptom descriptions and generating accurate diagnoses. The custom LSTM model processes text-based symptom descriptions but faces challenges with limited datasets, impacting its diagnostic accuracy. On the other hand, the BERT model leverages transfer learning, achieving over 92% accuracy in predictions, thanks to its ability to understand complex language inputs. An enhanced LSTM model is also presented, incorporating Dropout and Batch Normalization layers, which help increase the robustness and accuracy of the predictions while managing the model's complexity. The article also discusses the technical challenges of training these neural networks, including the variability of symptom descriptions and the need for large annotated medical datasets. Proposed solutions include domain-specific enhancements and data augmentation to improve the models' performance. In comparing the three models, the study identifies that while the custom LSTM model is more flexible and simple, it struggles with generalization. The BERT model, by contrast, excels in understanding and accuracy but requires more computational resources. The enhanced LSTM model offers a balance between complexity and performance. In conclusion, neural networks offer promising advances in telemedicine by providing accurate, AI-assisted self-diagnosis. Although challenges remain, such as data requirements and computing power, models like BERT are key to improving healthcare accessibility and accuracy in telemedicine applications. Among the main advantages of using artificial intelligence for the healthcare system are in-depth diagnostic analysis, individual treatment regimens, and optimization of medical processes. However, modern technologies in the medical field are not yet able to completely replace experienced specialists.

Investor attitude is an important factor in driving the behavior of the stock market, usually dwarfing conventional financial markets. Although previous study has investigated the mental side of investing, this has been concentrated mostly on the developed world and has had no real-time behavioral incorporation. This study fulfills this requirement by investigating the roles played by investor mentality, media reporting, and macroeconomic triggers in determining stock market volatility during bear markets with a special emphasis on Kathmandu. The goal is to measure the effect of sentiment on market performance based on both structured survey data and unstructured social media and news text. A hybrid methodological design is used, combining statistical modeling (Chi-Square test) and a machine learning-based sentiment classification approach utilizing a fine-tuned BERT (Bidirectional Encoder Representations from Transformers) model. Social media and financial news information are both drawn from a 2022–2025 Kaggle Stock Market Sentiment dataset, and NVivo provides thematic analysis of investor interviews. The following BERT model, coded through Python's Hugging Face Transformers package, resulted in a 9.2% improvement on accuracy compared to standard sentiment analysis models such as logistic regression and SVM, providing richer contextual knowledge of investor emotion. Empirical findings indicate a statistically significant relationship between sentiment and investment behavior (χ² = 23.250, p = 0.006), validating market change behavior. This study not only emphasizes sentiment as an important predictive variable but also proposes a scalable, real-time predictive framework for risk-sensitive investment strategy. The application of this model can improve the early identification of volatility and guide sentiment-based trading systems for investors, analysts, and policymakers in developing markets.

It is essential to understand how environmental factors and cultural backgrounds affect emotional responses to build Human-Computer Interaction (HCI) technologies. This study uses the BERT (Bidirectional Encoder Representations from Transformers) model to investigate the emotional landscape of interactions on the X platform across various cultural contexts. The aims of the study are as follows: (1) Gather and prepare X platform data from COVID-19 sources in India, Pakistan, Malaysia, the US, the UK, Australia, and Canada. (2) Use a BERT model to qualitatively analyze tweet sentiment. (3) Assess the accuracy of the model and look for sentiment trends in tweets from nations throughout the pandemic. The BERT model successfully classified sentiment, as seen by its 80.29% accuracy rate on test data. Sentiment research showed that positive sentiment was far more prevalent in the US, Canada, and Australia, indicating that these countries were better able to adjust to the COVID-19 situation. Stability was seen in the balanced sentiment distributions displayed by Pakistan, India, and the United Kingdom. Despite having fewer data points, Pakistan and Malaysia continued to have largely positive attitudes. This study provides the basis for a comparative analysis of emotional responses by taking contextual and cultural aspects into account.

Many cutting-edge language models have been used in the past to forecast election results. Sentiment analysis aids in opinion mining – a common experiment used to detect public opinions – on a given topic. Twitter has gained popularity and established itself as a crucial instrument for analyzing public opinion on elections and other trending issues. The unexpected but interesting results of recently held Nigeria's presidential election shifted attention to the upcoming governorship race in Lagos State. In this work, we propose a Google’s Bidirectional Encoder Representations from Transformers (BERT) model for the sentiment analysis of governorship election in Lagos State, Nigeria, using Twitter data. A total of 800,000 personal and public tweets were scraped from twitter concerning the three prominent contesting Lagos State Gubernatorial candidates using carefully selected search queries. The tweets were preprocessed to avoid noise and inconsistencies and the preprocessed tweets were parsed into the pre-trained and finetuned BERT model. The result was analyzed to establish the sentiments of the public about the candidates. The social networks of the candidates were also presented and the effect of parameter using different learning rates (LR) was also considered. The BERT model achieved the maximum performance under varied learning rate and epoch sizes of 88% precision, 92% recall and 91% F1-Measure. Results also showed that the learning rate at 1e-7 gave the best performance. Also, the smaller the learning rate, the higher the accuracy but the larger the epoch size, the higher the accuracy. Applying the developed BERT model to the public’s tweet showed that the election will be a two-party race between the Labour Party and All Progressives Congress party, thereby challenging the status quo. The results of the experiment demonstrated that sentiment analysis and other Natural Language Processing activities can help with comprehension of the social media environment. Results also showed how much influence each candidate has over the outcome of the election. We come to the conclusion that estimating election results and providing insights for electoral parties can benefit from sentiment analysis and other language models.

BackgroundDeaths related to physical trauma impose a heavy burden on society, and the Abbreviated Injury Scale (AIS) is an important tool for injury research. AIS covers injuries to various parts of the human body and scores them based on the severity of the injury. In practical applications, the complex AIS coding rules require experts to encode by consulting patient medical records, which inevitably increases the difficulty, time, and cost of evaluation of patient and also puts higher demands on the workload of information collection and processing. In some cases, the sheer number of patients or the inability to access detailed medical records necessary for coding further complicates independent AIS codes.ObjectiveThis study aims to use advanced deep learning techniques to predict AIS codes based on easily accessible diagnostic information of patients to improve the accuracy of trauma assessment.MethodsWe used a dataset of patients with trauma (n=26,810) collected by the Chongqing Daping Hospital between October 2013 and June 2024. We mainly selected the patient’s diagnostic information, injury description, cause of injury, injury region, injury types, and present illness history as the key feature inputs. We used a robust optimization Bidirectional Encoder Representations from Transformers (BERT) pretraining method to embed these features and constructed a prediction model based on BERT. This model aims to predict AIS codes and comprehensively evaluate its performance through a 5-fold cross-validation. We compared the BERT model with previous research results and current mainstream machine learning methods to verify its advantages in prediction tasks. In addition, we also conducted external validation of the model using 244 external data points from the Chongqing Emergency Center.ResultsThe BERT model proposed in this paper performs significantly better than the comparison model on independent test datasets with an accuracy of 0.8971, which surpassed the previous study by 10 % points. In addition, the area under the curve (AUC value of the BERT model is 0.9970, and the F1-score is 0.8434. In the external dataset, the accuracy, AUC, and F1-score results of the model are 0.7131, 0.8586, and 0.6801, respectively. These results indicate that our model has high generalization ability and prediction accuracy.ConclusionsThe BERT model we proposed is mainly based on diagnostic information to predict AIS codes, and its prediction accuracy is superior to previous investigations and current mainstream machine learning methods. It has a high generalization ability in external datasets.

Bidirectional Encoder Representations from Transformers (BERT) has gained increasing attention from researchers and practitioners as it has proven to be an invaluable technique in natural languages processing. This is mainly due to its unique features, including its ability to predict words conditioned on both the left and the right context, and its ability to be pretrained using the plain text corpus that is enormously available on the web. As BERT gained more interest, more BERT models were introduced to support different languages, including Arabic. The current state of knowledge and practice in applying BERT models to Arabic text classification is limited. In an attempt to begin remedying this gap, this review synthesizes the different Arabic BERT models that have been applied to text classification. It investigates the differences between them and compares their performance. It also examines how effective they are compared to the original English BERT models. It concludes by offering insight into aspects that need further improvements and future work.

Bali is one of the provinces with the most abundant natural and cultural wealth in Indonesia. One commodity that supports it is coffee. Bali Coffee is not only a gastronomic identity, but also a cultural identity which makes it have added value to be developed into various business lines. One business derivative that is quite promising is a coffee shop. However, these favorable conditions also need to be maintained to ensure good quality reaches consumers. One thing that can do is analyze reviews from customers. One of the most popular methods is Sentiment Analysis. This technique allows business to analyze customer reviews on social media. It can be a feedback to maintaining and improving quality and good relationships with customers. This research aims to create a machine learning model to analyze customer reviews at several coffee shops in Bali which are divided into three labels, namely: positive, negative and neutral. The methods used are: scraping, cleaning, stopword removal, embedding, undersampling, and modeling. The algorithms used are Bidirectional Encoder Representation from Transformer (BERT) and Generative Pre-trained Transformers (GPT). The performance metrics used in this research are precision, recall, accuracy and loss. This research succeeded in creating a sentiment analysis model for coffee shop customers in Bali. The BERT model obtained an accuracy value of 78% without undersampling with a loss in the 10th iteration of 0.27. Meanwhile, the BERT model with undersampling obtained an accuracy value of 32.85% with a loss in the 10th iteration of 0.16. The GPT2 model without undersampling gets an accuracy of 78% with a loss in the 10th iteration of 0.25. Meanwhile, the GPT model with undersampling obtained an accuracy value of 32.85% with a loss in the 10th iteration of 0.15.

Large language models (LLMs) have emerged as a milestone in the realm of machine learning, offering groundbreaking capabilities in understanding human language with remarkable coherence and versatility. Their application spans a multitude of domains, ranging from text classification to information extraction. Such models are first pre-trained on vast data corpora to learn general language patterns and then are fine-tuned on specific downstream tasks. Despite their impressive capabilities, LLMs encounter several limitations, including sub-optimal performance in understanding the domain-specific language of drug labels, limited performance in analyzing class-imbalanced data, and high computational complexity of training. Human prescription drug labeling contains a summary of the essential scientific information needed for the safe and effective use of the drug and includes critical information about drug products, such as pharmacokinetics and adverse events. Automatic information extraction from drug labels can facilitate processes such as finding the adverse reaction of the drugs or finding the interaction of one drug with another drug. Although LLMs are an interesting solution to perform this task, because of their training on generic data, they may not possess the depth of knowledge or the ability required to understand the language in such a specialized field. To address this issue, this work presents the developed PharmBERT, which is a BERT model specifically pretrained on the drug labels, publicly available online for the NLP community. We demonstrate that our model outperforms models such as vanilla BERT, ClinicalBERT, and BioBERT in multiple NLP tasks in the drug label domain. Another shortcoming of LLMs lies in their handling of class-imbalanced data, leading to a subpar performance in identifying and accurately responding to minority classes. This class-imbalance can skew the model's learning process, favoring majority classes, and thus diminishing its effectiveness in recognizing and appropriately addressing less represented categories. To cope with these challenges, we present a simple modification of standard fine-tuning. Specifically, we propose a two-stage fine-tuning. In Stage 1, we fine-tune the final layer of the pretrained model with class-balanced augmented data, generated using ChatGPT. As a large generative language model, ChatGPT is capable of generating novel yet contextually similar responses to a given text sample, which makes it an excellent candidate for data augmentation. In Stage 2, we perform the standard fine-tuning. Our modification has several benefits: (1) it leverages pretrained representations by only fine-tuning a small portion of the model parameters while keeping the rest untouched; (2) it allows the model to learn an initial representation of the specific task; and importantly (3) it protects the learning of tail classes from being at a disadvantage during the model updating. The experimental results show that the proposed two-stage fine-tuning outperforms vanilla fine-tuning and state-of-the-art methods on the above datasets. The third issue of LLMs is the high computational complexity of training. Due to the large number of parameters in many state-of-the-art LLMs, such as Bidirectional Encoder Representations from Transformers (BERT), the process of fine-tuning exhibits a high level of intricacy. One attractive solution to this issue is parameter-efficient fine-tuning, which means only fine-tuning a small portion of the model and not changing the rest. However, the question that which portion of the BERT model is the most important one for fine-tuning remains unanswered. Here, we first analyze different components in the BERT model to see which one undergoes the most change after fine-tuning. We show that output LayerNorm changes more than other components when finetuned for different General Language Understanding Evaluation (GLUE) tasks. Then we show that only fine-tuning this component reaches a comparable, or in some cases better, performance to full fine-tuning and other parameter-efficient fine-tuning methods. Moreover, we use Fisher information to find out which parameters in LayerNorm are the most important ones and show that with a small performance degradation, many NLP tasks in the GLUE benchmark can be solved by fine-tuning only a small, selected portion of LayerNorm. In summary, LLMs hold immense potential to redefine and enhance various fields, including drug-label analysis, but they also suffer from limitations like sub-optimal performance in understanding domain-specific texts of drug labels, not learning minority classes in the class-imbalanced data, and high computational complexity. In this thesis, for each one of these shortcomings, first, the problem is analyzed and then a solution is provided.

Background: Mobile app usage is increasing in the digital age, with Ride-Hailing app becoming the primary example of this trend. To obtain valuable understanding of how people perceive and interact with mobile app, user reviews on platforms such as Google Play are usually analyzed. This analysis can assist developers to identify areas for improvement in both Ride-hailing and Google Play App. A promising method that can be used to analyze user perception in this instance is Aspect-Based Sentiment Analysis (ABSA). Objective: This research aimed to apply ABSA to user reviews using Bidirectional Encoder Representations from Transformers (BERT) models. In this context, aspect identification and topic modeling were performed by using Latent Dirichlet Allocation (LDA). The model extracted topics from the reviews and used Generative Artificial Intelligence (GenAI) to define the aspects of the topics to further enhance the analysis. For consistency and accuracy, the method included sentiment annotation by a human annotator. Methods: A total of two datasets were used in this research, with the first collected by scraping user reviews of Ride-Hailing App while the second was obtained from Kaggle, and to identify relevant topics, modeling was performed using LDA. These topics were then categorized into aspects using GenAI, covering areas, such as customer experience, service, payment, app features, task management, and event management. Subsequently, sentiment labeling was conducted using human annotators to provide a reliable baseline. BERT model was then used to classify sentiment with aspect hints, and the evaluation included calculations of accuracy, precision, recall, and F1-score. Results: The results showed that BERT model achieved the highest accuracy of 97% in sentiment analysis across all datasets. Conclusion: This research provided valuable understanding of user experience and established a strong ABSA framework for analyzing user reviews using LDA, Aspect Annotation, GenAI, and BERT sentiment models. Future research could expand this method to other app categories and incorporate real-time ABSA for continuous monitoring and dynamic feedback. Keywords: User Reviews, Aspect-Based Sentiment Analysis (ABSA), Sentiment Analysis, Topic Modeling, Generative Artificial Intelligence (GenAI)

Semantic matching is one of the critical technologies for intelligent customer service. Since Bidirectional Encoder Representations from Transformers (BERT) is proposed, fine-tuning on a large-scale pre-training language model becomes a general method to implement text semantic matching. However, in practical application, the accuracy of the BERT model is limited by the quantity of pre-training corpus and proper nouns in the target domain. An enhancement method for knowledge based on domain dictionary to mask input is proposed to solve the problem. Firstly, for modul input, we use keyword matching to recognize and mask the word in domain. Secondly, using self-supervised learning to inject knowledge of the target domain into the BERT model. Thirdly, we fine-tune the BERT model with public datasets LCQMC and BQboost. Finally, we test the model’s performance with a financial company’s user data. The experimental results show that after using our method and BQboost, accuracy increases by 12.12% on average in practical applications.

In this study, the extractive summarization using sentence embeddings generated by the finetuned Bidirectional Encoder Representations from Transformers (BERT) models and the k-means clustering method has been investigated. To show how the BERT model can capture the knowledge in specific domains like engineering design and what it can produce after being finetuned based on domain-specific data sets, several BERT models are trained, and the sentence embeddings extracted from the finetuned models are used to generate summaries of a set of papers. Different evaluation methods are then applied to measure the quality of summarization results. Both the machine evaluation method Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and a human-based evaluation method are used for the comparison study. The results indicate that the BERT model finetuned with a larger dataset can generate summaries with more domain terminologies than the pretrained BERT model. Moreover, the summaries generated by BERT models have more contents overlapping with original documents than those obtained through other popular non-BERT-based models. The experimental results indicate that the BERT-based method can provide better and more informative summaries to engineers. It has also been demonstrated that the contextualized representations generated by BERT-based models can capture information in text and have better performance in applications like text summarizations after being trained by domain-specific data sets.

Sentence Boundary Disambiguation (SBD) is crucial for building datasets for tasks such as machine translation, syntactic analysis, and semantic analysis. Currently, most automatic sentence segmentation in Tibetan adopts the methods of rule-based and statistical learning, as well as the combination of the two, which have high requirements on the corpus and the linguistic foundation of the researchers and are more costly to annotate manually. In this study, we explore Tibetan SBD using deep learning technology. Initially, we analyze Tibetan characteristics and various subword techniques, selecting Byte Pair Encoding (BPE) and Sentencepiece (SP) for text segmentation and training the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. Secondly, we studied the Tibetan SBD based on different BERT pre-trained language models, which mainly learns the ambiguity of the shad (“།”) in different positions in modern Tibetan texts and determines through the model whether the shad (“།”) in the texts has the function of segmenting sentences. Meanwhile, this study introduces four models, BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, based on the BERT model for performance comparison. Finally, to verify the performance of the pre-trained language models on the SBD task, this study conducts SBD experiments on both the publicly available Tibetan pre-trained language model TiBERT and the multilingual pre-trained language model (Multi-BERT). The experimental results show that the F1 score of the BERT (BPE) model trained in this study reaches 95.32% on 465,669 Tibetan sentences, nearly five percentage points higher than BERT (SP) and Multi-BERT. The SBD method based on pre-trained language models in this study lays the foundation for establishing datasets for the later tasks of Tibetan pre-training, summary extraction, and machine translation.

Traditional research on preschool language development often fails to capture the complex nonlinear relationships and high-dimensional characteristics of language growth, leading to low prediction accuracy and poor cross-cultural applicability. This paper introduces a novel BERT (Bidirectional Encoder Representations from Transformers)-based model to predict preschool language development and evaluate its cross-cultural effectiveness. Text data from preschool children’s language datasets across multiple cultural backgrounds is collected, cleaned, and preprocessed to create suitable training samples. Special attention is given to the unique grammatical structures and cultural expressions in each language to ensure compatibility with the model. The BERT model is used to encode the processed text, leveraging its bidirectional self-attention mechanism to extract contextual information and generate deep feature representations essential for understanding preschool language development. The model combines both grammatical and semantic features for meaningful representations in subsequent predictions. Fine-tuning the pre-trained BERT model using the Adam optimizer enhances prediction accuracy, while cross-validation and hyperparameter tuning further improve its performance. Culturally specific annotations and vocabularies are incorporated to ensure the model’s effective prediction of language development across different regions. Experimental results show that the BERT model achieves an MAE (Mean Absolute Error) between 0.20 and 0.25, an MSE (Mean Squared Error) between 0.05 and 0.08, and an average R 2 value of 0.84 across English, Chinese, Spanish, and Japanese. These results demonstrate the model’s high accuracy and strong cross-cultural stability in predicting preschool language development.

The rising complexity of cyberattacks poses significant challenges to traditional intrusion detection system (IDS). Approaches that rely mainly on feature engineering and rule matching often fail to address new and complex attack patterns effectively. To address these challenges, this paper proposes a network IDS based on bidirectional encoder representations from transformers (BERT), which aims to leverage recent advances in the field of natural language processing (NLP) to enhance the performance of IDS. The core of our approach lies in converting network traffic data into textual representations that can be processed by applying advanced NLP techniques. Specifically, we first convert various types of information in network traffic into a natural-language-like sequence representation, which allows us to utilize a pre-trained BERT model for feature extraction. The BERT model, with its bidirectional encoder structure, captures complex contextual information in sequences, producing high-quality feature representations. After feature extraction is completed, we further perform classification using a classifier based on a bidirectional gated recurrent unit (GRU) network, which is able to efficiently process sequence data and reduce computational complexity while capturing temporal dependencies in the sequences. By training the classifier on the basis of the extracted features, we are able to achieve an accurate distinction between normal and abnormal traffic. To validate the effectiveness of our approach, we conducted experiments on several public datasets, and the experimental results show that the BERT-based network IDS significantly outperforms traditional machine learning (ML) and deep learning (DL) methods in terms of detection performance. The BERT model enhances detection accuracy and effectively captures complex attack patterns, compared to traditional methods. Our research not only demonstrates the great potential of NLP technology in the field of network security but also provides a new research direction for cross-domain applications. By treating network traffic as a “language,” our approach resolves issues that traditional methods struggle with while being scalable and adaptable to various network environments and attack patterns.