Environment Sentiment Analysis of Bali Coffee Shop Visitors Using Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer 2 (GPT2) Model

Twitter has emerged as a critical platform for capturing public sentiment, offering a valuable source for sentiment analysis. This study presents a comparative evaluation of two BERT (Bidirectional Encoder Representations from Transformers) models—baseline and fine-tuned—targeted at analyzing Indonesian-language tweets. Employing the CRISP-DM framework, the methodology encompasses automated data crawling, comprehensive text pre-processing (including case folding, cleaning, tokenisation, normalisation, and data augmentation), and model development using the IndoBERT-base-p1 architecture. The experimental results reveal that the fine-tuned BERT model achieves significantly improved performance over the non-optimized model, with accuracy, precision, recall, and F1-score values reaching 91%, 0.91, 0.90, and 0.91, respectively. These findings indicate the fine-tuned model's superior ability to capture linguistic subtleties and contextual sentiment features within informal social media text. Furthermore, the model is deployed in a web-based application for real-time sentiment classification, demonstrating its practical applicability. This study underscores the effectiveness of Fine-tuning in enhancing BERT-based sentiment analysis for low-resource languages. It highlights its potential for informing decisionmaking in digital communication, marketing, and policy research contexts.

With the development of Internet technology and the transformation of news media to informatization, news images, texts, and other information have exploded. The news image and text classification can effectively solve the disorder problem of news information. The early news image text classification is to establish artificial classifiers according to specific classification rules, but the classification error rate is high and the classification speed is slow. Later, machine learning technology replaced manual classification of news image texts. Although the classification efficiency has been greatly improved, the classification time is still a bit long. The bidirectional encoder representations from transformers (BERT) model uses transformer and encoder to pretrain news image text to improve classification efficiency. By comparing the differences between machine learning and BERT models in news image text classification, the experiments showed that the average precision, recall, and F1 values of the news image text classification algorithm using the BERT model were 96.6%, 95.7%, and 96.1%, respectively. All three evaluation criteria were about 5% more than the classification algorithm of the machine learning model. The classification speed of the news image text classification algorithm using the BERT model was 1.8 times that of the news image text classification algorithm based on support vector machine. Therefore, the news image text classification algorithm using the BERT model can improve the classification accuracy and efficiency of news image text.

Natural Language Processing (NLP) is a pivotal area in artificial intelligence, aiming to make computers capable of understanding and generating human language. This study evaluates and compares three prominent NLP models—the Generative Pre-trained Transformer (GPT) model, Bidirectional Encoder Representations from Transformers (BERT) model, and Generalized Autoregressive Pretraining for Language Understanding (XLNet)—to determine their strengths, limitations, and suitability for various tasks. The research involves a comprehensive analysis of these models, utilizing well-established datasets such as the Stanford Question Answering Dataset (SQuAD), General Language Understanding Evaluation (GLUE), Reading Comprehension from Examinations (RACE), and the Situations with Adversarial Generations (SWAG). The study explores each model's architecture, pre-training, and fine-tuning processes: GPT’s unidirectional approach is assessed for its language generation and handling of long-range dependencies; Bidirectional encoding is examined for its effectiveness in context understanding, and XLNet permutation-based training is analyzed for its robust contextual comprehension. The experimental results reveal that GPT excels in generative tasks but is constrained by its unidirectional nature. BERT achieves superior accuracy in comprehension tasks but is computationally demanding and susceptible to pre-training bias. XLNet outperforms both GPT and BERT in accuracy and contextual understanding, though at the cost of increased complexity. The results offer a significant understanding of the effectiveness and applicability of these models, suggesting future research directions such as hybrid models and improvements in efficiency.

Investor attitude is an important factor in driving the behavior of the stock market, usually dwarfing conventional financial markets. Although previous study has investigated the mental side of investing, this has been concentrated mostly on the developed world and has had no real-time behavioral incorporation. This study fulfills this requirement by investigating the roles played by investor mentality, media reporting, and macroeconomic triggers in determining stock market volatility during bear markets with a special emphasis on Kathmandu. The goal is to measure the effect of sentiment on market performance based on both structured survey data and unstructured social media and news text. A hybrid methodological design is used, combining statistical modeling (Chi-Square test) and a machine learning-based sentiment classification approach utilizing a fine-tuned BERT (Bidirectional Encoder Representations from Transformers) model. Social media and financial news information are both drawn from a 2022–2025 Kaggle Stock Market Sentiment dataset, and NVivo provides thematic analysis of investor interviews. The following BERT model, coded through Python's Hugging Face Transformers package, resulted in a 9.2% improvement on accuracy compared to standard sentiment analysis models such as logistic regression and SVM, providing richer contextual knowledge of investor emotion. Empirical findings indicate a statistically significant relationship between sentiment and investment behavior (χ² = 23.250, p = 0.006), validating market change behavior. This study not only emphasizes sentiment as an important predictive variable but also proposes a scalable, real-time predictive framework for risk-sensitive investment strategy. The application of this model can improve the early identification of volatility and guide sentiment-based trading systems for investors, analysts, and policymakers in developing markets.

The article examines the use of neural networks in telemedicine for patient self-diagnosis, focusing on custom LSTM (Long Short-Term Memory) models and pre-trained BERT (Bidirectional Encoder Representations from Transformers) models. Telemedicine has become essential in modern healthcare, particularly due to the rise in remote consultations and diagnostics during global health crises. The study reviews research in this area, highlighting contributions from international and Ukrainian experts. It explores the effectiveness of neural networks in interpreting patient symptom descriptions and generating accurate diagnoses. The custom LSTM model processes text-based symptom descriptions but faces challenges with limited datasets, impacting its diagnostic accuracy. On the other hand, the BERT model leverages transfer learning, achieving over 92% accuracy in predictions, thanks to its ability to understand complex language inputs. An enhanced LSTM model is also presented, incorporating Dropout and Batch Normalization layers, which help increase the robustness and accuracy of the predictions while managing the model's complexity. The article also discusses the technical challenges of training these neural networks, including the variability of symptom descriptions and the need for large annotated medical datasets. Proposed solutions include domain-specific enhancements and data augmentation to improve the models' performance. In comparing the three models, the study identifies that while the custom LSTM model is more flexible and simple, it struggles with generalization. The BERT model, by contrast, excels in understanding and accuracy but requires more computational resources. The enhanced LSTM model offers a balance between complexity and performance. In conclusion, neural networks offer promising advances in telemedicine by providing accurate, AI-assisted self-diagnosis. Although challenges remain, such as data requirements and computing power, models like BERT are key to improving healthcare accessibility and accuracy in telemedicine applications. Among the main advantages of using artificial intelligence for the healthcare system are in-depth diagnostic analysis, individual treatment regimens, and optimization of medical processes. However, modern technologies in the medical field are not yet able to completely replace experienced specialists.

Many cutting-edge language models have been used in the past to forecast election results. Sentiment analysis aids in opinion mining – a common experiment used to detect public opinions – on a given topic. Twitter has gained popularity and established itself as a crucial instrument for analyzing public opinion on elections and other trending issues. The unexpected but interesting results of recently held Nigeria's presidential election shifted attention to the upcoming governorship race in Lagos State. In this work, we propose a Google’s Bidirectional Encoder Representations from Transformers (BERT) model for the sentiment analysis of governorship election in Lagos State, Nigeria, using Twitter data. A total of 800,000 personal and public tweets were scraped from twitter concerning the three prominent contesting Lagos State Gubernatorial candidates using carefully selected search queries. The tweets were preprocessed to avoid noise and inconsistencies and the preprocessed tweets were parsed into the pre-trained and finetuned BERT model. The result was analyzed to establish the sentiments of the public about the candidates. The social networks of the candidates were also presented and the effect of parameter using different learning rates (LR) was also considered. The BERT model achieved the maximum performance under varied learning rate and epoch sizes of 88% precision, 92% recall and 91% F1-Measure. Results also showed that the learning rate at 1e-7 gave the best performance. Also, the smaller the learning rate, the higher the accuracy but the larger the epoch size, the higher the accuracy. Applying the developed BERT model to the public’s tweet showed that the election will be a two-party race between the Labour Party and All Progressives Congress party, thereby challenging the status quo. The results of the experiment demonstrated that sentiment analysis and other Natural Language Processing activities can help with comprehension of the social media environment. Results also showed how much influence each candidate has over the outcome of the election. We come to the conclusion that estimating election results and providing insights for electoral parties can benefit from sentiment analysis and other language models.

Background: Mobile app usage is increasing in the digital age, with Ride-Hailing app becoming the primary example of this trend. To obtain valuable understanding of how people perceive and interact with mobile app, user reviews on platforms such as Google Play are usually analyzed. This analysis can assist developers to identify areas for improvement in both Ride-hailing and Google Play App. A promising method that can be used to analyze user perception in this instance is Aspect-Based Sentiment Analysis (ABSA). Objective: This research aimed to apply ABSA to user reviews using Bidirectional Encoder Representations from Transformers (BERT) models. In this context, aspect identification and topic modeling were performed by using Latent Dirichlet Allocation (LDA). The model extracted topics from the reviews and used Generative Artificial Intelligence (GenAI) to define the aspects of the topics to further enhance the analysis. For consistency and accuracy, the method included sentiment annotation by a human annotator. Methods: A total of two datasets were used in this research, with the first collected by scraping user reviews of Ride-Hailing App while the second was obtained from Kaggle, and to identify relevant topics, modeling was performed using LDA. These topics were then categorized into aspects using GenAI, covering areas, such as customer experience, service, payment, app features, task management, and event management. Subsequently, sentiment labeling was conducted using human annotators to provide a reliable baseline. BERT model was then used to classify sentiment with aspect hints, and the evaluation included calculations of accuracy, precision, recall, and F1-score. Results: The results showed that BERT model achieved the highest accuracy of 97% in sentiment analysis across all datasets. Conclusion: This research provided valuable understanding of user experience and established a strong ABSA framework for analyzing user reviews using LDA, Aspect Annotation, GenAI, and BERT sentiment models. Future research could expand this method to other app categories and incorporate real-time ABSA for continuous monitoring and dynamic feedback. Keywords: User Reviews, Aspect-Based Sentiment Analysis (ABSA), Sentiment Analysis, Topic Modeling, Generative Artificial Intelligence (GenAI)

BackgroundDeaths related to physical trauma impose a heavy burden on society, and the Abbreviated Injury Scale (AIS) is an important tool for injury research. AIS covers injuries to various parts of the human body and scores them based on the severity of the injury. In practical applications, the complex AIS coding rules require experts to encode by consulting patient medical records, which inevitably increases the difficulty, time, and cost of evaluation of patient and also puts higher demands on the workload of information collection and processing. In some cases, the sheer number of patients or the inability to access detailed medical records necessary for coding further complicates independent AIS codes.ObjectiveThis study aims to use advanced deep learning techniques to predict AIS codes based on easily accessible diagnostic information of patients to improve the accuracy of trauma assessment.MethodsWe used a dataset of patients with trauma (n=26,810) collected by the Chongqing Daping Hospital between October 2013 and June 2024. We mainly selected the patient’s diagnostic information, injury description, cause of injury, injury region, injury types, and present illness history as the key feature inputs. We used a robust optimization Bidirectional Encoder Representations from Transformers (BERT) pretraining method to embed these features and constructed a prediction model based on BERT. This model aims to predict AIS codes and comprehensively evaluate its performance through a 5-fold cross-validation. We compared the BERT model with previous research results and current mainstream machine learning methods to verify its advantages in prediction tasks. In addition, we also conducted external validation of the model using 244 external data points from the Chongqing Emergency Center.ResultsThe BERT model proposed in this paper performs significantly better than the comparison model on independent test datasets with an accuracy of 0.8971, which surpassed the previous study by 10 % points. In addition, the area under the curve (AUC value of the BERT model is 0.9970, and the F1-score is 0.8434. In the external dataset, the accuracy, AUC, and F1-score results of the model are 0.7131, 0.8586, and 0.6801, respectively. These results indicate that our model has high generalization ability and prediction accuracy.ConclusionsThe BERT model we proposed is mainly based on diagnostic information to predict AIS codes, and its prediction accuracy is superior to previous investigations and current mainstream machine learning methods. It has a high generalization ability in external datasets.

Bidirectional Encoder Representations from Transformers (BERT) has gained increasing attention from researchers and practitioners as it has proven to be an invaluable technique in natural languages processing. This is mainly due to its unique features, including its ability to predict words conditioned on both the left and the right context, and its ability to be pretrained using the plain text corpus that is enormously available on the web. As BERT gained more interest, more BERT models were introduced to support different languages, including Arabic. The current state of knowledge and practice in applying BERT models to Arabic text classification is limited. In an attempt to begin remedying this gap, this review synthesizes the different Arabic BERT models that have been applied to text classification. It investigates the differences between them and compares their performance. It also examines how effective they are compared to the original English BERT models. It concludes by offering insight into aspects that need further improvements and future work.

Sentence Boundary Disambiguation (SBD) is crucial for building datasets for tasks such as machine translation, syntactic analysis, and semantic analysis. Currently, most automatic sentence segmentation in Tibetan adopts the methods of rule-based and statistical learning, as well as the combination of the two, which have high requirements on the corpus and the linguistic foundation of the researchers and are more costly to annotate manually. In this study, we explore Tibetan SBD using deep learning technology. Initially, we analyze Tibetan characteristics and various subword techniques, selecting Byte Pair Encoding (BPE) and Sentencepiece (SP) for text segmentation and training the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. Secondly, we studied the Tibetan SBD based on different BERT pre-trained language models, which mainly learns the ambiguity of the shad (“།”) in different positions in modern Tibetan texts and determines through the model whether the shad (“།”) in the texts has the function of segmenting sentences. Meanwhile, this study introduces four models, BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, based on the BERT model for performance comparison. Finally, to verify the performance of the pre-trained language models on the SBD task, this study conducts SBD experiments on both the publicly available Tibetan pre-trained language model TiBERT and the multilingual pre-trained language model (Multi-BERT). The experimental results show that the F1 score of the BERT (BPE) model trained in this study reaches 95.32% on 465,669 Tibetan sentences, nearly five percentage points higher than BERT (SP) and Multi-BERT. The SBD method based on pre-trained language models in this study lays the foundation for establishing datasets for the later tasks of Tibetan pre-training, summary extraction, and machine translation.

The MotoGP One race in West Nusa Tenggara Lombok, Mandalika which was held on March 18 2022, received many responses or reactions from the public on social media, especially Twitter. There are those who agree and disagree about the holding of MotoGP in Mandalika, to find out the responses of the people who agree or disagree is needed that can process tweets data using the sentiment analysis method. The use of BERT (Bidirectional Encoder Representations from Transformers) for sentiment analysis produces a bidirectional language model that can understand the context of all words from a sentence. The dataset used goes through preprocessing stages such as case folding, data cleaning, tokenization, normalization, and removal of stopwords before sentiment analysis is carried out. This study uses several hyperparameters, namely a batch size of 32, the optimizer uses Adam with a learning rate of 3e-6 or 0.000003, and an epoch of 25. The evaluation results of the model obtain an accuracy of 55%. Precision for positive by 56%, neutral by 59%, and negative by 44%. Recall for positive is 74%, neutral is 29%, and negative is 54%. F1-score for positive is 64%, neutral is 38%, and negative is 48%.

Recently, transformer-based models have reshaped the landscape of Natural Language Processing (NLP), particularly in the domain of Machine Translation (MT). this study explores three revolutionary transformer models: Bidirectional Encoder Representations from Transformers (BERT), Generative Pretrained Transformer (GPT), and Text-to-Text Transfer Transformer (T5). The study delves into their architecture, capabilities, and applications in the context of translation technology. The study begins by discussing the evolution of machine translation from rule-based to statistical machine translation and finally to transformer models. The models have distinct architectures and purposes which pushed the limits of MT and have been instrumental in revolutionising the field. The study found significant contributions of the models in the advancement of NLP tasks including translation technology. Using comparative approach, the study further elaborates on each model’s design and utility. BERT is strong in excelling in tasks requiring a deep understanding of the context. GPT is excellent for tasks such as text generation, translation and creative writing. While the strengths of T5 is text-to-text framework by simplifying the taskspecific architectures, making it easy to perform different NLP tasks. Recognising these models’ unique features allows translators to select the best one for particular translation tasks and adjust them for better accuracy, fluency, and cultural relevance in translations. The study concludes that the models bridge language barriers, improve cross-cultural communication and pave way for more accurate and natural translations in the future. The study also points out that language processing models are continually evolving but understanding BERT, GPT, and T5’s specific features is key for ongoing development in translation technology.

Fenomena bullying di media sosial, khususnya di Twitter, telah menjadi isu yang semakin memprihatinkan dengan dampak signifikan terhadap kesehatan mental pengguna. Dalam rangka mengatasi masalah ini, deteksi otomatis tweet yang mengandung konten bullying menjadi sangat penting. Penelitian ini bertujuan untuk membandingkan performa dua model pemrosesan bahasa alami terbaru, yaitu BERT (Bidirectional Encoder Representations from Transformers) dan XLNet, dalam klasifikasi tweet yang mengandung bullying. Metodologi penelitian ini melibatkan pengumpulan dataset tweet yang telah dilabeli sebagai bullying atau non-bullying. Proses preprocessing teks dilakukan untuk membersihkan dan menyiapkan data sebelum digunakan dalam pelatihan model. Kedua model, BERT dan XLNet, dilatih dan diuji menggunakan dataset yang sama. Evaluasi performa dilakukan dengan menggunakan metrik akurasi, presisi, recall, dan F1-score. Hasil penelitian menunjukkan bahwa kedua model memiliki kemampuan yang baik dalam mengidentifikasi tweet bullying, akan tetapi XLNet menunjukkan performa yang lebih unggul dibandingkan BERT dengan tingkat akurasi sebesar 95%. Dengan nilai presisi = 100%, recall = 0,87%, dan F1-score = 0,88%. XLNet mampu menangkap konteks dan nuansa bahasa yang lebih kompleks dalam tweet, yang berkontribusi pada akurasi klasifikasi yang lebih tinggi. Penelitian ini memberikan kontribusi penting dalam bidang deteksi bullying di media sosial dengan menunjukkan bahwa penggunaan model XLNet lebih efektif dibandingkan BERT. Temuan ini dapat membantu platform seperti Twitter dalam mengidentifikasi dan mencegah konten bullying, sehingga menciptakan lingkungan online yang lebih aman bagi pengguna, serta dapat digunakan sebagai dasar untuk pengembangan sistem deteksi bullying yang lebih canggih dan efisien di masa depan. Abstract The phenomenon of bullying on social media, particularly on Twitter, has become an increasingly concerning issue with significant impacts on users' mental health. In order to address this issue, automatic detection of tweets containing bullying content is crucial. This study aims to compare the performance of two recent natural language processing models, namely BERT (Bidirectional Encoder Representations from Transformers) and XLNet, in the classification of tweets containing bullying. The research methodology involves collecting a dataset of tweets that have been labelled as bullying or non-bullying. Text preprocessing is done to clean and prepare the data before it is used in model training. Both models, BERT and XLNet, were trained and tested using the same dataset. Performance evaluation was conducted using accuracy, precision, recall, and F1-score metrics. The results show that both models have a good ability to identify bullying tweets, but XLNet shows superior performance compared to BERT with an accuracy rate of 95%. With precision = 100%, recall = 0.87%, and F1-score = 0.88%. XLNet is able to capture more complex context and language nuances in tweets, which contributes to higher classification accuracy. This research makes an important contribution to the field of bullying detection on social media by showing that the use of the XLNet model is more effective than BERT. These findings can help platforms like Twitter identify and prevent bullying content, thereby creating a safer online environment for users, and can be used as a basis for the development of more sophisticated and efficient bullying detection systems in the future.

It is essential to understand how environmental factors and cultural backgrounds affect emotional responses to build Human-Computer Interaction (HCI) technologies. This study uses the BERT (Bidirectional Encoder Representations from Transformers) model to investigate the emotional landscape of interactions on the X platform across various cultural contexts. The aims of the study are as follows: (1) Gather and prepare X platform data from COVID-19 sources in India, Pakistan, Malaysia, the US, the UK, Australia, and Canada. (2) Use a BERT model to qualitatively analyze tweet sentiment. (3) Assess the accuracy of the model and look for sentiment trends in tweets from nations throughout the pandemic. The BERT model successfully classified sentiment, as seen by its 80.29% accuracy rate on test data. Sentiment research showed that positive sentiment was far more prevalent in the US, Canada, and Australia, indicating that these countries were better able to adjust to the COVID-19 situation. Stability was seen in the balanced sentiment distributions displayed by Pakistan, India, and the United Kingdom. Despite having fewer data points, Pakistan and Malaysia continued to have largely positive attitudes. This study provides the basis for a comparative analysis of emotional responses by taking contextual and cultural aspects into account.

The coronavirus pandemic hit the worldwide population to a large extent. But, one of the subtle effects of the COVID-19 pandemic was the depletion of the mental health of the people. Social media has become an efficient platform to express oneself, and Twitter is one of the most used platforms. There has been work where machine and deep learning were employed for tweet sentimental analysis for different applications including mental depression. Most of tweets sentimental analysis were focussed on positive and negative. There has been some research where neutral tweets were taken into consideration. In this research work, we have focussed on predicting depression of people, i.e. depressed, non-depressed and neutral from tweets during lock down period by employing deep learning models like bidirection long short-term memory (Bi-LSTM), bi-directional encoder representations from transformers (BERT), and XLNET. Also, the BERT model has been modified by adding classification layer for tweet classification. In addition, the exploratory data analysis was performed for postlockdown tweets.KeywordsCOVIDBi-LSTMBERTXLNETTweets