(Retracted) News image text classification algorithm with bidirectional encoder representations from transformers model

The article examines the use of neural networks in telemedicine for patient self-diagnosis, focusing on custom LSTM (Long Short-Term Memory) models and pre-trained BERT (Bidirectional Encoder Representations from Transformers) models. Telemedicine has become essential in modern healthcare, particularly due to the rise in remote consultations and diagnostics during global health crises. The study reviews research in this area, highlighting contributions from international and Ukrainian experts. It explores the effectiveness of neural networks in interpreting patient symptom descriptions and generating accurate diagnoses. The custom LSTM model processes text-based symptom descriptions but faces challenges with limited datasets, impacting its diagnostic accuracy. On the other hand, the BERT model leverages transfer learning, achieving over 92% accuracy in predictions, thanks to its ability to understand complex language inputs. An enhanced LSTM model is also presented, incorporating Dropout and Batch Normalization layers, which help increase the robustness and accuracy of the predictions while managing the model's complexity. The article also discusses the technical challenges of training these neural networks, including the variability of symptom descriptions and the need for large annotated medical datasets. Proposed solutions include domain-specific enhancements and data augmentation to improve the models' performance. In comparing the three models, the study identifies that while the custom LSTM model is more flexible and simple, it struggles with generalization. The BERT model, by contrast, excels in understanding and accuracy but requires more computational resources. The enhanced LSTM model offers a balance between complexity and performance. In conclusion, neural networks offer promising advances in telemedicine by providing accurate, AI-assisted self-diagnosis. Although challenges remain, such as data requirements and computing power, models like BERT are key to improving healthcare accessibility and accuracy in telemedicine applications. Among the main advantages of using artificial intelligence for the healthcare system are in-depth diagnostic analysis, individual treatment regimens, and optimization of medical processes. However, modern technologies in the medical field are not yet able to completely replace experienced specialists.

Bidirectional Encoder Representations from Transformers (BERT) has gained increasing attention from researchers and practitioners as it has proven to be an invaluable technique in natural languages processing. This is mainly due to its unique features, including its ability to predict words conditioned on both the left and the right context, and its ability to be pretrained using the plain text corpus that is enormously available on the web. As BERT gained more interest, more BERT models were introduced to support different languages, including Arabic. The current state of knowledge and practice in applying BERT models to Arabic text classification is limited. In an attempt to begin remedying this gap, this review synthesizes the different Arabic BERT models that have been applied to text classification. It investigates the differences between them and compares their performance. It also examines how effective they are compared to the original English BERT models. It concludes by offering insight into aspects that need further improvements and future work.

BackgroundDeaths related to physical trauma impose a heavy burden on society, and the Abbreviated Injury Scale (AIS) is an important tool for injury research. AIS covers injuries to various parts of the human body and scores them based on the severity of the injury. In practical applications, the complex AIS coding rules require experts to encode by consulting patient medical records, which inevitably increases the difficulty, time, and cost of evaluation of patient and also puts higher demands on the workload of information collection and processing. In some cases, the sheer number of patients or the inability to access detailed medical records necessary for coding further complicates independent AIS codes.ObjectiveThis study aims to use advanced deep learning techniques to predict AIS codes based on easily accessible diagnostic information of patients to improve the accuracy of trauma assessment.MethodsWe used a dataset of patients with trauma (n=26,810) collected by the Chongqing Daping Hospital between October 2013 and June 2024. We mainly selected the patient’s diagnostic information, injury description, cause of injury, injury region, injury types, and present illness history as the key feature inputs. We used a robust optimization Bidirectional Encoder Representations from Transformers (BERT) pretraining method to embed these features and constructed a prediction model based on BERT. This model aims to predict AIS codes and comprehensively evaluate its performance through a 5-fold cross-validation. We compared the BERT model with previous research results and current mainstream machine learning methods to verify its advantages in prediction tasks. In addition, we also conducted external validation of the model using 244 external data points from the Chongqing Emergency Center.ResultsThe BERT model proposed in this paper performs significantly better than the comparison model on independent test datasets with an accuracy of 0.8971, which surpassed the previous study by 10 % points. In addition, the area under the curve (AUC value of the BERT model is 0.9970, and the F1-score is 0.8434. In the external dataset, the accuracy, AUC, and F1-score results of the model are 0.7131, 0.8586, and 0.6801, respectively. These results indicate that our model has high generalization ability and prediction accuracy.ConclusionsThe BERT model we proposed is mainly based on diagnostic information to predict AIS codes, and its prediction accuracy is superior to previous investigations and current mainstream machine learning methods. It has a high generalization ability in external datasets.

It is essential to understand how environmental factors and cultural backgrounds affect emotional responses to build Human-Computer Interaction (HCI) technologies. This study uses the BERT (Bidirectional Encoder Representations from Transformers) model to investigate the emotional landscape of interactions on the X platform across various cultural contexts. The aims of the study are as follows: (1) Gather and prepare X platform data from COVID-19 sources in India, Pakistan, Malaysia, the US, the UK, Australia, and Canada. (2) Use a BERT model to qualitatively analyze tweet sentiment. (3) Assess the accuracy of the model and look for sentiment trends in tweets from nations throughout the pandemic. The BERT model successfully classified sentiment, as seen by its 80.29% accuracy rate on test data. Sentiment research showed that positive sentiment was far more prevalent in the US, Canada, and Australia, indicating that these countries were better able to adjust to the COVID-19 situation. Stability was seen in the balanced sentiment distributions displayed by Pakistan, India, and the United Kingdom. Despite having fewer data points, Pakistan and Malaysia continued to have largely positive attitudes. This study provides the basis for a comparative analysis of emotional responses by taking contextual and cultural aspects into account.

Twitter has emerged as a critical platform for capturing public sentiment, offering a valuable source for sentiment analysis. This study presents a comparative evaluation of two BERT (Bidirectional Encoder Representations from Transformers) models—baseline and fine-tuned—targeted at analyzing Indonesian-language tweets. Employing the CRISP-DM framework, the methodology encompasses automated data crawling, comprehensive text pre-processing (including case folding, cleaning, tokenisation, normalisation, and data augmentation), and model development using the IndoBERT-base-p1 architecture. The experimental results reveal that the fine-tuned BERT model achieves significantly improved performance over the non-optimized model, with accuracy, precision, recall, and F1-score values reaching 91%, 0.91, 0.90, and 0.91, respectively. These findings indicate the fine-tuned model's superior ability to capture linguistic subtleties and contextual sentiment features within informal social media text. Furthermore, the model is deployed in a web-based application for real-time sentiment classification, demonstrating its practical applicability. This study underscores the effectiveness of Fine-tuning in enhancing BERT-based sentiment analysis for low-resource languages. It highlights its potential for informing decisionmaking in digital communication, marketing, and policy research contexts.

Investor attitude is an important factor in driving the behavior of the stock market, usually dwarfing conventional financial markets. Although previous study has investigated the mental side of investing, this has been concentrated mostly on the developed world and has had no real-time behavioral incorporation. This study fulfills this requirement by investigating the roles played by investor mentality, media reporting, and macroeconomic triggers in determining stock market volatility during bear markets with a special emphasis on Kathmandu. The goal is to measure the effect of sentiment on market performance based on both structured survey data and unstructured social media and news text. A hybrid methodological design is used, combining statistical modeling (Chi-Square test) and a machine learning-based sentiment classification approach utilizing a fine-tuned BERT (Bidirectional Encoder Representations from Transformers) model. Social media and financial news information are both drawn from a 2022–2025 Kaggle Stock Market Sentiment dataset, and NVivo provides thematic analysis of investor interviews. The following BERT model, coded through Python's Hugging Face Transformers package, resulted in a 9.2% improvement on accuracy compared to standard sentiment analysis models such as logistic regression and SVM, providing richer contextual knowledge of investor emotion. Empirical findings indicate a statistically significant relationship between sentiment and investment behavior (χ² = 23.250, p = 0.006), validating market change behavior. This study not only emphasizes sentiment as an important predictive variable but also proposes a scalable, real-time predictive framework for risk-sensitive investment strategy. The application of this model can improve the early identification of volatility and guide sentiment-based trading systems for investors, analysts, and policymakers in developing markets.

Bidirectional Encoder Representations from Transformers (BERT) models achieve state-of-the-art performance on a number of Natural Language Processing tasks. However, their model size on disk often exceeds 1 GB and the process of fine-tuning them and using them to run inference consumes significant hardware resources and runtime. This makes them hard to deploy to production environments. This paper fine-tunes DistilBERT, a lightweight deep learning model, on medical text for the named entity recognition task of Protected Health Information (PHI) and medical concepts. This work provides a full assessment of the performance of DistilBERT in comparison with BERT models that were pre-trained on medical text. For Named Entity Recognition task of PHI, DistilBERT achieved almost the same results as medical versions of BERT in terms of F1 score at almost half the runtime and consuming approximately half the disk space. On the other hand, for the detection of medical concepts, DistilBERT's F1 score was lower by 4 points on average than medical BERT variants.

Large language models (LLMs) have emerged as a milestone in the realm of machine learning, offering groundbreaking capabilities in understanding human language with remarkable coherence and versatility. Their application spans a multitude of domains, ranging from text classification to information extraction. Such models are first pre-trained on vast data corpora to learn general language patterns and then are fine-tuned on specific downstream tasks. Despite their impressive capabilities, LLMs encounter several limitations, including sub-optimal performance in understanding the domain-specific language of drug labels, limited performance in analyzing class-imbalanced data, and high computational complexity of training. Human prescription drug labeling contains a summary of the essential scientific information needed for the safe and effective use of the drug and includes critical information about drug products, such as pharmacokinetics and adverse events. Automatic information extraction from drug labels can facilitate processes such as finding the adverse reaction of the drugs or finding the interaction of one drug with another drug. Although LLMs are an interesting solution to perform this task, because of their training on generic data, they may not possess the depth of knowledge or the ability required to understand the language in such a specialized field. To address this issue, this work presents the developed PharmBERT, which is a BERT model specifically pretrained on the drug labels, publicly available online for the NLP community. We demonstrate that our model outperforms models such as vanilla BERT, ClinicalBERT, and BioBERT in multiple NLP tasks in the drug label domain. Another shortcoming of LLMs lies in their handling of class-imbalanced data, leading to a subpar performance in identifying and accurately responding to minority classes. This class-imbalance can skew the model's learning process, favoring majority classes, and thus diminishing its effectiveness in recognizing and appropriately addressing less represented categories. To cope with these challenges, we present a simple modification of standard fine-tuning. Specifically, we propose a two-stage fine-tuning. In Stage 1, we fine-tune the final layer of the pretrained model with class-balanced augmented data, generated using ChatGPT. As a large generative language model, ChatGPT is capable of generating novel yet contextually similar responses to a given text sample, which makes it an excellent candidate for data augmentation. In Stage 2, we perform the standard fine-tuning. Our modification has several benefits: (1) it leverages pretrained representations by only fine-tuning a small portion of the model parameters while keeping the rest untouched; (2) it allows the model to learn an initial representation of the specific task; and importantly (3) it protects the learning of tail classes from being at a disadvantage during the model updating. The experimental results show that the proposed two-stage fine-tuning outperforms vanilla fine-tuning and state-of-the-art methods on the above datasets. The third issue of LLMs is the high computational complexity of training. Due to the large number of parameters in many state-of-the-art LLMs, such as Bidirectional Encoder Representations from Transformers (BERT), the process of fine-tuning exhibits a high level of intricacy. One attractive solution to this issue is parameter-efficient fine-tuning, which means only fine-tuning a small portion of the model and not changing the rest. However, the question that which portion of the BERT model is the most important one for fine-tuning remains unanswered. Here, we first analyze different components in the BERT model to see which one undergoes the most change after fine-tuning. We show that output LayerNorm changes more than other components when finetuned for different General Language Understanding Evaluation (GLUE) tasks. Then we show that only fine-tuning this component reaches a comparable, or in some cases better, performance to full fine-tuning and other parameter-efficient fine-tuning methods. Moreover, we use Fisher information to find out which parameters in LayerNorm are the most important ones and show that with a small performance degradation, many NLP tasks in the GLUE benchmark can be solved by fine-tuning only a small, selected portion of LayerNorm. In summary, LLMs hold immense potential to redefine and enhance various fields, including drug-label analysis, but they also suffer from limitations like sub-optimal performance in understanding domain-specific texts of drug labels, not learning minority classes in the class-imbalanced data, and high computational complexity. In this thesis, for each one of these shortcomings, first, the problem is analyzed and then a solution is provided.

Many cutting-edge language models have been used in the past to forecast election results. Sentiment analysis aids in opinion mining – a common experiment used to detect public opinions – on a given topic. Twitter has gained popularity and established itself as a crucial instrument for analyzing public opinion on elections and other trending issues. The unexpected but interesting results of recently held Nigeria's presidential election shifted attention to the upcoming governorship race in Lagos State. In this work, we propose a Google’s Bidirectional Encoder Representations from Transformers (BERT) model for the sentiment analysis of governorship election in Lagos State, Nigeria, using Twitter data. A total of 800,000 personal and public tweets were scraped from twitter concerning the three prominent contesting Lagos State Gubernatorial candidates using carefully selected search queries. The tweets were preprocessed to avoid noise and inconsistencies and the preprocessed tweets were parsed into the pre-trained and finetuned BERT model. The result was analyzed to establish the sentiments of the public about the candidates. The social networks of the candidates were also presented and the effect of parameter using different learning rates (LR) was also considered. The BERT model achieved the maximum performance under varied learning rate and epoch sizes of 88% precision, 92% recall and 91% F1-Measure. Results also showed that the learning rate at 1e-7 gave the best performance. Also, the smaller the learning rate, the higher the accuracy but the larger the epoch size, the higher the accuracy. Applying the developed BERT model to the public’s tweet showed that the election will be a two-party race between the Labour Party and All Progressives Congress party, thereby challenging the status quo. The results of the experiment demonstrated that sentiment analysis and other Natural Language Processing activities can help with comprehension of the social media environment. Results also showed how much influence each candidate has over the outcome of the election. We come to the conclusion that estimating election results and providing insights for electoral parties can benefit from sentiment analysis and other language models.

In this study, the extractive summarization using sentence embeddings generated by the finetuned Bidirectional Encoder Representations from Transformers (BERT) models and the k-means clustering method has been investigated. To show how the BERT model can capture the knowledge in specific domains like engineering design and what it can produce after being finetuned based on domain-specific data sets, several BERT models are trained, and the sentence embeddings extracted from the finetuned models are used to generate summaries of a set of papers. Different evaluation methods are then applied to measure the quality of summarization results. Both the machine evaluation method Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and a human-based evaluation method are used for the comparison study. The results indicate that the BERT model finetuned with a larger dataset can generate summaries with more domain terminologies than the pretrained BERT model. Moreover, the summaries generated by BERT models have more contents overlapping with original documents than those obtained through other popular non-BERT-based models. The experimental results indicate that the BERT-based method can provide better and more informative summaries to engineers. It has also been demonstrated that the contextualized representations generated by BERT-based models can capture information in text and have better performance in applications like text summarizations after being trained by domain-specific data sets.

In recent years, with the rapid development of the Internet and multimedia technology, English translation text classification has played an important role in various industries. However, English translation remains a complex and difficult problem. Seeking an efficient and accurate English translation method has become an urgent problem to be solved. The study first elucidated the possibility of the development of transfer learning technology in multimedia environments, which was recognized. Then, previous research on this issue, as well as the Bidirectional Encoder Representations from Transformers (BERT) model, the attention mechanism and bidirectional long short-term memory (Att-BILSTM) model, and the transfer learning based cross domain model (TLCM) and their theoretical foundations, were comprehensively explained. Through the application of transfer learning in multimedia network technology, we deconstructed and integrated these methods. A new text classification technology fusion model, the BATCL transfer learning model, has been established. We analyzed its requirements and label classification methods, proposed a data preprocessing method, and completed experiments to analyze different influencing factors. The research results indicate that the classification system obtained from the study has a similar trend to the BERT model at the macro level, and the classification method proposed in this study can surpass the BERT model by up to 28%. The classification accuracy of the Att-BILSTM model improves over time, but it does not exceed the classification accuracy of the method proposed in this study. This study not only helps to improve the accuracy of English translation, but also enhances the efficiency of machine learning algorithms, providing a new approach for solving English translation problems.

The fast move to digital for HR tasks has led to the creation of a lot of unstructured documents, especially offer letters, which provide important information about employees. Using usual approaches, this has not been processed well for extracting information. This study presents a comprehensive system for the automatic extraction of information from offer letters utilizing optical character recognition (OCR) and named entity recognition. A comparison investigation has been conducted about the performance of the conventional Conditional Random Field (CRF) model and the advanced Bidirectional Encoder Representations from Transformers (BERT) model. This paper presents a system for generating synthetic data to address the challenges of data scarcity and privacy. This framework may create fake data for the process of getting information from offer letters, such as alternative layouts, semantic content, and noise. This paper evaluated the efficacy of the models on a demanding dataset of 500 documents. The findings indicate that, while the traditional Conditional Random Field model yielded satisfactory outcomes, the advanced Bidirectional Encoder Representations from Transformers model produced significantly superior results in comparison to the conventional model. This experimental evaluation also proves the efficiency of the proposed system in extracting key entities such as candidate name, designation, salary, and date of joining accurately from the scanned offer letters. The comparative analysis proves that, although the proposed system using the CRF model works accurately in structured data, the BERT model has a better capacity for understanding the contextual relationship within the document text. Thus, the proposed transformer model proves to be effective in dealing with the variability in the document layouts, linguistic expressions, and document conditions.

As one of the most important posttranslational modifications (PTMs), protein lysine glycation changes the characteristics of the proteins and leads to the dysfunction of the proteins, which may cause diseases. Accurately detecting the glycation sites is of great benefit for understanding the biological function and potential mechanism of glycation in the treatment of diseases. However, experimental methods are expensive and time-consuming for lysine glycation site identification. Instead, computational methods, with their higher efficiency and lower cost, could be an important supplement to the experimental methods. In this study, we proposed a novel predictor, BERT-Kgly, for protein lysine glycation site prediction, which was developed by extracting embedding features of protein segments from pretrained Bidirectional Encoder Representations from Transformers (BERT) models. Three pretrained BERT models were explored to get the embeddings with optimal representability, and three downstream deep networks were employed to build our models. Our results showed that the model based on embeddings extracted from the BERT model pretrained on 556,603 protein sequences of UniProt outperforms other models. In addition, an independent test set was used to evaluate and compare our model with other existing methods, which indicated that our model was superior to other existing models.

Semantic matching is one of the critical technologies for intelligent customer service. Since Bidirectional Encoder Representations from Transformers (BERT) is proposed, fine-tuning on a large-scale pre-training language model becomes a general method to implement text semantic matching. However, in practical application, the accuracy of the BERT model is limited by the quantity of pre-training corpus and proper nouns in the target domain. An enhancement method for knowledge based on domain dictionary to mask input is proposed to solve the problem. Firstly, for modul input, we use keyword matching to recognize and mask the word in domain. Secondly, using self-supervised learning to inject knowledge of the target domain into the BERT model. Thirdly, we fine-tune the BERT model with public datasets LCQMC and BQboost. Finally, we test the model’s performance with a financial company’s user data. The experimental results show that after using our method and BQboost, accuracy increases by 12.12% on average in practical applications.

Bali is one of the provinces with the most abundant natural and cultural wealth in Indonesia. One commodity that supports it is coffee. Bali Coffee is not only a gastronomic identity, but also a cultural identity which makes it have added value to be developed into various business lines. One business derivative that is quite promising is a coffee shop. However, these favorable conditions also need to be maintained to ensure good quality reaches consumers. One thing that can do is analyze reviews from customers. One of the most popular methods is Sentiment Analysis. This technique allows business to analyze customer reviews on social media. It can be a feedback to maintaining and improving quality and good relationships with customers. This research aims to create a machine learning model to analyze customer reviews at several coffee shops in Bali which are divided into three labels, namely: positive, negative and neutral. The methods used are: scraping, cleaning, stopword removal, embedding, undersampling, and modeling. The algorithms used are Bidirectional Encoder Representation from Transformer (BERT) and Generative Pre-trained Transformers (GPT). The performance metrics used in this research are precision, recall, accuracy and loss. This research succeeded in creating a sentiment analysis model for coffee shop customers in Bali. The BERT model obtained an accuracy value of 78% without undersampling with a loss in the 10th iteration of 0.27. Meanwhile, the BERT model with undersampling obtained an accuracy value of 32.85% with a loss in the 10th iteration of 0.16. The GPT2 model without undersampling gets an accuracy of 78% with a loss in the 10th iteration of 0.25. Meanwhile, the GPT model with undersampling obtained an accuracy value of 32.85% with a loss in the 10th iteration of 0.15.