Stance-level Sarcasm Detection with BERT and Stance-centered Graph Attention Networks

In this study, we explore the potential of machine learning (ML) models after different text representation methods on the balanced IMDB dataset, which is widely regarded as a gold standard in sentiment analysis, one of the Natural Language processing (NLP) tasks. On the open source IMDB movie reviews dataset, we first undertake data cleaning and text representation with data preprocessing steps. Then, we apply sentiment classification using different ML models. In order to evaluate the models, we used precision (P), recall (R), F1-score (F1), and area under curve (AUC), as well as receiver operating characteristic (ROC). It is worth noting that text feature extraction with Bidirectional Encoder Representations from Transformers (BERT) provided the highest performance in all models, with the SVM model offering particularly promising results. In this model, we observed the following results: ACC 0.9033, F1 0.9308, R 0.9015, R 0.9015, P 0.9072, AUC 0.9638, and ROC 0.96. These findings suggest that NLP techniques and, in particular, machine learning models that employ BERT may offer high levels of accuracy and reliability in text classification problems. It would be beneficial for future studies to validate these findings using BERT on different NLP tasks. This would help to evaluate the effectiveness and applicability of the models in practice.

Artificial Intelligence (AI) functionalities have helped improve the working lives of the human population in sectors like the health sector, education sector, industries and tourism. Emerging economies, in particular, have been exploring the potential of AI to transform their tourism and hospitality sectors. This research paper aimed to examine the use of pre-trained AI models in the tourism and hospitality industries in emerging economies. The study presented a literature review of the current state of AI in the tourism and hospitality industries, including its opportunities and challenges. It then proposed the use of a pre-trained AI model, specifically the Bidirectional Encoder Representations from Transformers (BERT) model, to help address some of the challenges facing the tourism and hospitality sectors in emerging economies. BERT is a language model that can be fine-tuned for a range of natural language processing (NLP) tasks such as sentiment analysis, entity recognition and text classification. The paper discussed the potential applications of BERT in the tourism and hospitality sectors, such as analysing customer feedback, enhancing chatbot interactions and improving search engine optimization (SEO) for tourism websites. The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) was used to evaluate the accuracy of the chatbot model that was used in this study. In conclusion, this research paper suggested that pre-trained AI models, such as BERT, can offer valuable insights and solutions in improving customer satisfaction in the tourism and hospitality industries in emerging economies.

Natural Language Processing (NLP) is a pivotal area in artificial intelligence, aiming to make computers capable of understanding and generating human language. This study evaluates and compares three prominent NLP models—the Generative Pre-trained Transformer (GPT) model, Bidirectional Encoder Representations from Transformers (BERT) model, and Generalized Autoregressive Pretraining for Language Understanding (XLNet)—to determine their strengths, limitations, and suitability for various tasks. The research involves a comprehensive analysis of these models, utilizing well-established datasets such as the Stanford Question Answering Dataset (SQuAD), General Language Understanding Evaluation (GLUE), Reading Comprehension from Examinations (RACE), and the Situations with Adversarial Generations (SWAG). The study explores each model's architecture, pre-training, and fine-tuning processes: GPT’s unidirectional approach is assessed for its language generation and handling of long-range dependencies; Bidirectional encoding is examined for its effectiveness in context understanding, and XLNet permutation-based training is analyzed for its robust contextual comprehension. The experimental results reveal that GPT excels in generative tasks but is constrained by its unidirectional nature. BERT achieves superior accuracy in comprehension tasks but is computationally demanding and susceptible to pre-training bias. XLNet outperforms both GPT and BERT in accuracy and contextual understanding, though at the cost of increased complexity. The results offer a significant understanding of the effectiveness and applicability of these models, suggesting future research directions such as hybrid models and improvements in efficiency.

In this paper we focus on an Natural Language Inference task. Being given two sentences, we classify their relation as NEUTRAL, ENTAILMENT or CONTRADICTION. Considering the achievements of BERT (Bidirectional Encoder Representations from Transformers) in many Natural Language Processing tasks, we use BERT features to create our base model for this task. However, several questions arise: can other features improve the performance obtained with BERT? If we are able to predict the situations in which BERT will fail, can we improve the performance by providing alternative models for these situations? We test several strategies and models, as alternatives to the standalone BERT model in the possible failure situations, and we take advantage of semantic features extracted from Discourse Representation Structures.

Bidirectional Encoder Representations from Transformers (BERT) and BERT-based approaches are the current state-of-the-art in many natural language processing (NLP) tasks; however, their application to document classification on long clinical texts is limited. In this work, we introduce four methods to scale BERT, which by default can only handle input sequences up to approximately 400 words long, to perform document classification on clinical texts several thousand words long. We compare these methods against two much simpler architectures - a word-level convolutional neural network and a hierarchical self-attention network - and show that BERT often cannot beat these simpler baselines when classifying MIMIC-III discharge summaries and SEER cancer pathology reports. In our analysis, we show that two key components of BERT - pretraining and WordPiece tokenization - may actually be inhibiting BERT's performance on clinical text classification tasks where the input document is several thousand words long and where correctly identifying labels may depend more on identifying a few key words or phrases rather than understanding the contextual meaning of sequences of text.

Natural Language Processing (NLP) has been profoundly transformed by Artificial Intelligence (AI), particularly in developing academic chatbot systems. This study optimizes NLP for academic chatbots by implementing a domain-specific fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model. The primary objective is to enhance the chatbot's comprehension of conversational context, its accuracy in delivering academic information, and its pertinence in responding to inquiries from students and parents. The methodology involved collecting a corpus of 6,000 academic queries, followed by text pre-processing and fine-tuning the BERT model, with performance evaluated against traditional TF-IDF-SVM and LSTM baselines using accuracy and macro F1-score. The results demonstrate that the fine-tuned BERT model (Model C) achieved a superior accuracy of 95.8% and a macro F1-score of 0.95, significantly outperforming the LSTM (84.2% accuracy, 0.82 F1-score) and TF-IDF-SVM (61.5% accuracy, 0.59 F1-score) models. Furthermore, the model exhibited remarkable robustness, maintaining 93.5% accuracy on a challenging subset of queries with spelling irregularities, informal language, and grammatical errors. These findings indicate that the application of a domain-optimized BERT architecture effectively handles diverse and imperfect linguistic patterns, bridging the gap between generic language models and the specialized needs of the academic domain. The novelty of this study lies in its domain-specific fine-tuning of BERT for academic chatbot intent recognition, providing a replicable framework that can enhance institutional information systems and improve the quality of student and parent engagement.

Text classification algorithms investigate the intricate relationships between words or phrases and attempt to deduce the document's interpretation. In the last few years, these algorithms have progressed tremendously. Transformer architecture and sentence encoders have proven to give superior results on natural language processing tasks. But a major limitation of these architectures is their applicability for text no longer than a few hundred words. In this paper, we explore hierarchical transfer learning approaches for long document classification. We employ pre-trained Universal Sentence Encoder (USE) and Bidirectional Encoder Representations from Transformers (BERT) in a hierarchical setup to capture better representations efficiently. Our proposed models are conceptually simple where we divide the input data into chunks and then pass this through base models of BERT and USE. Then output representation for each chunk is then propagated through a shallow neural network comprising of LSTMs or CNNs for classifying the text data. These extensions are evaluated on 6 benchmark datasets. We show that USE + CNN/LSTM performs better than its stand-alone baseline. Whereas the BERT + CNN/LSTM performs on par with its stand-alone counterpart. However, the hierarchical BERT models are still desirable as it avoids the quadratic complexity of the attention mechanism in BERT. Along with the hierarchical approaches, this work also provides a comparison of different deep learning algorithms like USE, BERT, HAN, Longformer, and BigBird for long document classification. The Longformer approach consistently performs well on most of the datasets.

- Marathi is one of the popular oldest languages of India and possesses the greater syntactic complexity. Nouns, verbs and compound words of this language have very clear and simple rules that make the learning of the language an easy task for anybody. However, a more effective tool is required to consolidate the particularity of the higher-level grammar, especially the regional dialects. In the last few years there has been a lot of work done in the era of Artificial Intelligence (AI) and Natural Language Processing (NLP) where various languages related complicated tasks have been made easy. Pre-trained generative AI models like BERT (Bidirectional Encoder Representations from Transformers), GPT-3 (Generative Pre-trained Transformer), T5 (Text-to-Text Transfer Transformer) and many others hold much promise in different language uses like text generation, translation, and grammar checks. These models can formulate, interpreting and modifying any linguistic behaviour that might be useful in handling other challenges of Marathi language such as noun declensions and verb conjugations. AI model built from the transformer architecture can be adapted to handle several of linguistic problems in Marathi language that involves syntax analysis and error detection. This paper presents a generative AI model for learning the Marathi grammar which is further categorized into two parts. The first part of the study is devoted to the elementary grammar training, and the second part is dedicated to the intermediate and advanced levels. Using generative AI, this current model gives higher accuracy than rule-based system and presents effective ideas towards modern grammar. Also, this model became improving tools in computational linguistics for regional languages and for promoting language education. Keywords: Marathi language, generative AI, BERT, GPT-3, T5, computational linguistics

Given the foundational role of system requirements in design projects, designers can benefit from classifying, comparing, and observing connections between requirements. Manually undertaking these processes, however, can be laborious and time-consuming. Previous studies have employed Bidirectional Encoder Representations from Transformers (BERT), a state-of-the-art natural language processing (NLP) deep neural network model, to automatically analyze written requirements. Yet, it remains unclear whether BERT can sufficiently capture the nuances that differentiate requirements between and within design documents. This work evaluates BERT’s performance on two requirement classification tasks (one inter- document and one intra-document) executed on a corpus of 1,303 requirements sourced from five system design projects. First, in the “parent document classification” task, a BERT model is fine-tuned to classify requirements according to their originating project. A separate BERT model is then fine-tuned on a “functional classification” task where each requirement is classified as either functional or nonfunctional. Our results also include a comparison with a baseline model, Word2Vec, and demonstrate that our model achieves higher classification accuracy. When evaluated on test sets, the former model receives a Matthews correlation coefficient (MCC) of 0.95, while the latter receives an MCC of 0.82, indicating BERT’s ability to reliably distinguish requirements. This work then explores the application of BERT’s representations, known as embeddings, to identify similar requirements and predict requirement change.

To address the insufficient expressive capabilities of traditional methods in assessing the development level of rural tourism, this study explores the fusion application of the Bidirectional Encoder Representations from Transformers (BERT) and the Back Propagation (BP) algorithm to enhance the accuracy and comprehensiveness of rural tourism development assessment. Firstly, this study introduces the BERT deep learning model and its applications in natural language processing, alongside the role of the BP algorithm in pattern recognition and predictive analysis. Subsequently, a framework for assessing rural tourism development levels, integrating BERT and the BP algorithm, is proposed. This framework collects multidimensional rural tourism-related data and utilizes the BERT model for sentiment analysis and topic extraction from textual data. Empirical analysis of rural tourism development in a specific region validates the effectiveness of the proposed approach. Experimental results demonstrate: (1) The model achieves an accuracy of 84.33% and an F1 score of 85.33% on the publicly available Laptop dataset, with a processing time of 20 s, significantly outperforming other methods. Compared to traditional approaches, the proposed method accurately captures correlations between textual information and numerical data, thereby enhancing the credibility and accuracy of assessment results. (2) From the ablation study results, it is evident that removing any component from the model leads to performance degradation. Specifically, removing the Bidirectional Gated Recurrent Unit (BiGRU) reduces accuracy and F1 scores to 78.21% and 76.33% on the Laptop dataset, and to 85.10% and 70.45% on the Tourist_F dataset. Removing Text Convolutional Neural Network (CNN) reduces accuracy and F1 scores to 79.34% and 77.56% on the Laptop dataset, and to 86.25% and 72.11% on the Tourist_F dataset. The most significant performance decline occurs upon removing BERT, with accuracy and F1 scores decreasing to 70.14% and 66.43% on the Laptop dataset, and to 78.45% and 60.33% on the Tourist_F dataset. These results indicate that BiGcRU, TextCNN, and BERT each contribute significantly to the model’s performance. This study provides substantial support for advancing sustainable development in rural tourism, offering significant practical and innovative value.

(1) Background: Poor adherence to management behaviors in Chinese Type 2 diabetes mellitus (T2DM) patients leads to an uncontrolled prognosis of diabetes, which results in significant economic costs for China. It is imperative to quickly locate vulnerability factors in the management behavior of patients with T2DM. (2) Methods: In this study, a thematic analysis of the collected interview materials was conducted to construct the themes of T2DM management vulnerability. We explored the applicability of the pre-trained models based on the evaluation metrics in text classification. (3) Results: We constructed 12 themes of vulnerability related to the health and well-being of people with T2DM in Tianjin. We considered that Bidirectional Encoder Representation from Transformers (BERT) performed better in this Natural Language Processing (NLP) task with a shorter completion time. With the splitting ratio of 6:3:1 and batch size of 64 for BERT, the test accuracy was 97.71%, the completion time was 10 min 24 s, and the macro-F1 score was 0.9752. (4) Conclusions: Our results proved the applicability of NLP techniques in this specific Chinese-language medical environment. We filled the knowledge gap in the application of NLP technologies in diabetes management. Our study provided strong support for using NLP techniques to rapidly locate vulnerability factors in T2DM management.

With the popularity of large-scale corpora, statistics-based models have become mainstream model in Natural Language Processing (NLP). The Bidirectional Encoder Representations from Transformers (BERT), as one of those models, has achieved excellent results in various tasks of NLP since its emergence. But it still has shortcomings, such as poor capability of extracting local features and exploding of training gradients. After analyzing the shortcomings of BERT, this paper proposed an Information-gain Association Vector Semi-supervised Bidirectional Encoder Representations from Transformers (IAS-BERT) model, which improves the capability of capturing local features. Considering the influence of feature's polarity to overall sentiment and association between two word-embeddings, we use information gain on the training corpus. And then, the information gain results are used as an annotation of training corpus to generate a new word embedding. At the same time, we use forward-matching to optimize the computational overhead of IAS-BERT. We experiment the model on dataset of sentiment analysis, and it have achieved good results.

Relying on large pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) for encoding and adding a simple prediction layer has led to impressive performance in many clinical natural language processing (NLP) tasks. In this work, we present a novel extension to the Transformer architecture, by incorporating signature transform with the self-attention model. This architecture is added between embedding and prediction layers. Experiments on a new Swedish prescription data show the proposed architecture to be superior in two of the three information extraction tasks, comparing to baseline models. Finally, we evaluate two different embedding approaches between applying Multilingual BERT and translating the Swedish text to English then encode with a BERT model pretrained on clinical notes.

Online medias are currently the dominant source of Information due to not being limited by time and place, fast and wide distributions. However, inaccurate news, or often referred as fake news is a major problem in news dissemination for online medias. Inaccurate news is information that is not true, that is engineered to cover the real information and has no factual basis. Usually, inaccurate news is made in the form of news that has mass appeal and is presented in the guise of genuine and legitimate news nuances to deceive or change the reader's mind or opinion. Identification of inaccurate news from real news can be done with natural language processing (NLP) technologies. In this paper, we proposed bidirectional encoder representations from transformers (BERT) for inaccurate news identification. BERT is a language model based on deep learning technologies and it has found effective for many NLP tasks. In this study, we use transfer learning and fine-tuning to adapt BERT for inaccurate news identification. The experiments show that our method could achieve accuracy of 99.23%, recall 99.46%, precision 98.86%, and F-Score of 99.15%. It is largely better than traditional method for the same tasks.

Recently, transformer-based models have reshaped the landscape of Natural Language Processing (NLP), particularly in the domain of Machine Translation (MT). this study explores three revolutionary transformer models: Bidirectional Encoder Representations from Transformers (BERT), Generative Pretrained Transformer (GPT), and Text-to-Text Transfer Transformer (T5). The study delves into their architecture, capabilities, and applications in the context of translation technology. The study begins by discussing the evolution of machine translation from rule-based to statistical machine translation and finally to transformer models. The models have distinct architectures and purposes which pushed the limits of MT and have been instrumental in revolutionising the field. The study found significant contributions of the models in the advancement of NLP tasks including translation technology. Using comparative approach, the study further elaborates on each model’s design and utility. BERT is strong in excelling in tasks requiring a deep understanding of the context. GPT is excellent for tasks such as text generation, translation and creative writing. While the strengths of T5 is text-to-text framework by simplifying the taskspecific architectures, making it easy to perform different NLP tasks. Recognising these models’ unique features allows translators to select the best one for particular translation tasks and adjust them for better accuracy, fluency, and cultural relevance in translations. The study concludes that the models bridge language barriers, improve cross-cultural communication and pave way for more accurate and natural translations in the future. The study also points out that language processing models are continually evolving but understanding BERT, GPT, and T5’s specific features is key for ongoing development in translation technology.