Exploring BERT for Aspect-based Sentiment Analysis in Portuguese Language

In the ever-expanding digital landscape, the abundance of user-generated content on consumer platforms such as Booking and TripAdvisor offers a rich source of information for both travellers and hoteliers. Sentiment analysis, a fundamental research task of natural language processing (NLP) is used for mining sentiments and opinions within this vast reservoir of text reviews. A more specific type of sentiment analysis, i.e., aspect-based sentiment analysis (ABSA), is used when processing customer reviews is required. In ABSA, we aim to capture aspect-level sentiments and intricate relationships between various aspects within reviews. This article proposes a novel approach to ABSA by introducing a novel technique of word sense disambiguation (WSD) and integrating it with the Transformer architecture bidirectional encoder representations from Transformers (BERT) and graph convolutional networks (GCNs). The proposed approach resolves the intriguing ambiguities of the words and represents the review data as a complex graph structure, facilitating the modeling of intricate relationships between different aspects. The combination of bidirectional long short-term memory (BiLSTM) and GCN proves effective in capturing inter-dependencies among various aspects, providing a nuanced understanding of customer sentiments. The experiments are conducted on the RABSA dataset (an enhanced and richer hotel review data collection), and results demonstrate that our approach outperforms previous baselines, showcasing the effectiveness of integrating WSD in ABSA. Furthermore, an ablation study confirms the significant contribution of the WSD module to the overall performance. Moreover, we explore different similarity measures and find that cosine similarity yields the best results when identifying the real sense of a word in a given sentence using WordNet. The findings of our work and future work related to our work create lots of interest for people in the tourism and hospitality industry. This research gives another boost to the concept of the potential of NLP techniques in sentiment analysis. It emphasizes that if we combine the potential of NLP techniques along with state-of-the-art machine learning frameworks, we can shape the future of this field.

According to data from the U.S. Center for Disease Control and Prevention, as of June 2020, a significant number of African Americans had been infected with the coronavirus disease, experiencing disproportionately higher death rates compared to other demographic groups. These disparities highlight the urgent need to examine the experiences, behaviors, and opinions of the African American population in relation to the COVID-19 pandemic. By understanding their unique challenges in navigating matters of health and well-being, we can work towards promoting health equity, eliminating disparities, and addressing persistent barriers to care. Since Twitter data has shown significant promise as a representation of human behavior and for opinion mining, this study leverages Twitter data published in 2020 to characterize the pandemic-related experiences of the United States’ African American population using aspect-based sentiment analysis. Sentiment analysis is a common task in natural language processing that identifies the emotional tone (i.e., positive, negative, or neutral) of a text sample. Aspect-based sentiment analysis increases the granularity of sentiment analysis by also extracting the aspect for which sentiment is expressed. We developed a machine learning pipeline consisting of image and language-based classification models to filter out tweets not related to COVID-19 and those unlikely published by African American Twitter subscribers, leading to an analysis of nearly 4 million tweets. Overall, our results show that the majority of tweets had a negative tone, and that the days with larger numbers of published tweets often coincided with major U.S. events related to the pandemic as suggested by major news headlines (e.g., vaccine rollout). We also show how word usage evolved throughout the year (e.g., outbreak to pandemic and coronavirus to covid). This work also points to important issues like food insecurity and vaccine hesitation, along with exposing semantic relationships between words, such as covid and exhausted. As such, this work furthers understanding of how the nationwide progression of the pandemic may have impacted the narratives of African American Twitter users.

Aspect-Based Sentiment Analysis (ABSA) is one of the highly challenging tasks in natural language processing. It extracts fine-grained sentiment information in user-generated reviews, as it aims at predicting the polarities towards predefined aspect categories or relevant entities in free text. Previous deep learning approaches usually rely on large-scale pre-trained language models and the attention mechanism, which applies the complete computed attention weights and does not place any restriction on the attention assignment. We argue that the original attention mechanism is not the ideal configuration for ABSA, as for most of the time only a small portion of terms are strongly related to the sentiment polarity of an aspect or entity. In this paper, we propose a masked attention mechanism customized for ABSA, with two different approaches to generate the mask. The first method sets an attention weight threshold that is determined by the maximum of all weights, and keeps only attention scores above the threshold. The second selects the top words with the highest weights. Both remove the lower score parts that are assumed to be less relevant to the aspect of focus. By ignoring part of input that is claimed irrelevant, a large proportion of input noise is removed, keeping the downstream model more focused and reducing calculation cost. Experiments on the Multi-Aspect Multi-Sentiment (MAMS) and SemEval-2014 datasets show significant improvements over state-of-the-art pre-trained language models with full attention, which displays the value of the masked attention mechanism. Recent work shows that simple self-attention in Transformer quickly degenerates to a rank-1 matrix, and masked attention may be another cure for that trend.

Aspect Sentiment Triplet Extraction (ASTE) is a complex and challenging task in Natural Language Processing (NLP). It aims to extract the triplet of aspect term, opinion term, and their associated sentiment polarity, which is a more fine-grained study in Aspect Based Sentiment Analysis. Furthermore, there have been a large number of approaches being proposed to handle this relevant task. However, existing methods for ASTE suffer from powerless interactions between different sources of textual features, and they usually exert an equal impact on each type of feature, which is quite unreasonable while building contextual representation. Therefore, in this paper, we propose a novel Multi-Branch GCN (MBGCN)-based ASTE model to solve this problem. Specifically, our model first generates the enhanced semantic features via the structure-biased BERT, which takes the position of tokens into the transformation of self-attention. Then, a biaffine attention module is utilized to further obtain the specific semantic feature maps. In addition, to enhance the dependency among words in the sentence, four types of linguistic relations are defined, namely part-of-speech combination, syntactic dependency type, tree-based distance, and relative position distance of each word pair, which are further embedded as adjacent matrices. Then, the widely used Graph Convolutional Network (GCN) module is utilized to complete the work of integrating the semantic feature and linguistic feature, which is operated on four types of dependency relations repeatedly. Additionally, an effective refining strategy is employed to detect whether word pairs match or not, which is conducted after the operation of each branch GCN. At last, a shallow interaction layer is designed to achieve the final textual representation by fusing the four branch features with different weights. To validate the effectiveness of MBGCNs, extensive experiments have been conducted on four public and available datasets. Furthermore, the results demonstrate the effectiveness and robustness of MBGCNs, which obviously outperform state-of-the-art approaches.

Natural Language Inference (NLI) is a crucial task in natural language processing that involves determining the relationship between two sentences, typically referred to as the premise and the hypothesis. However, traditional NLI models solely rely on the semantic information inherent in independent sentences and lack relevant situational visual information, which can hinder a complete understanding of the intended meaning of the sentences due to the ambiguity and vagueness of language. To address this challenge, we propose an innovative ScenaFuse adapter that simultaneously integrates large-scale pre-trained linguistic knowledge and relevant visual information for NLI tasks. Specifically, we first design an image-sentence interaction module to incorporate visuals into the attention mechanism of the pre-trained model, allowing the two modalities to interact comprehensively. Furthermore, we introduce an image-sentence fusion module that can adaptively integrate visual information from images and semantic information from sentences. By incorporating relevant visual information and leveraging linguistic knowledge, our approach bridges the gap between language and vision, leading to improved understanding and inference capabilities in NLI tasks. Extensive benchmark experiments demonstrate that our proposed ScenaFuse, a scenario-guided approach, consistently boosts NLI performance.

The BERT model is de facto state-of-the-art for aspect-based sentiment analysis (ABSA), an important task in natural language processing. Similar to every other model based on deep learning, BERT is vulnerable to so-called adversarial examples: strategically modified inputs that cause a change in the model's prediction of the underlying input. In this paper we propose three new methods to create character-level adversarial examples against BERT and evaluate their effectiveness on the ABSA task. Specifically, our attack methods mimic human behavior and use leetspeak, common misspellings, or misplaced commas. By concentrating these changes on important words, we are able to maximize misclassification rates with minimal changes. To the best of our knowledge, we are the first to look into adversarial examples for the ABSA task and the first to propose these attacks.

Nowadays word embeddings, also known as word vectors, play an important role for many natural language processing (NLP) tasks. In general, these word embeddings are learned from unsupervised learning models (e.g. Word2Vec, GloVe) with a large unannotated corpus and they are independent with the task of their application. In this paper we aim to enrich word embeddings by adding more information from a specific task that is the aspect based sentiment analysis. We propose a model using a convolutional neural network that takes a labeled data set, the learned word embeddings from an unsupervised learning model (e.g. Word2Vec) as input and fine-tunes word embeddings to capture aspect category and sentiment information. We conduct experiments on restaurant review data (http://spidr-ursa.rutgers.edu/datasets/). Experimental results show that fine-tuned word embeddings outperform unsupervisedly learned word embeddings.

Sentiment analysis is becoming rapidly important for exploring social media Bangla text. The lack of sufficient resources is considered to be an important challenge for Aspect Based Sentiment Analysis (ABSA) of the Bangla language. The ABSA is a technique that divides the text and defines its sentiment based on its aspects. In this paper, we developed a high-quality Bangla ABSA annotated dataset namely BANGLA_ABSA. The datasets are labelled with aspects category and their respective sentiment polarity to do the ABSA task in Bangla. Four open domains namely Restaurant, Movie, Mobile phone, and Car are considered to make the dataset. The datasets are called Restaurant_ABSA, Movie_ABSA, Mobile_phone_ABSA, and Car_ABSA respectively that contain 801, 800, 975, and 1149 comments. All the comments are either complex or compound sentences. We created the dataset manually and annotated the same by exerting opinions. We organized the dataset as three tuples in Excel format namely 〈Id, Comment, {Aspect category, Sentiment Polarity}〉. These data are very important that facilitate the efficient handling of sentiment for any machine learning and deep learning research, especially for Bangla text.

Natural Language Inference (NLI) is a fundamental task in Natural Language Processing (NLP).However, adapting NLI models to new domains remains challenging due to the high cost of collecting domain-specific training data.While prior work proposed 15 sentence transformation rules to automate training data generation, these rules do not sufficiently capture the diversity of natural language.We propose a novel framework that combines Out-of-Distribution (OOD) detection and BERT-based clustering to identify premise-hypothesis pairs in the SNLI dataset that are not covered by existing rules and discover four new transformation rules from them.Using these rules with Chain-of-Thought (CoT) prompting and Large Language Models (LLMs), we generate high-quality training data and augment the SNLI dataset.Our method yields consistent performance improvements across dataset sizes, achieving +0.85%p accuracy on 2k samples and +0.15%p on 550k samples.Furthermore, a distribution-aware augmentation strategy enhances performance across all scales.Beyond manual explanations, we extend our framework to automatically-generated explanations (CoT-Ex), demonstrating that they provide a scalable alternative to human-written explanations and enable reliable rule discovery.

Semantic textual similarity(STS) and natural language inference(NLI) are important tasks in natural language processing(NLP) such as information retrieval, text classification, subject extraction, text summarization, machine translation and plagiarism detection. Lack of appropriate datasets in the Persian language is a major obstacle to progress in this area. Therefore, in this paper, we present FarSick, a new dataset for STS and NLI tasks in the Persian language. FarSick is the first relatively large-scale STS dataset for the Persian language. It includes 9804 pairs of Persian sentences with labels for similarity and inference for each pair of sentences. This dataset is collected by translating and editing the sentences of SICK dataset. We also measured the performance of traditional, statistical and deep learning models on it, e.g. transformers, Convolution Neural Networks, Bidirectional LSTMs, weighted average of word vectors, etc. We used different pre-trained embeddings, word2vec, glove, fastText and Bert sentence transformer. We used accuracy metric to test NLI tasks and Pearson metric to test STS tasks. The dataset is available at https://github.com/ZahraGhasemi-AI/FarSick.

Natural language inference (NLI) is an essential subtask in many natural language processing applications. It is a directional relationship from premise to hypothesis. A pair of texts is defined as entailed if a text infers its meaning from the other text. The NLI is also known as textual entailment recognition, and it recognizes entailed and contradictory sentences in various NLP systems like Question Answering, Summarization and Information retrieval systems. This paper describes the NLI problem attempted for a low resource Indian language Malayalam, the regional language of Kerala. More than 30 million people speak this language. The paper is about the Malayalam NLI dataset, named MaNLI dataset, and its application of NLI in Malayalam language using different models, namely Doc2Vec (paragraph vector), fastText, BERT (Bidirectional Encoder Representation from Transformers), and LASER (Language Agnostic Sentence Representation). Our work attempts NLI in two ways, as binary classification and as multiclass classification. For both the classifications, LASER outperformed the other techniques. For multiclass classification, NLI using LASER based sentence embedding technique outperformed the other techniques by a significant margin of 12% accuracy. There was also an accuracy improvement of 9% for LASER based NLI system for binary classification over the other techniques.

This paper presents UniBERT, a compact multilingual language model that uses an innovative training framework that integrates three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on a meticulously curated Wikipedia corpus spanning 107 languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks. Comprehensive evaluations on four tasks, named entity recognition, natural language inference, question answering, and semantic textual similarity, demonstrate that our multilingual training strategy, enhanced by an adversarial objective, significantly improves cross-lingual generalization. Specifically, UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.12%, and statistical analysis confirms the significance of these gains (p value = 0.0184). This work highlights the benefits of combining adversarial training and knowledge distillation to build robust and scalable language models, thus advancing the field of multilingual and cross-lingual natural language processing.

Learning effective representations of text words has long been a research focus in natural language processing and other machine learning tasks. In many early tasks, a text word is often represented by a one-hot vector in a discrete manner. Such a solution is not only restricted by the dimension curse, but also unable to reflect the semantic relationships between words. Recent developments focus on the learning of low-dimension and continuous vector representations of text words, known as word embedding, which can be easily applied to downstream tasks such as machine translation, natural language inference, semantic analysis and so on. In this paper, we will introduce the development of word embedding, describe the representative methods, and report its recent research trend. This paper can provide a quick guide for understanding the principle of word embedding and its development.

Sentiment analysis (SA) is one of the most active and rapidly developing research areas in both information retrieval and text mining fields of computer science. SA is the process of analysing, handling, generalizing, and reasoning about user sentiments, opinions, and emotions hidden within the text by using sentiment. There are various categories of levels of SA: (1) document-level, (2) sentence-level, and (3) aspect-level. This paper is focused on sorting out some Aspect-based sentiment analysis (ABSA) works. ABSA is one of the primary tasks in sentiment analysis, which involves identifying the expression of an opinion or sensation of an object in a particular dimension or feature. ABSA involves two primary subtasks: (1) aspect extraction and (2) aspect-based sentiment classification. In contrast to document-and sentence-based sentiment analysis, ABSA simultaneously considers sentiment and target information. Generally, the ABSA methods can be categorized into Transformer based ABSA and Graph Neural Network based ABSA. This paper introduces some popular frameworks, datasets, and evaluation metrics of ABSA tasks based on the above-proposed classification and methodology.

Question answering over tables is a very popular semantic parsing task in natural language processing (NLP). However, few existing methods focus on table images, even though there are usually large-scale unstructured tables in practice (e.g., table images). Table parsing from images is nontrivial since it is closely related to not only NLP but also computer vision (CV) to parse the tabular structure from an image. In this demo, we present a question answering system for unstructured table images. The proposed system mainly consists of 1) a table recognizer to recognize the tabular structure from an image and 2) a table parser to generate the answer to a natural language question over the table. In addition, to train the model, we further provide table images and structure annotations for two widely used semantic parsing datasets. Specifically, the test set is used for this demo, from where the users can either choose from default questions or enter a new custom question.