Sentiment Analysis for Software Engineering: How Far Can Pre-trained Transformer Models Go?

Pre-Trained Language Models and Their Applications

The emergence of Pre-trained Language Models (PLMs) has achieved tremendous success in the field of Natural Language Processing (NLP) by learning universal representations on large corpora in a self-supervised manner. The pre-trained models and the learned representations can be beneficial to a series of downstream NLP tasks. This training paradigm has recently been adapted to the recommendation domain and is considered a promising approach by both academia and industry. In this paper, we systematically investigate how to extract and transfer knowledge from pre-trained models learned by different PLM-related training paradigms to improve recommendation performance from various perspectives, such as generality, sparsity, efficiency and effectiveness. Specifically, we propose a comprehensive taxonomy to divide existing PLM-based recommender systems w.r.t. their training strategies and objectives. Then, we analyze and summarize the connection between PLM-based training paradigms and different input data types for recommender systems. Finally, we elaborate on open issues and future research directions in this vibrant field.

Coffee bean production can encounter challenges due to fluctuations in global coffee prices, impacting the economic stability of some countries that heavily depend on coffee production. The primary objective is to evaluate how effectively various pre-trained models can predict coffee types using advanced deep learning techniques. The selection of an optimal pre-trained model is crucial, given the growing popularity of specialty coffee and the necessity for precise classification. We conducted a comprehensive comparison of several pre-trained models, including AlexNet, LeNet, HRNet, Google Net, Mobile V2 Net, ResNet (50), VGG, Efficient, Darknet, and DenseNet, utilizing a coffee-type dataset. By leveraging transfer learning and fine-tuning, we assess the generalization capabilities of the models for the coffee classification task. Our findings emphasize the substantial impact of the pre-trained model choice on the model's performance, with certain models demonstrating higher accuracy and faster convergence than conventional alternatives. This study offers a thorough evaluation of pre-trained architectural models regarding their effectiveness in coffee classification. Through the evaluation of result metrics, including sensitivity (1.0000), specificity (0.9917), precision (0.9924), negative predictive value (1.0000), accuracy (1.0000), and F1 score (0.9962), our analysis provides nuanced insights into the intricate landscape of pre-trained models.

Ensemble learning-based approach for improving generalization capability of machine reading comprehension systems

Concept mapping is a knowledge representation method used to represent and understand concepts, entities, and the relationships between them, which are referred to as hyponymy–hypernymy semantic relations. These relations are primarily used to describe the hierarchical and categorical relationships between different concepts or entities. The detection of hyponymy–hypernymy semantic relations is an important task in the field of natural language processing, crucial to many downstream tasks such as information extraction, automatic reasoning, and personalized recommendations. These tasks often require understanding the semantic relations between concepts or entities in text for more accurate analysis and reasoning. Currently, algorithms for identifying and detecting hyponymy–hypernymy semantic relations face two main challenges: first, candidate hyponymy–hypernymy relation tuples do not exist in the same contextual sentence, failing to meet the co-occurrence requirement; second, distributed algorithms have issues with lexical memory. To address these issues, this paper proposes an improved algorithm for detecting hyponymy–hypernymy relations based on heterogeneous graph neural networks, aiming to detect hyponymy–hypernymy relations in various candidate word sets and then construct a hierarchical system. To meet the co-occurrence requirement of candidate word pairs, an open-source large model is utilized to generate contextual sentences for the candidate pairs. Sub-word features are adopted to capture the intrinsic semantic connections between nested phrases, thus alleviating the issue of reversed predictions for nested phrases. The representation of relation nodes is modified by encoding relation definitions through a pre-trained model, enabling the model to understand the semantic relations between concept nodes. To address the problem of overfitting in traditional graph attention networks, the calculation order of adjacency node aggregation is changed in the heterogeneous graph to capture dynamic attention features. In addition, a pipeline for hierarchical system construction is designed and implemented, combining the divide-and-conquer approach with loop detection algorithms. Compared to baseline metrics, the proposed method achieves a 4.14% improvement in accuracy and a 0.62% increase in F1 score on the EVALution dataset; a 4.89% increase in accuracy and a 0.71% improvement in F1 score on the Bansal dataset; and a 1.05% increase in accuracy, a 2.21% increase in recall, and a 1.79% improvement in F1 score on a self-annotated Chinese dictionary dataset.

Recent advances in neural architectures, such as the Transformer, coupled with the emergence of large-scale pre-trained models such as BERT, have revolutionized the field of Natural Language Processing (NLP), pushing the state of the art for a number of NLP tasks. A rich family of variations of these models has been proposed, such as RoBERTa, ALBERT, and XLNet, but fundamentally, they all remain limited in their ability to model certain kinds of information, and they cannot cope with certain information sources, which was easy for pre-existing models. Thus, here we aim to shed light on some important theoretical limitations of pre-trained BERT-style models that are inherent in the general Transformer architecture. First, we demonstrate in practice on two general types of tasks—segmentation and segment labeling—and on four datasets that these limitations are indeed harmful and that addressing them, even in some very simple and naive ways, can yield sizable improvements over vanilla RoBERTa and XLNet models. Then, we offer a more general discussion on desiderata for future additions to the Transformer architecture that would increase its expressiveness, which we hope could help in the design of the next generation of deep NLP architectures.

The 21st century is presenting humankind with unprecedented environmental and medical challenges. The ability to design novel proteins tailored for specific purposes could transform our ability to respond timely to these issues. Recent advances in the field of artificial intelligence are now setting the stage to make this goal achievable. Protein sequences are inherently similar to natural languages: Amino acids arrange in a multitude of combinations to form structures that carry function, the same way as letters form words and sentences that carry meaning. Therefore, it is not surprising that throughout the history of Natural Language Processing (NLP), many of its techniques have been applied to protein research problems. In the last few years, we have witnessed revolutionary breakthroughs in the field of NLP. The implementation of Transformer pre-trained models has enabled text generation with human-like capabilities, including texts with specific properties such as style or subject. Motivated by its considerable success in NLP tasks, we expect dedicated Transformers to dominate custom protein sequence generation in the near future. Finetuning pre-trained models on protein families will enable the extension of their repertoires with novel sequences that could be highly divergent but still potentially functional. The combination of control tags such as cellular compartment or function will further enable the controllable design of novel protein functions. Moreover, recent model interpretability methods will allow us to open the 'black box' and thus enhance our understanding of folding principles. While early initiatives show the enormous potential of generative language models to design functional sequences, the field is still in its infancy. We believe that protein language models are a promising and largely unexplored field and discuss their foreseeable impact on protein design.

BERT is a pre-trained language model that achieves state-of-the-art performance on natural language processing (NLP) tasks. Once it was published, it quickly became one of the most popular models in the NLP field. The official recommended method for applying BERT to downstream tasks is fine-tuning. However, we argue that transfer learning is also a very practical approach to applying BERT, and this approach has its own advantages compared to fine-tuning. In this paper, we explore the advantages of the transfer learning approach through the method that uses transfer learning and fine-tuning approaches separately to apply BERT to the same eight GLUE benchmark tasks and compare the training time spent and model performance scores, such as accuracy or F1 score, of the two approaches. Finally, this paper finds that among all eight GLUE tasks and on small training sets, transfer learning saves 30% to 50% of the time compared to a fine-tuning approach to train the models on the data of downstream tasks and can achieve very similar performance. Besides, for the same amount of time, transfer learning can obtain higher performance scores than fine-tuning on larger training sets. In conclusion, on small training sets, transfer learning has a huge advantage in time consumption under the premise of approximate performance, and it also performs better on large training sets under the same length of time, compared to fine-tuning.

WhisPAr: Transferring pre-trained audio models to fine-grained classification via Prompt and Adapter

In the field of natural language processing, pre-training can effectively improve the performance of downstream tasks. In recent years, pre-training has been continuously developed in Tibetan NLP. We built three pre-trained models of Tibetan Word2Vec, Tibetan ELMo, and Tibetan ALBERT, and applied them to the two downstream tasks of Tibetan text classification and Tibetan part-of-speech tagging. Comparing them with the baseline models of these two downstream tasks, it is found that the performance of the downstream tasks using the pre-training is significantly better than the baseline model. The three pre-trained models have also brought a gradual improvement in performance for Tibetan downstream tasks.

Natural language processing (NLP) is a popular technology after the rise of big data and machine learning in recent years. With the development of deep learning, the field of natural language processing has also undergone a landmark transformation, including the emergence of the BERT large-scale language training model. The emergence of this model makes text mining a qualitative leap, meets more practical needs, and solves the related problems of feature vectorization of unstructured data. This article will sort out the connotation, task application, and main optimization and improvement methods of the BERT pre-training model released by Google, and provide a reference for subsequent related research and development based on BERT.

Abstractive single document summarization is considered as a challenging problem in the field of artificial intelligence and natural language processing. Meanwhile and specifically in the last two years, several deep learning summarization approaches were proposed that once again attracted the attention of researchers to this field.It is a well-known issue that deep learning approaches do not work well with small amounts of data. With some exceptions, this is, unfortunately, the case for most of the datasets available for the summarization task. Besides this problem, it should be considered that phonetic, morphological, semantic and syntactic features of the language are constantly changing over the time and unfortunately most of the summarization corpora are constructed from old resources. Another problem is the language of the corpora. Not only in the summarization field, but also in other fields of natural language processing, most of the corpora are only available in English. In addition to the above problems, license terms, and fees of the corpora are obstacles that prevent many academics and specifically non-academics from accessing these data.This work describes an open source framework to create an extendable multilingual corpus for abstractive single document summarization that addresses the above-mentioned problems. We describe a tool consisted of a scalable crawler and a centralized key-value store database to construct a corpus of an arbitrary size using a news aggregator service.

The availability of different pre-trained semantic models has enabled the quick development of machine learning components for downstream applications. However, even if texts are abundant for low-resource languages, there are very few semantic models publicly available. Most of the publicly available pre-trained models are usually built as a multilingual version of semantic models that will not fit well with the need for low-resource languages. We introduce different semantic models for Amharic, a morphologically complex Ethio-Semitic language. After we investigate the publicly available pre-trained semantic models, we fine-tune two pre-trained models and train seven new different models. The models include Word2Vec embeddings, distributional thesaurus (DT), BERT-like contextual embeddings, and DT embeddings obtained via network embedding algorithms. Moreover, we employ these models for different NLP tasks and study their impact. We find that newly-trained models perform better than pre-trained multilingual models. Furthermore, models based on contextual embeddings from FLAIR and RoBERTa perform better than word2Vec models for the NER and POS tagging tasks. DT-based network embeddings are suitable for the sentiment classification task. We publicly release all the semantic models, machine learning components, and several benchmark datasets such as NER, POS tagging, sentiment classification, as well as Amharic versions of WordSim353 and SimLex999.

The rising complexity of cyberattacks poses significant challenges to traditional intrusion detection system (IDS). Approaches that rely mainly on feature engineering and rule matching often fail to address new and complex attack patterns effectively. To address these challenges, this paper proposes a network IDS based on bidirectional encoder representations from transformers (BERT), which aims to leverage recent advances in the field of natural language processing (NLP) to enhance the performance of IDS. The core of our approach lies in converting network traffic data into textual representations that can be processed by applying advanced NLP techniques. Specifically, we first convert various types of information in network traffic into a natural-language-like sequence representation, which allows us to utilize a pre-trained BERT model for feature extraction. The BERT model, with its bidirectional encoder structure, captures complex contextual information in sequences, producing high-quality feature representations. After feature extraction is completed, we further perform classification using a classifier based on a bidirectional gated recurrent unit (GRU) network, which is able to efficiently process sequence data and reduce computational complexity while capturing temporal dependencies in the sequences. By training the classifier on the basis of the extracted features, we are able to achieve an accurate distinction between normal and abnormal traffic. To validate the effectiveness of our approach, we conducted experiments on several public datasets, and the experimental results show that the BERT-based network IDS significantly outperforms traditional machine learning (ML) and deep learning (DL) methods in terms of detection performance. The BERT model enhances detection accuracy and effectively captures complex attack patterns, compared to traditional methods. Our research not only demonstrates the great potential of NLP technology in the field of network security but also provides a new research direction for cross-domain applications. By treating network traffic as a “language,” our approach resolves issues that traditional methods struggle with while being scalable and adaptable to various network environments and attack patterns.

The pre-training and fine-tuning paradigm has demonstrated its effectiveness and has become the standard approach for tailoring language models to various tasks. Currently, community-based platforms offer easy access to various pre-trained models, as anyone can publish without strict validation processes. However, a released pre-trained model can be a privacy trap for fine-tuning datasets if it is carefully designed. In this work, we propose PreCurious framework to reveal the new attack surface where the attacker releases the pre-trained model and gets a black-box access to the final fine-tuned model. PreCurious aims to escalate the general privacy risk of both membership inference and data extraction on the fine-tuning dataset. The key intuition behind PreCurious is to manipulate the memorization stage of the pre-trained model and guide fine-tuning with a seemingly legitimate configuration. While empirical and theoretical evidence suggests that parameter-efficient and differentially private fine-tuning techniques can defend against privacy attacks on a fine-tuned model, PreCurious demonstrates the possibility of breaking up this invulnerability in a stealthy manner compared to fine-tuning on a benign pre-trained model. While DP provides some mitigation for membership inference attack, by further leveraging a sanitized dataset, PreCurious demonstrates potential vulnerabilities for targeted data extraction even under differentially private tuning with a strict privacy budget e.g. . Thus, PreCurious raises warnings for users on the potential risks of downloading pre-trained models from unknown sources, relying solely on tutorials or common-sense defenses, and releasing sanitized datasets even after perfect scrubbing.