Transfer Learning in Natural Language Processing (NLP)

TERFDA: Tensor Embedding RF Domain Adaptation for varying noise interference

Active Learning is a machine learning and data mining technique that selects the most informative samples for labeling and uses them as training data; it is especially useful when there are large amount of unlabeled data and labeling them is expensive. Recently, batch-mode active learning, where a set of samples are selected concurrently for labeling, based on their collective merit, has attracted a lot of attention. The objective of batch-mode active learning is to select a set of informative samples so that a classifier learned on these samples has good generalization performance on the unlabeled data. Most of the existing batch-mode active learning methodologies try to achieve this by selecting samples based on certain criteria. In this article we propose a novel criterion which achieves good generalization performance of a classifier by specifically selecting a set of query samples that minimize the difference in distribution between the labeled and the unlabeled data, after annotation. We explicitly measure this difference based on all candidate subsets of the unlabeled data and select the best subset. The proposed objective is an NP-hard integer programming optimization problem. We provide two optimization techniques to solve this problem. In the first one, the problem is transformed into a convex quadratic programming problem and in the second method the problem is transformed into a linear programming problem. Our empirical studies using publicly available UCI datasets and two biomedical image databases demonstrate the effectiveness of the proposed approach in comparison with the state-of-the-art batch-mode active learning methods. We also present two extensions of the proposed approach, which incorporate uncertainty of the predicted labels of the unlabeled data and transfer learning in the proposed formulation. In addition, we present a joint optimization framework for performing both transfer and active learning simultaneously unlike the existing approaches of learning in two separate stages, that is, typically, transfer learning followed by active learning. We specifically minimize a common objective of reducing distribution difference between the domain adapted source, the queried and labeled samples and the rest of the unlabeled target domain data. Our empirical studies on two biomedical image databases and on a publicly available 20 Newsgroups dataset show that incorporation of uncertainty information and transfer learning further improves the performance of the proposed active learning based classifier. Our empirical studies also show that the proposed transfer-active method based on the joint optimization framework performs significantly better than a framework which implements transfer and active learning in two separate stages.

Document classification is a challenging task to the data being high-dimensional and sparse. Many transfer learning methods have been investigated for improving the classification performance by effectively transferring knowledge from a source domain to a target domain, which is similar to but different from the source domain. However, most of the existing methods cannot handle the case that the training data of the target domain does not have labels. In this study, we propose a transductive transfer learning system, utilizing solutions evolved by genetic programming (GP) on a source domain to automatically pseudolabel the training data in the target domain in order to train classifiers. Different from many other transfer learning techniques, the proposed system pseudolabels target-domain training data to retrains classifiers using all target-domain features. The proposed method is examined on nine transfer learning tasks, and the results show that the proposed transductive GP system has better prediction accuracy on the test data in the target domain than existing transfer learning approaches including subspace alignment-domain adaptation methods, feature-level-domain adaptation methods, and one latest pseudolabeling strategy-based method.

Named Entity Recognition (NER) is a fundamental task in information extraction from unstructured text. Most previous machine-learning-based NER systems are domain-specific, which implies that they may only perform well on some specific domains (e.g., Newswire ) but tend to adapt poorly to other related but different domains (e.g., Weblog ). Recently, transfer learning techniques have been proposed to NER. However, most transfer learning approaches to NER are developed for binary classification, while NER is a multiclass classification problem in nature. Therefore, one has to first reduce the NER task to multiple binary classification tasks and solve them independently. In this article, we propose a new transfer learning method, named Transfer Joint Embedding (TJE), for cross-domain multiclass classification, which can fully exploit the relationships between classes (labels), and reduce domain difference in data distributions for transfer learning. More specifically, we aim to embed both labels (outputs) and high-dimensional features (inputs) from different domains (e.g., a source domain and a target domain) into a unified low-dimensional latent space, where 1) each label is represented by a prototype and the intrinsic relationships between labels can be measured by Euclidean distance; 2) the distance in data distributions between the source and target domains can be reduced; 3) the source domain labeled data are closer to their corresponding label-prototypes than others. After the latent space is learned, classification on the target domain data can be done with the simple nearest neighbor rule in the latent space. Furthermore, in order to scale up TJE, we propose an efficient algorithm based on stochastic gradient descent (SGD). Finally, we apply the proposed TJE method for NER across different domains on the ACE 2005 dataset, which is a benchmark in Natural Language Processing (NLP). Experimental results demonstrate the effectiveness of TJE and show that TJE can outperform state-of-the-art transfer learning approaches to NER.

Language model (LM) has become a common method of transfer learning in Natural Language Processing (NLP) tasks when working with small labeled datasets. An LM is pretrained using an easily available large unlabelled text corpus and is fine-tuned with the labelled data to apply to the target (i.e., downstream) task. As an LM is designed to capture the linguistic aspects of semantics, it can be biased to linguistic features. We argue that exposing an LM model during fine-tuning to instances that capture diverse semantic aspects (e.g., topical, linguistic, semantic relations) present in the dataset will improve its performance on the underlying task. We propose a Mixed Aspect Sampling (MAS) framework to sample instances that capture different semantic aspects of the dataset and use the ensemble classifier to improve the classification performance. Experimental results show that MAS performs better than random sampling as well as the state-of-the-art active learning models to abuse detection tasks where it is hard to collect the labelled data for building an accurate classifier.

Deep learning has emerged as a powerful tool in various domains, revolutionising machine learning research. However, one persistent challenge is the scarcity of labelled training data, which hampers the performance and generalisation of deep learning models. To address this limitation, researchers have developed innovative methods to overcome data scarcity and enhance deep model learning capabilities. Two prevalent techniques that have gained significant attention are transfer learning and self-supervised learning. Transfer learning leverages knowledge learned from pre-training on a large-scale dataset, such as ImageNet, and applies it to a target task with limited labelled data. This approach allows models to benefit from the learned representations and effectively transfer knowledge to new tasks, resulting in improved learning performance and generalisation. On the other hand, self-supervised learning focuses on training models using pretext tasks that do not require manual annotation, allowing them to learn valuable representations from large amounts of unlabelled data. These learned representations can then be fine-tuned for downstream tasks, mitigating the need for extensive labelled data. In recent years, transfer and self-supervised learning have found applications in various fields, including medical image processing, video recognition, and natural language processing. These approaches have demonstrated remarkable achievements, enabling breakthroughs in areas such as disease diagnosis, object recognition, and language understanding. However, while these methods offer numerous advantages, they also have limitations. For example, transfer learning may face domain mismatch issues between the pre-training and target domains, while self-supervised learning requires careful design of pretext tasks to ensure meaningful representations. This review paper explores the recent applications of these pre-training methods in various fields within the past three years. It delves into the advantages and limitations of each approach, assesses the performance of models employing these techniques, and identifies potential directions for future research. By providing a comprehensive review of current pre-training methods, this article offers guidance for selecting the best technique for specific deep learning applications to address the data scarcity issue.

Machine learning techniques have long been the foundations of speech processing. Bayesian classiflcation, decision trees, unsupervised clustering, the EM algorithm, maximum entropy, etc. are all part of existing speech recognition systems. The success of statistical speech recognition has led to the rise of statistical and empirical methods in natural language processing. Indeed, many of the machine learning techniques used in language processing, from statistical part-of-speech tagging to the noisy channel model for machine translation have roots in work conducted in the speech fleld. However, advances in learning theory and algorithmic machine learning approaches in recent years have led to signiflcant changes in the direction and emphasis of the statistical and learning centered research in natural language processing and made a mark on natural language and speech processing. Approaches such as memory based learning, a range of linear classiflers such as Boosting, SVMs and SNoW and others have been successfully applied to a broad range of natural language problems, and these now inspire new research in speech retrieval and recognition. We have seen an increasingly close collaboration between voice and language processing researchers in some of the shared tasks such as spontaneous speech recognition and understanding, voice data information extraction, and machine translation. The purpose of this special issue was to invite speech and language researchers to communicate with each other, and with the machine learning community on the latest machine learning advances in their work. The call for papers was met with great enthusiasm from the speech and natural language community. Thirty six submissions were received; each paper was reviewed by at least three reviewers. Only ten papers were selected re∞ecting not only some of the best work on machine learning in the areas of natural language and spoken language processing but also what we view as a collection of papers that represent current trends in these areas of research both from the perspective of

<p dir="ltr">Recent advancements in Natural Language Processing (NLP), specifically Large Language Models (LLMs), have demonstrated their transformative power in various domains, including software engineering. This thesis focuses on addressing the critical problem of conflict and duplicate requirement identification in requirement engineering, leveraging the potential of structured and non-structured data in software engineering for automation. For this purpose, first, a diverse set of requirements are annotated for entity recognition tasks, enabling the creation of machine learning and transformer-based models which provides 95.8% accuracy in identifying software-specific entities from requirement texts present in DOORS dataset. Second, building upon this foundation, a two-phase algorithm is developed for conflict detection in requirement documents. The algorithm utilizes supervised learning on requirement data to identify potential conflicts and subsequently validates them semantically using entity extraction techniques. Third, to further refine the conflict and duplicate identification process, the problem is formulated as a sentence pair classification task. Transfer learning techniques, such as sequential transfer learning and cross-domain transfer learning, are employed, and a novel architecture called Software-Requirement Bi-directional Encoder Representation Transformer (SR-BERT) is proposed for requirement pair classification. SR-BERT achieves 95.3% F1-score in determining conflicts, duplicate, and neutral requirement pairs. Additionally, misclassifications flagged by cross-domain transfer learning are reevaluated using Actor-Action (AA) extraction, Part of Speech (POS) tagging, and Semantic Role Labeling (SRL) techniques. The requirement pair classification approach demonstrates great potential in conflict detection tasks, and its performance is enhanced by integrating bi-directional encoders with a contrastive learning framework and cross-encoders. Fourth, inspired by recent advancements in prompt-based learning, a Prompt-based Text-to-Text Models for Software Requirement Pairs (PT2-SRP) framework is also developed, leveraging specially crafted prompts for requirement pair datasets and training encoder-decoder models to identify conflicts in requirement pairs. Lastly, the impact of Data Augmentation (DA) on requirement conflict identification is examined. Specif ically, during the curation of datasets for requirement pair classification task, a scarcity of conflict requirement pairs is noted. To overcome this limitation, novel data augmentation techniques are devised, which ensure the generation of meaningful data instances while preserving the class labels in requirement pair datasets.</p>

Over the years, drone usage have become an increasing part of the ever-connected society that we are currently living in. Its usages have proliferated beyond the military sector to various commercial and consumer activities such as package delivery, disaster relief, agriculture and filming. With the rise of Wi-Fi controlled drone. Wi-Fi controlled drone has increased its popularity for personal use due to its affordability, and the ease of operating the drone through smart-devices like mobile phone, tablets and computers. As such, this increases the likelihood of drone presence in various environments, especially in critical government infrastructure, leading to various privacy and security concern by the authorities and the public with malicious intent. Therefore, various signature-based methodology of drone detection has emerged such as the visual and Radio Frequency (RF) signature-based detection method. Visual signature-based detection relies on camera capture and image processing but this is an expensive approach. Whereas, RF signature- based detection relies on the identification of the emission of RF signal by the drone. However, since most commercial electronics devices were built based on Wi-Fi technology, the differentiation of the RF signals transmitted between a drone or a standard Wi-Fi device in a crowded Wi-Fi environment such as a school campus or city area is an challenging task. In this paper, we propose a novel machine learning approach that leverages on the identified unique signatures of Wi-Fi devices in terms of Radio Frequency (RF) and network packets measurement to differentiate the presence of Wi-Fi drone and standard Wi-Fi devices in an urban setting. Furthermore, we also carried out a meticulous pre-processing procedure and a better training scheme of using Stratified K-Fold Cross-Validation (SKFCV), to enhance the richness in the data signature and fully exploit the permutation of the data during training respectively for better performance of the ML models. Two supervised classification Machine Learning (ML) models, namely the Logistic Regression (LR), and Artificial Neural Network (ANN) were applied using the joint data measurements to identify the presence of drone in dense Wi-Fi environment. The experimental results have shown that the proposed novel ML approach of using both RF and network measurement signatures coupled with the pre-processing and training methodology on LR and ANN ML models have outperformed the traditional RF signature-based drone detection ML accuracy results by 15.1% and 21.63% respectively in a crowded Wi-Fi environment.

Currently, numerous machine learning (ML) techniques are being applied in the field of renewable energy (RE). These techniques may not perform well if they do not have enough training data. Additionally, the main assumption in most of the ML algorithms is that the training and testing data are from the same feature space and have similar distributions. However, in many practical applications, this assumption is false. Recently, transfer learning (TL) has been introduced as a promising machine-learning framework to mitigate these issues by preparing extra-domain data so that knowledge may be transferred across domains. This learning technique improves performance and avoids the resource expensive collection and labeling of domain-centric datasets; furthermore, it saves computing resources that are needed for re-training new ML models from scratch. Lately, TL has drawn the attention of researchers in the field of RE in terms of forecasting and fault diagnosis tasks. Owing to the rapid progress of this technique, a comprehensive survey of the related advances in RE is needed to show the critical issues that have been solved and the challenges that remain unsolved. To the best of our knowledge, few or no comprehensive surveys have reviewed the applications of TL in the RE field, especially those pertaining to forecasting solar and wind power, load forecasting, and predicting failures in power systems. This survey fills this gap in RE classification and forecasting problems, and helps researchers and practitioners better understand the state of the art technology in the field while identifying areas for more focused study. In addition, this survey identifies the main issues and challenges of using TL for REs, and concludes with a discussion of future perspectives.

There is a huge necessity for tools that implement accessibility in Software Engineering (SE) education. The use of diagrams to teach software development is a very common practice, and there are a lot of UML diagrams represented as images in didactic materials that need an accessible version for visually impaired or blind students. Machine learning techniques, such as deep learning, can be used to automate this task. The practical application of deep learning in many classification problems in the context of SE is problematic due to the large volumes of labeled data required for training. Transfer learning techniques can help in this type of task by taking advantage of pre-trained models based on Convolutional Neural Networks (CNN), so that better results may be achieved even with few images. In this work, we applied transfer learning and data augmentation for UML diagrams classification on a dataset specially created for the development of this work, containing six types of UML diagrams. The dataset was also made available as a contribution of this work. We experimented three widely-known CNN architectures: VGG16, RestNet50, and InceptionV3. The results demonstrated that the use of transfer learning contributes for achieving good results even using scarce data. However, there is still a room for improvement regarding the successful classification of the UML diagrams addressed in this work.

Many existing data mining and machine learning techniques are based on the assumption that training and test data fit the same distribution. This assumption does not hold, however, as in many cases of Web mining and wireless computing when labeled data becomes outdated or test data are from a different domain with training data. In these cases, most machine learning methods would fail in correctly classifying new and future data. It would be very costly and infeasible to collect and label enough new training data. Instead, we would like to recoup as much useful knowledge as possible from the old data. This problem is known as transfer learning. In this talk, I will give an overview of the transfer learning problem, present a number of important directions in this research, and discuss our own novel solutions to this problem.

A city-wide experimental testbed for the next generation wireless networks

<p class="VARAbstract">The growing availability of three-dimensional (3D) data, such as point clouds, coming from Light Detection and Ranging (LiDAR), Mobile Mapping Systems (MMSs) or Unmanned Aerial Vehicles (UAVs), provides the opportunity to rapidly generate 3D models to support the restoration, conservation, and safeguarding activities of cultural heritage (CH). The so-called scan-to-BIM process can, in fact, benefit from such data, and they can themselves be a source for further analyses or activities on the archaeological and built heritage. There are several ways to exploit this type of data, such as Historic Building Information Modelling (HBIM), mesh creation, rasterisation, classification, and semantic segmentation. The latter, referring to point clouds, is a trending topic not only in the CH domain but also in other fields like autonomous navigation, medicine or retail. Precisely in these sectors, the task of semantic segmentation has been mainly exploited and developed with artificial intelligence techniques. In particular, machine learning (ML) algorithms, and their deep learning (DL) subset, are increasingly applied and have established a solid state-of-the-art in the last half-decade. However, applications of DL techniques on heritage point clouds are still scarce; therefore, we propose to tackle this framework within the built heritage field. Starting from some previous tests with the Dynamic Graph Convolutional Neural Network (DGCNN), in this contribution close attention is paid to: i) the investigation of fine-tuned models, used as a transfer learning technique, ii) the combination of external classifiers, such as Random Forest (RF), with the artificial neural network, and iii) the evaluation of the data augmentation results for the domain-specific ArCH dataset. Finally, after taking into account the main advantages and criticalities, considerations are made on the possibility to profit by this methodology also for non-programming or domain experts.</p><p>Highlights:</p><ul><li><p>Semantic segmentation of built heritage point clouds through deep neural networks can provide performances comparable to those of more consolidated state-of-the-art ML classifiers.</p></li><li><p>Transfer learning approaches, as fine-tuning, can considerably reduce computational time also for CH domain-specific datasets, as well as improve metrics for some challenging categories (i.e. windows or mouldings).</p></li><li><p>Data augmentation techniques do not significantly improve overall performances.</p></li></ul>

Due to the latest advancements in monitoring technologies, interest in the possibility of early-detection of quality issues in components has grown considerably in the manufacturing industry. However, implementation of such techniques has been limited outside of the research environment due to the more demanding scenarios posed by production environments. This paper proposes a method of assessing the health of a machining process and the machine tool itself by applying a range of machine learning (ML) techniques to sensor data. The aim of this work is not to provide complete diagnosis of a condition, but to provide a rapid indication that the machine tool or process has changed beyond acceptable limits; making for a more realistic solution for production environments. Prior research by the authors found good visibility of simulated failure modes in a number of machining operations and machine tool fingerprint routines, through the defined sensor suite. The current research set out to utilise this system, and streamline the test procedure to obtain a large dataset to test ML techniques upon. Various supervised and unsupervised ML techniques were implemented using a range of features extracted from the raw sensor signals, principal component analysis and continuous wavelet transform. The latter were classified using convolutional neural networks (CNN); both custom-made networks, and pre-trained networks through transfer learning. The detection and classification accuracies of the simulated failure modes across all classical ML and CNN techniques tested were promising, with all approaching 100% under certain conditions.