Learning to Generate Examples for Semantic Processing Tasks

Based on recent advances in natural language modeling and those in text generation capabilities, we propose a novel data augmentation method for text classification tasks. We use a powerful pre-trained neural network model to artificially synthesize new labeled data for supervised learning. We mainly focus on cases with scarce labeled data. Our method, referred to as language-model-based data augmentation (LAMBADA), involves fine-tuning a state-of-the-art language generator to a specific task through an initial training phase on the existing (usually small) labeled data. Using the fine-tuned model and given a class label, new sentences for the class are generated. Our process then filters these new sentences by using a classifier trained on the original data. In a series of experiments, we show that LAMBADA improves classifiers' performance on a variety of datasets. Moreover, LAMBADA significantly improves upon the state-of-the-art techniques for data augmentation, specifically those applicable to text classification tasks with little data.

Knowledge enhanced pre-trained language models (K-PLMs) are shown to be effective for many public tasks in the literature, but few of them have been successfully applied in practice. To address this problem, we propose K-AID, a systematic approach that includes a low-cost knowledge acquisition process for acquiring domain knowledge, an effective knowledge infusion module for improving model performance, and a knowledge distillation component for reducing the model size and deploying K-PLMs on resource-restricted devices (e.g., CPU) for real-world application. Importantly, instead of capturing entity knowledge like the majority of existing K-PLMs, our approach captures relational knowledge, which contributes to better improving sentence-level text classification and text matching tasks that play a key role in question answering (QA). We conducted a set of experiments on five text classification tasks and three text matching tasks from three domains, namely E-commerce, Government, and Film&TV, and performed online A/B tests in E-commerce. Experimental results show that our approach is able to achieve substantial improvement on sentence-level question answering tasks and bring beneficial business value in industrial settings.

Labelling multiple training examples for text classification models is usually time‐consuming and complex. Data augmentation can be used to automatically expand the dataset by transforming the original data. However, it may cause semantic changes without modifying the labels, which reduces the effectiveness of the classifiers. In this paper, we propose a data‐augmentation method called the virtual word insertion technique, which generates new sentences by randomly inserting virtual words into existing sentences. Two methods are used to achieve virtual word embedding: unweighted average and weighted average. Furthermore, a new concept of weight is proposed: the class deviation factor, which demonstrates the correlation between words and classes. Based on this new concept, different weights are assigned to words of different classes. Experiments are conducted on five different classification tasks. Ablation experiments are also performed to explore the effects of random operation and number of augmented sentences for classification results. The results of these experiments show that our method improves the classification performance and outperforms two other contrasting data‐augmentation methods in automatically augmenting the dataset. Compared to raw datasets, the average accuracy improvement of our method is 3.5% for a small‐scale dataset and 1% for a large‐scale dataset.

Improving the prediction of extreme wind speed events with generative data augmentation techniques

How to use advanced deep learning technology to build an effective and powerful text classification model, extract text semantic attributes and achieve good classification results on large-scale test datasets has always been the main problem and severe challenge of text classification tasks. The purpose of this paper is to study the application optimization of NLP system in text semantics and text classification based on deep learning technology. A WC-GCN classification algorithm combining GCN and Attention mechanism is proposed. The related tasks of text classification are discussed, including three typical text classification tasks, including sentiment analysis, topic analysis, and news classification. The algorithm model proposed in this paper is verified from the experimental point of view. Compared with other deep learning-based text classification methods, the proposed algorithm model performs better on the public datasets of three text classification tasks. The news text also achieved a high accuracy rate.

This paper presents an Attention-based Data Augmentation (ADA) approach that extracts keywords from minority class data points using a vector similarity-based mechanism, uses the extracted keywords to extract significant contextual words from minority class documents using an attention mechanism, and uses the significant contextual words to enrich the minority class dataset. By creating new documents based on significant contextual words and adding them to the minority class dataset, we oversample the dataset for the minority class. On the classification job, we compare the original and oversampled versions of the datasets. We also compare ADA over the augmented datasets with two popular state-of-the-art text data augmentation methods. According to the experimental findings, classification algorithms perform better when used to augmented datasets produced by any data augmentation technique than when applied to the datasets’ original versions. Additionally, the classifiers trained over the augmented datasets generated by ADA are more effective than those generated by state-of-the-art data augmentation techniques.

Most Facial Expression Recognition (FER) systems rely on machine learning approaches that require large databases for an effective training. As these are not easily available, a good solution is to augment the databases with appropriate data augmentation (DA) techniques, which are typically based on either geometric transformation or oversampling augmentations (e.g., generative adversarial networks (GANs)). However, it is not always easy to understand which DA technique may be more convenient for FER systems because most state-of-the-art experiments use different settings which makes the impact of DA techniques not comparable. To advance in this respect, in this paper, we evaluate and compare the impact of using well-established DA techniques on the emotion recognition accuracy of a FER system based on the well-known VGG16 convolutional neural network (CNN). In particular, we consider both geometric transformations and GAN to increase the amount of training images. We performed cross-database evaluations: training with the "augmented" KDEF database and testing with two different databases (CK+ and ExpW). The best results were obtained combining horizontal reflection, translation and GAN, bringing an accuracy increase of approximately 30%. This outperforms alternative approaches, except for the one technique that could however rely on a quite bigger database.

In recent years, contrastive learning has been very successful in unsupervised tasks of representation learning and has received a lot of attention in supervised tasks. In supervised tasks, the discrete nature of natural language makes the construction of sample pairs difficult and the models are poorly robust to adversarial samples, so it remains a challenge to make contrastive learning effective for text classification tasks and to guarantee the robustness of the models. This paper presents a contrastive adversarial learning framework built using data augmentation with labeled insertion data. Specifically,By adding perturbation to the word-embedding matrix, adversarial samples are generated as positive examples of contrastive learning, and external semantic information is introduced to construct negative examples. Contrastive learning is used to improve the sensitivity and generalization ability of the model, and adversarial training is used to improve robustness, thereby improving the classification accuracy. In addition, the momentum contrast from unsupervised tasks is also introduced into the text classification task to increase the number of sample pairs. Experimental results on several datasets show that the proposed approach outperforms the baseline comparison approach, and in addition some experiments are conducted to verify the effectiveness of the proposed framework under low-resource conditions.

In recent years, with the rapid development of large language models, optimizing BERT's performance on small-scale datasets using large language models has gradually become a research hotspot. To this end, this paper proposes a data augmentation method based on large language models to enhance BERT's performance in text classification tasks. Specifically, we first use large language models to back-translate the training data. By translating the text into other languages and then back into the original language, we generate new samples that are semantically consistent but have diverse expressions, thereby increasing the diversity of the training data. Subsequently, the augmented training set is used to train the BERT model, which significantly improves classification accuracy on the Reuters News Classification dataset. Experimental results show that this method effectively mitigates the limitations of small-scale datasets and significantly enhances the model's generalization ability, providing a novel and efficient solution for text classification tasks.

Recent deep neural network models trained on smaller and less diverse datasets use data augmentation to alleviate limitations such as overfitting, reduced robustness, and lower generalization. Methods using 3D datasets are among the most common to use data augmentation techniques such as random point drop, scaling, translation, rotations, and jittering. However, these data augmentation techniques are fixed and are often applied to the entire object, ignoring the object’s local geometry. Different local neighborhoods on the object surface hold a different amount of geometric complexity. Applying the same data augmentation techniques at the object level is less effective in augmenting local neighborhoods with complex structures. This paper presents PatchAugment, a data augmentation framework to apply different augmentation techniques to the local neighborhoods. Our experimental studies on PointNet++ and DGCNN models demonstrate the effectiveness of PatchAugment on the task of 3D Point Cloud Classification. We evaluated our technique against these models using four benchmark datasets, ModelNet40 (synthetic), ModelNetlO (synthetic), SHREC’16 (synthetic) and ScanObjectNN (real-world).

One of the common challenges in medical artificial intelligence (AI) applications using echocardiography is the lack of image data harmonization. This study aims to improve the prediction accuracy of left ventricular ejection fraction (LVEF) AI models by incorporating data augmentation (DA) techniques to address the variability in image data across different vendor machines. A database comprising 15,770 echocardiographic videos from 3154 patients across five different centers was utilized, with the data acquired using various vendor machines. We prepared datasets specific to GE healthcare (GE), philips (PH), and canon medical systems (CA) vendors, including 1911, 804, and 427 cohorts, respectively. A three-dimensional convolutional neural network (3D-CNN) was trained to predict LVEF, using videos consisting of 20 images per heartbeat as input, with training performed exclusively on data from GE machines (GE-based model). DA techniques, including gamma correction, scaling, median filtering, unsharp masking, translation, rotation, noise correction, and image conversion using a cycle generative adversarial network, were applied. A regression analysis using five different chamber views was performed on the GE test data. The accuracy of LVEF prediction using the GE-based model with DA—specifically gamma correction, scaling, and translation corrections—was excellent. The mean absolute error (root mean square error) in test cohorts was 4.33 (5.58) for GE, 4.42 (5.57) for PH, and 4.89 (6.57) for CA, which were comparable to the results of a model developed using data from all vendors, demonstrating the effectiveness of DA in harmonizing images. This study demonstrates that echocardiographic videos are transferable across vendors and that DA is highly effective in improving LVEF predictions from data acquired using different vendor machines.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-21277-y.

Text classification is one of the fundamental tasks in natural language processing. Recently, deep neural networks have achieved promising performance in the text classification task compared to shallow models. Despite of the significance of deep models, they ignore the fine-grained (matching signals between words and classes) classification clues since their classifications mainly rely on the text-level representations. To address this problem, we introduce the interaction mechanism to incorporate word-level matching signals into the text classification task. In particular, we design a novel framework, EXplicit interAction Model (dubbed as EXAM), equipped with the interaction mechanism. We justified the proposed approach on several benchmark datasets including both multilabel and multi-class text classification tasks. Extensive experimental results demonstrate the superiority of the proposed method. As a byproduct, we have released the codes and parameter settings to facilitate other researches.

This investigation investigates the influence of a variety of data augmentation techniques on sentiment analysis in low-resource languages, with a particular emphasis on Bulgarian, Croatian, Slovak, and Slovene. The following primary research topic is addressed: is it possible to improve sentiment analysis efficacy in low-resource languages through data augmentation? Our sub-questions look at how different augmentation methods affect performance, how effective WordNet-based augmentation is compared to other methods, and whether lemma-based augmentation techniques can be used, especially for Croatian sentiment tasks. The sentiment-labelled evaluations in the selected languages are included in our data sources, which were curated with additional annotations to standardise labels and mitigate ambiguities. Our findings show that techniques like replacing words with synonyms, masked language model (MLM)-based generation, and permuting and combining sentences can only make training datasets slightly bigger. However, they provide limited improvements in model accuracy for low-resource language sentiment classification. WordNet-based techniques, in particular, exhibit a marginally superior performance compared to other methods; however, they fail to substantially improve classification scores. From a practical perspective, this study emphasises that conventional augmentation techniques may require refinement to address the complex linguistic features that are inherent to low-resource languages, particularly in mixed-sentiment and context-rich instances. Theoretically, our results indicate that future research should concentrate on the development of augmentation strategies that introduce novel syntactic structures rather than solely relying on lexical variations, as current models may not effectively leverage synonymic or lemmatised data. These insights emphasise the nuanced requirements for meaningful data augmentation in low-resource linguistic settings and contribute to the advancement of sentiment analysis approaches.

Area under the curve may hide poor generalisation to external datasets

The emergent of machine learning (ML) and deep learning (DL) methods have created a substantial window of chance for their use in the industry. Given that both ML and DL algorithms are capable of identifying links between enormous amounts of data, one of the jobs for which these techniques have the most potential is visual inspection. These methods, nevertheless, call for a lot of photographs, which are not always possible to capture. Techniques for data augmentation are used as a remedy for this. Many datasets are created utilizing data augmentation approaches based on geometric transformations, starting from a very unbalanced dataset containing photos of various medical images. Many datasets are created utilizing data augmentation approaches based on geometric transformations, starting from a very unbalanced dataset containing photos of various medical images. The geometric transformation-based data augmentation segments the infected area and the classification process is proposed to highlight the severity of the disease. The proposed suggests an impartial and all-encompassing framework of evaluation for various information augmentation techniques. With this cutting-edge procedure, various augmented techniques are thoroughly and accurately assessed regarding data diversity and classification accurateness utilizing massive public datasets already available. In order to replicate the typical moderate size datasets used in deep learning, a regular-interval sampling strategy constructed on similarity ranking is described that can be used to choice samples from enormous open datasets and produce a subset that can accurately represent the original set. Then, using a variety of data augmentation methods, the augmented datasets are created from the small-sized datasets.