SAR-HUB: Pre-Training, Fine-Tuning, and Explaining

Prompting techniques unlock valuable insights from pre-trained models, leading to a remarkable performance in Natural Language Processing (NLP). Inspired by NLP advancements, various graph prompt methods have emerged to transfer knowledge from pre-trained graph models to diverse downstream graph tasks. However, designing appropriate prompts for different downstream tasks remains a significant challenge due to substantial differences between pre-training graph tasks and downstream graph tasks. Moreover, existing graph prompt methods prevent transferability across different domains as they independently pre-train a model for each dataset. To address these challenges, we propose Graph Pretraining and Task-Specific Prompting (GTSP). To the best of our knowledge, GTSP is the first unified pre-training framework to handle cross-domain graph data and multiple types of graph tasks simultaneously. The key idea of our framework is to utilize similar label features between pre-training tasks and downstream tasks to seamlessly integrate the downstream task prompt into the pretrained model. We then propose a task-specific prompting architecture and design a label similarity matrix as prompt input. With GTSP, the pre-trained model can adapt to various types of downstream graph tasks, effectively avoiding the challenges of forcibly altering graph task forms. Experimental analysis conducted on benchmark datasets demonstrates the superiority of our method.

Pre-trained language models with structured semantic knowledge have demonstrated remarkable performance in a variety of downstream natural language processing tasks. The typical methods of integrating knowledge are designing different pre-training tasks and training from scratch, which requires high-end hardware, massive storage resources, and long computing times. Prompt learning is an effective approach to tuning language models for specific tasks, and it can also be used to infuse knowledge. However, most prompt learning methods accept one token as the answer, instead of multiple tokens. To tackle this problem, we propose the long-answer prompt learning method (KLAPrompt), with three different long-answer strategies, to incorporate semantic knowledge into pre-trained language models, and we compare the performance of these three strategies through experiments. We also explore the effectiveness of the KLAPrompt method in the medical field. Additionally, we generate a word sense prediction dataset (WSP) based on the Xinhua Dictionary and a disease and category prediction dataset (DCP) based on MedicalKG. Experimental results show that discrete answers with the answer space partitioning strategy achieve the best results, and introducing structured semantic information can consistently improve language modeling and downstream tasks.

WhisPAr: Transferring pre-trained audio models to fine-grained classification via Prompt and Adapter

In the realm of computer vision, image classification is a critical issue with many applications, including multimedia content analysis, security and surveillance, and medical imaging. The accuracy of image classification algorithms has considerably increased with the development of deep learning. The discipline of image classification has dramatically benefited from the usage of pre-trained Convolutional Neural Network (CNN) models. In this study, we perform image classification on the Wang dataset composed of 1000 images separated into ten categories, using six different pre-trained CNN models. The research made use of pre-trained versions of VGG16, Densenet, Mobilenet, Inception V3, Resnet50, and Xception models. We assessed the model performances in terms of training and testing accuracy. Testing accuracies for ten unique batches of data were calculated and averaged out. Densenet outperformed state-of-the-art models like Xception by a small margin, mainly due to low data availability. The findings of this study show how pre-trained models have advanced the field of image classification and shed light on how well these models perform in diverse image classification tasks. This study can serve as a valuable reference for researchers and practitioners in the field of computer vision and deep learning and help inform their choices when selecting pre-trained models for image classification, depending on their needs, while also considering data availability and computational constraints.

Pre-trained vision-language models ( e.g ., CLIP) have shown impressive success in various computer vision tasks with their generalization capability. Recently, parameter-efficient fine-tuning (PEFT) approaches have been actively explored to effectively and efficiently adapt the pre-trained vision-language models to a variety of downstream tasks. However, most existing PEFT approaches suffer from a task overfitting issue since the general knowledge of the pre-trained models is forgotten while a small number of learnable parameters in soft prompts/adapters are fine-tuned on a small data set from a specific target task. Thus, we propose a P arameter- E fficient F ine- T uning via Meta - R egularization (PEFT-MetaR) to improve the generalizability of parameter-efficient fine-tuning methods for vision-language models. Specifically, PEFT-MetaR meta-learns both the regularizer and learnable parameters to harness the task-specific knowledge from the downstream tasks and task-agnostic general knowledge from the pretrained models. Further, PEFT-MetaR augments the task to generate multiple virtual tasks to alleviate the meta-overfitting. In addition, we provide the analysis to comprehend how PEFT-MetaR improves the generalizability from the perspective of the gradient alignment. Our experiments demonstrate that PEFT-MetaR improves the generalizability of parameter-efficient fine-tuning methods on various datasets.

Deep neural networks such as Convolutional Neural Networks (CNNs) have achieved several significant milestones in visual data analytics. Benefited from transfer learning, many researchers use pre-trained CNN models to accelerate the training process. However, there is still uncertainty about the deep learning models, structures, and applications. For instance, the diversity of the datasets may affect the performance of each pre-trained model. Therefore, in this paper, we proposed a new approach based on genetic algorithms to select or regenerate the best pre-trained CNN models for different visual datasets. A new genetic encoding model is presented which denotes different pre-trained models in our population. During the evolutionary process, the optimal genetic code that represents the best model is selected, or new competitive individuals are generated using the genetic operations. The experimental results illustrate the effectiveness of the proposed framework which outperforms several existing approaches in visual data classification.

Exploiting pretrained CNN models for the development of an EEG-based robust BCI framework

Sport performance analysis in sports practice cannot be separable. It is important to help coach analyse and improve the performance of their athletes through training or game session. Due to the advancement of technology nowadays, the notational analysis of the video content using various software packages has become possible. Unluckily, the coach needs to recognize the actions manually before doing further analysis. The purpose of this study is to formulate an automated system for badminton smash recognition on widely available broadcasted videos using pre-trained Convolutional Neural Network (CNN) method. Smash and other badminton actions such as clear, drop, lift and net from the video were used to formulate the CNN models. Therefore, two experiments were conducted in this study. The first experiment is the study on the performance between four different existing pre-trained models which is AlexNet, GoogleNet, Vgg-16 Net and Vgg-19 Net in recognizing five actions. The results show that the pre-trained AlexNet model has the highest performance accuracy and fastest training period among the other models. The second experiment is the study on the performance of two different pre-trained models which is AlexNet and GoogleNet to recognize smash and non-smash action only. The results show that the pre-trained GoogleNet model produces the best performance in recognizing smash action. In conclusion, pre-trained AlexNet model is suitable to be used to automatically recognize the five badminton actions while GoogleNet model is excellent at recognizing smash action from the broadcasted video for further notational analysis.

The availability of different pre-trained semantic models has enabled the quick development of machine learning components for downstream applications. However, even if texts are abundant for low-resource languages, there are very few semantic models publicly available. Most of the publicly available pre-trained models are usually built as a multilingual version of semantic models that will not fit well with the need for low-resource languages. We introduce different semantic models for Amharic, a morphologically complex Ethio-Semitic language. After we investigate the publicly available pre-trained semantic models, we fine-tune two pre-trained models and train seven new different models. The models include Word2Vec embeddings, distributional thesaurus (DT), BERT-like contextual embeddings, and DT embeddings obtained via network embedding algorithms. Moreover, we employ these models for different NLP tasks and study their impact. We find that newly-trained models perform better than pre-trained multilingual models. Furthermore, models based on contextual embeddings from FLAIR and RoBERTa perform better than word2Vec models for the NER and POS tagging tasks. DT-based network embeddings are suitable for the sentiment classification task. We publicly release all the semantic models, machine learning components, and several benchmark datasets such as NER, POS tagging, sentiment classification, as well as Amharic versions of WordSim353 and SimLex999.

Coffee bean production can encounter challenges due to fluctuations in global coffee prices, impacting the economic stability of some countries that heavily depend on coffee production. The primary objective is to evaluate how effectively various pre-trained models can predict coffee types using advanced deep learning techniques. The selection of an optimal pre-trained model is crucial, given the growing popularity of specialty coffee and the necessity for precise classification. We conducted a comprehensive comparison of several pre-trained models, including AlexNet, LeNet, HRNet, Google Net, Mobile V2 Net, ResNet (50), VGG, Efficient, Darknet, and DenseNet, utilizing a coffee-type dataset. By leveraging transfer learning and fine-tuning, we assess the generalization capabilities of the models for the coffee classification task. Our findings emphasize the substantial impact of the pre-trained model choice on the model's performance, with certain models demonstrating higher accuracy and faster convergence than conventional alternatives. This study offers a thorough evaluation of pre-trained architectural models regarding their effectiveness in coffee classification. Through the evaluation of result metrics, including sensitivity (1.0000), specificity (0.9917), precision (0.9924), negative predictive value (1.0000), accuracy (1.0000), and F1 score (0.9962), our analysis provides nuanced insights into the intricate landscape of pre-trained models.

Alignment between image and text has shown promising improvements on patch-level pre-trained document image models. However, investigating more effective or finer-grained alignment techniques during pre-training requires a large amount of computation cost and time. Thus, a question naturally arises: Could we fine-tune the pre-trained models adaptive to downstream tasks with alignment objectives and achieve comparable or better performance? In this paper, we propose a new model architecture with alignment-enriched tuning (dubbed AETNet) upon pre-trained document image models, to adapt downstream tasks with the joint task-specific supervised and alignment-aware contrastive objective. Specifically, we introduce an extra visual transformer as the alignment-ware image encoder and an extra text transformer as the alignment-ware text encoder before multimodal fusion. We consider alignment in the following three aspects: 1) document-level alignment by leveraging the cross-modal and intra-modal contrastive loss; 2) global-local alignment for modeling localized and structural information in document images; and 3) local-level alignment for more accurate patch-level information. Experiments on various downstream tasks show that AETNet can achieve state-of-the-art performance on various downstream tasks. Notably, AETNet consistently outperforms state-of-the-art pre-trained models, such as LayoutLMv3 with fine-tuning techniques, on three different downstream tasks. Code is available at https://github.com/MAEHCM/AET.

In recent years, pre-training with the transformer architecture has gained significant attention. While this approach has led to notable performance improvements across a variety of downstream tasks, the underlying mechanisms by which pre-training models influence these tasks, particularly in the context of biological data, are not yet fully elucidated. In this study, focusing on the pre-training on nucleotide sequences, we decompose a pre-training model of Bidirectional Encoder Representations from Transformers (BERT) into its embedding and encoding modules to analyze what a pre-trained model learns from nucleotide sequences. Through a comparative study of non-standard pre-training at both the data and model levels, we find that a typical BERT model learns to capture overlapping-consistent k-mer embeddings for its token representation within its embedding module. Interestingly, using the k-mer embeddings pre-trained on random data can yield similar performance in downstream tasks, when compared with those using the k-mer embeddings pre-trained on real biological sequences. We further compare the learned k-mer embeddings with other established k-mer representations in downstream tasks of sequence-based functional prediction. Our experimental results demonstrate that the dense representation of k-mers learned from pre-training can be used as a viable alternative to one-hot encoding for representing nucleotide sequences. Furthermore, integrating the pre-trained k-mer embeddings with simpler models can achieve competitive performance in two typical downstream tasks. The source code and associated data can be accessed at https://github.com/yaozhong/bert_investigation.

Assess the efficacy of deep convolutional neural networks (DCNNs) in detection of critical enteric feeding tube malpositions on radiographs. 5475 de-identified HIPAA compliant frontal view chest and abdominal radiographs were obtained, consisting of 174 x-rays of bronchial insertions and 5301 non-critical radiographs, including normal course, normal chest, and normal abdominal x-rays. The ground-truth classification for enteric feeding tube placement was performed by two board-certified radiologists. Untrained and pretrained deep convolutional neural network models for Inception V3, ResNet50, and DenseNet 121 were each employed. The radiographs were fed into each deep convolutional neural network, which included untrained and pretrained models. The Tensorflow framework was used for Inception V3, ResNet50, and DenseNet. Images were split into training (4745), validation (630), and test (100). Both real-time and preprocessing image augmentation strategies were performed. Receiver operating characteristic (ROC) and area under the curve (AUC) on the test data were used to assess the models. Statistical differences among the AUCs were obtained. p < 0.05 was considered statistically significant. The pretrained Inception V3, which had an AUC of 0.87 (95 CI; 0.80–0.94), performed statistically significantly better (p < .001) than the untrained Inception V3, with an AUC of 0.60 (95 CI; 0.52–0.68). The pretrained Inception V3 also had the highest AUC overall, as compared with ResNet50 and DenseNet121, with AUC values ranging from 0.82 to 0.85. Each pretrained network outperformed its untrained counterpart. (p < 0.05). Deep learning demonstrates promise in differentiating critical vs. non-critical placement with an AUC of 0.87. Pretrained networks outperformed untrained ones in all cases. DCNNs may allow for more rapid identification and communication of critical feeding tube malpositions.

Diabetic retinopathy (DR) is a common eye disease that people get from diabetes. About 33.7% of the people with diabetes have DR. With our datas, which are pictures of the eyeball with and without DR, we tried different convolutional neural network (CNN) models to get the best accuracy score. We tested our datas with a default CNN model, and 5 different pre-trained models: MobileNet, VGG16, VGG19, Inception V3, and Inception ResNet V2. The default CNN model didn’t perform very well, getting only 10.4%. The pre-trained model also didn’t perform as good as expected, so we decided to use GRU with the models, which increases the score. For the higher accuracy, we added bidirectional GRU to train the whole parameters in the model. The 5 different pre-trained models scored an average of 74.2% accuracy score, and Inception ResNet V2 with bidirectional GRU included scored the highest accuracy, achieving 83.57%. For additional study, we used a class activation map to spot the abnormal parts of the eyeball with DR, and we could spot abnormal veins and bleeding on the eyeball. However, our research has limitations on that we did not use the segmentation methods, which is more advanced technique compared to classification, such as U-net, Fully Convolutional Network (FCN), Deep Lab V3, and Feature Pyramid Network. Furthermore, even though our model classified 5 different classes, the fact that the highest accuracy score was lower than 90% is also a limitation. For further study, we would prepare a masking method for applying segmentation methods to our dataset.

Using low-rank adaptation to fine-tune pre-trained neural network models has attracted widespread attention due to its advantages of low resource requirements, high precision, and no additional inference delay. However, most existing methods are designed for large language models based on the Transformer structure and lack adaptation to convolutional neural networks (CNN) widely used on Internet of Things (IoT) devices. In addition, IoT devices are usually deployed outdoors and collect large amounts of data that are affected by environmental conditions. Providing a highly robust CNN model is a prerequisite for providing high-quality services. To this end, this paper proposes a highly robust seed-filter adaptation method (SeFA) for pre-trained CNNs. SeFA introduces an adaptation branch with the same structure as the backbone network. In the adaptation branch, some filters are first designated seed filters and grouped. Then, additional filters are generated from the grouped seed filters and nonlinear transformation functions (NLFs) with different hyperparameters. The seed filters' parameters are updated during model training, and the NLF hyperparameters are randomly initialized and frozen. Both grouping seed filters and configuring NLFs with non-learnable hyperparameters can improve the model's robustness. This is because grouping seed filters can generate diverse filters on demand without increasing the model's complexity, and the NLFs' rules can regularize the model. It is worth mentioning that the number of fine-tuning parameters of the pre-trained model that can adapt to downstream tasks can be flexibly controlled by specifying the number of seed filters. The key idea of this paper is to propose SeFA with flexible, controllable, learnable parameters, high robustness, and adaptability to pretrained CNN models, thereby facilitating fine-tuning on resource-constrained IoT devices and providing highly robust visual services. Experimental results on the CIFAR-10, CIFAR-10-C, CIFAR-100, CIFAR-100-C, and Icons-50 datasets demonstrate that the proposed SeFA outperforms other state-of-the-art methods. Specifically, on the ResNet-152 model and the CIFAR-10-C dataset, the accuracy of our SeFA is about 7% higher than that of the full fine-tuning method. In addition, we verify the efficiency of SeFA-based models for fine-tuning and inference on the device.