Introducing Various Semantic Models for Amharic: Experimentation and Evaluation with Multiple Tasks and Datasets

Coffee bean production can encounter challenges due to fluctuations in global coffee prices, impacting the economic stability of some countries that heavily depend on coffee production. The primary objective is to evaluate how effectively various pre-trained models can predict coffee types using advanced deep learning techniques. The selection of an optimal pre-trained model is crucial, given the growing popularity of specialty coffee and the necessity for precise classification. We conducted a comprehensive comparison of several pre-trained models, including AlexNet, LeNet, HRNet, Google Net, Mobile V2 Net, ResNet (50), VGG, Efficient, Darknet, and DenseNet, utilizing a coffee-type dataset. By leveraging transfer learning and fine-tuning, we assess the generalization capabilities of the models for the coffee classification task. Our findings emphasize the substantial impact of the pre-trained model choice on the model's performance, with certain models demonstrating higher accuracy and faster convergence than conventional alternatives. This study offers a thorough evaluation of pre-trained architectural models regarding their effectiveness in coffee classification. Through the evaluation of result metrics, including sensitivity (1.0000), specificity (0.9917), precision (0.9924), negative predictive value (1.0000), accuracy (1.0000), and F1 score (0.9962), our analysis provides nuanced insights into the intricate landscape of pre-trained models.

Prognostics and Health Management (PHM) is vital for complex engineering systems, yet its data-driven solutions are often hampered by the "small data problem"—a scarcity of labeled fault data in industrial settings. This limitation restricts the training and generalization of machine learning models and is compounded by varying operational conditions that reduce the relevance of historical data and pre-trained models. This research introduces a research framework to tackle these small data challenges in PHM. The primary objective is to develop a robust and adaptable PHM methodology by enhancing and synergistically integrating similarity-based Few-Shot Learning (FSL) with large-scale pre-trained time-series models. The research will focus on two main thrusts. First, it aims to improve the generalization capabilities of FSL frameworks by addressing limitations such as noise robustness, domain shift adaptability, and generalization to novel faults across diverse PHM domains. This involves developing noise-robust feature extraction, integrating domain adaptation techniques, and exploring expressive similarity metrics. Second, the study will investigate the effective adaptation of state-of-the-art pre-trained time-series models (e.g., TimesNet) for PHM tasks under data scarcity, focusing on efficient fine-tuning and synergistic integration with the enhanced FSL approaches. The author's prior success in a PHM data challenge using a similarity-based method for spacecraft systems provides preliminary validation. This research is expected to deliver an enhanced PHM framework for high-accuracy diagnostics with limited data, contributing generalized FSL models, systematic methods for leveraging pre-trained models in PHM, and advancing the practical deployment of intelligent PHM solutions.

WhisPAr: Transferring pre-trained audio models to fine-grained classification via Prompt and Adapter

Prompting techniques unlock valuable insights from pre-trained models, leading to a remarkable performance in Natural Language Processing (NLP). Inspired by NLP advancements, various graph prompt methods have emerged to transfer knowledge from pre-trained graph models to diverse downstream graph tasks. However, designing appropriate prompts for different downstream tasks remains a significant challenge due to substantial differences between pre-training graph tasks and downstream graph tasks. Moreover, existing graph prompt methods prevent transferability across different domains as they independently pre-train a model for each dataset. To address these challenges, we propose Graph Pretraining and Task-Specific Prompting (GTSP). To the best of our knowledge, GTSP is the first unified pre-training framework to handle cross-domain graph data and multiple types of graph tasks simultaneously. The key idea of our framework is to utilize similar label features between pre-training tasks and downstream tasks to seamlessly integrate the downstream task prompt into the pretrained model. We then propose a task-specific prompting architecture and design a label similarity matrix as prompt input. With GTSP, the pre-trained model can adapt to various types of downstream graph tasks, effectively avoiding the challenges of forcibly altering graph task forms. Experimental analysis conducted on benchmark datasets demonstrates the superiority of our method.

Pre-trained vision-language models ( e.g ., CLIP) have shown impressive success in various computer vision tasks with their generalization capability. Recently, parameter-efficient fine-tuning (PEFT) approaches have been actively explored to effectively and efficiently adapt the pre-trained vision-language models to a variety of downstream tasks. However, most existing PEFT approaches suffer from a task overfitting issue since the general knowledge of the pre-trained models is forgotten while a small number of learnable parameters in soft prompts/adapters are fine-tuned on a small data set from a specific target task. Thus, we propose a P arameter- E fficient F ine- T uning via Meta - R egularization (PEFT-MetaR) to improve the generalizability of parameter-efficient fine-tuning methods for vision-language models. Specifically, PEFT-MetaR meta-learns both the regularizer and learnable parameters to harness the task-specific knowledge from the downstream tasks and task-agnostic general knowledge from the pretrained models. Further, PEFT-MetaR augments the task to generate multiple virtual tasks to alleviate the meta-overfitting. In addition, we provide the analysis to comprehend how PEFT-MetaR improves the generalizability from the perspective of the gradient alignment. Our experiments demonstrate that PEFT-MetaR improves the generalizability of parameter-efficient fine-tuning methods on various datasets.

This paper introduces an approach to employ deep features for person re-identification. In contrast to existing works, we focus on using pre-trained deep models and their concept-based output to enhance attribute presentations of person images. There are two main contributions. First, we investigate recent state-of-the-art deep learning models for the task and provide a comprehensive evaluation. Second, we present an approach to improve identification accuracy of a standard attribute-based person re-identification approach. By using pre-trained models, we avoid re-training new deep models which always require a huge amount of training data and high computational cost. The idea is to utilize the correlation between generic concepts learned by the deep models and specific pre-defined attributes commonly used for person re-identification. We employ the deeply learned features from generic concepts to represent person images. These images and their manually annotated attributes are then used to train attribute classifiers. Given the classifiers, attributes of the probe and gallery images can be automatically extracted to compose attribute-based feature vectors. Re-identification is done by matching the vectors. Experiments conducted on several benchmark datasets demonstrated the effectiveness of the proposed approach.

The advent of pre-trained language models has directed a new era of Natural Language Processing (NLP), enabling us to create powerful language models. Among these models, Transformer-based models like BERT have grown in popularity due to their cutting-edge effectiveness. However, these models heavily rely on resource-intensive languages, forcing other languages into multilingual models(mBERT). The two fundamental challenges with mBERT become significantly more challenging in a resource-constrained language like Bangla. It was trained on a limited and organized dataset and contained weights for all other languages. Besides, current research on other languages suggests that a language-specific BERT model will exceed multilingual ones. This paper introduces Bangla-BERT^a, a monolingual BERT model for the Bangla language. Despite the limited data available for NLP tasks in Bangla, we perform pre-training on the largest Bangla language model dataset, BanglaLM, which we constructed using 40 GB of text data. Bangla-BERT achieves the highest results in all datasets and vastly improves the state-of-the-art performance in binary linguistic classification, multilabel extraction, and named entity recognition, outperforming multilingual BERT and other previous research. The pre-trained model is assessed against several non-contextual models such as Bangla fasttext and word2vec the downstream tasks. Finally, this model is evaluated by transfer learning based on hybrid deep learning models such as LSTM, CNN, and CRF in NER, and it is observed that Bangla-BERT outperforms state-of-the-art methods. The proposed Bangla-BERT model is assessed by using benchmark datasets, including Banfakenews, Sentiment Analysis on Bengali News Comments, and Cross-lingual Sentiment Analysis in Bengali. Finally, it is concluded that Bangla-BERT surpasses all prior state-of-the-art results by 3.52%, 2.2%, and 5.3%.

Word segmentation and POS tagging are two important problems included in many NLP tasks. They, however, have not drawn much attention of Vietnamese researchers all over the world. In this paper, we focus on the integration of advantages from several resourses to improve the accuracy of Vietnamese word segmentation as well as POS tagging task. For word segmentation, we propose a solution in which we try to utilize multiple knowledge resources including dictionary-based model, N-gram model, and named entity recognition model and then integrate them into a Maximum Entropy model. The result of experiments on a public corpus has shown its effectiveness in comparison with the best current models. We got 95.30% F1 measure. For POS tagging, motivated from Chinese research and Vietnamese characteristics, we present a new kind of features based on the idea of word composition. We call it morpheme-based features. Our experiments based on two POS-tagged corpora showed that morpheme-based features always give promising results. In the best case, we got 89.64% precision on a Vietnamese POS-tagged corpus when using Maximum Entropy model.

SEA: Hierarchically searching efficient adapters for pre-trained models.

Korean is the native and official language spoken by Chinese-Korean people, and Weibo is a social media software with a huge number of users in China. Currently, there is few studies related to sentiment analysis of Korean-language Weibo texts posted by Chinese-Korean users. In this paper, we propose a sentiment classification method for Chinese-Korean Weibo based on pre-trained language model and transfer learning. Firstly, we crawled the Chinese-Korean Weibo data from Sina Weibo and label them with sentiment to get the Chinese-Korean Weibo sentiment analysis (CKWSA) dataset. Secondly, to solve the problem of few training samples of the Chinese-Korean Weibo sentiment analysis dataset, we fine-tune the classifier based on the pre-trained Korean language model on the Korean Twitter sentiment analysis dataset to obtain the Korean Twitter sentiment classification model; and further fine-tune the model on CKWSA dataset to get Chinese-Korean Weibo sentiment classification model. The experiments show that the proposed classification method based on pre-trained language model and transfer learning has great performance, and there is an improvement compared other baselines on the Chinese-Korean Weibo sentiment analysis dataset.

This study aims to enhance object detection systems by comparing pre-trained classification models with custom-trained ones, focusing on task-based deep learning for image recognition. The problem addressed is the challenge of accurately detecting and classifying objects in complex environments where traditional recognition systems may fall short. The proposed solution leverages transfer learning utilizing pre-trained models like ResNet or VGGNet as feature extractors. By exploiting the convolutional layers of these models, the system captures common features for specific detection tasks. Experimental analyses on benchmark datasets confirm the efficacy of this approach, demonstrating improved detection accuracy and efficiency in various scenarios. Specifically, FasterRCNN achieves a mean Average Precision (mAP) of 78% on synthetic datasets and 74% on real datasets at an Intersection over Union (IoU) threshold of 0.5. This indicates FasterRCNN's superior performance in terms of accuracy, making it a strong candidate for applications requiring high detection accuracy.

BackgroundThe emergence of genomic language models (gLMs) offers an unsupervised approach to learning a wide diversity of cis-regulatory patterns in the non-coding genome without requiring labels of functional activity generated by wet-lab experiments. Previous evaluations have shown that pre-trained gLMs can be leveraged to improve predictive performance across a broad range of regulatory genomics tasks, albeit using relatively simple benchmark datasets and baseline models. Since the gLMs in these studies were tested upon fine-tuning their weights for each downstream task, determining whether gLM representations embody a foundational understanding of cis-regulatory biology remains an open question.ResultsHere, we evaluate the representational power of pre-trained gLMs to predict and interpret cell-type-specific functional genomics data that span DNA and RNA regulation for six major functional genomics prediction tasks. Our findings suggest that probing the representations of current pre-trained gLMs do not offer substantial advantages over conventional machine learning approaches that use one-hot encoded sequences. Nevertheless, highly tuned supervised models trained from scratch using one-hot encoded sequences can achieve performance competitive with or better than pre-trained models across the datasets explored in this study.DiscussionThis work highlights a major gap with current gLMs, raising potential issues in conventional pre-training strategies for the non-coding genome.Supplementary informationThe online version contains supplementary material available at 10.1186/s13059-025-03674-8.

FastPTM: Fast weights loading of pre-trained models for parallel inference service provisioning

Since the current remote sensing pre-trained models trained on optical images are not as effective when applied to SAR image tasks, it is crucial to create sensor-specific SAR models with generalized feature representations and to demonstrate with evidence the limitations of optical pre-trained models in downstream SAR tasks. The following aspects are the focus of this study: pre-training, fine-tuning, and explaining. First, we collect the current large-scale open-source SAR scene image classification datasets to pre-train a series of deep neural networks, including convolutional neural networks (CNNs) and vision transformers (ViT). A novel dynamic range adaptive enhancement method and a mini-batch class-balanced loss are proposed to tackle the challenges in SAR scene image classification. Second, the pre-trained models are transferred to various SAR downstream tasks compared with optical ones. Lastly, we propose a novel knowledge point interpretation method to reveal the benefits of the SAR pre-trained model with comprehensive and quantifiable explanations. This study is reproducible using open-source code and datasets, demonstrates generalization through extensive experiments on a variety of tasks, and is interpretable through qualitative and quantitative analyses. The codes and models are open source.

In this investigation, we propose a solution for the author’s gender identification task called AGI-P. This task has several real-world applications across different fields, such as marketing and advertising, forensic linguistics, sociology, recommendation systems, language processing, historical analysis, education, and language learning. We created a new dataset to evaluate our proposed method. The dataset is balanced in terms of gender using a random sampling method and consists of 1944 samples in total. We use accuracy as an evaluation measure and compare the performance of the proposed solution (AGI-P) against state-of-the-art machine learning classifiers and fine-tuned pre-trained multilingual language models such as DistilBERT, mBERT, XLM-RoBERTa, and Multilingual DEBERTa. In this regard, we also propose a customized fine-tuning strategy that improves the accuracy of the pre-trained language models for the author gender identification task. Our extensive experimental studies reveal that our solution (AGI-P) outperforms the well-known machine learning classifiers and fine-tuned pre-trained multilingual language models with an accuracy level of 92.03%. Moreover, the pre-trained multilingual language models, fine-tuned with the proposed customized strategy, outperform the fine-tuned pre-trained language models using an out-of-the-box fine-tuning strategy. The codebase and the corpus will be publicly available for scholars in this area upon publication.