Improving low-resource machine transliteration by using 3-way transfer learning

Dynamic multiobjective optimization problems (DMOPs) aim to optimize multiple (often conflicting) objectives that are changing over time. Recently, there are a number of promising algorithms proposed based on transfer learning methods to solve DMOPs. However, it is very challenging to reduce the negative effect in transfer learning and find more effective transferred solutions. To fill this research gap, this article proposes a clustering-based transfer (CBT) learning method to solve DMOPs. When the environment changes, two novel operations (clustering-based selection (CBS) and CBT) are used to guide knowledge transfer. Specifically, CBS aims to find a population with nondominated solutions and dominated solutions as the training data for the new environment. Then, CBT further collects the previous Pareto-optimal solutions and some noise solutions as the training data for the previous environment. Two training data sets from different environments are, respectively, divided into multiple clusters and transfer learning is conducted on two similar clusters with high probability to reduce the negative effect, which can train an accurate prediction model to identify the promising solutions for the new environment. Empirical studies have been conducted on 14 benchmark DMOPs and one real-life path planning problem of unmanned air/ground vehicles, which validate the effectiveness of our proposed method. Especially, our method can significantly reduce negative transfer on 12 out of 14 cases when compared with direct transfer learning.

Surface electromyogram (sEMG)-based human-computer interface (HCI) is an effective tool for detecting human movements. Because sEMG-based motion recognition usually requires prolonged data measurements from the user (target), transfer learning reusing pre-measured (source) data from other users and pre-trained classifiers can be applied to sEMG data to reduce the measurement time. However, little knowledge is available regarding the combination of transfer learning methods and classifiers in sEMG data applications. Thus, we investigated the classification accuracy of data- and parameter-space-based transfer learning with shallow or deep classifiers in cross-subject sEMG classification. The dataset contains eight classes of forearm motions recorded from 25 volunteer participants. We used a support vector machine (SVM) as a shallow classifier as well as a deep neural network architecture, referred to as an artificial neural network (ANN), as a deep classifier. In addition, we used style transfer mapping (STM) as a data-space-based transfer learning method and fine-tuning (FT) as a parameter-space-based transfer learning method. Consequently, the classification accuracy of the ANN was higher than that of the SVM, regardless of the combinational use of transfer learning. STM and FT significantly improved the classification accuracy compared with non-transfer cases regardless of the classifier (note that FT can only be used with the ANN). In particular, the combined use of FT and the ANN yielded the best accuracy. These findings suggest that parameter-space-based transfer learning and deep classifiers are suitable for cross-subject sEMG classification. The combined use of parameter-space-based transfer learning and deep classifiers can effectively reduce the data measurement time of sEMG-based HCI applications.

Getting accurate lifetime predictions for a particular cell chemistry remains a challenging process, largely dependent on time and cost-intensive experimental battery testing. This paper proposes a transfer learning (TL) method to develop LIB ageing models, which allow for the leveraging of experimental laboratory testing data previously obtained for a different cell technology. The TL method is implemented through Neural Networks models, using LiNiMnCoO2/C laboratory ageing data as a baseline model. The obtained TL model achieves an 1.01% overall error for a broad range of operating conditions, using for retraining only two experimental ageing tests of LiFePO4/C cells.

Transfer learning using computational intelligence: A survey

Production forecasting is vital in the oil and gas sector, empowering engineers with insights for effective reservoir management. This paper introduces the concept of Transfer Learning as a powerful tool in the domain of machine-learning-assisted production forecasting that accounts of 3D spatial distributions of three geological properties, namely porosity, permeability, and saturation, two completion parameters, namely hydraulic fracture height and length, and production constraint. Transfer learning efficiently leverages knowledge from one problem to improve generalization on another, especially when data is scarce and computational resources are limited. To demonstrate the utility of transfer learning, we evaluate two scenarios of transfer learning. The first transfer learning scenario demonstrates the generalization of the forecasting to cases with variable hydraulic fracture spacing using limited training data. The second transfer learning scenario demonstrates the generalization of the forecasting to cases with variable natural fracture spacing and natural fracture permeability using limited training data. Source dataset contained 2000 realizations, while the target dataset contained 20, 40, 60, 80, 100, 250, 500, or 1000 realizations to represent the scenarios of limited training-data availability. The study confirms the benefits of transfer learning when the training dataset size is small (generally less than 100 training realizations); however, under large training dataset size (around 500 or more training realizations), transfer learning is not needed. For the first scenario involving variable hydraulic fracture spacing, the use of transfer learning ensured that the source model can be trained on target dataset with 80 realizations for gas rate forecasting at an accuracy of 25% in terms of MAPE, and with 500 realizations for condensate rate forecasting at an accuracy of 12% in terms of MAPE. Similarly, for the second scenario involving variable natural fracture spacing and natural fracture permeability, the use of transfer learning ensured that the source model can be trained on target dataset with 250 realizations for gas rate forecasting at an accuracy of 24% in terms of MAPE, and with 500 realizations for condensate rate forecasting at an accuracy of 23% in terms of MAPE. This illustrates the potential of transfer learning in improving forecasting models with limited data using a well pre-trained model and enhanced hyperparameter tuning of the transfer learning model. For cases with 500 or more training realizations, transfer learning severely underperforms as compared to training a conventional machine-learning model from scratch. The paper explores two cases of transfer learning.

The transfer learning methodology leverages knowledge from the source domain with abundant training data to the insufficient target domain. Recently, new approaches continue to be developed and used to solve different classification tasks, ranging from public news to videos and to many others. Most transfer learning methods are based on the assumption that both source and target data are in the same feature space or with the same data distribution, which however is not always true in real applications where it would lead to a negative transfer. In order to overcome this hurdle, the multiple-source transfer learning framework is useful. Since many real systems can be represented by networks, how to utilize the structural similarity between different networks so as to increase the transfer effectiveness becomes important. In this paper, the NES specification index is used to quantitatively measure the structural similarity between two networks, based on which a new transfer learning method (named NES-TL) is developed. Experiments on tag popularity prediction in StackExchange Q&A communities verify the effectiveness of the proposed approach, showing that it behaves better than existing baseline methods.

China has one of the widest distributions of carbonate rocks in the world. Karst wetland is a special and important ecosystem of carbonate rock regions. Chlorophyll-a (Chla) concentration is a key indicator of eutrophication, and could quantitatively evaluate water quality status of karst wetland. However, the spectral reflectance characteristics of the water bodies of karst wetland are not yet clear, resulting in remote sensing retrieval of Chla with great challenges. This study is a pioneer in utilizing field-based full-spectrum hyperspectral data to reveal the spectral response characteristics of karst wetland water body and determine the sensitive spectral bands of Chla. We further evaluated the Chla retrieval performance of multi-platform spectral data between Analytical Spectral Device (ASD), Unmanned aerial vehicle (UAV), and PlanetScope (Planet). We proposed two multi-sensor weighted integration strategies and two transfer learning frameworks for estimating water Chla from the largest karst wetland in China by combing a partial least square with adaptive ensemble algorithms. The results showed that: (1) In the range of 500–850 nm, the spectral reflectance of water bodies in the karst wetland was overall 0.001–0.105 higher than the inland water bodies, and the sensitive spectral ranges of water Chla focus on 603–778 nm; (2) UAV images outperformed ASD and Planet data, and produced the highest inversion accuracy (R2 = 0.670) for water Chla in karst wetland; (3) Multi-sensor weighted integration retrieval methods improved the Chla estimation accuracy (R2 = 0.716). Integration retrieval methods with the different weights produced the better Chla estimation accuracy than that of methods with the equal weights; (4) The transfer learning methods from ASD to UAV platform provided the better retrieval performance (the average R2 = 0.669) than that of methods from UAV to Planet platform. The transfer learning methods obtained the highest estimation accuracy of Chla (R2 = 0.814) when the ratio of the training and test data in the target domain was 7:3. The transfer learning methods produced the higher estimation accuracies with the distribution of the absolute residuals between predicted and measured values <20.957 mg/m3 compared to the multi-sensor weighted integration retrieval methods, which demonstrated that transfer learning is more suitable for estimating Chla in karst wetland water bodies using multi-platform and multi-sensor data. The results provide a scientific basis for the protection and sustainable development of karst wetlands.

Thermal imaging has been used for decades to monitor the health status of neonates as an non-invasive and non-ionizing imaging technique. Applications such as thermal asymmetry and disease analysis can be performed by applying deep learning methods to thermal imaging technique. However, thousands of different images are needed to perform analyzes with deep learning methods. It takes many years to create data sets with thousands of different images due to feeding time, medication time and instant baby care in the neonatal intensive care unit. In this study, a unhealthy-healthy classification was performed using thermal images obtained from the Selcuk University, Faculty of Medicine, Neonatal Intensive Care Unit for one year. Transfer learning method has been used to overcome the lack of data problem. When VGG16 model was used for transfer learning, the results were obtained as 100% sensitivity and 94.73% specificity. This result shows that thermal imaging and transfer learning method can be used in early diagnosis of diseases.

Transfer Learning Based Neural Machine Translation of English-Khasi on Low-Resource Settings

Tropical cyclones (TCs) are among the most severe and destructive natural events, and they have a major detrimental impact on both the economy and society. This study used Geo-KOMPSAT-2A (GK2A) satellite data to estimate TC intensity in the western North Pacific based on a convolutional neural network (CNN) model. Given that the GK2A data cover only the period since 2019, we applied transfer learning to the model using information learned from previous Communication, Ocean, and Meteorological Satellite (COMS) data, which cover a considerably longer period (2011&amp;#8211;2019). Transfer learning is a powerful technique that can improve the performance of a model even if the target task is based on a small amount of data. Experiments with various transfer learning methods using the GK2A and COMS data showed that the frozen&amp;#8211;fine-tuning method had the best performance due to the high similarity between the two datasets. The test results for 2021 showed that employing transfer learning led to a 20% reduction in the root mean square error (RMSE) compared to models using only GK2A data. For the operational model, which additionally used TC images and intensities from six hours earlier, transfer learning reduced the RMSE by 5.5%. Because continuous long-term data are not always available for TC intensity estimation based on geostationary satellite images, these results imply that transfer learning may constitute a new advance in this area. Acknowledgement. This research was supported by Korea Institute of Marine Science &amp;amp; Technology Promotion (KIMST) funded by the Korea Coast Guard (RS-2023-00238652, Integrated Satellite-based Applications Development for Korea Coast Guard). &amp;#160;

Neural transfer learning for assigning diagnosis codes to EMRs

Transfer learning improves quality for lowresource machine translation, but it is unclear what exactly it transfers. We perform several ablation studies that limit information transfer, then measure the quality impact across three language pairs to gain a black-box understanding of transfer learning. Word embeddings play an important role in transfer learning, particularly if they are properly aligned. Although transfer learning can be performed without embeddings, results are sub-optimal. In contrast, transferring only the embeddings but nothing else yields catastrophic results. We then investigate diagonal alignments with auto-encoders over real languages and randomly generated sequences, finding even randomly generated sequences as parents yield noticeable but smaller gains. Finally, transfer learning can eliminate the need for a warmup phase when training transformer models in high resource language pairs.

Recently, transfer learning has gained popularity in the machine learning community. Transfer Learning (TL) has emerged as a promising paradigm that leverages knowledge learned from one or more related domains to improve prediction accuracy in a target domain with limited data. However, for time series forecasting (TSF) applications, transfer learning is relatively new. This paper addresses the need for empirical studies as identified in recent reviews advocating the need for practical guidelines for Transfer Learning approaches and method designs for time series forecasting. The main contribution of this paper is the suggestion of a comprehensive framework for Transfer Learning Sensitivity Analysis (SA) for time series forecasting. We achieve this by identifying various parameters seen from various angles of transfer learning applied to time series, aiming to uncover factors and insights that influence the performance of transfer learning in time series forecasting. Undoubtedly, symmetry appears to be a core aspect in the consideration of these factors and insights. A further contribution is the introduction of four TL performance metrics encompassed in our framework. These TL performance metrics provide insight into the extent of the transferability between the source and the target domains. Analyzing whether the benefits of transferred knowledge are equally or unequally accessible and applicable across different domains or tasks speaks to the requirement of symmetry or asymmetry in transfer learning. Moreover, these TL performance metrics inform on the possibility of the occurrence of negative transfers and also provide insight into the possible vulnerability of the network to catastrophic forgetting. Finally, we discuss a sensitivity analysis of an Ensemble TL technique use case (with Multilayer Perceptron models) as a proof of concept to validate the suggested framework. While the results from the experiments offer empirical insights into various parameters that impact the transfer learning gain, they also raise the question of network dimensioning requirements when designing, specifically, a neural network for transfer learning.

Machine learning and deep learning advancements have boosted Brain-Computer Interface (BCI) performance, but their wide-scale applicability is limited due to factors like individual health, hardware variations, and cultural differences affecting neural data. Studies often focus on uniform single-site experiments in uniform settings, leading to high performance that may not translate well to real-world diversity. Deep learning models aim to enhance BCI classification accuracy, and transfer learning has been suggested to adapt models to individual neural patterns using a base model trained on others' data. This approach promises better generalizability and reduced overfitting, yet challenges remain in handling diverse and imbalanced datasets from different equipment, subjects, multiple centres in different countries, and both healthy and patient populations for effective model transfer and tuning. In a setting characterized by maximal heterogeneity, we proposed P300 wave detection in BCIs employing a convolutional neural network fitted with adaptive transfer learning based on Poison Sampling Disk (PDS) called Active Sampling (AS), which flexibly adjusts the transition from source data to the target domain. Our results reported for subject adaptive with 40% of adaptive fine-tuning that the averaged classification accuracy improved by 5.36% and standard deviation reduced by 12.22% using two distinct, internationally replicated datasets. These results outperformed in classification accuracy, computational time, and training efficiency, mainly due to the proposed Active Sampling (AS) method for transfer learning.

Ensuring the best quality and performance of modern speech technologies, today, is possible based on the widespread use of machine learning methods. The idea of this project is to study and implement an end-to-end system of automatic speech recognition using machine learning methods, as well as to develop new mathematical models and algorithms for solving the problem of automatic speech recognition for agglutinative (Turkic) languages. Many research papers have shown that deep learning methods make it easier to train automatic speech recognition systems that use an end-to-end approach. This method can also train an automatic speech recognition system directly, that is, without manual work with raw signals. Despite the good recognition quality, this model has some drawbacks. These disadvantages are based on the need for a large amount of data for training. This is a serious problem for low-data languages, especially Turkic languages such as Kazakh and Azerbaijani. To solve this problem, various methods are needed to apply. Some methods are used for end-to-end speech recognition of languages belonging to the group of languages of the same family (agglutinative languages). Method for low-resource languages is transfer learning, and for large resources – multi-task learning. To increase efficiency and quickly solve the problem associated with a limited resource, transfer learning was used for the end-to-end model. The transfer learning method helped to fit a model trained on the Kazakh dataset to the Azerbaijani dataset. Thereby, two language corpora were trained simultaneously. Conducted experiments with two corpora show that transfer learning can reduce the symbol error rate, phoneme error rate (PER), by 14.23 % compared to baseline models (DNN+HMM, WaveNet, and CNC+LM). Therefore, the realized model with the transfer method can be used to recognize other low-resource languages.