Forward Compatible Training for Large-Scale Embedding Retrieval Systems

The rising machine learning (ML) models have become the preferred way for landslide detection based on remote sensing images, but the performance of these models in a sample-free area are rarely concerned in many studies. In this study, we used a cross-validation method (training model in one area and validation in another) to compare the model portability of trained ML models applied in an “off-site” area, as a consideration of the landslide detection ability of these models in sample-free areas. We integrate nighttime light imagery, multi-seasonal optical Landsat time-series and digital elevation data, and we employed support vector machines (SVM), artificial neural networks (ANN) and random forest (RF) models to classify the satellite imagery and identify landslides. Samples of two scenarios generated from two subareas of the Jiuzhaigou disaster-stricken region are used for the cross-application and accuracy evaluation of three ML models. The results revealed that when the trained models are applied in areas outside those in which they were developed, the landslide identification accuracy of these three models has declined. Especially for the SVM and ANN models, the accuracy is greatly reduced and there appears a seriously imbalanced user’s and producer’s accuracy. However, although the performance of the RF model is lower than that of SVM and ANN models in their local area, the RF model exhibits stable portability, and retains the original performance and achieves a satisfactory balance between overestimation and underestimation in “off-site” areas. An additional validation from a new area proved that the landslide detection performance of the RF model with stable portability is higher than that of the SVM and ANN models in “off-site” areas. The results suggest that evaluating the model portability through cross-application can be a useful way to determine the most suitable model for landslide detection in “off-site” areas with a similar geographic environment to model development areas, so as to maximize the accuracy of landslide detection based on limited samples.

The application of transfer learning to T1ρ MRI tibiofemoral cartilage segmentation.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Background and objectivePositron emission tomography/computed tomography (PET/CT) is recommended as the standard imaging modality for diffuse large B-cell lymphoma (DLBCL) staging. However, many studies have neglected the role of patients' prognostic factors with respect to imaging PET/CT of quantitative features. In this paper, a multi-view learning (MVL) model is established to make full use of both clinical and imaging data to predict the prognosis of DLBCL patients and thereby assist doctors in decision-making.MethodsFeature engineering, including feature extraction, feature screening by recursive feature elimination, and dimensionality reduction by principal component analysis, are successively performed on the clinical data and imaging data of the research subjects to obtain the study data. After dividing the data into training and test sets, an instance weighting method is applied to the training data. Subsequently, kernel mapping is performed on the imaging features and clinical features separately, and this kernel mapping is processed in the new kernel feature space using kernel canonical correlation analysis (KCCA). Lastly, model training is performed on the obtained common kernel subspace using a support vector machine (SVM). The final overall model, named SVM-2view-KCCA (SVM-2 K), was compared with three other multi-view models (Ensemble-SVM, Multi-view maximum entropy discrimination, and canonical correlation analysis). The performance of the model was evaluated on the test data with respect to several dichotomous metrics: accuracy, sensitivity, F1 score, the area under the curve (AUC), and G-mean.ResultsThe SVM model improved AUC by 10.5%, sensitivity by 11.9%, accuracy by 9.8%, F1 score by 9.2%, and G-mean by 7.8% for the DLBCL test data after feature engineering based on dimensionality reduction and instance weighting. In the performance comparison of single-view learning models, the SVM-based integration of clinical and imaging features achieved the best overall performance (AUC = 86.3%, accuracy = 91.6%, sensitivity = 83.2%, F1 = 85.7%, and G-mean = 86.1%). In the comparison of MVL models, SVM-2 K achieved the best overall performance (AUC = 92.1%, accuracy = 96.9%, sensitivity = 90.9%, F1 = 92.8%, and G-mean = 91.4%), and the performance of each MVL model was better than that of the best single-view learning model.ConclusionsMVL models outperformed single-view learning models. Of the MVL models, the proposed SVM-2 K achieved the best overall performance and could accurately predict patient prognosis.

Random feature mapping (RFM) is the core operation in the random weight neural network (RWNN). Its quality has a significant impact on the performance of a RWNN model. However, there has been no good way to evaluate the quality of RFM. In this paper, we introduce a new concept called dispersion degree of matrix information distribution (DDMID), which can be used to measure the quality of RFM. We used DDMID in our experiments to explain the relationship between the rank of input data and the performance of the RWNN model and got some interesting results. We found that: (1) when the rank of input data reaches a certain threshold, the model’s performance increases with the increase in the rank; (2) the impact of the rank on the model performance is insensitive to the type of activation functions and the number of hidden nodes; (3) if the DDMID of an RFM matrix is very small, it implies that the first $$\mathbf{k}$$ singular values in the singular value matrix of the RFM matrix contain too much information, which usually has a negative impact on the final closed-form solution of the RWNN model. Besides, we verified the improvement effect of intrinsic plasticity (IP) algorithm on RFM by using DDMID. The experimental results showed that DDMID allows researchers evaluate the mapping quality of data features before model training, so as to predict the effect of data preprocessing or network initialization without model training. We believe that our findings could provide useful guidance when constructing and analyzing a RWNN model.

Low precision training can significantly reduce the computational overhead of training deep neural networks (DNNs). Though many such techniques exist, cyclic precision training (CPT), which dynamically adjusts precision throughout training according to a cyclic schedule, achieves particularly impressive improvements in training efficiency, while actually improving DNN performance. Existing CPT implementations take common learning rate schedules (e.g., cyclical cosine schedules) and use them for low precision training without adequate comparisons to alternative scheduling options. We define a diverse suite of CPT schedules and analyze their performance across a variety of DNN training regimes, some of which are unexplored in the low precision training literature (e.g., node classification with graph neural networks). From these experiments, we discover alternative CPT schedules that offer further improvements in training efficiency and model performance, as well as derive a set of best practices for choosing CPT schedules. Going further, we find that a correlation exists between model performance and training cost, and that changing the underlying CPT schedule can control the tradeoff between these two variables. To explain the direct correlation between model performance and training cost, we draw a connection between quantized training and critical learning periods, suggesting that aggressive quantization is a form of learning impairment that can permanently damage model performance.

Glacial cirque identification based on Convolutional Neural Networks

Compressional and shear travel time logs (DTC and DTS) acquired using sonic logging tools are crucial for subsurface geomechanical characterization. In this study, 13 ‘easy-to-acquire’ conventional logs were processed using 6 shallow learning models, namely ordinary least squares (OLS), partial least squares (PLS), elastic net (EN), LASSO, multivariate adaptive regression splines (MARS), and artificial neural network (ANN), to successfully synthesize DTC and DTS logs. Among the 6 models, ANN outperforms other models with R2 of 0.87 and 0.85 for the syntheses of DTC and DTS logs, respectively. The 6 shallow learning models are trained and tested with 8481 data points acquired from a 4240-feet depth interval of a shale reservoir in Well 1, and the trained models are deployed in Well 2 for purposes of blind testing against 2920 data points from 1460-feet depth interval. Following that, 5 clustering algorithms are applied on the 13 ‘easy-to-acquire’ logs to identify clusters and compare them with the prediction performance of the shallow learning models used for log synthesis. Dimensionality reduction algorithm is used to visualize the characteristics of the clustering algorithm. Hierarchical clustering, DBSCAN, and self-organizing map (SOM) algorithms are sensitive to outliers and did not effectively differentiate the input data into consistent clusters. Gaussian mixture model can well differentiate the various formations, but the clusters do not have a strong correlation with the prediction performance of the log-synthesis models. Clusters identified using K-means method have a strong correlation with the prediction performance of the shallow learning models. By combining the predictive shallow learning models for log synthesis with the K-means clustering algorithm, we propose a reliable workflow that can synthesize the DTC and DTS logs, as well as generate a reliability indicator for the predicted logs to help an user better understand the performance of the shallow learning models during deployment.

Training large-scale Neural Networks requires substantial computational power and energy. Federated Learning enables distributed model training across geospatially distributed data centers, leveraging renewable energy sources to reduce the carbon footprint of AI training. Various client selection strategies have been developed to align the volatility of renewable energy with stable and fair model training in a federated system. However, due to the privacy-preserving nature of Federated Learning, the quality of data on client devices remains unknown, posing challenges for effective model training. In this paper, we introduce a modular approach on top to state-of-the-art client selection strategies for carbon-efficient Federated Learning. Our method enhances robustness by incorporating a noisy client data filtering, improving both model performance and sustainability in scenarios with unknown data quality. Additionally, we explore the impact of carbon budgets on model convergence, balancing efficiency and sustainability. Through extensive evaluations, we demonstrate that modern client selection strategies based on local client loss tend to select clients with noisy data, ultimately degrading model performance. To address this, we propose a gradient norm thresholding mechanism using probing rounds for more effective client selection and noise detection, contributing to the practical deployment of carbon-efficient Federated Learning.

Honey is a naturally sweet and viscous product for which the addition of any substance is prohibited by international regulation. Detection of adulteration in honey is a technical problem: adulteration of honey with invert sugar and syrup may not be reliably detected by direct sugar analysis because its constituents are identical to the major natural components of the honey. Therefore, it is important to develop a rapid and reliable analytical method to detect such additions. We used near-infrared spectroscopy (NIR) combined with principle component analysis (PCA) and artificial neural networks (ANN) modelling to discriminate between honey and corn syrup in adulterated honey. Fifteen honey samples from north-west Croatia (Krapina-Zagorje County) were intentionally supplemented with differing proportions of corn syrup ranging from 10-90%. We collected a total of 460 NIR spectra using the Control Development NIR128L-1.7 spectrophotometer (Control Development, South Bend, Indiana, USA) with their software Spec32 software anda HL-2000 halogen light source. For each of the prepared samples, we measured water content by refractometer (Brouwland, Belgium), conductivity byconductometer (SevenCompact, MettlerToledo, Switzerland), and colour using a PCE-CSM3 colorimeter (PCE Instruments, Germany). Prior to ANN modelling, PCA was used to identify patterns and highlight similarities and differences in data of the individual set of the experiment. The goal of PCA is to extract important information from the data table and to express this information as a set of new orthogonal variables called principal components or factors (PCs or Fs). We conducted PCA of raw spectra using the Unscrambler® X 10.4 software (CAMO software, Norway). Data were divided into ANN model training, test, and validation datasets at a 70:15:15 ratio using the first five PCs. ANNs were calibrated using model training data, and evaluated using model test and model validation datasets for their ability to predict: i) the amount of added adultering substance in honey, ii) water content, iii) conductivity and iv) colour of the adulterated honey. Multiple layer perception (MLP) networks were developed in Statistica v.10.0 software (StatSoft, Tulsa, USA). Back error propagation algorithm available in Statistica v.10.0 was applied for the model training. Model performance was evaluated using R2 and root mean squared error (RMSE) values for model training, test, and validation datasets. Results show that network MLP 5-8-6 with five neurons in the input layer, 8 neurons in the hidden layer and 6 neurons in the output layer predicts the analysed output variables with high precision (R2validation,concentration = 0.995, R2validation,water content = 0.993, R2validation,conductivity = 0.992, R2validation,L = 0.939, R2validation,a = 0.895, R2validation,b = 0.924).

We propose cooperative edge-assisted dynamic federated learning ( CE-FL). CE-FL introduces a distributed machine learning (ML) architecture, where data collection is carried out at the end devices, while the model training is conducted <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">cooperatively</i> at the end devices and the edge servers, enabled via data offloading from the end devices to the edge servers through base stations. CE-FL also introduces floating aggregation point, where the local models generated at the devices and the servers are aggregated at an edge server, which varies from one model training round to another to cope with the network evolution in terms of data distribution and users’ mobility. CE-FL considers the heterogeneity of network elements in terms of communication/computation models and the proximity to one another. CE-FL further presumes a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic</i> environment with online variation of data at the network devices which causes a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">drift</i> at the ML model performance. We model the processes taken during CE-FL, and conduct analytical convergence analysis of its ML model training. We then formulate network-aware CE-FL which aims to adaptively optimize all the network elements via tuning their contribution to the learning process, which turns out to be a non-convex mixed integer problem. Motivated by the large scale of the system, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">distributed optimization solver</i> to break down the computation of the solution across the network elements. We finally demonstrate the effectiveness of our framework with the data collected from a real-world testbed.

In early design phases and during software evolution, design-time energy efficiency analyses enable software architects to reason on the effect of design decisions on energy efficiency. Energy efficiency analyses rely on accurate power models to estimate power consumption. Deriving power models that are both accurate and usable for design time predictions requires extensive measurements and manual analysis. Existing approaches that aim to automate the extraction of power models focus on the construction of models for runtime estimation of power consumption. Power models constructed by these approaches do not allow users to identify the central set of system metrics that impact energy efficiency prediction accuracy. The identification of these central metrics is important for design time analyses, as an accurate prediction of each metric incurs modeling effort. We propose a methodology for the automated construction of multi-metric power models using systematic experimentation. Our approach enables the automated training and selection of power models for the design time prediction of power consumption. We validate our approach by evaluating the prediction accuracy of derived power models for a set of enterprise and data-intensive application benchmarks.

This study proposes a flexible parametric proportional odds regression model that incorporates the exponentiated-Weibull distribution as a baseline for analyzing censored lifetime data. The proposed model is referred to as the exponentiated-Weibull proportional odds regression model. This model provides greater flexibility in capturing a wider range of hazard shapes and survival patterns. The paper discusses the theoretical framework as well as estimation methods for the model parameters. Additionally, extensive simulation studies are conducted to evaluate the proposed model's performance under different scenarios. The results demonstrate that the model effectively accommodates the unique characteristics of the exponentiated-Weibull distribution. Furthermore, two real-world datasets are presented to illustrate and compare the model's practical application and performance with existing proportional odds regression models. The findings highlight the advantages of using the proposed model and its potential to enhance the analysis of survival data and capture complex survival patterns.

Machine unlearning aims to eliminate the influence of a subset of training samples (i.e., unlearning samples) from a trained model. Effectively and efficiently removing the unlearning samples without negatively impacting the overall model performance is challenging. Existing works mainly exploit input and output space and classification loss, which can result in ineffective unlearning or performance loss. In addition, they utilize unlearning or remaining samples ineffectively, sacrificing either unlearning efficacy or efficiency. Our main insight is that the direct optimization on the representation space utilizing both unlearning and remaining samples can effectively remove influence of unlearning samples while maintaining representations learned from remaining samples. We propose a contrastive unlearning framework, leveraging the concept of representation learning for more effective unlearning. It removes the influence of unlearning samples by contrasting their embeddings against the remaining samples' embeddings so that their embeddings are closer to the embeddings of unseen samples. Experiments on a variety of datasets and models on both class unlearning and sample unlearning showed that contrastive unlearning achieves the best unlearning effects and efficiency with the lowest performance loss compared with the state-of-the-art algorithms. In addition, it is generalizable to different contrastive frameworks and other models such as vision-language models. Our main code is available on github.com/Emory-AIMS/Contrastive-Unlearning

Froth flotation is a popular ground ore separation method with a wide range of applications in the mining industry due to its versatility and simplicity. In this method, a separation with a certain grade and recovery is achieved thanks to the difference in the surface hydrophobicity of the different insoluble compounds that are present in an industrial ground ore. Under a process intensification scope, the study of froth flotation columns is of remarkable interest due to their higher gas phase surface-to-volume ratios and longer bubble residence times that result in higher grades and recoveries while reducing the space requirements in a plant layout and the associated capital and operating costs. However, despite the progress made in flotation physics throughout the years, its complex nature has forced many controloriented modelling efforts to rely on empirical or mixed approaches with limited generalizability for a broad range of operating conditions. In contrast, while the use of high spatial resolution methods has improved our comprehension of flotation physics at both microscopic and macroscopic levels, the computational cost constitutes a drawback when developing tools for monitoring and control purposes. In view of this, a rework of a fundamental model originally developed in our group to describe a single batch flotation cell into a spatially-finite framework is proposed. Such rework features the use of discrete bubble size classes and population balance equations to track the evolution of gas holdup in the pulp and froth zones, and combines this discretized scheme with first-order attachment and detachment mechanisms to describe bubble-particle interactions as a function of time and height. Training data gathered in-situ from experiments using a conductivity probe inserted in the pulp zone, a machine vision-based measurement system (VisioFroth™) mounted on top of the froth zone, and experimental protocols to measure the cumulative cumulative solids recovery were used to train the model parameters in the pulp, interface and froth zones. Once trained, the predictive capability of the model was qualitatively assessed by carrying out parametric studies using two independent operating variables: the air inlet flow rate and the stirring rate. Subsequently, quantitative assessments were conducted by contrasting model predictions with testing data, both under operating conditions different to those of the training data. These procedures were conducted using experimental data gathered under a hybrid operation mode, where agitation was provided by a mechanical impeller inside the JKTech cell located at the base of the column flotation device. The parametric fitting revealed that the bubble-particle attachment phenomena were considered the driving force in the modelling of solids recovery, with a reasonable match between model trends and training experimental data (SSE for cumulative solids recovery of 690.4, 289.6 and 117.9 in the experimental datasets HY 900RPM - 5LMIN, HY 900RPM 11LMIN 200PPM and HY 1500RPM 11LMIN, respectively). The parametric studies and the Normalized Forward Sensitivity Index analysis revealed that the solids recovery predictions made by the trained model had a higher sensitivity to variations in the air inlet flow rate, which was evidenced in the predictive studies with opposing performances in the prediction of the testing experimental data (SSE for cumulative solids recovery of 102.8 and 3909.5 in the experimental datasets HY 900RPM 11LMIN and HY 1500RPM 5LMIN, respectively) despite the acceptable performance of the trained model in the prediction of the bubble size distribution dynamics in both zones. The proposed strategies to improve the predictive capability of the model included the retraining of the model under higher training and testing data volumes, the use of alternative data split ratios and the use of estimation methods to refine the accuracy of the predictions in real time.