Learning rate and outlier analysis of linear learning algorithms

<p><b>Metric Learning has proved valuable in information retrieval and classification problems, with many algorithms using statistical formulations to their advantage. Optimal Transport, a powerful framework for computing distances between probability distributions, has seen significant uptake in Machine Learning and some applications in Metric Learning. The Bures-Wasserstein distance, a particular formulation of Optimal Transport, has seen limited application in Metric Learning despite being well suited to the field. In this thesis Linear Metric Learning algorithms incorporating the Bures-Wasserstein distance are developed, utilising its novel properties. The first set of algorithms we contribute use the Bures-Wasserstein distance for learning linear metrics using Cyclic Projections, Riemannian barycenters and gradient-based algorithms. The convergence properties and classification performance of these algorithms are then compared to benchmark Linear Metric Learning algorithms. Our algorithms achieve competitive classification performance on feature and image datasets and converge reliability.</b></p> <p>The second set of contributions extend Linear Metric Learning with the Bures-Wasserstein distance to a low-rank context to address issues related to scalability. An extension of the previous Cyclic Projections algorithm and a Riemannian optimisation method are developed. Convergence and classification experiments analogous to the full-rank case are conducted to examine how the reduced rank affects both of these performance metrics. Both algorithms converge reliably under a variety of synthetic conditions. Reduction in rank generally decreased the training time for algorithms until effects caused by the size of the training data dominated. Reducing rank either left classification error unchanged or increased it. With respect to other low-rank Linear Metric Learning algorithms our algorithms again performed competitively with respect to classification error. </p> <p>The last set of contributions consider Linear Metric Learning on Symmetric Positive Definite (SPD) matrices. Some datasets are expressed as or are suited to a SPD representation. As such, Metric Learning algorithms that respect the structure of these matrices have found success when adapted to this kind of data. Another extension of the Cyclic Projections algorithm is developed to use the Generalised Bures-Wasserstein distance on SPD matrices. Two other methods based on a ground metric formulation of the Bures-Wasserstein distance for learning linear metrics on SPD data are also introduced. One ground metric formulation uses an approximation (inspired by the Sinkhorn distance) to increase its scalability. Convergence experiments on synthetic problems adapted to SPD matrices are conducted on the ground metric algorithms to establish their convergence properties. Classification experiments are also conducted with respect to several types of SPD data. While the Cyclic Projections method is competitive, the ground metric algorithms have poor performance. Despite their poor classification performance, the approximation method does evaluate faster than the Generalised Bures-Wasserstein distance, potentially offering some scalability improvements.</p>

Speller brain-computer interface (BCI) systems can help neuromuscular disorders patients write their thoughts by using the electroencephalogram (EEG) signals by just focusing on the speller tasks. For practical speller-based BCI systems, the P300 event-related brain potential is measured by using the EEG signal. In this paper, we design a robust machine-learning algorithm for P300 target detection. The novel spatial-temporal linear feature learning (STLFL) algorithm is proposed to extract high-level P300 features. The STLFL method is a modified linear discriminant analysis technique focusing on the spatial-temporal aspects of information extraction. A new P300 detection structure is then proposed based on the combination of the novel STLFL feature extraction and discriminative restricted Boltzmann machine (DRBM) for the classification approach (STLFL + DRBM). The effectiveness of the proposed technique is evaluated using two state-of-the-art P300 BCI datasets. Across the two available databases, we show that in terms of average target recognition accuracy and standard deviation values, the proposed STLFL + DRBM method outperforms traditional methods by 33.5, 78.5, 93.5, and 98.5% for 1, 5, 10, and 15 repetitions, respectively, in BCI competition III datasets II and by 71.3, 100, 100, and 100% for 1, 5, 10, and 15 repetitions, respectively, in BCI competition II datasets II and by 67.5 ± 4, 84.2 ± 2.5, 93.5 ± 1, 96.3 ± 1, and 98.4 ± 0.5% for rapid serial visual presentation (RSVP) based dataset in repetitions 1–5. The method has some advantages over the existing variants including its efficiency, robustness with a small number of training samples, and a high ability to create discriminative features between classes.

An unsupervised perceptron algorithm and several generalizations are presented in this paper. Based on stochastic approximation theory, some general analysis of neural network learning algorithms is provided. Also, the definitions of convergence speed and robustness of a learning algorithm are given. It is shown that the unsupervised perceptron algorithms converge to the principal component of the input data under some conditions. In addition, the convergence speeds and robustness of the unsupervised perceptrons, the Oja (1982, 1983) and the Widrow-Hoff algorithms are given in explicit forms.

SummaryThis paper describes a learning parallel constraint programming (CP) solver designed for solving CP problems with several instances on massively parallel computing platforms comprising multi‐core parallel machines or Many Integrated Cores. The CP solver proposed in this work is based on aPortfolioparallelization that employs a linear reward inaction learning algorithm in order to obtain the best possible performance for a large set of instances of the same problem. The linear reward inaction algorithm enables the prediction of the number of cores to be assigned to each search strategy based on previous experiments, reducing the computing time required to solve constraint satisfaction and optimization problems. The underlying principle of thePortfolioapproach is to runNsequential search strategies usingNcomputing cores (NToNPortfolio) where each core uses its own strategy in order to perform a search that is different from strategies used by the other cores. The first strategy that finds a solution stops all other strategies. The problem with theNToNPortfolioapproach is that the number of search strategies is very small compared with the current number of computing cores used by the parallel machines. However, using an internal parallelization for each search strategy, it is possible to runNparallel search usingPcomputing cores withP≫N(NToPPortfolio). ThisNToPPortfolioperforms suboptimally for solving different CP problems because many computing resources are wasted. To improve thisPortfoliomodel, an adaptiveNToPPortfoliowas proposed, which tries to privilege the strategy that is most likely to find a solution first in order to give it more computing cores than the other strategies. However, the main problem with the adaptivePortfoliois that it loses all the learned information at the end of each search; it is designed to solve just one CP problem. Furthermore, many computational resources are wasted by bothPortfoliosolvers, the non‐adaptive (NToNandNToP) and the adaptiveNToPPortfolio, when employed in some industrial projects, such as thePAJEROproject, which always solves different instances of the same CP problem. To minimize the amount of wasted resources and to learn the most efficient search strategies, we propose a new learningPortfoliosolver that uses a learning algorithm that configures automatically number of cores to each search. The performance obtained using the differentPortfoliosolvers is compared and illustrated by solving CP problems using as example the Google OR‐Tools solver. Copyright © 2016 John Wiley & Sons, Ltd.

<p>Information processing requires handwritten digit recognition, but methods of writing and image defects like brightness changes, blurring, and noise make image recognition challenging. This paper presents a strategy for categorizing offline handwritten digits in both Devanagari script and Roman script (English numbers) using Deep Learning (DL) algorithms, a branch of Machine Learning (ML) that uses Neural Networks (NN) with multiple layers to acquire hierarchical representations of input autonomously. The research study develops classification algorithms for recognising handwritten digits in numerical characters (0–9), analyzing classifier combination approaches, and determining their accuracy. The study aims to optimize recognition results when working with multiple scripts simultaneously. It proposes a simple profiling technique, Linear Discriminant Analysis (LDA) implementation, and a NN structure for numerical character classification. However, testing shows inconsistent outcomes from the LDA classifier. The approach, which combines profile-based Feature Extraction (FE) with advanced classification algorithms, can significantly improve the field of HWR numerical characters, as evidenced by the diverse outcomes it produces. The model performed 98.98% on the MNIST dataset. In the CPAR database, we completed a cross-dataset evaluation with 98.19% accuracy.</p>

Kernel methods for classification is a well-studied area in which data are implicitly mapped from a lower-dimensional space to a higher dimensional space to improve classification accuracy. However, for most kernel methods, one must still choose a kernel to use for the problem. Since there is, in general, no way of knowing which kernel is the best, multiple kernel learning (MKL) is a technique used to learn the aggregation of a set of valid kernels into a single (ideally) superior kernel. The aggregation can be done using weighted sums of the precomputed kernels, but determining the summation weights is not a trivial task. Furthermore, MKL does not work well with large datasets because of limited storage space and prediction speed. In this paper, we address all three of these multiple kernel challenges. First, we introduce a new linear feature level fusion technique and learning algorithm, GAMKLp. Second, we put forth three new algorithms, DeFIMKL, DeGAMKL, and DeLSMKL, for nonlinear fusion of kernels at the decision level. To address MKL's storage and speed drawbacks, we apply the Nystrom approximation to the kernel matrices. We compare our methods to a successful and state-of-the-art technique called MKL-group lasso (MKLGL), and experiments on several benchmark datasets show that some of our proposed algorithms outperform MKLGL when applied to support vector machine (SVM)-based classification. However, to no surprise, there does not seem to be a global winner but instead different strategies that a user can employ. Experiments with our kernel approximation method show that we can routinely discard most of the training data and at least double prediction speed without sacrificing classification accuracy. These results suggest that MKL-based classification techniques can be applied to big data efficiently, which is confirmed by an experiment using a large dataset.

The paper presents a compositional hybrid fuzzy–neural cognitive model designed to support intelligent decision-making in two critical application domains: (i) fault diagnosis of wind turbine components and (ii) operational switching control in electrical substations. The proposed model integrates fuzzy cognitive maps with neural learning algorithms, thereby ensuring high interpretability, robustness to exogenous disturbances, and suitability for real-time deployment within a problem-oriented intelligent decision support system (IDSS). The objective of the study is to evaluate the applicability and methodological advantages of the compositional model as a functional element of a real-time IDSS architecture for the diagnosis and control of complex technical systems (CTS) in the power industry. The considered CTS include wind turbine assemblies, where timely fault detection and location are required, as well as substation equipment, where operational switching must be executed under strict safety and reliability constraints. The research outcomes comprise the development of a multilayer interpretable model based on fuzzy cognitive maps and augmented with adaptive linear and bilinear learning algorithms. The model enables continuous online adaptation, captures nonlinear dependencies among system variables, and explicitly incorporates external factors into the decision-making process. Experimental evaluation using real and synthetic datasets, including wind turbine condition-monitoring data and the IEEE 118-bus benchmark, demonstrates that the proposed model outperforms state-of-the-art approaches (LFCM, RNN, CNN) in diagnostic accuracy, decision latency, and computational efficiency. The practical significance of the study lies in enhancing the reliability and operational resilience of IDSS for managing and diagnosing CTS under uncertainty. The model supports the ongoing digitalization of power-system operations and contributes to reducing failure risks and economic losses by enabling timely, context-aware decision support in real-time environments.

We propose a novel algorithm for the task of supervised discriminative distance learning by nonlinearly embedding vectors into a low dimensional Euclidean space. We work in the challenging setting where supervision is with constraints on similar and dissimilar pairs while training. The proposed method is derived by an approximate kernelization of a linear Mahalanobis-like distance metric learning algorithm and can also be seen as a kernel neural network. The number of model parameters and test time evaluation complexity of the proposed method are O(dD) where D is the dimensionality of the input features and d is the dimension of the projection space -- this is in contrast to the usual kernelization methods as, unlike them, the complexity does not scale linearly with the number of training examples. We propose a stochastic gradient based learning algorithm which makes the method scalable (w.r.t. the number of training examples), while being nonlinear. We train the method with up to half a million training pairs of 4096 dimensional CNN features. We give empirical comparisons with relevant baselines on seven challenging datasets for the task of low dimensional semantic category based image retrieval.

Internet and computer systems are an indispensable part of daily life. The number of web applications have increased with the development of technology and digital transformation. Web applications have high risk for security since the applications is not developed securely, contains vulnerabilities and easily accessible by hackers. Web application firewall is used to protect web applications from attacks. Signature-based and anomaly-based methods are used in web application firewalls. In this research, anomaly-based web application firewall model was developed using natural language processing techniques and linear support vector machine learning algorithm. Word n-gram and character n-gram natural language processing techniques were compared by performing separate models. Proposed model achieve higher detection performance with using the character n-gram compared to other studies. According to the results of the experiment proposed model is capable of detection web attacks effectively with the overall detection accuracy rate of 99.53%.

Biofilms are intricate bacterial communities encased in a self-produced extracellular matrix (ECM) of DNA, lipids, proteins, and polysaccharides. The diverse ECM composition across bacterial species significantly influences the progression of biofilm-associated infections, making precise identification crucial for effective treatment. Traditional methods such as biochemical assays, MALDI-TOF mass spectrometry, DNA sequencing and culturing provide valuable insights but have notable drawbacks, including time-consuming procedures, high costs, and the need for specialized equipment and trained personnel. These limitations hinder the rapid and widespread adoption of biofilm identification in clinical settings, underscoring the need for more streamlined, accurate, and accessible methods. In this study, we employed a paper-based ratiometric sensor array with fluorescent dyes (3-hydroxyflavone derivatives) pre-adsorbed onto paper microzone plates to identify bacterial biofilms. The fluorescence signals from the sensor upon interaction with biofilms were analyzed using linear discriminant analysis and different machine learning algorithms, including neural networks, support vector machines, and naïve Bayes classifiers. Our results show that the sensor array accurately distinguishes between biofilms of eight species with 97.5% classification accuracy. It effectively identifies individual bacteria at OD600 as low as 0.002 o.u. Additionally, using neural networks, the sensor array achieves more than 95% accuracy in distinguishing planktonic bacteria from biofilms and shows over 85% accuracy in identifying clinical bacterial species and biofilms. These findings highlight the sensor's potential for high-precision biofilm identification in laboratory and clinical settings, offering a valuable tool for advancing biofilm research and enhancing clinical diagnostics.

Background: Increasing youths’ physical activity is mandatory to reduce the risk of non-communicable diseases (NCCDs). Basketball is a team sport that is potentially positive in increasing teenagers’ physical performance, health indicators, and well-being. Objective: The objective was to compare the physical activity level (PAL), body composition, strength, and flexibility of teen male basketball players (BG) (n = 15) and adolescent non-practitioners of sport (NS: n = 14). Methodology: All participants were healthy and free from any health disability from a Brazilian high school. A linear regression machine learning algorithm was applied to predict the adolescent´s physical components. In a quasi-experimental analysis, data were extracted by PAL, body fat percentage (BF%), handgrip strength (HG), back extensor muscle’s’ strength (BMS), lower limb power (LLP), and lower limb flexibility (LLF). Parametric (independent T-test) and non-parametric (Mann-Whitney U test) were employed to compare the variable’s average and chi-square was applied to compare categorical data. Results: BG presented an upper number of adolescents classified with high PAL than the NS group (p = 0.0002, large ES, V = 0.73) and a lower number of adolescents classified with low PAL than the NS group (p = 0.0002, V = 0.73), less BF% (p = 0.02, r = 0.85), greater values of HGS (p = 0.005, r = 0.34), greater values of BMSLS (p = 0.005, r = 0.33), greater values of LLP (p = 0.007, r = 0.30), and greater values for LLF (p = 0.02, r = 0.17). Therefore, there was a positive effect of high PAL compared with low PAL in HG, (p = 0.005, r = 0.24) and also for high PAL in LLF, (High PAL: (p = 0.006, r = 0.23). Regarding machine learning analysis, the four models (linear regression, Ridge regression, random forest regression, and Bayesian regression) expressed good generalization performance, with a coefficient of determination (R2) ranging from 0.77 to 0.88, root mean square error (RMSE) from 1.01 to 3.92, with an average mean difference of four points between the predicted and real values. The worst model was random forest regression R2 = 0.77, RMSE = 3.92, and the best model was Bayesian regression (R2 = 0.88, RMSE = 1.01). Conclusion: The BG group presented better results than the NS group for PAL, BF%, HG, BMS, LLP, and LLF. Body fat percentage precisely predicted the player’s’ vertical jump (VJ). In addition to the physical superiority of the BG, this study revealed the importance of managing body composition for both health and performance improvements.

The changes in the housing market are not only related to human beings’ daily life, but also have an important impact on the national economy. The prediction of housing price is one of the most widely concerned topics, which is linked to the formulation of national real estate policies and the analysis of the economic situation. In this context, this paper takes housing price prediction as the topic, selects the Eames housing price dataset in Iowa, and uses supervised multiple linear regression and machine learning algorithm to train and test the real estate price prediction model. Among them, there are 79 explanatory variables, which are related to housing attributes and the explanatory variable is housing price. 1460 data is included in the training set and 1459 in the test set. In the part of machine learning algorithm, PaddlePaddle deep learning framework is used in this paper to train and test the models with the help of AI Studio platform. The experimental results show that the scatter plots of the real values are clustered and distributed on both sides of the predicted line, and their direct differences are within 30 points. According to the analysis, the real estate price prediction model based on linear regression and machine learning is reliable and stable. This paper aims to provide some suggestions for the housing price prediction. These results shed light on guiding the reference direction for investors, so as to guide the formulation of relevant policies.

Feature selection is an essential step in the development of empirical flood damage models based on machine learning techniques. So far, most models of this type were developed using data from a single region or country, and few of them utilize harmonized transboundary datasets. Here, we have harmonized 38 variables present in the datasets of three flood damage surveys conducted in Germany (n = 516), the Netherlands (n = 409) and Belgium (n = 320) after the 2021 mega-floods in Europe. After performing data imputation and multicollinearity check, we used linear and non-linear machine learning algorithms to assess permutation importance and identify features most influencing flood damage. The results of the four models suggest that besides the hazard variables such as water depth and human stability, the location of the heating system (in the basement or at a higher floor) appears among the topmost important features for both building and contents damage.Subsequently, we did an analysis for a low and high range of water depths using the median value (0.6 m) as splitting criteria. In the lower range, for both types of damage, water depth appears to be the dominating driver, and specifically for the building damage, it exceeds by far the importance of any other variable. In contrast, for water depths above 0.6 m other factors outweigh water depth. In the case of content damage, building footprint area becomes the most important factor across all the models. For the building damage some hazard (e.g. human stability), exposure (e.g. building size) and vulnerability (e.g. hazard knowledge) variables have a comparable importance with that of water depth. Hence, our results show that multivariable models appear particularly necessary for modelling flood damage induced by high and extreme hazard conditions.

The Algerian automobile insurance market faces significant challenges in pricing insurance policies due to the lack of reliable predictions for insurance losses. In this paper, we introduce a new ratemaking system that leverages advanced data analysis techniques, including Generalized Linear Models and machine learning algorithms like Neural Networks, boosting, and stacking algorithms, to model claims frequency. By analyzing data and statistics of drivers in the Algerian market, this system offers a data-driven solution that helps insurers to better understand their risk exposure and make informed pricing decisions. The proposed system has implications for both insurers and policyholders in terms of fairer and more accurate pricing, which will ultimately benefit the Algerian economy.

Due to a growing need in visualizing human pluripotent stem cell-derived organoids from recent advancements in the field, an efficient bulk-processing application is necessary to provide preprocessing and image analysis services. In this study, we developed Organalysis, a high-accuracy, multifunctional, and accessible application that meets these needs by providing the functionality of image manipulation and enhancement, organoid area and intensity calculation, fractal analysis, noise removal, and feature importance computation. The image manipulation feature includes brightness and contrast adjustment. The area and intensity calculation computes six values for each image: organoid area, total image area, percentage of the image covered by organoid, the total intensity of organoid, the total intensity of organoid-by-organoid area, and total intensity of organoid by total image area. The fractal analysis function computes the fractal dimension value for each image. The noise removal function removes superfluous marks from the input images, such as bubbles and other unwanted noise. The feature importance function trains a lasso-regularized linear regression machine learning algorithm to identify cardiac growth factors that are the strongest determinants for cell differentiation. The batch processing of this application further builds on existing services like ImageJ to provide a more convenient way to process multiple images. Collectively, the versatility and preciseness of Organalysis demonstrate novelty, since no other current imaging software combines the capability of batch processing and the breadth of feature analysis. Therefore, Organalysis provides unique functions in cardiac organoid research and proves to be invaluable in regenerative medicine.