Specialization in Hierarchical Learning Systems

We consider the problem of nonparametric estimation of unknown smooth functions in the presence of restrictions on the shape of the estimator and on its support using polynomial splines. We provide a general computational framework that treats these estimation problems in a unified manner, without the limitations of the existing methods. Applications of our approach include computing optimal spline estimators for regression, density estimation, and arrival rate estimation problems in the presence of various shape constraints. Our approach can also handle multiple simultaneous shape constraints. The approach is based on a characterization of nonnegative polynomials that leads to semidefinite programming (SDP) and second-order cone programming (SOCP) formulations of the problems. These formulations extend and generalize a number of previous approaches in the literature, including those with piecewise linear and B-spline estimators. We also consider a simpler approach in which nonnegative splines are approximated by splines whose pieces are polynomials with nonnegative coefficients in a nonnegative basis. A condition is presented to test whether a given nonnegative basis gives rise to a spline cone that is dense in the space of nonnegative continuous functions. The optimization models formulated in the article are solvable with minimal running time using off-the-shelf software. We provide numerical illustrations for density estimation and regression problems. These examples show that the proposed approach requires minimal computational time, and that the estimators obtained using our approach often match and frequently outperform kernel methods and spline smoothing without shape constraints. Supplementary materials for this article are provided online.

Introduction: Achievement of specified qualitative indicators in machine learning solutions depends not only on the efficiency of algorithms, but also on data properties. One of the lines for the development of classification and regression models is the specification of local properties of data. Purpose: To improve the qualitative predictors when solving classification and regression problems based on the adaptive selection of various machine learning models on separate local segments of data sample. Results: We propose a method that uses a combination of different models and machine learning algorithms on subsamples in regression and classification problems. The method is based on the calculation of qualitative predictors and the selection of the best models on the local segments of data sample. The finding of transformations of data and time series allows to create sample sets, with the data having different properties (for example, variance, sampling fraction, data range, etc.). We consider the data segmentation based on the change point detection algorithm in time series trends and on analytical information. On the example of the real dataset, we show the experimental values of the loss function for the proposed method with different classifiers on separate segments and on the whole sample. Practical relevance: The results can be used in classification and regression problems for the development of machine learning models and methods. The proposed method allows to improve classification and regression qualitative predictors by assigning models that have the best performance on separate segments.

Neuroevolution is a powerful and general technique for evolving the structure and weights of artificial neural networks. Though neuroevolutionary approaches such as NeuroEvolution of Augmenting Topologies (NEAT) have been successfully applied to various problems including classification, regression, and reinforcement learning problems, little work has explored application of these techniques to larger-scale multiclass classification problems. In this paper, NEAT is evaluated in several multiclass classification problems, and then extended via two ensemble approaches: One-vs-All and One-vs-One. These approaches decompose multiclass classification problems into a set of binary classification problems, in which each binary problem is solved by an instance of NEAT. These ensemble models exhibit reduced variance and increasingly superior accuracy as the number of classes increases. Additionally, higher accuracy is achieved early in training, even when artificially constrained for the sake of fair comparison with standard NEAT. However, because the approach can be trivially distributed, it can be applied quickly at large scale to solve real problems. In fact, these approaches are incorporated into DarwinTM, an enterprise automatic machine learning solution that also incorporates various other algorithmic enhancements to NEAT. The resulting complete system has proven robust to a wide variety of client datasets.

Multivariate density estimation is a fundamental problem in Applied Statistics and Machine Learning. Given a collection of data sampled from an unknown distribution, the task is to approximately reconstruct the generative distribution. There are two different approaches to the problem, the parametric approach and the non-parametric approach. In the parametric approach, the approximate distribution is represented by a model from a predetermined family. In this thesis, we adopt the parametric approach and investigate the use of a model family called latent tree models for the task of density estimation. Latent tree models are tree-structured Bayesian networks in which leaf nodes represent observed variables, while internal nodes represent hidden variables. Such models can represent complex relationships among observed variables, and in the meantime, admit efficient inference among them. Consequently, they are a desirable tool for density estimation. While latent tree models are studied for the first time in this thesis for the purpose of density estimation, they have been investigated earlier for clustering and latent structure discovery. Several algorithms for learning latent tree models have been proposed. The state-of-the-art is an algorithm called EAST. EAST determines model structures through principled and systematic search, and determines model parameters using the EM algorithm. It has been shown to be capable of achieving good trade-off between fit to data and model complexity. It is also capable of discovering latent structures behind data. Unfortunately, it has a high computational complexity, which limits its applicability to density estimation problems. In this thesis, we propose two latent tree model learning algorithms specifically for density estimation. The two algorithms have distinct characteristics and are suitable for different applications. The first algorithm is called HCL. HCL assumes a predetermined bound on model complexity and restricts to binary model structures. It first builds a binary tree structure based on mutual information and then runs the EM algorithm once on the resulting structure to determine the parameters. As such, it is efficient and can deal with large applications. The second algorithm is called Pyramid. Pyramid does not assume predetermined bounds on model complexity and does not restrict to binary tree structures. It builds model structures using heuristics based on mutual information and local search. It is slower than HCL. However, it is faster than EAST and is only slightly inferior to EAST in terms of the quality of the resulting models. In this thesis, we also study two applications of the density estimation techniques that we develop. The first application is to approximate probabilistic inference in Bayesian networks. A Bayesian network represents a joint distribution over a set of random variables. It often happens that the network structure is very complex and making inference directly on the network is computational intractable. We propose to approximate the joint distribution using a latent tree model and exploit the latent tree model for faster inference. The idea is to sample data from the Bayesian network, learn a latent tree model from the data offline, and when online, make inference with the latent tree model instead of the original Bayesian network. HCL is used here because the sample size needs to be large to produce accurate approximation and it is possible to predetermine a bound on the online running. Empirical evidence shows that this method can achieve good approximation accuracy at low online computational cost. The second application is classification. A common approach to this task is to formulate it as a density estimation problem: One constructs the class-conditional density for each class and then uses the Bayes rule to make classification. We propose to estimate those class-conditional densities using either EAST or Pyramid. Empiricalevidence shows that this method yields good classification performances. Moreover, the latent tree models built for the class-conditional densities are often meaningful, which is conducive to user confidence. A comparison between EAST and Pyramid reveals that Pyramid is significantly more efficient than EAST, while it results in more or less the same classification performance as the latter.

A learning strategy in Learning Classifier Systems (LCSs) defines how classifiers cover a state-action space in a problem. Previous analyses in classification problems have empirically claimed an adequate learning strategy can be decided depending on the types of noise in the problem. This issue is still arguable from two aspects. First, there lacks comparison of learning strategies in reinforcement learning problems with different types of noise. Second, when we can claim so, a further issue is how should classifiers cover the state-action space in order to improve the stability of LCS performance on as many types of noise as possible? This paper first attempts to empirically conclude these issues on a version of LCSs (i.e., the XCS classifier system). That is, we present a new concept of learning strategy for LCSs, and complement that claim by comparing it with the existing learning strategies on a reinforcement learning problem. Our learning strategy covers all state-action pairs but assigns more classifiers to the highest-return action at each state than other actions. Our results support that claim that existing learning strategies have dependencies on the types of noise in reinforcement learning problems. However, our learning strategy improves the stability of XCS performance compared with the existing strategies on all types of noise employed in this paper.

This research aims to improve quality indicators in solving classification and regression problems based on the adaptive selection of various machine learning models on separate data samples from local segments. The proposed method combines different models and machine learning algorithms on individual subsamples in regression and classification problems based on calculating qualitative indicators and selecting the best models on local sample segments. Detecting data changes and time sequences makes it possible to form samples where the data have different properties (for example, variance, sample fraction, data span, and others). Data segmentation is used to search for trend changes in an algorithm for points in a time series and to provide analytical information. The experiment performance used actual data samples and, as a result, obtained experimental values of the loss function for various classifiers on individual segments and the entire sample. In terms of practical novelty, it is possible to use the obtained results to increase quality indicators in classification and regression problem solutions while developing models and machine learning methods. The proposed method makes it possible to increase classification quality indicators (F-measure, Accuracy, AUC) and forecasting (RMSE) by 1%–8% on average due to segmentation and the assignment of models with the best performance in individual segments. Doi: 10.28991/ESJ-2023-07-03-03 Full Text: PDF

In this paper, we study the problem of density estimation from data that contains small measurement errors. The only assumption on these errors is that the maximal measurement error is bounded by some real number converging to zero for sample size tending to infinity. In particular, we do not assume that the measurement errors are independent with expectation zero. We estimate the density by a standard kernel density estimate applied to data with measurement errors and derive a data-driven method to choose its bandwidth. We derive an adaptation result for this estimate and analyze the expected L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> error of our density estimate depending on the smoothness of the density and the size of the maximal measurement error. The results are applied in a density estimation problem in a simulation model, where we show under suitable assumptions that the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> error of our newly proposed estimate converges to zero much faster than the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> error of the standard kernel density estimate if both are based on the same number of observations in the simulation model. The performance of the method in case of finite sample size is analyzed using simulated data.

Problems with censored data arise frequently in survival analyses and reliability applications. The estimation of the density function of the failure times is often of interest. Two inherent problems in density estimation for lifetime data are the spill-over at the origin and the smoothing parameter selection. To address these issues, we propose the use of asymmetric kernels (like inverse Gaussian) with bandwidths selected by a Bayes criterion. We show strong pointwise consistency of the density estimator, and for suitable choices of the prior, we show that one can obtain meaningful bandwidths with the same rates of convergence as for the classical asymptotically optimal bandwidths.

A fast training algorithm for extreme learning machine based on matrix decomposition

This paper considers the problem of nonparametric deconvolution density estimation when sample observa-tions are contaminated by double exponentially distributed errors. Three different deconvolution density estima-tors are introduced: a weighted kernel density estimator, a kernel density estimator based on the support vector regression method in a RKHS, and a classical kernel density estimator. The performance of these deconvolution density estimators is compared by means of a simulation study.

Cytotoxicity of the CD37 antibody BI 836826 against chronic lymphocytic leukaemia cells in combination with chemotherapeutic agents or PI3K inhibitors.

We deal with deconvolution problems in density estimation. Assume that the data follow a density, which is a convolution of the original density f being of interest with a noise density f ϵ. In order to estimate the density f, one usually should know f ϵ completely and then uses some technique for deconvolution. In contrast, the so-called TAYLEX and SIMEX methods introduced by Carroll and Hall and Cook and Stefanski, respectively use partial information on f ϵ only and correct the naive density estimator towards the deconvoluted one. In the present paper, we assume that we have more and more information on the noise density when the sample size increases. We show that by applying these methods, one can achieve almost optimal rates and optimal rates respectively for densities f belonging to certain Sobolev classes.

Density estimation in the presence of noise

Bandwidth selection is the main problem of kernel density estimation, the most popular method of density estimation. The classical normal reference bandwidth usually oversmoothes the density estimate. The existing hi‐tech bandwidths have computational problems (even may not exist) and are not robust against outliers in the sample. A highly robust normal reference bandwidth is proposed, which adapts to different types of densities.

Convolutional Neural Networks have been widely used in remote sensing scene classification. Since this kind of model needs a large number of training samples containing data category information, a Generative Adversarial Network (GAN) is usually used to address the problem of lack of samples. However, GAN mainly generates scene data samples that do not contain category information. To address this problem, a novel supervised adversarial Lie Group feature learning network is proposed. In the case of limited data samples, the model can effectively generate data samples with category information. There are two main differences between our method and the traditional GAN. First, our model takes category information and data samples as the input of the model and optimizes the constraint of category information in the loss function, so that data samples containing category information can be generated. Secondly, the object scale sample generation strategy is introduced, which can generate data samples of different scales and ensure that the generated data samples contain richer feature information. After large-scale experiments on two publicly available and challenging datasets, it is found that our method can achieve better scene classification accuracy even with limited data samples.