Sparse Boosting Based Machine Learning Methods for High-Dimensional Data

There are a large number of high-dimensional data in the Internet of Brain Things, and the data is reduced from the high-dimensional to low-dimensional to maintain the similarity between the data, thereby effectively ensuring the operating speed of the Internet of Brain Things. In the traditional method, the distributed dimensionality reduction reconstruction algorithm of the High dimensional data has poor dimensionality reduction effect and serious data distortion after reconstruction. A method for dimensionality reconstruction of high-dimensional data in the Internet of Brain Things is proposed. By using the algorithm of linear discriminant analysis, the projection matrix of high dimensional data is constructed to solve it. According to the solution results, using improved implicit variable model to establish a high dimensional large data dimensionality reduction model for the Internet of Brain Things. The fitness value of data after dimensionality reduction is calculated by quantum immune clonal algorithm, and the optimal individual and optimal solution are determined. The data reconfiguration is realized through the optimal solution marshalling. Experimental results show that the proposed algorithm can effectively improve the dimensionality reduction of High dimensional data in the Internet of Brain Things. After reconstruction, the reconstructed data retain accurate data information, the reliability of reconstructed data is high, and the computational complexity is not high, the need for small storage space, and the advantages of strong promotion ability.

<p style='text-indent:20px;'>Variable selection and parameter estimation are fundamental and important problems in high dimensional data analysis. In this paper, we employ the hard thresholding regularization method [<xref ref-type="bibr" rid="b1">1</xref>] to handle these issues under the framework of high-dimensional and sparse linear regression model. Theoretically, we establish a sharp non-asymptotic estimation error for the global solution and further show that the support of the global solution coincides with the target support with high probability. Motivated by the KKT condition, we propose a primal dual active set algorithm (PDAS) to solve the minimization problem, and show that the proposed PDAS algorithm is essentially a generalized Newton method, which guarantees that the proposed PDAS algorithm will converge fast if a good initial value is provided. Furthermore, we propose a sequential version of the PDAS algorithm (SPDAS) with a warm-start strategy to choose the initial value adaptively. The most significant advantage of the proposed procedure is its fast calculation speed. Extensive numerical studies demonstrate that the proposed method performs well on variable selection and estimation accuracy. It has favorable exhibition over the existing methods in terms of computational speed. As an illustration, we apply the proposed method to a breast cancer gene expression data set.</p>

The Birnbaum–Saunders (BS) distribution is broadly used to model failure times in reliability and survival analysis. In this article, we propose a simultaneous parameter estimation and variable selection procedure in a log–linear BS regression model for high-dimensional survival data. To deal with censored survival data, we iteratively run a combination of the stochastic EM algorithm (SEM) and variable selection procedure to generate pseudo-complete data and select variables until convergence. Treating pseudo-complete data as uncensored data via SEM makes it possible to incorporate iterative penalized least squares and simplify computation. We demonstrate the efficacy of our method using simulated and real datasets.

High dimension and complexity of network high-dimensional data lead to poor feature selection effect network high-dimensional data. To effectively solve this problem, feature selection algorithms for high-dimensional network data based on supervised discriminant projection (SDP) have been designed. The sparse representation problem of high-dimensional network data is transformed into an Lp norm optimization problem, and the sparse subspace clustering method is used to cluster high-dimensional network data. Dimensionless processing is carried out for the clustering processing results. Based on the linear projection matrix and the best transformation matrix, the dimensionless processing results are reduced by combining the SDP. The sparse constraint method is used to achieve feature selection of high-dimensional data in the network, and the relevant feature selection results are obtained. The experimental findings demonstrate that the suggested algorithm can effectively cluster seven different types of data and converges when the number of iterations approaches 24. The F1 value, recall, and precision are all kept at high levels. High-dimensional network data feature selection accuracy on average is 96.9%, and feature selection time on average is 65.1 milliseconds. The selection effect for network high-dimensional data features is good.

In high dimensional setting, componentwise L 2boosting method has been used to construct sparse model of high prediction, but it tends to select many ineffective variables. Several sparse boosting methods, such as, SparseL 2Boosting and Twin Boosting, have been proposed to improve the variable selection of L 2boosting algorithm. In this paper, we propose a new general sparse boosting method (GSBoosting). The relations are established between GSBoosting and other well known regularized variable selection methods in orthogonal linear model, such as adaptive Lasso, hard thresholds etc. Simulations results show that GSBoosting has good performance in both prediction and variable selection.

We consider the problem of simultaneous variable selection and parameter estimation in an ultra-high dimensional linear regression model. The adaptive penalty functions are used in this regard to achieve the oracle variable selection property with simpler assumptions and lesser computational burden. Noting the non-robust nature of the usual adaptive procedures (e.g. adaptive LASSO) based on the squared error loss function against data contamination, quite frequent with modern large-scale data sets (e.g. noisy gene expression data, spectra and spectral data), in this paper, we present a new adaptive regularization procedure using a robust loss function based on the density power divergence (DPD) measure under a general class of error distributions. We theoretically prove that the proposed adaptive DPD-LASSO estimator of the regression coefficients is highly robust, consistent, asymptotically normal and leads to robust oracle-consistent variable selection under easily verifiable assumptions. Numerical illustrations are provided for the mostly used normal and heavy-tailed error densities. Finally, the proposal is applied to analyse an interesting spectral dataset, in the field of chemometrics, regarding the electron-probe X-ray microanalysis (EPXMA) of archaeological glass vessels from the 16th and 17th centuries.

Independent Component Analysis (ICA) has emerged as a useful method for separation of components, such as in removing noise from data. We examine one of the challenges of ICA - instability, particularly in high dimensions, when the independent components vary, each time when ICA is performed. This may be due to various causes including the stochastic nature of the algorithm and the additive noise. The objective of this study is to examine denoising and stability issues of ICA in high dimensions and make a comparative evaluation of select approaches. We take a challenging electrocardiogram dataset which is a high-dimensional time series of multiple sensors. We experiment with a mix of approaches and methods – for resampling, clustering, ICA algorithms and dimensionality. We check the internal validity using the Icasso stability index, the Amari separation performance index and the Minimum Distance (MD) index. The first key contribution of this work is that it finds counter-evidence to the claim that resampling (bootstrapping) tackles the question of stability. The second contribution is that it finds evidence of an important limitation of the Minimum Distance index when dealing with high dimensional data – the index may become highly concentrated and may remain sub-optimal at all dimensions. Selectively removing noise components by visual inspection can improve the Amari, the Icasso or the MD index values. Some automated tools exist, but in high dimensions, visual inspection of the individual components is still required for effective denoising – data driven methods are not good enough.

Using the traditional linear model to implement variable selection can perform very effectively in some cases, provided the response to relevant components is approximately monotone and its gradient changes only slowly. In other circumstances, nonlinearity of response can result in significant vector components being overlooked. Even if good results are obtained by linear model fitting, they can sometimes be bettered by using a nonlinear approach. These circumstances can arise in practice, with real data, and they motivate alternative methodologies. We suggest an approach based on ranking generalized empirical correlations between the response variable and components of the explanatory vector. This technique is not prediction-based, and can identify variables that are influential but not explicitly part of a predictive model. We explore the method’s performance for real and simulated data, and give a theoretical argument demonstrating its validity. The method can also be used in conjunction with, rather than as an alternative to, conventional prediction-based variable selections, by providing a preliminary “massive dimension reduction” step as a prelude to using alternative techniques (e.g., the adaptive lasso) that do not always cope well with very high dimensions. Supplemental materials relating to the numerical sections of this paper are available online.

Gaussian process (GP) regression or kriging has been extensively applied in the engineering literature for the purposes of building a cheap-to-evaluate surrogate, within the contexts of multi-fidelity modeling, model calibration, and design optimization. With the ongoing automation of manufacturing and industrial practices as a part of Industry 4.0, there has been a greater need for advancing GP regression techniques to handle challenges such as high input dimensionality, data paucity or big data problems, these consist primarily of proposing efficient design of experiments, optimal data acquisition strategies, sparsifying covariance kernels, and other mathematical tricks. In this work, our attention is focused on the challenges of efficiently training a GP model, which, to the authors opinion, has attracted very little attention and is to-date poorly addressed. The performance of widely used training approaches such as maximum likelihood estimation and Markov Chain Monte Carlo (MCMC) sampling can deteriorate significantly in high-dimensional and big data problems and can lead to cost deficient implementations of critical importance to many industrial applications. Here, we compare an Adaptive Sequential Monte Carlo (ASMC) sampling algorithm to classic MCMC sampling strategies and we demonstrate the effectiveness of our implementation on several mathematical problems and challenging industry applications of varying complexity. The computational time savings of the ASMC approach manifest in large-scale problems helping us to push the boundaries of applicability and scalability of GPs for model calibration in various domains of the industry, including but not limited to design automation, design engineering, smart manufacturing, predictive maintenance, and supply chain manufacturing.

With the rapid development of hardware and software technology, modern industry has produced a large amount of high-dimensional unlabeled data, such as pictures and videos. As clusters of these data sets may exist in some subspaces, traditional algorithms are no longer applicable. The related algorithms based on sparse subspace can find clusters in the subspace, which solves the problem of high data dimension. However, their clustering process based on only one single feature, which results in their performance being particularly sensitive to this single view. Affected by the integrated algorithm, a large number of multi-view methods began to emerge. These methods improve the clustering performance by integrating the subspace expressions of multiple views, but the problem is that the complementary information of multiple views cannot be fully considered. In addition, the problem of non-uniform distribution in clusters also exists in high-dimensional data sets. In this paper, based on the multi-view subspace clustering method, a clustering algorithm based on tensor low rank expression is proposed to solve the clustering problem of high-dimensional datasets. On the one hand, this paper solves the problem of noise and data corruption by combining the method of $\ell _{2,1} $ norms, which transform the optimization problem of solving multi-view subspace expression into a low-rank expression problem of tensor to fully consider the complementarities between views. On the other hand, the proposed scheme solves the problem of non- uniform distribution in clusters in high-dimensional data by combining with the density peak algorithm based on hierarchical strategy. Experiment results show the effectiveness of the algorithm.

We introduce a fast approach to classification and clustering applicable to high-dimensional continuous data, based on Bayesian mixture models for which explicit computations are available. This permits us to treat classification and clustering in a single framework, and allows calculation of unobserved class probability. The new classifier is robust to adding noise variables as a drawback of the built-in spike-and-slab structure of the proposed Bayesian model. The usefulness of classification using our method is shown on metabololomic example, and on the Iris data with and without noise variables. Agglomerative hierarchical clustering is used to construct a dendrogram based on the posterior probabilities of particular partitions, to provide a dendrogram with a probabilistic interpretation. An extension to variable selection is proposed which summarises the importance of variables for classification or clustering and has probabilistic interpretation. Having a simple model provides estimation of the model parameters using maximum likelihood and therefore yields a fully automatic algorithm. The new clustering method is applied to metabolomic, microarray, and image data and is studied using simulated data motivated by real datasets. The computational difficulties of the new approach are discussed, solutions for algorithm acceleration are proposed, and the written computer code is briefly analysed. Simulations shows that the quality of the estimated model parameters depends on the parametric distribution assumed for effects, but after fixing the model parameters to reasonable values, the distribution of the effects influences clustering very little. Simulations confirms that the clustering algorithm and the proposed variable selection method is reliable when the model assumptions are wrong. The new approach is compared with the popular Bayesian clustering alternative, MCLUST, fitted on the principal components using two loss functions in which our proposed approach is found to be more efficient in almost every situation.

ABS TRACT This study was motivated by classification problem in Functional Magnetic Resonance Imaging (fMRI), a noninvasive imaging technique which allows an experimenter to take images of a subject’s brain over time. As fMRI studies usually have a small number of subjects and we assume that there is a smooth, underlying curve describing the observations in fMRI data, this results in incredibly high-dimensional datasets that are functional in nature. High dimensionality is one of the biggest problems in statistical analysis of fMRI data. There is also a need for the development of better classification methods. One of the best things about fMRI technique is its noninvasiveness. If statistical classification methods are improved, it could aid the advancement of noninvasive diagnostic techniques for mental illness or even degenerative diseases such as Alzheimer’s. In this paper, we develop a variable selection technique, which tackles high dimensionality and correlation problems in fMRI data, based on L1 regularization-group lasso for the functional logistic regression model where the response is binary and represent two separate classes; the predictors are functional. We assess our method with a simulation study and an application to a real fMRI dataset.

The seamless-L0 (SELO) penalty is a smooth function that very closely resembles the L0 penalty, which has been demonstrated theoretically and practically to be effective in nonconvex penalization for variable selection. In this paper, the authors first generalize the SELO penalty to a class of penalties retaining good features of SELO, and then develop variable selection and parameter estimation in Cox models using the proposed generalized SELO (GSELO) penalized log partial likelihood (PPL) approach. The authors show that the GSELO-PPL procedure possesses the oracle property with a diverging number of predictors under certain mild, interpretable regularity conditions. The entire path of GSELO-PPL estimates can be efficiently computed through a smoothing quasi-Newton (SQN) with continuation algorithm. The authors propose a consistent modified BIC (MBIC) tuning parameter selector for GSELO-PPL, and show that under some regularity conditions, the GSELOPPL- MBIC procedure consistently identifies the true model. Simulation studies and real data analysis are conducted to evaluate the finite sample performance of the proposed method.

In high-dimensional data analysis, penalized likelihood estimators are shown to provide superior results in both variable selection and parameter estimation. A new algorithm, APPLE, is proposed for calculating the Approximate Path for Penalized Likelihood Estimators. Both convex penalties (such as LASSO) and folded concave penalties (such as MCP) are considered. APPLE efficiently computes the solution path for the penalized likelihood estimator using a hybrid of the modified predictor-corrector method and the coordinate-descent algorithm. APPLE is compared with several well-known packages via simulation and analysis of two gene expression data sets.

The high dimensionality of electronic nose data increases the difficulty of their use in classification models. Reducing this high dimensionality helps reduce variable numbers, improve classification accuracy, and reduce computation time and sensor cost. In this research, the Cyranose 320 electronic nose, which was used for apple defect detection, was optimized by selecting only the most relevant of its internal 32 sensors using different selection methods. Two robust heuristic optimization algorithms, genetic algorithm (GA) and covariance matrix adaptation evolutionary strategy (CMAES), were applied and compared. Although both algorithms searched the optimal sensors resulting in a best classification error rate of 4.4%, the average classification error rate of CMA over 30 random seed runs was 5.0% (s.d.=0.006) which was better than 5.2% (s.d.=0.004) from the GA. The final optimal solution sets obtained by integer GA showed that including more sensors did not guarantee better classification performance. The best reduction in classification error rate was 10% while the number of sensors was reduced 78%. This study provided a robust and efficient optimization approach to reduce high data dimensionality of the electronic nose data, which substantially improved electronic nose performance in apple defect detection while potentially reducing the overall electronic nose cost for future specific applications.