Scalable Partial Least Squares Regression on Grammar-Compressed Data Matrices

With massive high-dimensional data now common-place in research and industry, there is a strong and growing demand for more scalable computational techniques for data analysis and knowledge discovery. In this paper, we review scalable algorithms for learning statistical models on high-dimensional data. Especially, we introduce two techniques of lossless and lossy compressions. The first one is a method using grammar compression. Grammar compression is a lossless compression for texts and has been successfully applied to binary data matrices for scalable learning of statistical models. The second one is a method of lossy compressions named feature maps (FMs). Recently, quite a few number of FMs for kernel approximations have been proposed and have been used in practical applications. Those methods, of which we present a brief survey in this paper, open the door for large-scale analyses of massive and high-dimensional data.

Massive high dimensional data has a large sample size and high dimensionality. However, all the features of the massive high dimensional data are used for identification or classification, which will increase the calculation time and reduce the accuracy of identification or classification. Therefore, extracting features with strong expressive ability is very important for processing massive high-dimensional data. In order to solve this problem, this paper presents a feature extraction method which based on Stacked Denoising Autoencoder (SDAE). SDAE trains each layer of neural network by unsupervised layer-by-layer greedy training. Then supervised training Softmax classifier. And finally uses Back Propagation (BP) algorithm to optimize the entire model. In this paper, the ISOLET data set is taken as experimental data. The experimental results demonstrated that our proposed method can extract feature subsets with strong expressive ability. And using this feature subset for classification, the classification accuracy is significantly improved.

Quantitative structure-activity relationship (QSAR) studies have been performed on piperidine derivatives (n = 119) as CCR5 antagonists. The whole data set was divided into a training set (75% of the dataset) and a test set (remaining 25%) on the basis of K-means clustering technique. Models developed from the training set were used to assess the predictive potential of the models using test set compounds. Initially classical type QSAR models were developed using structural, spatial, electronic, physicochemical and/or topological parameters using statistical methods like stepwise regression, partial least squares (PLS) and factor analysis followed by multiple linear regression (FA-MLR). Using topological and structural parameters, FA-MLR provided the best equation based on internal validation (Q2 = 0.514) but the best externally validated model was obtained with PLS ( = 0.565). When structural, physicochemical, spatial and electronic descriptors were used, the best Q2 value (0.562) was obtained from the stepwise regression derived model whereas the best value (0.571) came from the PLS model. When topological descriptors were used in combination with the structural, physicochemical, spatial and electronic descriptors, the best Q2 and values obtained were 0.530 (stepwise regression) and 0.580 (PLS) respectively. Attempt was made to develop 3D-QSAR models using molecular shape analysis descriptors in combination with structural, physicochemical, spatial and electronic parameters. Linear models were developed using genetic function algorithm coupled with multiple linear regression. However, the results from the 3D-QSAR study were not superior to those of the classical QSAR models. Finally, artificial neural network was employed for development of nonlinear models. The ANN models showed acceptable values of squared correlation coefficient for the observed and predicted values of the test set compounds. From the view point of external predictability, selected ANN models were superior to the linear QSAR models. All reported models satisfy the criteria of external validation as recommended by Golbraikh and Tropsha (J Mol Graphics Mod 2002; 20: 269–276), whereas the majority of the models have modified r2 () value of the test set for external validation more than 0.5 as suggested by Roy and Roy (QSAR Comb Sci 2008; 27: 302-313).

Objective. Partial Least Squares (PLS) regression is a suitable linear decoder model for correlated and high dimensional neural data. This algorithm has been widely used in the application of brain–computer interface (BCI) for the decoding of motor parameters. PLS does not consider nonlinear relations between brain signal features. The nonlinear version of PLS that considers a nonlinear relationship between the latent variables has not been proposed for the decoding of intracranial data. This nonlinear model may cause overfitting in some cases due to a larger number of free parameters. In this paper, we develop a new version of nonlinear PLS, namely nonlinear sparse PLS (NLS PLS) and test it in BCI applications. Approach. In motor related BCI systems, improving the decoding accuracy of both kinetic and kinematic parameters of movement is crucial. To do this, two BCI datasets were chosen to decode the force amplitude and position of hand trajectory using the nonlinear and sparse versions of PLS algorithm. In our new NLS PLS method, we considered a polynomial relationship between the latent variables and used the lasso penalization in the latent space to avoid overfitting and to improve the decoding accuracy. Main results. Some linear and nonlinear based PLS models and our new proposed method, NLS PLS, were applied to the two datasets. According to our results, significant improvement from the NLS PLS method is confirmed over other methods. Our results show that nonlinear PLS outperforms generic PLS in the force decoding but it has lower accuracy in the hand trajectory decoding because of high dimensional feature space. By using lasso penalization, we presented a sparse nonlinear PLS-based model that outperforms generic PLS in both datasets and improves the coefficient of determination, 34% in the force decoding and 10% in the hand trajectory decoding. Significance. We constructed a simple PLS-based model that considers a nonlinear relationship between features and it is also robust to overfitting because of using the lasso penalty in the latent space. This model is suitable for a high dimensional and correlated datasets, like intracranial data and can improve the accuracy of estimation.

Decision analysis and optimization are examples of inverse management problems which can be effectively handled by means of Artificial Intelligence (AT) and simulation techniques. However, the application of these techniques in specific industrial contexts finds major constraints in the lack of formalized methodologies and systematic procedures to retrieve and handle the data required for their implementation. The paper proposes Knowledge Discovery (KD) techniques as means of data pre-processing for the development of Al-based decision support systems (DSSs). This use of Knowledge Discovery techniques is illustrated in relation to the design of a hybrid DSS - combining AI and simulation techniques - for the management of maintenance services in the public transport sector.

A novel formulation of the wide kernel algorithm for partial least squares regression (PLSR) is proposed. We show how the elimination of redundant calculations in the traditional applications of PLSR helps in speeding up any choice of cross‐validation strategy by utilizing precalculated lookup matrices.The proposed lookup approach is combined with some additional computational shortcuts resulting in highly effective and numerically accurate cross‐validation results. The computational advantages of the proposed method are demonstrated by comparisons to the classical NIPALS and the bidiag2 algorithms for calculating cross‐validated PLSR models. Problems including both one and several responses, double/nested cross‐validated, and one‐vs‐all classification are among the considered applications.

This paper investigated the feasibility of Fourier transform infrared transmission (FT-IR) spectroscopy to detect talcum powder illegally added in tea based on chemometric methods. Firstly, 210 samples of tea powder with 13 dose levels of talcum powder were prepared for FT-IR spectra acquirement. In order to highlight the slight variations in FT-IR spectra, smoothing, normalize and standard normal variate (SNV) were employed to preprocess the raw spectra. Among them, SNV preprocessing had the best performance with high correlation of prediction (RP = 0.948) and low root mean square error of prediction (RMSEP = 0.108) of partial least squares (PLS) model. Then 18 characteristic wavenumbers were selected based on a hybrid of backward interval partial least squares (biPLS) regression, competitive adaptive reweighted sampling (CARS) algorithm and successive projections algorithm (SPA). These characteristic wavenumbers only accounted for 0.64% of the full wavenumbers. Following that, 18 characteristic wavenumbers were used to build linear and nonlinear determination models by PLS regression and extreme learning machine (ELM), respectively. The optimal model with RP = 0.963 and RMSEP = 0.137 was achieved by ELM algorithm. These results demonstrated that FT-IR spectroscopy with chemometrics could be used successfully to detect talcum powder in tea.

The rapid growth of healthcare data in recent years calls for more advanced and efficient analytic techniques. Artificial intelligence facilitates finding insightful patterns in massive high-dimensional data. Considering the latest movements towards using machine learning and deep learning techniques in the medical domain, in this study, we focused on the publications in which researchers employed artificial intelligence techniques for cancer diagnosis and treatment. Using dynamic topic modeling and natural language processing techniques, we analyzed the contents and trends of more than 12,000 scientific publications within the period of 2000 to 2018, extracted from two different sources, i.e., Elsevier’s Scopus and PubMed. While drawing the landscape of cancer research, our results also shed light on the evolution of artificial intelligence techniques and algorithms used for cancer diagnosis and treatment. Our findings confirm that modern computer science algorithms are being widely applied to extract patterns from large-scale medical images to cure different types of cancer with a special focus on deep learning techniques in recent years.

Highly correlated predictors and nonlinear relationships between response and predictors potentially affected the performance of predictive modeling, especially when using the ordinary least square (OLS) method. The simple technique to solve this problem is by using another method such as Partial Least Square Regression (PLSR), Support Vector Regression with kernel Radial Basis Function (SVR-RBF), and Random Forest Regression (RFR). The purpose of this study is to compare OLS, PLSR, SVR-RBF, and RFR using simulation data. The methods were evaluated by the root mean square error prediction (RMSEP). The result showed that in the linear model, SVR-RBF and RFR have large RMSEP; OLS and PLSR are better than SVR-RBF and RFR, and PLSR provides much more stable prediction than OLS in case of highly correlated predictors and small sample size. In nonlinear data, RFR produced the smallest RMSEP when data contains high correlated predictors.

Knowledge of subject-specific muscle volumes may be used as surrogates for evaluating muscle strength and power generated by 'fat-free' muscle mass. This study presents population-based statistical learning approaches for predicting 'fat-free' muscle volume from known anthropometric measurements. Using computed tomography (CT) imaging data to obtain lower-limb muscle volumes from 50 men and women, this study evaluated six statistical learning methods for predicting muscle volumes from anthropometric measurements: (i) stepwise regression, (ii) linear support vector machine (SVM), (iii) 2nd-order polynomial SVM, (iv) linear partial least squares regression (PLSR), (v) quadratic PLSR, and (vi) 3rd-order spline fit PLSR. These techniques have successfully been demonstrated in bioengineering applications and freely available in open-source toolkits. Analysis revealed that separating a general population into sexes and/or cohorts based on adipose level may improve prediction accuracies. The most important measures that statistically influence muscle volume predictions were shank girth, followed by sex and finally leg length, as identified using stepwise regression. SVM learning predicted muscle volume with an accuracy of 85 ± 4% when using linear interpolation, but performed poorly with an accuracy of 59 ± 6% using polynomial interpolation. The simpler linear PLSR exhibited muscle volume prediction accuracy of 87 ± 2%, while quadratic PLSR was slightly reduced at 82 ± 3%. For the spline fit PLSR, high accuracy was observed on the training data set (~ 99%) but over-fitting (a drawback of high-interpolation methods) resulted in erroneous predictions on testing data, and hence, the model was deemed unsuitable. In conclusion, use of linear PLSR models with variables of sex, leg length, and shank girth is a useful tool for predicting 'fat-free' muscle volume.

Simultaneous determination of enalapril maleate (ENA) and nitrendipine (NIT) in pharmaceutical preparations was performed using liquid chromatography (LC) and the partial least-squares-1 (PLS-1) method. In LC, the separation was achieved on a C8 column and the optimum mobile phase for good separation in a gradient elution programme was found to be acetonitrile-water (φ r = 81: 19) and optimum flow-rate, temperature, injection volume, and detection wavelength were set at 1.0 mL min−1, 25°C, 10 μL, and 210 nm, respectively. Dienogest was selected as an internal standard. In the spectrophotometry, a PLS-1 chemometric method was used. The absorbance data matrix related to the concentration data matrix was established by measurement of absorbances in their zero order spectra with an increment of Δλ = 1 nm in the 220–290 nm range for ENA and with Δλ = 1 nm in the 230–290 nm range for NIT in the PLS-1 method. Following this step, calibration was established by using this data matrix to predict the unknown concentrations of ENA and NIT in their binary mixture. These optimised methods were validated and successfully applied to a pharmaceutical preparation in tablet form and the results were subjected to comparison.

Partial least squares regression - or PLS regression - is a multivariate method in which the model parameters are estimated using either the SIMPLS or NIPALS algorithm. PLS regression has been extensively used in applied research because of its effectiveness in analyzing relationships between an outcome and one or several components. Note that the NIPALS algorithm can provide estimates parameters on incomplete data. The selection of the number of components used to build a representative model in PLS regression is a central issue. However, how to deal with missing data when using PLS regression remains a matter of debate. Several approaches have been proposed in the literature, including the Q2 criterion, and the AIC and BIC criteria. Here we study the behavior of the NIPALS algorithm when used to fit a PLS regression for various proportions of missing data and different types of missingness. We compare criteria to select the number of components for a PLS regression on incomplete data set and on imputed data set using three imputation methods: multiple imputation by chained equations, k-nearest neighbour imputation, and singular value decomposition imputation. We tested various criteria with different proportions of missing data (ranging from 5% to 50%) under different missingness assumptions. Q2-leave-one-out component selection methods gave more reliable results than AIC and BIC-based ones.

Present day computational fluid dynamics (CFD) simulations generate considerable amounts of data, sometimes on the order of TB/s. Often, a significant fraction of this data is discarded because current storage systems are unable to keep pace. To address this, data compression algorithms can be applied to data arrays containing flow quantities of interest (QoIs) to reduce the overall required storage. The matrix column interpolative decomposition (ID) can be implemented as a type of lossy compression for data matrices that factors the original data matrix into a product of two smaller factor matrices. One of these matrices consists of a subset of the columns of the original data matrix, while the other is a coefficient matrix which approximates the original data matrix columns as linear combinations of the selected columns. Motivating this work is the observation that the structure of ID algorithms makes them well suited for the asynchronous nature of task-based parallelism; they can operate independently on subdomains of the system of interest and, as a result, provide varied levels of compression. Using the task-based Legion programming model, a single-pass ID algorithm (SPID) for CFD applications is implemented. Performance studies, scalability, and the accuracy of the compression algorithm are presented for a benchmark analytical Taylor-Green vortex problem, as well as large-scale implementations of both low and high Reynolds number ( Re) compressible Taylor-Green vortices using a high-order Navier-Stokes solver. In the case of the analytical solution, the resulting compressed solution was rank-one, with error on the order of machine precision. For the low- Re vortex, compression factors between 1000 and 10,000 were achieved for errors in the range 10−2–10−3. Similar error values were seen for the high- Re vortex, this time with compression factors between 100 and 1000. Moreover, strong and weak scaling results demonstrate that introducing SPID to solvers leads to negligible increases in runtime.

Short-wave near-infrared (NIR) spectra of raw milk have been analyzed in the 800–1100 nm region by two-dimensional (2D) correlation spectroscopy. In this study, we have used both the well-known generalized 2D correlation spectroscopy method, which yields correlation coefficients among spectral variances on all the wavelength points (wavelength-wavelength correlation), and a novel sample-sample correlation spectroscopy method, which gives correlation coefficients among the concentration dynamics of the species in the system. The sample-sample correlation spectroscopy develops correlation maps with the samples on the axes. First, a set of 34 spectra was ordered according to increasing fat content, while all other milk components varied freely. Both synchronous and asynchronous wavelength-wavelength correlation maps have shown strong baseline changes despite the use of multiplicative scatter correction as a pretreatment. Bands at 930 and 970 nm due to fat and water, respectively, have been found to be the most significant spectral features. The synchronous sample-sample correlation map was calculated from the original data matrix and was compared with the outer product of the fat concentration vector. The comparison has revealed that the main spectral variances in the raw milk spectra in the short-wave NIR region are due to fat. The same procedure was repeated for a set of 15 samples that contained a constant fat content and were ordered according to increasing protein content. Poor agreement was found between the outer product of the protein concentration vector and the synchronous sample-sample correlation map. This result suggests that the spectral variances in raw milk spectra that have a constant fat content are due not exclusively to proteins but also due to other milk components. Comparisons of partial least-squares (PLS) regression analysis of fat and proteins and both sample-sample and wavelength-wavelength 2D correlation analyses of raw milk spectra have been made.

A computerised algorithm to select automatically appropriate wavelength (or 2θ) variables for subsequent multivariate calibration modelling was applied to the determination of iron, manganese, potassium, calcium, titanium, silicon, aluminium, magnesium and sodium in a range of certified geological materials by XRF spectrometry. The application of partial least-squares (PLS) regression is shown to be superior in terms of predictive performance to univariate linear regression modelling and multiple linear regression analysis. The combined process of automated variable selection and PLS modelling is amenable to providing an automated XRF quantitative analysis software system.