Multiclass Classification by Sparse Multinomial Logistic Regression

We consider high-dimensional binary classification by sparse logistic regression. We propose a model/feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the non-asymptotic bounds for its misclassification excess risk. To assess its tightness, we establish the corresponding minimax lower bounds. The bounds can be reduced under the additional low-noise condition. The proposed complexity penalty is remarkably related to the Vapnik-Chervonenkis-dimension of a set of sparse linear classifiers. Implementation of any complexity penalty-based criterion, however, requires a combinatorial search over all possible models. To find a model selection procedure computationally feasible for high-dimensional data, we extend the Slope estimator for logistic regression and show that under an additional weighted restricted eigenvalue condition it is rate-optimal in the minimax sense.

The estimating the chlorophyll contents in the crops helps to identify the condition of crops and different classification of crops with soil characteristics in order to assist the farmer or others with agriculture growth. In this paper, a hybrid approach is introduced to estimate the Chlorophyll contents in the crops using hyperspectral image segmentation with active learning, which consists of two main steps. First, we use a sparse multinomial logistic regression (SMLR) model to learn the class posterior probability distributions with Quadratic Programming or joint probability distribution. Second, we use the information acquired in the previous step to segment the hyper spectral image using a Markov Random field segments to estimate the dependencies using spatial information and edge Information by minimum spanning forest rooted on markers. In order to reduce the cost of acquiring large training sets, PSO optimization is performed based on the SMLR posterior probabilities on the Normalized difference vegetation index (NDVI). The state-of-the-art performance of the proposed approach is illustrated using real hyper spectral data sets collected from the North Karnataka in a number of experimental comparisons with recently developed or statistical hyperspectral image analysis methods in terms of precision, recall and f—measure.

Robust inference on average treatment effects with possibly more covariates than observations

Event Abstract Back to Event A decoded neurofeedback method for MEG/SSVEF Hiroki Ora1, 2 and Kenji Kansaku1, 2* 1 Research Institute of National Rehabilitation Center for Persons with Disabilities, Department of Rehabilitation for Brain Function, Japan 2 The University of Electro-Communications, Brain Science Inspired Life Support Research Center, Japan Decoded neurofeedback method was recently proposed to lead brain activity to a target state (Shibata, et al., 2011). While researchers utilized functional magnetic resonance imaging in the former studies, little is focused on magnetoencephalography (MEG). Real-time magnetoencephalography (rtMEG) is an emerging neurofeedback technology that could potentially benefit multiple areas of basic and clinical neuroscience, and in this study, we used an rtMEG system to perform decoded neurofeedback training for MEG/steady-state visual evoked field (SSVEF). Five able-bodied participants (age 33.2 years old, 5 females) participated in this study. The visual stimuli were displayed on a screen in front of the participant, and consisted of a circular checkerboard patch on the left and a circular checkerboard patch on the right. The checkerboard patches flickered at 5 (left) or 6 (right) Hz. They were asked to attend to the left, right, or middle of the screen in a SSVEF task. The used MEG scanner was a 306-channel Elekta Neuromag system (Elekta Oy, Helsinki, Finland). We constructed a sparse multinomial logistic regression (SMLR) decoder from MEG signals during the SSVEF task in each participant. We then conducted 3-day MEG neurofeedback training. In a trial, a white fixation cross was turned green for 5 second, then a green solid circle, whose radius indicated a score of the SMLR decoder, was presented. Participants were asked to “somehow regulate your brain activity to make the green solid circle bigger while the fixation cross is green.” During the trainings, no flickering visual stimuli were presented. After the trainings, the SSVEF task was used again to evaluate the training effects. Furthermore, we analyzed weights of the SMLR decoders to determine a subset of MEG sensors that contributed to decode the attended orientation. The SMLR decoder was able to classify the MEG signals into 3 attentional directions (left, right or middle) (88.2 %). The participants were able to increase decoder scores through the trainings (p<0.05). In the post-training SSVEF tasks, accuracy for the target orientation was higher than that for the non-target orientation (p<0.05). Analysis of weights of the decoders demonstrated that sensors on the occipital region contributed to decode the attended orientation (p<0.05, Bonferroni corrected). The results suggest that the decoded neurofeedback training was effective for MEG/SSVEF, and the method may enhance robustness of steady-state visual evoked potential (SSVEP)-based Brain-Computer Interface (BCI). Keywords: Magnetic Resonance Imaging, decoded neurofeedback, Magnetoencephalography (MEG), Real-time magnetoencephalography, Steady-state visual evoked field Conference: 2015 International Workshop on Clinical Brain-Machine Interfaces (CBMI2015), Tokyo, Japan, 13 Mar - 15 Mar, 2015. Presentation Type: Poster 1-6 Topic: Clinical Brain-Machine Interfaces Citation: Ora H and Kansaku K (2015). A decoded neurofeedback method for MEG/SSVEF. Conference Abstract: 2015 International Workshop on Clinical Brain-Machine Interfaces (CBMI2015). doi: 10.3389/conf.fnhum.2015.218.00007 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 23 Apr 2015; Published Online: 29 Apr 2015. * Correspondence: Dr. Kenji Kansaku, Research Institute of National Rehabilitation Center for Persons with Disabilities, Department of Rehabilitation for Brain Function, Tokorozawa, Saitama, Japan, kansakuk@dokkyomed.ac.jp Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Hiroki Ora Kenji Kansaku Google Hiroki Ora Kenji Kansaku Google Scholar Hiroki Ora Kenji Kansaku PubMed Hiroki Ora Kenji Kansaku Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Digital soil maps can be used to depict the ability of soil to fulfill certain functions. Digital maps offer reliable information that can be used in spatial planning programs. Several broad types of data mining approaches through Digital Soil Mapping (DSM) have been tested. The usual approach is to select a model that produces the best validation statistics. However, instead of choosing the best model, it is possible to combine all models realizing their strengths and weaknesses. We applied seven different techniques for the prediction of soil classes based on 194 sites located in Isfahan region. The mapping exercise aims to produce a soil class map that can be used for better understanding and management of soil resources. The models used in this study include Multinomial Logistic Regression (MnLR), Artificial Neural Networks (ANN), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), Bayesian Networks (BN), and Sparse Multinomial Logistic Regression (SMnLR). Two ensemble models based on majority votes (Ensemble.1) and MnLR (Ensemble.2) were implemented for integrating the optimal aspects of the individual techniques. The overall accuracy (OA), Cohen's kappa coefficient index (κ) and the area under the curve (AUC) were calculated based on 10-fold-cross validation with 100 repeats at four soil taxonomic levels. The Ensemble.2 model was able to achieve larger OA, κ coefficient and AUC compared to the best performing individual model (i.e., RF). Results of the ensemble model showed a decreasing trend in OA from Order (0.90) to Subgroup (0.53). This was also the case for the κ statistic, which was the largest for the Order (0.66) and smallest for the Subgroup (0.43). Same decrease was observed for AUC from Order (0.81) to Subgroup (0.67). The improvement in κ was substantial (43 to 60%) at all soil taxonomic levels, except the Order level. We conclude that the application of the ensemble model using the MnLR was optimal, as it provided a highly accurate prediction for all soil taxonomic levels over and above the individual models. It also used information from all models, and thus this method can be recommended for improved soil class modelling. Soil maps created by this DSM approach showed soils that are prone to degradation and need to be carefully managed and conserved to avoid further land degradation.

This paper presents a constructive method for deriving an updated discriminant eigenspace for classification when bursts of data that contains new classes is being added to an initial discriminant eigenspace in the form of random chunks. Basically, we propose an incremental linear discriminant analysis (ILDA) in its two forms: a sequential ILDA and a Chunk ILDA. In experiments, we have tested ILDA using datasets with a small number of classes and small-dimensional features, as well as datasets with a large number of classes and large-dimensional features. We have compared the proposed ILDA against the traditional batch LDA in terms of discriminability, execution time and memory usage with the increasing volume of data addition. The results show that the proposed ILDA can effectively evolve a discriminant eigenspace over a fast and large data stream, and extract features with superior discriminability in classification, when compared with other methods.

On the usefulness of one-class classifier ensembles for decomposition of multi-class problems

We consider model selection in generalized linear models (GLM) for high-dimensional data and propose a wide class of model selection criteria based on penalized maximum likelihood with a complexity penalty on the model size. We derive a general nonasymptotic upper bound for the Kullback–Leibler risk of the resulting estimators and establish the corresponding minimax lower bounds for the sparse GLM. For the properly chosen (nonlinear) penalty, the resulting penalized maximum likelihood estimator is shown to be asymptotically minimax and adaptive to the unknown sparsity. We also discuss possible extensions of the proposed approach to model selection in the GLM under additional structural constraints and aggregation.

The risk factors for stunting incidence involve categorical data in both the response and predictor variables. Therefore, we developed a sparse categorical principal component logistic regression model capable of handling data with multicollinearity. The parameters of the sparse categorical principal component logistic regression model were estimated using the maximum likelihood method and the Newton-Raphson iterative approach. The analysis yielded a likelihood ratio value of 144.81 and a chi-square statistic value of 11.07, indicating that all factors included in the model are statistically significant. The results highlight that medical history, inadequate complementary feeding, formula feeding, lack of complementary feeding programs, and lack of iron supplementation for mothers are highly associated with the risk of stunting in toddlers. This emphasizes the need for attention to maternal nutrition from pregnancy through breastfeeding, as well as the nutrition of the toddler. Some important points proposed in this method are:•Stunting data consists of categorical variables containing multicollinearity.•The method applied is sparse logistic regression combined with categorical principal component analysis.•Analysis of risk factors for stunting in toddlers is based on the child's own condition, as well as parental factors, namely age, education, and intake of additional food and supplementary tablets during pregnancy.

BackgroundMicroarray technology is widely used in cancer diagnosis. Successfully identifying gene biomarkers will significantly help to classify different cancer types and improve the prediction accuracy. The regularization approach is one of the effective methods for gene selection in microarray data, which generally contain a large number of genes and have a small number of samples. In recent years, various approaches have been developed for gene selection of microarray data. Generally, they are divided into three categories: filter, wrapper and embedded methods. Regularization methods are an important embedded technique and perform both continuous shrinkage and automatic gene selection simultaneously. Recently, there is growing interest in applying the regularization techniques in gene selection. The popular regularization technique is Lasso (L1), and many L1 type regularization terms have been proposed in the recent years. Theoretically, the Lq type regularization with the lower value of q would lead to better solutions with more sparsity. Moreover, the L1/2 regularization can be taken as a representative of Lq (0 <q < 1) regularizations and has been demonstrated many attractive properties.ResultsIn this work, we investigate a sparse logistic regression with the L1/2 penalty for gene selection in cancer classification problems, and propose a coordinate descent algorithm with a new univariate half thresholding operator to solve the L1/2 penalized logistic regression. Experimental results on artificial and microarray data demonstrate the effectiveness of our proposed approach compared with other regularization methods. Especially, for 4 publicly available gene expression datasets, the L1/2 regularization method achieved its success using only about 2 to 14 predictors (genes), compared to about 6 to 38 genes for ordinary L1 and elastic net regularization approaches.ConclusionsFrom our evaluations, it is clear that the sparse logistic regression with the L1/2 penalty achieves higher classification accuracy than those of ordinary L1 and elastic net regularization approaches, while fewer but informative genes are selected. This is an important consideration for screening and diagnostic applications, where the goal is often to develop an accurate test using as few features as possible in order to control cost. Therefore, the sparse logistic regression with the L1/2 penalty is effective technique for gene selection in real classification problems.

Hyperspectral image classification impose challenging requirements to a classifier. It is well known that more spectral bands can be difficult to process and introduce problems such as the Hughes phenomenon. Nevertheless, user requirements are very demanding, as expectations grow with the available number of spectral bands: subtle differences in a large number of classes must be distinguished. As multiclass classifiers become rather complex for a large number of classes, a combination of binary classification results are often used to come to a class decision. In this approach, the posterior probability is retained for each of the binary classifiers. From these, a combined posterior probability for the multiclass case is obtained. The proposed technique is applied to map the highly diverse Belgian coastline. In total, 17 vegetation types are defined. Additionally, bare soil, shadow, water and urban area are also classified. The posterior probabilities are used for unmixing. This is demonstrated for 4 classes: bare soil and 3 vegetation classes. Results are very promosing, outperforming other approaches such as linear unmixing.

The high dimensionality of microarray datasets endows the task of multiclass tissue classification with various difficulties--the main challenge being the selection of features deemed relevant and non-redundant to form the predictor set for classifier training. The necessity of varying the emphases on relevance and redundancy, through the use of the degree of differential prioritization (DDP) during the search for the predictor set is also of no small importance. Furthermore, there are several types of decomposition technique for the feature selection (FS) problem--all-classes-at-once, one-vs.-all (OVA) or pairwise (PW). Also, in multiclass problems, there is the need to consider the type of classifier aggregation used--whether non-aggregated (a single machine), or aggregated (OVA or PW). From here, first we propose a systematic approach to combining the distinct problems of FS and classification. Then, using eight well-known multiclass microarray datasets, we empirically demonstrate the effectiveness of the DDP in various combinations of FS decomposition types and classifier aggregation methods. Aided by the variable DDP, feature selection leads to classification performance which is better than that of rank-based or equal-priorities scoring methods and accuracies higher than previously reported for benchmark datasets with large number of classes. Finally, based on several criteria, we make general recommendations on the optimal choice of the combination of FS decomposition type and classifier aggregation method for multiclass microarray datasets.

Accurate classification of Alzheimer’s Disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI) are critical for the effective treatment of AD. However, compared with AD classification tasks, predicting the conversion of MCI to AD is relatively difficult. as there are only minor differences among MCI groups. What’s more, in brain imaging analysis, the high dimensionality and relatively small number of subjects brings challenges to computer-aided diagnosis of AD and MCI. Many previous researches focused on the identification of imaging biomarkers for AD diagnosis. In this paper, we introduce sparse logistic regression for the early diagnosis of AD. Sparse logistic regression (SLR) uses L1/2 regularization to impose a sparsity constraint on logistic regression. The L1/2 regularization is considered a representative of Lq regularization, where fewer but informative key brain regions are applied for the classification of AD/MCI. We evaluated the SLR on 197 subjects from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Experimental results showed that the SLR improves the classification performance of AD/MCI compared other classical methods.

This paper introduces logistic regression with sparse and smooth regularizations (LR-SS), a novel framework that simultaneously enhances both classification and feature extraction capabilities of standard logistic regression. By incorporating a family of symmetric smoothness constraints into sparse logistic regression, LR-SS uniquely preserves underlying structures inherent in structured data, distinguishing it from existing approaches. Within the minorization–maximization (MM) framework, we develop an efficient optimization algorithm that combines coordinate descent with soft-thresholding techniques. Through extensive experiments on both simulated and real-world datasets, including time series and image data, we demonstrate that LR-SS significantly outperforms conventional sparse logistic regression in classification tasks while providing more interpretable feature extraction. The results highlight LR-SS’s ability to leverage sparse and symmetric smooth regularizations for capturing intrinsic data structures, making it particularly valuable for machine learning applications requiring both predictive accuracy and model interpretability.

In this work we propose to fit a sparse logistic regression model by a weakly\nconvex regularized nonconvex optimization problem. The idea is based on the\nfinding that a weakly convex function as an approximation of the $\\ell_0$\npseudo norm is able to better induce sparsity than the commonly used $\\ell_1$\nnorm. For a class of weakly convex sparsity inducing functions, we prove the\nnonconvexity of the corresponding sparse logistic regression problem, and study\nits local optimality conditions and the choice of the regularization parameter\nto exclude trivial solutions. Despite the nonconvexity, a method based on\nproximal gradient descent is used to solve the general weakly convex sparse\nlogistic regression, and its convergence behavior is studied theoretically.\nThen the general framework is applied to a specific weakly convex function, and\na necessary and sufficient local optimality condition is provided. The solution\nmethod is instantiated in this case as an iterative firm-shrinkage algorithm,\nand its effectiveness is demonstrated in numerical experiments by both randomly\ngenerated and real datasets.\n