Polynomial Network Classifier with Discriminative Feature Extraction

The multinomial logistic regression (MLR) model is widely used in statistics and machine learning. On the one hand, stochastic gradient descent (SGD) is the most common approach for determining the parameters of a such model in big data scenarios, due to its simplicity and low computational complexity property. Furthermore, SGD has proven convergence under reasonable conditions. However, SGD has slow sub-linear rates of convergence and it often reduces convergence speed due to the plateau phenomenon. On the other hand, stochastic natural gradient descent (SNGD), proposed by Amari, is a manifold optimization method shown to be Fisher efficient when it converges, but its convergence properties remain unproven and it is often computationally prohibitive for models with a large number of parameters. Here, we propose dual stochastic natural gradient descent (DSNGD), a stochastic optimization method for MLR based on manifold optimization concepts. In the discrete scenario, DSNGD (i) has linear per-iteration computational complexity in the number of parameters, and (ii) is proven to converge. To achieve (i) we leverage the dual flatness of the family of joint distributions for MLR to simplify computations. To ensure (ii) DSNGD builds on the foundational ideas of convergent stochastic natural gradient descent (CSNGD), a variant of SNGD with guaranteed convergence, using an independent sequence to construct a bounded approximation of the natural gradient. By generalizing a result from Sunehag et al., we prove that DSNGD converges in the discrete case and maintains linear computational complexity per iteration. Beyond its convergence property and linear computational complexity, DSNGD empirically demonstrates fast convergence comparable to SNGD, improves upon SGD performance, and exhibits stability where SNGD does not.

Ensuring safety in geotechnical engineering has consistently posed challenges due to the inherent variability of soil. In the case of slope stability problems, performing on-site tests is both costly and time-intensive due to the need for sophisticated equipment (to acquire and move) and logistics. Consequently, the analysis of simulation models based on soft computing proves to be a practical and invaluable alternative. In this research work, learning abilities of the Class Noise Two (CN2), Stochastic Gradient Descent (SGD), Group Method of Data Handling (GMDH) and artificial neural network (ANN) have been investigated in the prediction of the factor of safety (FOS) of slopes. This has been successfully done through literature search, data curation and data sorting. A total of three hundred and forty-nine (349) data entries on the FOS of slopes were collected from literature and sorted to remove odd values and unlogic results, which had been used together in a previous research work. After the sorting process, the remainder of the realistic data entries was 296. The previous work which had included unrealistic data entries had unit weight, γ (kN/m3), cohesion, C (kPa),angle of internal friction (Φ°), slope angle (°), slope height H (m), and pore water pressure ratio, ru as the studied parameters, which formed the independent variables. After careful checks, the initial results showed poor correlation with the individual factors and the factors were collected into three non-dimensional parameters based on the understanding of the physics of flows, which are: C/γ.h-Cohesion/unit weight x slope height, tan(ϕ)/tan(β)-the tangent of internal friction angle/Tangent of slope angle, and ρ/γ.h-Water pressure/unit weight x slope height, which are deployed as inputs and FOS-the safety factor of the slope as the output. At the end of the exercise, the ANN outclassed the other techniques with SSE of 62%, MAE of 0.27, MSE of 0.21, RMSE of 0.46, average total error of 24%, and R2 of 0.946 thereby becoming the decisive intelligent model in this exercise. However, there is an advantage the deployment of GMDH, which comes second in order of superiority, has over the ANN. This is the development of a closed-form equation that allows its model to be applied manually in the design of slope stability problems. Overall, the present research models outperformed the eleven (11) models of the previous work due to sorting and elimination of unrealistic data entries deposited in the literature, the application of dimensionless combination of the studied slope stability parameters and the superiority of the selected machine learning techniques.

Image classification is a very popular machine learning domain in which deep convolutional neural networks have mainly emerged on such applications. These networks manage to achieve remarkable performance in terms of prediction accuracy but they are considered as black box models since they lack the ability to interpret their inner working mechanism and explain the main reasoning of their predictions. There is a variety of real world tasks, such as medical applications, in which interpretability and explainability play a significant role. Making decisions on critical issues such as cancer prediction utilizing black box models in order to achieve high prediction accuracy but without provision for any sort of explanation for its prediction, accuracy cannot be considered as sufficient and ethnically acceptable. Reasoning and explanation is essential in order to trust these models and support such critical predictions. Nevertheless, the definition and the validation of the quality of a prediction model’s explanation can be considered in general extremely subjective and unclear. In this work, an accurate and interpretable machine learning framework is proposed, for image classification problems able to make high quality explanations. For this task, it is developed a feature extraction and explanation extraction framework, proposing also three basic general conditions which validate the quality of any model’s prediction explanation for any application domain. The feature extraction framework will extract and create transparent and meaningful high level features for images, while the explanation extraction framework will be responsible for creating good explanations relying on these extracted features and the prediction model’s inner function with respect to the proposed conditions. As a case study application, brain tumor magnetic resonance images were utilized for predicting glioma cancer. Our results demonstrate the efficiency of the proposed model since it managed to achieve sufficient prediction accuracy being also interpretable and explainable in simple human terms.

Using the Group Method of Data Handling (GMDH) a polynomial network is designed in this paper for learning the nonlinear image distortion of a camera. The GMDH network designed can effectively learn image distortion in various camera systems of different optical features unlike most existing techniques that assume a physical model explicitly. Compared to multilayer perceptrons (MLPs), which are popularly used to learn a nonlinear relation without modeling, a GMDH network is self-organizing and its learning is faster. We prove the advantages of the proposed technique with various simulated data sets and in a real experiment.

Biased outcomes in machine learning models can arise due to various factors, including limited training data, imbalanced class distribution, suboptimal training methodologies, and overfitting. In training neural networks with Stochastic Gradient Descent (SGD) and backpropagation, the choice of hyperparameters like learning rate and momentum is crucial to influencing the model’s performance. A comprehensive grid search study was conducted using static hyperparameters with standard SGD and dynamic hyperparameters with the Adam optimizer. The investigation focused on a multifaceted analysis across different tasks —- classification, segmentation, and detection — and was applied to four image-based applications: digit classification using the MNIST dataset, tuberculosis detection from chest X-ray images, lung segmentation in chest X-ray images, and the detection of malaria parasites in blood smear images. In the first comparative study on the MNIST dataset, the SGD algorithm consistently outperformed the Adam algorithm as noise levels increased. SGD held a slight advantage in accuracy over Adam in a noise-free environment. This advantage became more apparent as noise was introduced and increased to moderate levels. At high noise levels, both algorithms experienced a significant decline in performance, yet SGD maintained a relatively better accuracy compared to Adam. This trend underscored SGD’s superior ability to generalize across varying noise conditions. For TB detection in the second application, the DenseNet121 architecture was used, and it was found that Adam showed better performance on the larger TBX11K dataset. However, SGD outperformed Adam on a smaller subset of TBX11K. In the third comparative study for lung segmentation using COVID-19 and NLM datasets, SGD slightly outperformed Adam based on the mean and Hausdorff distances. In the case of malaria parasite detection using the YOLOv8 architecture, SGD and Adam optimizers showed varying performances across different conditions. Initially, both achieved high accuracies on the full dataset, with Adam slightly outperforming SGD. However, with decreasing dataset size, SGD maintained more consistent performance, while Adam’s accuracy fluctuated significantly. In noise tests, both showed equal accuracy on clean data, but under noise conditions, SGD maintained higher accuracies, suggesting better generalization capabilities. Further experiments assessed how well SGD and Adam could cope with domain shifts, mainly when using data from different countries. SGD’s generalization performance was superior to Adam’s performance in these experiments. In summary, although Adam is arguably the more popular of the two optimization techniques, SGD held its own in the experiments and showed superior performance when it came to generalization or classifying in noisy conditions.

As there has been a paradigm shift in the learning load from a human subject to a computer, machine learning has been considered as a useful tool for Brain-Computer Interfaces (BCIs). In this paper, we propose a novel Bayesian framework for discriminative feature extraction for motor imagery classification in an EEG-based BCI in which the class-discriminative frequency bands and the corresponding spatial filters are optimized by means of the probabilistic and information-theoretic approaches. In our framework, the problem of simultaneous spatiospectral filter optimization is formulated as the estimation of an unknown posterior probability density function (pdf) that represents the probability that a single-trial EEG of predefined mental tasks can be discriminated in a state. In order to estimate the posterior pdf, we propose a particle-based approximation method by extending a factored-sampling technique with a diffusion process. An information-theoretic observation model is also devised to measure discriminative power of features between classes. From the viewpoint of classifier design, the proposed method naturally allows us to construct a spectrally weighted label decision rule by linearly combining the outputs from multiple classifiers. We demonstrate the feasibility and effectiveness of the proposed method by analyzing the results and its success on three public databases.

In this paper, we consider constrained optimization problems subject to a convex set C. Stochastic gradient descent (SGD) is a simple and popular stochastic optimization algorithm that has been the workhorse of machine learning for many years. We show a new and surprising fact about SGD, in that depending on the constraint set C, one can operate SGD with much lower-dimensional stochastic gradients without affecting its performance. In particular, we design an optimization algorithm that operates with the lower-dimensional (compressed) stochastic gradients, and establish that with the right set of parameters it has the exact same dimension-free convergence guarantees as that of regular SGD for popular convex and nonconvex optimization settings. We also present two applications of these bounds, one for improving the empirical risk minimization bounds in differentially private nonconvex optimization, and other for reducing communication costs with distributed SGD. Additionally, we also show that this connection between constraint set structure and gradient compression also extends beyond SGD to the conditional gradient (Frank-Wolfe) method. The geometry of the constraint set, captured by its Gaussian width, plays an important role in all our results.

This paper provides a comprehensive introduction to a novel approach to pattern recognition which is based on the generalized probabilistic descent method (GPD) and its related design algorithms. The paper contains a survey of recent recognizer design techniques, the formulation of GPD, the concept of minimum classification error learning that is closely related to the GPD formalization, a relational analysis between GPD and other important design methods, and various embodiments of GPD-based design, including segmental-GPD, minimum spotting error training, discriminative utterance verification, and discriminative feature extraction. GPD development has its origins in basic pattern recognition and Bayes decision theory. It represents a simple but careful re-investigation of the classical theory and successfully leads to an innovative framework. For clarity of presentation, detailed discussions about its embodiments are provided for examples of speech pattern recognition tasks that use a distance-based classifier. Experimental results in speech pattern recognition tasks clearly demonstrate the remarkable utility of the family of GPD-based design algorithms.

Alzheimer's Disease (AD), the most common type of dementia, is characterized by memory issues that worsen with time. When evaluating single modalities, there are specific efficient learning techniques that produce a poor identification rate. The clinical significance of Mild Cognitive Impairment (MCI) has clinical importance, which is crucial for identifying and categorizing AD patients. This study includes the utilization of 3D T1 weighted /Magnetic Resonance Imaging (3D T1WI MRI) images where the structural changes in the brain are identified for predicting early AD. The classification accuracy for AD, MCI, and Normal Control (NC) has also increased due to cognitive evaluations when comparing three one vs one classifications, i.e. AD vs. NC, MCI vs. NC, AD vs. MCI, and multi-class classification. The proposed approach’s novelty is a binary and multi-class classification of structural MRI images using a Siamese 3D Convolutional Neural Network (Siamese 3D-CNN). The Stochastic Gradient Descent (SGD) has been employed to modify the proposed network's weights, and 5-Fold Cross-Validation (CV) has been used for each binary classification task. Our study used the neuropsychological data of 417 people from the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset to validate the suggested methodology. Experimental results demonstrate that accuracy, sensitivity, and specificity for three binary classifications produce the maximum percentage, in contrast to other relevant approaches found in the literature. In addition, the multi-class classification accuracy of 99.8% was attained.

There is an increased interest in building data analytics frameworks with advanced algebraic capabilities both in industry and academia. Many of these frameworks, e.g., TensorFlow, implement their compute-intensive primitives in two flavors—as multi-thread routines for multi-core CPUs and as highly-parallel kernels executed on GPU. Stochastic gradient descent (SGD) is the most popular optimization method for model training implemented extensively on modern data analytics platforms. While the data-intensive properties of SGD are well-known, there is an intense debate on which of the many SGD variants is better in practice. In this paper, we perform a comprehensive experimental study of parallel SGD for training machine learning models. We consider the impact of three factors – computing architecture (multi-core CPU or GPU), synchronous or asynchronous model updates, and data sparsity – on three measures—hardware efficiency, statistical efficiency, and time to convergence. We draw several interesting findings from our experiments with logistic regression (LR), support vector machines (SVM), and deep neural nets (MLP) on five real datasets. As expected, GPU always outperforms parallel CPU for synchronous SGD. The gap is, however, only 2-5X for simple models, and below 7X even for fully-connected deep nets. For asynchronous SGD, CPU is undoubtedly the optimal solution, outperforming GPU in time to convergence even when the GPU has a speedup of 10X or more. The choice between synchronous GPU and asynchronous CPU is not straightforward and depends on the task and the characteristics of the data. Thus, CPU should not be easily discarded for machine learning workloads. We hope that our insights provide a useful guide for applying parallel SGD in practice and – more importantly – choosing the appropriate computing architecture.

Stochastic gradient descent (SGD) usually samples training data based on the uniform distribution, which may not be a good choice because of the high variance of its stochastic gradient. Thus, importance sampling methods are considered in the literature to improve the performance. Most previous work on SGD-based methods with importance sampling requires the knowledge of Lipschitz constants of all component gradients, which are in general difficult to estimate. In this paper, we study an adaptive importance sampling method for common SGD-based methods by exploiting the local first-order information without knowing any Lipschitz constants. In particular, we periodically changes the sampling distribution by only utilizing the gradient norms in the past few iterations. We prove that our adaptive importance sampling non-asymptotically reduces the variance of the stochastic gradients in SGD, and thus better convergence bounds than that for vanilla SGD can be obtained. We extend this sampling method to several other widely used stochastic gradient algorithms including SGD with momentum and ADAM. Experiments on common convex learning problems and deep neural networks illustrate notably enhanced performance using the adaptive sampling strategy.

Two new methods for the optimization of probabilistic fuzzy classifiers are proposed. Probabilistic fuzzy systems are specially attractive due to their explicit and simultaneous modelling of two kinds of uncertainty, namely vagueness in linguistic terms (fuzziness) and probabilistic uncertainty. The current method uses the maximization of the likelihood with the stochastic gradient descent, which not only converges to local minima but also does not guarantee the minimization of the misclassification error. The proposed methods address this specific problem by incorporating global search techniques. The first algorithm proposed is a genetic algorithm with simple crossover and mutation operations. The other is a first generation memetic algorithm which combines the genetic algorithm with the stochastic gradient descent. A total of five benchmarks were used to compare the three algorithms. The results show that the proposed methods have an average relative improvement of 2% and 6% for the accuracy with the genetic and memetic algorithms, respectively.

Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in significantly more complicated methods than SGD. This paper proposes a new method to adaptively estimate a preconditioner, such that the amplitudes of perturbations of preconditioned stochastic gradient match that of the perturbations of parameters to be optimized in a way comparable to Newton method for deterministic optimization. Unlike the preconditioners based on secant equation fitting as done in deterministic quasi-Newton methods, which assume positive definite Hessian and approximate its inverse, the new preconditioner works equally well for both convex and nonconvex optimizations with exact or noisy gradients. When stochastic gradient is used, it can naturally damp the gradient noise to stabilize SGD. Efficient preconditioner estimation methods are developed, and with reasonable simplifications, they are applicable to large-scale problems. Experimental results demonstrate that equipped with the new preconditioner, without any tuning effort, preconditioned SGD can efficiently solve many challenging problems like the training of a deep neural network or a recurrent neural network requiring extremely long-term memories.

We propose a new method for fabric defect detection by incorporating the design of an adaptive wavelet-based feature extractor with the design of an Euclidean distance-based detector. The proposed method characterizes the fabric image with multiscale wavelet features by using undecimated discrete wavelet transforms. Each nonoverlapping window of the fabric image is then detected as defect or nondefect with an Euclidean distance-based detector. Instead of using the standard wavelet bases, an adaptive wavelet basis is designed for the detection of fabric defects. Minimization of the detection error Is achieved by incorporating the design of the adaptive wavelet with the design of the detector parameters using a discriminative feature extraction (DFE) training method. The proposed method has been evaluated on 480 defect samples from five types of defects, and 480 nondefect samples, where a 97.5% detection rate and 0.63% false alarm rate were achieved. The evaluations were also carried out on unknown types of defects, where a 93.3% detection rate and 3.97% false alarm rate were achieved in the detection of 180 defect samples and 780 nondefect samples. © 2002 Society of Photo-Optical Instrumentation Engineers.

Recently, Li et al. proposed a new auditory feature for robust speech recognition in noise environments. The new feature was derived by mimicking closely the function of human auditory process. Several filters were used to model the outer ear, middle ear, and cochlea, and the initial filter parameters and shapes were obtained from crude psychoacoustics results, experience, or experiments. Although one may adjust the feature parameters by hand to get better performance, the resulting feature parameters still may not be optimal in the sense of minimal recognition errors, especially for different tasks. To further improve the auditory feature, in this paper we apply discriminative training to optimize the auditory feature parameters with some guidance from psychoacoustic evidence but otherwise in a data-driven approach so as to minimize the recognition errors. One significant contribution over similar efforts in the past, such as discriminative feature extraction, is that we make no assumption on the parametric form of the auditory filters. Instead, we only require the filters to be smooth and triangular-like as suggested by psychoacoustics research. Our approach is evaluated on the Aurora database and achieves a word error reduction of 19.2%.