The neural tree classifier: Complex decision boundaries with single-layer nets

We investigate the geometric complexity of decision boundaries for robust training compared to standard training. By considering the local geometry of nearest neighbour sets, we study them in a model-agnostic way and theoretically derive a lower-bound \(R^*\in \mathbb {R}\) on the perturbation magnitude \(\delta \in \mathbb {R}\) for which robust training provably requires a geometrically more complex decision boundary than accurate training. We show that state-of-the-art robust models learn more complex decision boundaries than their non-robust counterparts, confirming previous hypotheses. Then, we compute \(R^*\) for common image benchmarks and find that it also empirically serves as an upper bound over which label noise is introduced. We demonstrate for deep neural network classifiers that perturbation magnitudes \(\delta \ge R^*\) lead to reduced robustness and generalization performance. Therefore, \(R^*\) bounds the maximum feasible perturbation magnitude for norm-bounded robust training and data augmentation. Finally, we show that \(R^*< 0.5R\) for common benchmarks, where R is a distribution’s minimum nearest neighbour distance. Thus, we improve previous work on determining a distribution’s maximum robust radius.

Previous research demonstrates the sensitivity of adults and infants to the statistical regularity of input distributions defining speech categories [D. L. Grieser and P. K. Kuhl, Dev. Psychol. 25, 577–588 (1989)] and even non-human animals exhibit such sensitivity [Kluender et al., J. Acoust. Soc. Am. 104, 3568–3580 (1998)]. Speech categories’ structure also possesses information to support the use of decision boundaries in categorization. To investigate the interaction of distribution versus decision-boundary information in auditory category learning, the current research tracked listeners learning novel non-speech categories defined by two acoustic dimensions, the center frequency and modulation frequency, via explicit training with feedback. Early in learning, listeners exhibited sensitivity to distributional regularities of categories by robustly responding to more densely sampled regions of acoustic space. However, evidence of listeners’ reliance on a decision boundary emerged once learning plateaued. The point in learning at which decision boundaries predicted categorization response was determined not only by the type of listeners’ prior experience with the sounds but also by the perceptual salience of the acoustic dimensions. [Work supported by NSF.]

Compressed signal processing (CSP) is a branch of compressive sensing (CS), which gives a direction to solve a class of signal processing problems directly from the compressive measurements of a signal. CSP utilizes the information preserved in the compressive measurements of a signal to solve certain inference problems like: classification, detection, and estimation, without reconstructing the original signal. It further simplifies the signal processing compared to conventional CS by omitting their complex reconstruction stage. This, in turn, reduces the implementation complexity of signal processing systems. This paper investigates the performance of CSP for classification application. After extracting the features from compressive measurements, these features or the data instances are used for classification purpose. Through experimental analysis, it has been found that as the CS undersampling factor is increased, the overlapping among the data instances predominates. This results in a complex decision boundary, which in turn degrades the classification accuracy at higher undersampling factors. To overcome the above issue, this paper proposes the use of a machine learning method known as overlap aware learning along with CSP. This generates a smoother decision boundary and hence improves the classification accuracy at higher undersampling factors. The simulation results show the trend of improved classification accuracy using the proposed method. An analysis of the proposed method has been done on different datasets and based on run-time complexity and complexity vs gain analysis to verify the effectiveness of proposed method.

A two-level learning algorithm that decomposes multilayer neural networks into a set of sub-networks is presented. Many popular optimization methods, such as conjugate-gradient and quasi-Newton methods, can be utilized to train these sub-networks. In addition, if the activation functions are hard-limiting functions, the multilayer neural networks can be trained by the perceptron learning rule in this two-level learning algorithm. Two experimental problems are given as examples for this algorithm.

The paper proposes a method for the design optimization of electromagnetic devices. It utilizes an artificial neural network with the quasi-Newton algorithm. The proposed method can determine optimal weights and biases in the neural network more rapidly than the conventional method. A simple electromagnetic device is optimized by using this method. The paper reviews several optimization techniques for learning in multilayer neural networks. From the results of comparative numerical simulations, it proposes a method to construct the nonlinear mapping function more rapidly than the conventional algorithm, that is, the error back-propagation proposed by Rumelhart et al. (1986).

Artificial neural networks (ANNs) are widely used in applications with complex decision boundaries. A large number of activation functions have been proposed in the literature to achieve better representations of the observed data. However, only a few works employ Tsallis statistics, which has successfully been applied to various other fields. This paper presents a random neural network (RNN) with q -Gaussian activation functions [ q -generalized RNN (QRNN)] based on Tsallis statistics. The proposed method employs an additional parameter q (called the entropic index) which reflects the degree of nonextensivity. This approach has the flexibility to model complex decision boundaries of different shapes by varying the entropic index. We conduct numerical experiments to analyze the efficiency of QRNN compared with RNNs and several other classical methods. Statistical tests (Wilcoxon and Friedman) are used to validate our results and show that the QRNN performs significantly better than RNNs with different activation functions. In addition, we find that QRNN outperforms many of the compared classical methods, with the exception of support vector machines, in which case it still exhibits a substantial advantage in terms of implementation simplicity and speed.

It has been widely accepted that the classification accuracy can be improved by combining outputs of multiple classifiers. However, how to combine multiple classifiers with various (potentially conflicting) decisions is still an open problem. A rich collection of classifier combination procedures -- many of which are heuristic in nature -- have been developed for this goal. In this brief, we describe a dynamic approach to combine classifiers that have expertise in different regions of the input space. To this end, we use local classifier accuracy estimates to weight classifier outputs. Specifically, we estimate local recognition accuracies of classifiers near a query sample by utilizing its nearest neighbors, and then use these estimates to find the best weights of classifiers to label the query. The problem is formulated as a convex quadratic optimization problem, which returns optimal nonnegative classifier weights with respect to the chosen objective function, and the weights ensure that locally most accurate classifiers are weighted more heavily for labeling the query sample. Experimental results on several data sets indicate that the proposed weighting scheme outperforms other popular classifier combination schemes, particularly on problems with complex decision boundaries. Hence, the results indicate that local classification-accuracy-based combination techniques are well suited for decision making when the classifiers are trained by focusing on different regions of the input space.

We investigate star-galaxy classification for astronomical surveys in the context of four methods enabling the interpretation of black-box machine learning systems. The first is outputting and exploring the decision boundaries as given by decision tree based methods, which enables the visualization of the classification categories. Secondly, we investigate how the Mutual Information based Transductive Feature Selection (MINT) algorithm can be used to perform feature pre-selection. If one would like to provide only a small number of input features to a machine learning classification algorithm, feature pre-selection provides a method to determine which of the many possible input properties should be selected. Third is the use of the tree-interpreter package to enable popular decision tree based ensemble methods to be opened, visualized, and understood. This is done by additional analysis of the tree based model, determining not only which features are important to the model, but how important a feature is for a particular classification given its value. Lastly, we use decision boundaries from the model to revise an already existing method of classification, essentially asking the tree based method where decision boundaries are best placed and defining a new classification method. We showcase these techniques by applying them to the problem of star-galaxy separation using data from the Sloan Digital Sky Survey (hereafter SDSS). We use the output of MINT and the ensemble methods to demonstrate how more complex decision boundaries improve star-galaxy classification accuracy over the standard SDSS frames approach (reducing misclassifications by up to $\approx33\%$). We then show how tree-interpreter can be used to explore how relevant each photometric feature is when making a classification on an object by object basis.

The objective of this work was to develop a classification system in order to confirm the authenticity of Galician potatoes with a Certified Brand of Origin and Quality (CBOQ) and to differentiate them from other potatoes that did not have this quality brand. Elemental analysis (K, Na, Rb, Li, Zn, Fe, Mn, Cu, Mg and Ca) of potatoes was performed by atomic spectroscopy in 307 samples belonging to two categories, CBOQ and Non-CBOQ potatoes. The 307 x 10 data set was evaluated employing multivariate chemometric techniques, such as cluster analysis and principal component analysis in order to perform a preliminary study of the data structure. Different classification systems for the two categories on the basis of the chemical data were obtained applying several commonly supervised pattern recognition procedures [such as linear discriminant analysis, K-nearest neighbours (KNN), soft independent modelling of class analogy and multilayer feed-forward neural networks]. In spite of the fact that some of these classification methods produced satisfactory results, the particular data distribution in the 10-dimensional space led to the proposal of a new vector quantization-based classification procedure (VQBCP). The results achieved with this new approach (percentages of recognition and prediction abilities > 97%) were better than those attained by KNN and can be compared advantageously with those provided by LDA (linear discriminant analysis), SIMCA (soft independent modelling of class analogy) and MLF-ANN (multilayer feed-forward neural networks). The new VQBCP demonstrated good performance by carrying out adequate classifications in a data set in which the classes are subgrouped. The metal profiles of potatoes provided sufficient information to enable classification criteria to be developed for classifying samples on the basis of their origin and brand.

Multilayer neural networks are typical learning systems. Here we can find all the necessary components of a learning system: a performance index, a memory, and learning algorithms. Being designed according to the principles of their biological analogues, multilayer neural networks (MNN) are able to solve a wide range of problems in pattern recognition [1], identification [2], control of complex dynamical non-linear systems [3], [4], robot control [5], etc.

Many advanced methods proposed for control of robot manipulators are based on the dynamic models of the robot systems. Model-based control design needs a correct dynamic model and precise parameters of the system. Practically speaking, however, every dynamic model has some degrees of incorrectness and every parameter associates with some degrees of identification error. The incorrectness and errors eventually result positioning or trajectory tracking errors, and even cause the system to be unstable. In the past two decades, intensive research activities have been devoted on the design of robust control systems and adaptive control systems for the robot in order to overcome the control system drawback caused by the model errors and uncertain parameters, and a great number of research results have been reported, for example, (Hsia, 1989), (Kou, and Wang, 1989), (Slotine and Li, 1989), ( Spong, 1992), and (Cheah, Liu and Slotine, 2006). However, almost parts of results associate with complicated control system design approaches and difficulties in the control system implementation for industrial robot manipulators. Recently, neural network technology attracts many attentions in the design of robot controllers. It has been pointed out that multi-layered neural network can be used for the approximation of any nonlinear function. Other advantages of the neural networks often cited are parallel distributed structure, and learning ability. They make such the artificial intelligent technology attractive not only in the application areas such as pattern recognition, information and graphics processing, but also in intelligent control of nonlinear and complicated systems such as robot manipulators (Sanger, 1994), (Kim and Lewis, 1999), (Kwan and Lewis, 2000), (Jung and Yim, 2001) (Yu and Wang, 2001). A new field in robot control using neural network technology is beginning to emerge to deal with the issues related to the dynamics in the robot control design. A neural network based dynamics compensation method has been proposed for trajectory control of a robot system (Jung and Hsia, 1996). A combined approach of neural network and sliding mode technology for both feedback linearization and control error compensation has been presented (Barambones and Etxebarria, 2002). Sensitivity of a neural network performance to learning rate in robot control has been investigated (Clark and Mills, 2000). In the following, we present a simple control system consisting of a traditional controller and a neural network controller with parallel structure for trajectory tracking control of 16

In recent years, multilayer neural networks and recurrent networks have emerged as important components for representing nonlinear transformations and have proved particularly successful in pattern recognition and optimization problems. The authors explore methods for incorporating such networks in adaptive systems for the identification and control of complex nonlinear dynamical systems. >

Viewing pattern recognition as a problem in statistical classification wherein an n-dimensional hypercube is partitioned into category regions with decision boundaries, this paper focuses on a class of nonlinear boundary forms and describes a variable-structure system which is capable of evolving these boundaries adaptively. Most of the present adaptive recognition systems have a priori fixed structure, usually corresponding to a linear decision procedure, and adaptation is performed by parameter optimization. It is apparent that in practice the simple linear system frequently will be an inadequate approximation to the desired boundaries. The central problem in a more general recognition procedure is the selection and analysis of suitable nonlinear relations among measurements. In the system proposed in this paper, the structure adaptation develops as follows: the recognition network initially assumes a linear structure, a set of relations among the input measurements is generated and selected on the basis of a set of design data, and to accomodate these relations, the structure changes to nonlinear. By repetitively generating and selecting additional relations among the measurements, the structure gradually adjusts itself within the class of allowable structures toward an optimal configuration. By means of simulation on a digital computer, the procedure for structure adaptation was successfully applied to a number of practical problems, handwritten numerals, spoken vowels, and electroencephalograms.

A great amount of information about a classification problem is contained in those instances falling near the decision boundary. This intuition dates back to the earliest studies in pattern recognition, and in the more recent adaptive approaches to the so called boundary hunting, such as the work of Aha et alii on Instance Based Learning and the work of Vapnik et alii on Support Vector Machines. The last work is of particular interest, since theoretical and experimental results ensure the accuracy of boundary reconstruction. However, its optimization approach has heavy computational and memory requirements, which limits its application on huge amounts of data. In the paper we describe an alternative approach to boundary hunting based on adaptive labeled quantization architectures. The adaptation is performed by a stochastic gradient algorithm for the minimization of the error probability. Error probability minimization guarantees the accurate approximation of the optimal decision boundary, while the use of a stochastic gradient algorithm defines an efficient method to reach such approximation. In the paper comparisons to Support Vector Machines are considered.

Optimum path forest-based classifiers are a novel approach for supervised pattern recognition. The OPF classifier differs from traditional approaches by not estimating probability density functions of the classes neither assuming samples linearity, and creates a discrete optimal partition of the feature space, in which the decision boundary is obtained by the influence zones of the most representative samples of the training set. Due to the large number of applications in biomedical signal processing involving pattern recognition techniques, specially voice disorders identification, we propose here the laryngeal pathology detection by means of OPF. Experiments were performed in three public datasets against SVM, and a comparison in terms of accuracy rates and execution times was also regarded.