Local Classifier Weighting by Quadratic Programming

A new k-harmonic nearest neighbor classifier based on the multi-local means

We investigate the geometric complexity of decision boundaries for robust training compared to standard training. By considering the local geometry of nearest neighbour sets, we study them in a model-agnostic way and theoretically derive a lower-bound \(R^*\in \mathbb {R}\) on the perturbation magnitude \(\delta \in \mathbb {R}\) for which robust training provably requires a geometrically more complex decision boundary than accurate training. We show that state-of-the-art robust models learn more complex decision boundaries than their non-robust counterparts, confirming previous hypotheses. Then, we compute \(R^*\) for common image benchmarks and find that it also empirically serves as an upper bound over which label noise is introduced. We demonstrate for deep neural network classifiers that perturbation magnitudes \(\delta \ge R^*\) lead to reduced robustness and generalization performance. Therefore, \(R^*\) bounds the maximum feasible perturbation magnitude for norm-bounded robust training and data augmentation. Finally, we show that \(R^*< 0.5R\) for common benchmarks, where R is a distribution’s minimum nearest neighbour distance. Thus, we improve previous work on determining a distribution’s maximum robust radius.

Remote sensing technology is widely used in land surveying and earth science research, such as the study of the earth’s oceans, glaciers, hydrology, ecology, geology and so on, is still in the military, intelligence, commercial, economic and other aspects can also be used to select the appropriate training samples, supervised classification with five kinds of single classifier, the classification accuracy comparison of different methods. It was found that the maximum classification accuracy of the two plots was the highest with the maximum likelihood method, but the classification accuracy was not the highest for 1 of the cultivated areas. Therefore, two decision fusion based multiple classifiers combination algorithms are proposed, and the results of single classifier are processed by using ENVI and IDL software. Compared with the classification results of a single classifier, the overall classification accuracy of multiple classifiers is improved by 2.5%, and the accuracy of 1 of the cultivated land in the study area is increased by 15.5%. Methods the combination of multiple classifiers can use different characteristics of single classifier, so as to compensate for the lack of a single classifier.

This paper proposes a method for efficient nearest neighbor classification in non-Euclidean spaces with computationally expensive similarity/distance measures. Efficient approximations of such measures are obtained using the BoostMap algorithm, which produces embeddings into a real vector space. A modification to the BoostMap algorithm is proposed, which uses an optimization cost that is more appropriate when our goal is classification accuracy as opposed to nearest neighbor retrieval accuracy. Using the modified algorithm, multiple approximate nearest neighbor classifiers are obtained, that provide a wide range of trade-offs between accuracy and efficiency. The approximations are automatically combined to form a cascade classifier, which applies the slower and more accurate approximations only to the hardest cases. The proposed method is experimentally evaluated in the domain of handwritten digit recognition using shape context matching. Results on the MNIST database indicate that a speed-up of two to three orders of magnitude is gained over brute force search, with minimal losses in classification accuracy.

In this paper, we propose a multi-local means based nearest neighbor classifier (MLMNN). In the MLMNN, k categorical nearest neighbors of a query sample are first found and used to calculate the corresponding k categorical multi-local mean vectors which can represent different local class-specific sample distributions. Then, the query sample is represented by a linear combination of k categorical local mean vectors and the representation coefficient of each local mean vector as the contribution to representing and classifying the query sample is obtained. Finally, the class-specific representation-based distance (i.e. reconstruction residual) between the query sample and k categorical multi-local mean vectors is adopted to determine the class label of the query sample. The experimental results on three popular face databases show that the proposed MLMNN method outperforms the related competitive KNN-based methods.

Combining multiple global models (e.g. back-propagation based neural networks) is an effective technique for improving classification accuracy by reducing a variance through manipulating training data distributions. Standard combining methods do not improve local classifiers (e.g. k-nearest neighbors) due to their low sensitivity to data perturbation. Here, we propose an adaptive attribute boosting technique to coalesce multiple local classifiers each using different relevant attribute information. In addition, a modification of boosting method is developed for heterogeneous databases with unstable driving attributes by drawing blocks of data at each boosting round. To reduce the computational costs of k-nearest neighbor (k-NN) classifiers, a novel fast k-NN algorithm is designed. The adaptive attribute boosting applied to real life data and artificial data show observable improvements in prediction accuracy for both local and global classifiers when unstable driving attributes are present in the data. The spatial variant of boosting applied to the same data sets resulted in highly significant improvements for the k-NN classifier, making it competitive to boosted neural networks.

Perceptual relativity-based local hyperplane classification

Classification of imbalanced data sets is one of the basic challenges in the field of machine learning and data mining. There have been many proposed methods for classification of such data sets up to now. In algorithmic level methods, new algorithms are created which are adapted to the nature of imbalanced data sets. Gravitational fixed radius nearest neighbor algorithm (GFRNN) is an algorithmic level method developed with the aim of enhancing k nearest neighbor classifier to acquire the ability of dealing with imbalanced data sets. This algorithm, utilizes the sum of gravitational forces on a query sample from its nearest neighbors in a fixed radius to determine its label. Simplicity and no need for parameter setting during the run of algorithm are the main advantages of this method. In this paper, a method is proposed for improving the performance of GFRNN algorithm in which mass assigning of each training sample is done based on the sum of gravitational forces from other training samples on it. The results obtained from applying the proposed method on 10 data sets prove the superiority of it compared with 5 other algorithms.

Transductive learning is a special case of semi-supervised learning, where class labels to the test patterns alone are found. That is, the domain of the learner is the test set alone. Often, transductive learners achieve a better classification accuracy, since additional information in the form of test patterns location in the feature-space is used. For several inductive learners, there exists corresponding transductive learners; like for SVMs there exists transductive SVMs (TSVMs). For nearest neighbor based classifiers, their corresponding transductive methods are achieved through graph mincuts or spectral graph mincuts. It is shown that these solutions achieve low leave-one-out cross-validation (LOOCV) error with respect to nearest neighbor based classifiers. It is formally shown in the paper that, through a clustering method, it is possible to get various solutions having zero LOOCV error with respect to nearest neighbor based classifiers. Some solutions can have low classification accuracy. The paper proposes, instead of optimizing LOOCV error, to optimize a margin like criterion. This criterion is based on the observation that similar labeled patterns should be nearer to each other, while dissimilar labeled patterns should be far away. An approximate method to solve the proposed optimization problem is given in the paper which is called selective incremental transductive nearest neighbor classifier (SI-TNNC). SI-TNNC finds the test pattern from the test set which is very close to one class of training patterns and at the same time very much away from the other class of training examples. The selected test pattern is given its nearest neighbor's label and is added to the training set. This pattern is removed from the test set. The process is repeated with the next best test pattern, and is stopped only when the test set becomes empty. An algorithm to implement SI-TNNC method is given in the paper which has a quadratic time complexity. Other related solutions have either cubic time complexity or are NP-hard. Experimentally, using several standard data-sets, it is shown that the proposed transductive learner achieves on-par or better classification accuracy than its related competitors.

The characteristics of society in choosing contraceptive methods are also the crucial factors for the government to prepare the family planning services needed at Bulakamba District, Brebes Regency, Central Java. In this case, a classification process needs to be done to assist the process of classifying the characteristics of society in the selection of contraceptive methods. Multinomial Logistic Regression classification is good in exploring data information meanwhile Fuzzy K Nearest Neighbor (FK-NN) classification is good for handling big data and noise. These two methods used in this study because they are relevant to the data applied and will be compared their classification accuracy through APER and Press's Q calculations.The classification accuracy results obtained based on the APER calculation for Multinomial Logistic Regression is 88,25% and Fuzzy K Nearest Neighbor (FK-NN) is 88,92%. Meanwhile, the Press's Q value of both methods are 9600,945 and 9518,014 greater than χ 2𝛼,1 which is 3,841, so that it is statistically accurate. Based on the results obtained, it can be concluded that Multinomial Logistic Regression classification method has a better classification accuracy than Fuzzy K Nearest Neighbor (FK-NN) method.

In this paper, we propose the two-phase representation based classification called the two-phase linear reconstruction measure based classification (TPLRMC). It is inspired from the fact that the linear reconstruction measure (LRM) gauges the similarities among feature samples by decomposing each feature sample as a liner combination of the other feature samples with \(L_{p}\)-norm regularization. Since the linear reconstruction coefficients can fully reveal the feature’s neighborhood structure that is hidden in the data, the similarity measures among the training samples and the query sample are well provided in classifier design. In TPLRMC, it first coarsely seeks the K nearest neighbors for the query sample with LRM, and then finely represents the query sample as the linear combination of the determined K nearest neighbors and uses LRM to perform classification. The experimental results on face databases show that TPLRMC can significantly improve the classification performance.

In recent years, assessing human emotions through Electroencephalogram (EEG) is become one of the active research area in Brain Computer Interface (BCI) development. The combination of surface Laplacian filtering, time-frequency analysis (wavelet transform) and linear classifiers (K Nearest Neighbor (KNN) and Linear Discriminant Analysis (LDA)) are used to detect the discrete emotions (happy, surprise, fear, disgust, and neutral) of human through EEG signals. The database is generated with 20 subjects in the age group of 21~39 years using 64 channels with a sampling frequency of 256 Hz. An audio-visual induction (video clips) based protocol has been designed for evoking the discrete emotions. The raw EEG signals are preprocessed through surface Laplacian filtering method and decomposed into five different EEG frequency bands using Wavelet Transform (WT) and the statistical features from alpha frequency band is considered for classifying the emotions. In our work, there are four different wavelet functions (ldquodb4rdquo, ldquodb8rdquo, ldquosym8rdquo and ldquocoif5rdquo) are used to derive the linear and non linear features for classifying the emotions. The validation of statistical features is performed using 5 fold cross validation. In this work, KNN outperforms LDA by offering a maximum average classification rate of 78.04 % on 62 channels, 77.61% and 71.30% on 24 channels and 8 channels respectively. Finally we present the average classification accuracy and individual classification accuracy of two different classifiers for justifying the performance of our emotion recognition system.

Aiming at the shortcomings of single network classification model, this paper applies CNN-LSTM (convolutional neural networks-long short-term memory) combined network in the field of music emotion classification and proposes a multifeature combined network classifier based on CNN-LSTM which combines 2D (two-dimensional) feature input through CNN-LSTM and 1D (single-dimensional) feature input through DNN (deep neural networks) to make up for the deficiencies of original single feature models. The model uses multiple convolution kernels in CNN for 2D feature extraction, BiLSTM (bidirectional LSTM) for serialization processing and is used, respectively, for audio and lyrics single-modal emotion classification output. In the audio feature extraction, music audio is finely divided and the human voice is separated to obtain pure background sound clips; the spectrogram and LLDs (Low Level Descriptors) are extracted therefrom. In the lyrics feature extraction, the chi-squared test vector and word embedding extracted by Word2vec are, respectively, used as the feature representation of the lyrics. Combining the two types of heterogeneous features selected by audio and lyrics through the classification model can improve the classification performance. In order to fuse the emotional information of the two modals of music audio and lyrics, this paper proposes a multimodal ensemble learning method based on stacking, which is different from existing feature-level and decision-level fusion methods, the method avoids information loss caused by direct dimensionality reduction, and the original features are converted into label results for fusion, effectively solving the problem of feature heterogeneity. Experiments on million song dataset show that the audio classification accuracy of the multifeature combined network classifier in this paper reaches 68%, and the lyrics classification accuracy reaches 74%. The average classification accuracy of the multimodal reaches 78%, which is significantly improved compared with the single-modal.

Purpose The email is an important medium for sharing information rapidly. However, spam, being a nuisance in such communication, motivates the building of a robust filtering system with high classification accuracy and good sensitivity towards false positives. In that context, this paper aims to present a combined classifier technique using a committee selection mechanism where the main objective is to identify a set of classifiers so that their individual decisions can be combined by a committee selection procedure for accurate detection of spam. Design/methodology/approach For training and testing of the relevant machine learning classifiers, text mining approaches are used in this research. Three data sets (Enron, SpamAssassin and LingSpam) have been used to test the classifiers. Initially, pre-processing is performed to extract the features associated with the email files. In the next step, the extracted features are taken through a dimensionality reduction method where non-informative features are removed. Subsequently, an informative feature subset is selected using genetic feature search. Thereafter, the proposed classifiers are tested on those informative features and the results compared with those of other classifiers. Findings For building the proposed combined classifier, three different studies have been performed. The first study identifies the effect of boosting algorithms on two probabilistic classifiers: Bayesian and Naïve Bayes. In that study, AdaBoost has been found to be the best algorithm for performance boosting. The second study was on the effect of different Kernel functions on support vector machine (SVM) classifier, where SVM with normalized polynomial (NP) kernel was observed to be the best. The last study was on combining classifiers with committee selection where the committee members were the best classifiers identified by the first study i.e. Bayesian and Naïve bays with AdaBoost, and the committee president was selected from the second study i.e. SVM with NP kernel. Results show that combining of the identified classifiers to form a committee machine gives excellent performance accuracy with a low false positive rate. Research limitations/implications This research is focused on the classification of email spams written in English language. Only body (text) parts of the emails have been used. Image spam has not been included in this work. We have restricted our work to only emails messages. None of the other types of messages like short message service or multi-media messaging service were a part of this study. Practical implications This research proposes a method of dealing with the issues and challenges faced by internet service providers and organizations that use email. The proposed model provides not only better classification accuracy but also a low false positive rate. Originality/value The proposed combined classifier is a novel classifier designed for accurate classification of email spam.

Classifier combination based on confidence transformation