Bayesian adaptive nearest neighbor

We suggest a simple modification to the Kd-tree search algorithm for nearest neighbor search resulting in an improved performance. The Kd-tree data structure seems to work well in finding nearest neighbors in low dimensions but its performance degrades even if the number of dimensions increases to more than two. Since the exact nearest neighbor search problem suffers from the curse of dimensionality we focus on approximate solutions; a c-approximate nearest neighbor is any neighbor within distance at most c times the distance to the nearest neighbor. We show that for a randomly constructed database of points if the query point is chosen close to one of the points in the data base, the traditional Kd-tree search algorithm has a very low probability of finding an approximate nearest neighbor; the probability of success drops exponentially in the number of dimensions d as e − Ω(d/c). However, a simple change to the search algorithm results in a much higher chance of success. Instead of searching for the query point in the Kd-tree we search for a random set of points in the neighborhood of the query point. It turns out that searching for e Ω(d/c) such points can find the c-approximate nearest neighbor with a much higher chance of success.KeywordsSuccess ProbabilityRandom PointNeighbor SearchComputational GeometryQuery PointThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Currently, Nearest-Neighbor approaches (NN) have been widely applied to real world image data mining. These approaches have the following three disadvantages: (i) the performance is inferior on small datasets; (ii) the performance of approximated nearest neighbor search will degrade for data with high dimensions; (iii) they are heavily dependent on the chosen feature and distance measure. To overcome these intrinsic weaknesses, we propose a novel Nearest-Neighbor method, which improves the original NN approaches from three aspects. Firstly, we propose a novel neighborhood similarity measure, where the similarity between test images and labeled images in the database is calculated jointly by the original image-to-image similarity and the average similarity of their neighboring unlabeled data. Secondly, we adopt the kernelized locality sensitive hashing to effectively conduct the nearest neighbor search for high dimensional data. Finally, to enhance the robustness of the method on different genres of images, we propose to fuse the discrimination power of different features by considering all the retrieved nearest neighbors via hashing systems using different features/kernels. Experimental result shows the advantage over traditional Nearest-Neighbor methods using the labeled data only. Even when the ratio of labeled data is very small, our method could also achieve remarkable results, thanks to the help of unlabeled data and multiple features.

This paper proposes an approach based on fuzzy rough set theory to improve nearest neighbor based classification. Six measures are introduced to evaluate the quality of the nearest neighbors. This quality is combined with the frequency at which classes occur among the nearest neighbors and the similarity w.r.t. the nearest neighbor, to decide which class to pick among the neighbor's classes. The importance of each aspect is weighted using optimized weights. An experimental study shows that our method, Quality, Frequency and Similarity based Fuzzy Nearest Neighbor (QFSNN), outperforms state-of-the-art nearest neighbor classifiers.

The nearest neighbor classifier (NNC) is a non-parametric classification technique that classifies a test pattern to the class of its nearest neighbor in the training data. In this research, we applied the NNC to a standard letter recognition data and obtained a superior classification rate in comparison to extant approaches. A prime drawback to the NNC technique has been the relative inefficiency of the model. A modified NNC was implemented and applied to the same recognition problem. It is found that if we choose a suitable threshold minimum difference for classification, we can reduce the CPU time by half without lowering the performance of the classifier.

Machine learning (ML) applications have been thriving recently, largely attributed to the increasing availability of data. However, inconsistency and incomplete information are ubiquitous in real-world datasets, and their impact on ML applications remains elusive. In this paper, we present a formal study of this impact by extending the notion of Certain Answers for Codd tables, which has been explored by the database research community for decades, into the field of machine learning. Specifically, we focus on classification problems and propose the notion of "Certain Predictions" (CP) --- a test data example can be certainly predicted (CP'ed) if all possible classifiers trained on top of all possible worlds induced by the incompleteness of data would yield the same prediction. We study two fundamental CP queries: (Q1) checking query that determines whether a data example can be CP'ed; and (Q2) counting query that computes the number of classifiers that support a particular prediction (i.e., label). Given that general solutions to CP queries are, not surprisingly, hard without assumption over the type of classifier, we further present a case study in the context of nearest neighbor (NN) classifiers, where efficient solutions to CP queries can be developed --- we show that it is possible to answer both queries in linear or polynomial time over exponentially many possible worlds. We demonstrate one example use case of CP in the important application of "data cleaning for machine learning (DC for ML)." We show that our proposed CPClean approach built based on CP can often significantly outperform existing techniques, particularly on datasets with systematic missing values. For example, on 5 datasets with systematic missingness, CPClean (with early termination) closes 100% gap on average by cleaning 36% of dirty data on average, while the best automatic cleaning approach BoostClean can only close 14% gap on average.

The Nearest Neighbor (NN) classifier is one of the most used and well-known techniques for classification. However, NN suffers from several drawbacks such as high storage requirements, low efficiency in classification response, and noise sensitivity in the neighborhood. Prototype weighting as one the effective learning methods, assigns a weight to each prototype in order to determine its importance and simultaneously removes both redundant and noisy ones as a side effect. NN has an instance-based structure, and this is why; it is one of the rare methods for which the results of cross validation can be achieved for all possible combinations called as Complete-Cross-Validation (CCV) without need to construct and train exponential number of models. Using CCV as the objective function in learning the weights of prototypes can increase generalization capability of the classifier and also decrease the randomness of the learning method. Nevertheless, it cannot be computed such efficiently to be used as a fitness function during most of learning algorithms. In this paper, an iterative learning method has been proposed, in which the objective is incrementally improved without computing CCV even once. The proposed approach was assessed through a series of experiments with UCI in comparison with basic NN and related prototype weighting scheme and could outperform them.

A method to speed up the Bayes classifier

In this paper, an efficient finger vein recognition algorithm based on the combination of the nearest centroid neighbor and sparse representation classification techniques ( ${k}$ NCN-SRC) is presented. The previously proposed recognition algorithms are mainly based on distance computation. In the proposed method, the distance, as well as the spatial distribution, are considered to achieve a better recognition rate. The proposed method consists of two stages: first, the ${k}$ nearest neighbors of the test sample are selected based on the nearest centroid neighbor, and then, in the second stage, based on the selected number of closest nearest centroid neighbors ( ${k}$ ), the test sample is classified by sparse representation. Findings from the proposed method ${k}$ NCN-SRC demonstrated an increased recognition rate. This improvement can be attributed to the selection of the train samples, where the train samples are selected by considering the spatial and distance distribution. In addition, the complexity of SRC is reduced by reducing the number of train samples for the classification of the test sample by sparse representation and the processing speed of the proposed algorithm is significantly improved in comparison with the conventional SRC which is due to the reduced number of training samples. It can be concluded that the ${k}$ NCN-SRC classification method is efficient for finger vein recognition. An increase in the recognition rate of 3.35% , 9.07% , 20.23% , and 0.81% is obtained for the proposed ${k}$ NCN-SRC method in comparison with the conventional SRC for the four tested public finger vein databases.

One of the most popular queries in vehicle navigation, continuous k nearest neighbor, has been widely addressed. However, none of them focuses on continuous lookahead k nearest neighbor. Hence, in this paper, we propose a new approach, called continuous lookahead K nearest neighbor (CLKNN). CLKNN query is different from the traditional continuous k nearest neighbor, whereby in our CLKNN, mobile users concerns with only the interest points in the forward space of query point according to a predefined moving direction. Interest points, which are behind the moving query point, are not of interest anymore. We propose algorithms for lookahead KNN as well as continuous lookahead KNN. The former is used for static query point, whereas the latter is used for moving query point. Our experiments verify the applicability of the proposed approach to solve queries which involve lookahead k nearest neighbors continuously.

Passive polarization based imaging is a useful tool in computer vision and pattern recognition. A passive polarization imaging system forms a polarimetric image from the reflection of ambient light that contains useful information for computer vision tasks such as object detection (classification) and recognition. Applications of polarization based pattern recognition include material classification and automatic shape recognition. In this paper, we present two target detection algorithms for images captured by a passive polarimetric imaging system. The proposed detection algorithms are based on Bayesian decision theory. In these approaches, an object can belong to one of any given number classes and classification involves making decisions that minimize the average probability of making incorrect decisions. This minimum is achieved by assigning an object to the class that maximizes the a posteriori probability. Computing a posteriori probabilities requires estimates of class conditional probability density functions (likelihoods) and prior probabilities. A Probabilistic neural network (PNN), which is a nonparametric method that can compute Bayes optimal boundaries, and a -nearest neighbor (KNN) classifier, is used for density estimation and classification. The proposed algorithms are applied to polarimetric image data gathered in the laboratory with a liquid crystal-based system. The experimental results validate the effectiveness of the above algorithms for target detection from polarimetric data.

Forensic human identification with targeted microbiome markers using nearest neighbor classification

This paper is about a variant of k nearest neighbor classification on large many-class high dimensional datasets.K nearest neighbor remains a popular classification technique, especially in areas such as computer vision, drug activity prediction and astrophysics. Furthermore, many more modern classifiers, such as kernel-based Bayes classifiers or the prediction phase of SVMs, require computational regimes similar to k-NN. We believe that tractable k-NN algorithms therefore continue to be important.This paper relies on the insight that even with many classes, the task of finding the majority class among the k nearest neighbors of a query need not require us to explicitly find those k nearest neighbors. This insight was previously used in (Liu et al., 2003) in two algorithms called KNS2 and KNS3 which dealt with fast classification in the case of two classes. In this paper we show how a different approach, IOC (standing for the International Olympic Committee) can apply to the case of n classes where n > 2.IOC assumes a slightly different processing of the datapoints in the neighborhood of the query. This allows it to search a set of metric trees, one for each class. During the searches it is possible to quickly prune away classes that cannot possibly be the majority.We give experimental results on datasets of up to 5.8 x 105 records and 1.5 x 103 attributes, frequently showing an order of magnitude acceleration compared with each of (i) conventional linear scan, (ii) a well-known independent SR-tree implementation of conventional k-NN and (iii) a highly optimized conventional k-NN metric tree search.

This paper presents an algorithm, called the winner-update algorithm, for accelerating the nearest neighbor search. By constructing a hierarchical structure for each feature point in the l/sub p/ metric space, this algorithm can save a large amount of computation at the expense of moderate preprocessing and twice the memory storage. Given a query point, the cost for computing the distances from this point to all the sample points can be reduced by using a lower bound list of the distance established from Minkowski's inequality. Our experiments have shown that the proposed algorithm can save a large amount of computation, especially when the distance between the query point and its nearest neighbor is relatively small. With slight modification, the winner-update algorithm can also speed up the search for k nearest neighbors, neighbors within a specified distance threshold, and neighbors close to the nearest neighbor.

On multi-type reverse nearest neighbor search

Nearest neighbor search, a classic way of identifying similar data, can be applied to various areas, including database, machine learning, natural language processing, software engineering, etc. Secure nearest neighbor search aims to find nearest neighbors to a given query point over encrypted data without accessing data in plaintext. It provides privacy protection to datasets when nearest neighbor queries need to be operated by an untrusted party (e.g., a public server). While different solutions have been proposed to support nearest neighbor queries on encrypted data, these existing solutions still encounter critical drawbacks either in efficiency or privacy. In light of the limitations in the current literature, we propose a novel approximate nearest neighbor search solution, referred to as Sparser, by leveraging a combination of space-filling curves, perturbation, and Order-Preserving Encryption. The advantages of Sparser are twofold, strengthening privacy and improving efficiency. Specifically, Sparser pre-processes plaintext data with space-filling curves and perturbation, such that data is sparse, which mitigates leakage abuse attacks and renders stronger privacy. In addition to privacy enhancement, Sparser can efficiently find approximate nearest neighbors over encrypted data with logarithmic time. Through extensive experiments over real-world datasets, we demonstrate that Sparser can achieve strong privacy protection under leakage abuse attacks and minimize search time.