Large Scale Metric learning

Nearest Neighbor search is one of the simplest and most intuitive ideas in data mining. Due to it's simplicity and diverse utility, Nearest Neighbor search is often found to be the workhorse of a variety of data mining, machine learning, and computer vision algorithms. For very high dimensional data, the naive linear search tends to be optimal. This is due to the so called curse of dimensionality in database search. To accelerate the search time, may researchers have turned to the power and efficiency of GPGPU computing; mainly for computing the data intensive distance metrics. Though promising speedups are achieved through this method, it fails to capitalize on many of the recent advances in algorithms for bounded error approximate nearest neighbor searches. Approximate nearest neighbor methods give satisfactory results, especially in very large and possibly redundant datasets, while boasting sub-linear search complexities. In this paper we present a c-approximate r-nearest neighbor search algorithm for CUDA using Locality Sensitive Hash with Nearest Neighbor search (LSH-NN). We implement this system in CUDA and test it on real world image SIFT vector data. Our tests show that we are able to achieve significant speedup over the serial version, with good approximations on scalability, while achieving the near-optimal search complexity of LSH-NN.

Bootstrapping search margin-based nearest neighbor method for qualitative spectroscopic analysis.

Nearest neighbor search is a fundamental problem in computer vision and machine learning. The straightforward solution, linear scan, is both computationally and memory intensive in large scale high-dimensional cases, hence is not preferable in practice. Therefore, there have been a lot of interests in algorithms that perform approximate nearest neighbor (ANN) search. In this paper, we propose a novel addition-based vector quantization algorithm, Asymmetric Mapping Quantization (AMQ), to efficiently conduct ANN search. Unlike existing addition-based quantization methods that suffer from handling the problem caused by the norm of database vector, we map the query vector and database vector using different mapping functions to transform the computation of L-2 distance to inner product similarity, thus do not need to evaluate the norm of database vector. Moreover, we further propose Distributed Asymmetric Mapping Quantization (DAMQ) to enable AMQ to work on very large dataset by distributed learning. Extensive experiments on approximate nearest neighbor search and image retrieval validate the merits of the proposed AMQ and DAMQ.

ABSTRACTIn this article, speeded-up robust features (SURF) for each image have been calculated. Discrete Fourier transform (DFT) method has been applied to these SURF. High dimensions of these SURF–DFT feature vectors are reduced to low dimensions with large-margin nearest neighbor (LMNN), Gaussian process latent variable models (GPLVM), and neighborhood component analysis (NCA). When size reduction process was done, effect on the GPLVM, LMNN, and NCA of the 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 feature numbers has been examined. These features are classified by naive Bayes (NB) classifier. Thus, SURF_DFT_GPLVM_NB, SURF_DFT_NCA_NB, and SURF_DFT_LMNN_NB methods for gastric histopathological images have been developed. Classification results obtained with these methods have been compared. According to the obtained results, the highest classification result was obtained as 90.24% by using 4 features by SURF_DFT_GPLVM_NB method for second group images.

The rapid advancement of deepfake technology poses significant challenges to athletic celebrities, where altered or falsified media can impact athletes’ reputations, fan engagement, and the integrity of match broadcasting. This paper proposes a novel framework for deepfake image recognition for athletic celebrities using locality sensitive hashing (LSH). LSH, an efficient technique for high‐dimensional nearest neighbor searches, is employed to detect and differentiate deepfake images from authentic media. By extracting high‐dimensional features from images and videos using convolutional neural networks (CNNs), LSH is applied to hash similar content into clusters for quick and accurate deepfake detection. The proposed method is tested on real‐world dataset, showing promising results in terms of accuracy and computational efficiency. This research highlights the importance of integrating advanced hashing techniques like LSH in safeguarding the authenticity of digital content and provides insights into future directions for deepfake detection mechanisms.

Accurate automated classification of brain tumors from magnetic resonance imaging (MRI) is essential for early diagnosis and treatment. This study presents a hybrid framework combining Convolutional Neural Network (CNN) deep features, Large Margin Nearest Neighbor (LMNN) metric learning, and swarm-intelligence optimization for robust four-class classification. Five pretrained CNNs—DenseNet201, MobileNetV2, ResNet50, ResNet101, and InceptionV3—were evaluated on a dataset of 7,023 images categorized as glioma, meningioma, pituitary, healthy. Among these, DenseNet201 provided the highest baseline performance with 92.66% accuracy. LMNN improved feature separability, while Particle Swarm Optimization (PSO) and Grey Wolf Optimizer (GWO) selected compact subsets. The selected features were classified using k-Nearest Neighbor (KNN), Support Vector Machine (SVM), Artificial Neural Network (ANN), and Random Forest (RF). The DenseNet201–LMNN–GWO–KNN configuration, termed DenseWolf-K, achieved the best performance with 99.64% accuracy, establishing it as the optimal implementation of the framework. Robustness and generalizability were further confirmed using an independent external dataset. Model explainability was ensured through feature-level ranking of GWO-selected features and occlusion sensitivity maps, an Explainable Artifical Intelligence (XAI) method. Overall, the proposed DenseWolf-K framework delivers high accuracy, low false-negative rates, compact representation, and enhanced interpretability, representing a reliable and efficient solution for MRI-based brain tumor classification.

As known, the supervised feature extraction aims to search a discriminative low dimensional space where the new samples in the sample class cluster tightly and the samples in the different classes keep away from each other. For most of algorithms, how to push these samples located in class margin or in other class (called hard samples in this paper) towards the class is difficult during the transformation. Frequently, these hard samples affect the performance of most of methods. Therefore, for an efficient method, to deal with these hard samples is very important. However, fewer methods in the past few years have been specially proposed to solve the problem of hard samples. In this study, the large margin nearest neighbor (LMNN) and weighted local modularity (WLM) in complex network are introduced respectively to deal with these hard samples in order to push them towards the class quickly and the samples with the same labels as a whole shrink into the class, which both result in small within-class distance and large margin between classes. Combined WLM with LMNN, a novel feature extraction method named WLMLMNN is proposed, which takes into account both the global and local consistencies of input data in the projected space. Comparative experiments with other popular methods on various real-world data sets demonstrate the effectiveness of the proposed method.

In this paper we present a novel classification method of SPECT images for the development of a computer aided diagnosis (CAD) system aiming to improve the early detection of the Alzheimer’s Disease (AD). The system combines firstly template-based normalized mean square error (NMSE) features of tridimensional Regions of Interest (ROIs) t-test selected with secondly Large Margin Nearest Neighbors (LMNN), which is a distance metric technique aiming to separate examples from different classes (Controls and AD) by a Large Margin. LMNN uses a rectangular matrix (called RECT-LMNN) as an effective feature reduction technique. Moreover, the proposed system evaluates Support Vector Machine (SVM) classifier, yielding a 97.93% AD diagnosis accuracy, which reports clear improvements over existing techniques, for instance the Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA) or Normalized Minimum Squared Error (NMSE) evaluated with SVM.KeywordsSPECT Brain ImagingAlzheimer’s diseaseDistance Metric Learningfeature reductionSupport Vector Machines

Instance-based learning techniques use a set of stored training instances to classify new examples. The most common such learning technique is the nearest neighbor method, in which new instances are classified according to the closest training instance. A critical element of any such method is the metric used to determine distance between instances. Euclidean distance is by far the most commonly used metric; no one, however, has systematically considered whether a different metric, such as Manhattan distance, might perform equally well on naturally occurring data sets. Some evidence from psychological research indicates that Manhattan distance might be preferable in some circumstances. This paper examines three different distance metrics and presents experimental comparisons using data from three domains: malignant cancer classification, heart disease diagnosis, and diabetes prediction. The results of these studies indicate that the Manhattan distance metric works works quite well, although not better than the Euclidean metric that has become a standard for machine learning experiments. Because the nearest neighbor technique provides a good benchmark for comparisons with other learning algorithms, the results below include a number of such comparisons, which show that nearest neighbor, using any distance metric, compares quite well to other machine learning techniques.

Machine Learning (ML) algorithms, as the core technology in Artificial Intelligence (AI) applications, such as self-driving vehicles, make important decisions by performing a variety of data classification or prediction tasks. Attacks on data or algorithms in AI applications can lead to misclassification or misprediction, which can fail the applications. For each dataset separately, the parameters of ML algorithms should be tuned to reach a desirable classification or prediction accuracy. Typically, ML experts tune the parameters empirically, which can be time consuming and does not guarantee the optimal result. To this end, some research suggests an analytical approach to tune the ML parameters for maximum accuracy. However, none of the works consider the ML performance under attack in their tuning process. This paper proposes an analytical framework for tuning the ML parameters to be secure against attacks, while keeping its accuracy high. The framework finds the optimal set of parameters by defining a novel objective function, which takes into account the test results of both ML accuracy and its security against attacks. For validating the framework, an AI application is implemented to recognize whether a subject's eyes are open or closed, by applying k-Nearest Neighbors (kNN) algorithm on her Electroencephalogram (EEG) signals. In this application, the number of neighbors (k) and the distance metric type, as the two main parameters of kNN, are chosen for tuning. The input data perturbation attack, as one of the most common attacks on ML algorithms, is used for testing the security of the application. Exhaustive search approach is used to solve the optimization problem. The experiment results show k = 43 and cosine distance metric is the optimal configuration of kNN for the EEG dataset, which leads to 83.75% classification accuracy and reduces the attack success rate to 5.21%.

Nearest neighbor (NN) search is a fundamental issue in many computer applications, such as multimedia search, computer vision and machine learning. While this problem is trivial in low-dimensional search spaces, it becomes much more difficult in higher dimension because of the phenomenon known as the curse of dimensionality, where the complexity grows exponentially with dimension and the data tends to show strong correlations between dimensions. In this paper, we introduce a new hashing method to efficiently cope with this challenge. The idea is to split the search space into many subspaces based on a number of jointly-independent and uniformly-distributed circular random variables (CRVs) computed from the data points. Our method has been tested on datasets of local SIFT and global GIST features and was compared to locality sensitive hashing (LSH), Spherical Hashing methods (HD and SHD) and the fast library for approximate nearest neighbor (FLANN) matcher by using linear search as a baseline. The experimental results show that our method outperforms all state-of-the-art methods for the GIST features. For SIFT features, the results indicate that our method significantly reduces the search query time while preserving the search quality and outperforms FLANN for datasets of size less than 200 K points.

One central task in many visual surveillance scenarios is person re-identification, i.e., recognizing an individual person across a network of spatially disjoint cameras. Most successful recognition approaches are either based on direct modeling of the human appearance or on machine learning. In this work, we aim at taking advantage of both directions of research. On the one hand side, we compute a descriptive appearance representation encoding the vertical color structure of pedestrians. To improve the classification results, we additionally estimate the transition between two cameras using a pair-wisely estimated metric. In particular, we introduce 4D spatial color histograms and adopt Large Margin Nearest Neighbor (LMNN) metric learning. The approach is demonstrated for two publicly available datasets, showing competitive results, however, on lower computational costs.

We propose a novel method using Locality-Sensitive Hashing (LSH) for solving the optimization problem that arises in the training stage of support vector machines for large data sets, possibly in high dimensions. LSH was introduced as an efficient way to look for neighbors in high dimensional spaces. Random projections-based LSH functions create bins so that when great probability points belonging to the same bin are close, the points that are far will not be in the same bin. Based on these bins, it is not necessary to consider the whole original set but representatives in each one of them, thus reducing the effective size of the data set. A key of our proposal is that we work with the feature space and use only the projections to search for closeness in this space. Moreover, instead of choosing the projection directions at random, we sample a small subset and solve the associated SVM problem. Projections in this direction allows for a more precise sample in many cases and an approximation of the solution of the large problem is found in a fraction of the running time with small degradation of the classification error. We present two algorithms, theoretical support, and numerical experiments showing their performances on real life problems taken from the LIBSVM data base.

The great advances in deep learning enable human beings to extract the semantic information in ubiquitous unstructured data into high-dimensional vectors, leading to an urgent priority of managing vector data. Nearest neighbor (NN) query is a pivotal question in vector data management. Unfortunately, the NN query has been proved to be intrinsically hard. Researchers, thus, turn to approximate nearest neighbor (ANN) query which targets better efficiency by slightly compromising accuracy. Despite this, existing ANN methods still incur high time and space costs, which are mainly due to the high dimensionality. Specifically, in most in-memory ANN algorithms, the time cost is dominated by the time for comparing the distances among vectors. Each distance comparison operation (DCO) involves hundreds of arithmetic operations as each vector has hundreds of dimensions. Similarly, the space cost is dominated by the space for storing the vectors. Towards the goal of reducing the time and space costs, we present three methods in this thesis. First, we observe that, for a DCO, if an approximate distance can confirm that a vector is not a NN, then it can be discarded without computing exact distances. Based on this observation, we propose a method named ADSampling. For a DCO, it iteratively and incrementally samples a few dimensions and computes an approximate distance. When this distance is accurate enough to decide the result of the DCO, it terminates sampling. Second, to further improve the efficiency of DCOs, we design an efficient distance estimator. We propose a new quantization method named RaBitQ, which maps $D$-dimensional vectors into $D$-bit strings. RaBitQ provides an unbiased and asymptotically optimal estimator with good empirical accuracy. In addition, we introduce efficient implementations of RaBitQ based on bitwise operations or SIMD-based operations. Third, by extending the RaBitQ method, we develop an algorithm which compresses a floating-point vector into a $B$-bit unsigned integer vector. We prove that our algorithm is asymptotically optimal in terms of the space-accuracy trade-off.

Many large multimedia applications require efficient processing of nearest neighbor queries. Often, multimedia data are represented as a collection of important high-dimensional feature vectors. Existing Locality Sensitive Hashing (LSH) techniques require users to find top-k similar feature vectors for each of the feature vectors that represent the query object. This leads to wasted and redundant work due to two main reasons: 1) not all feature vectors may contribute equally in finding the top-k similar multimedia objects, and 2) feature vectors are treated independently during query processing. Additionally, there is no theoretical guarantee on the returned multimedia results. In this work, we propose a practical and efficient indexing approach for finding top-k approximate nearest neighbors for multimedia data using LSH called mmLSH, which can provide theoretical guarantees on the returned multimedia results. Additionally, we present a buffer-conscious strategy to speed up the query processing. Experimental evaluation shows significant gains in performance time and accuracy for different real multimedia datasets when compared against state-of-the-art LSH techniques.