Using Locality-Sensitive Hashing for SVM Classification of Large Data Sets

Application of distributed SVM architectures in classifying forest data cover types

Finding nearest neighbors in high-dimensional spaces is a fundamental operation in many multimedia retrieval applications. Exact tree-based indexing approaches are known to suffer from the notorious curse of dimensionality for high-dimensional data. Approximate searching techniques sacrifice some accuracy while returning good enough results for faster performance. Locality Sensitive Hashing (LSH) is a very popular technique for finding approximate nearest neighbors in high-dimensional spaces. Apart from providing theoretical guarantees on the query results, one of the main benefits of LSH techniques is their good scalability to large datasets because they are external memory based. The most dominant costs for existing LSH techniques are the algorithm time and the index I/Os required to find candidate points. Existing works do not compare both of these dominant costs in their evaluation. In this experimental survey paper, we show the impact of both these costs on the overall performance of the LSH technique. We compare three state-of-the-art techniques on four real-world datasets, and show that, in contrast to recent works, C2LSH is still the state-of-the-art algorithm in terms of performance while achieving similar accuracy as its recent competitors.

A two-dimensional spectral model of bathymetry developed by Fox and Hayes [Rev. Geophys. 23(1), 1–48 (1985)] is applied to gridded sea beam data from the central Juan de Fuca Ridge in the northeast Pacific Ocean. Parameters of the model corresponding to (1) isotropic (nondirectionally dependent) roughness, (2) anisotropic (directionally dependent) roughness, (3) orientation of the anisotropic component, and (4) spectral rolloff (∼fractal dimension) are derived by fitting the model surface to two-dimensional amplitude spectra of bathymetry determined on a regular grid over the study area. Each term of the model is individually mapped and compared to ground truth data from SeaMARC-I sidescan sonar imagery. Parameters of the model clearly define the contrast between constructional volcanic terrain (rough, isotropic, with high fractal dimension) and tectonic extensional terrain (smoother, anisotropic, with low fractal dimension). An agglomerative, hierarchical cluster analysis is applied to the data, independent of spatial information, to delineate groups of spectra with similar characteristics. Distinct, mappable regions, corresponding to volcanic and tectonic provinces, are objectively determined. Also, coherent subregions of consistent spectral properties occur within the larger volcanic/tectonic divisions. The classification of large data sets, such as the entire Juan de Fuca Ridge system, can best be accomplished through the use of apriori techniques, such as k-means clustering, which requires the input of predefined characteristics for bathymetric provinces.

In this paper, we propose a system for file classification in large data sets based on spiking neural networks (SNNs). File information contained in key-value metadata pairs is mapped by a novel correlative temporal encoding scheme to spike patterns that are input to an SNN. The correlation between input spike patterns is determined by a file similarity measure. Unsupervised training of such networks using spike-timing-dependent plasticity (STDP) is addressed first. Then, supervised SNN training is considered by backpropagation of an error signal that is obtained by comparing the spike pattern at the output neurons with a target pattern representing the desired class. The classification accuracy is measured for various publicly available data sets with tens of thousands of elements, and compared with other learning algorithms, including logistic regression and support-vector machines. Simulation results indicate that the proposed SNN-based system using memristive synapses may represent a valid alternative to classical machine learning algorithms for inference tasks, especially in environments with asynchronous ingest of input data and limited resources.

Many machine learning and pattern recognition algorithms rely heavily on good distance metrics to achieve competitive performance. While distance metrics can be learned, the computational expense of doing so is currently infeasible on large datasets. In this paper, we propose two efficient-and-effective approaches for selecting the training dataset using Locality-Sensitive Hashing (LSH) with discriminative information, and with K-Means clustering inside LSH buckets, for accelerating metric learning. Our methods yield a speedup factor of (N/C)2, where N is training set size and C ≪ N is the user-selected compressed set size, achieving quadratic speedup to metric learning often realized as a 1–2 or more orders of magnitude improvement with little degradation to accuracy. For example, our generic filter approach enables the current overall fastest Large Margin Nearest Neighbor (LMNN) to learn metrics on one million samples in 6.8 minutes down from 5.4hrs—a 48x speedup. LMNN and similar state-of-the-art methods use tree data structures to speed up nearest-neighbor queries—an advantage that degrades at higher dimensions. Our approach does not share this limitation.

Data Mining deals with efficient algorithms for dealing with large data. When such algorithms are combined with data compaction, they would lead to superior performance. Approaches to deal with large data include working with representatives of data instead of entire data. The representatives should preferably be generated with minimal data scans. In the current chapter we discuss working with methods of lossy and non-lossy data compression methods combined with clustering and classification of large datasets. We demonstrate the working of such schemes on two large data sets.

Locality-sensitive hashing (LSH) has emerged as the dominant algorithmic technique for similarity search with strong performance guarantees in high-dimensional spaces. A drawback of traditional LSH schemes is that they may have false negatives, i.e., the recall is less than 100%. This limits the applicability of LSH in settings requiring precise performance guarantees. Building on the recent theoretical construction that eliminates false negatives, we propose a fast and practical covering LSH scheme for Hamming space called Fast CoveringLSH (fcLSH). Inheriting the design benefits of CoveringLSH our method avoids false negatives and always reports all near neighbors. Compared to CoveringLSH we achieve an asymptotic improvement to the hash function computation time from O(dL) to O(d + (LlogL), where d is the dimensionality of data and L is the number of hash tables. Our experiments on synthetic and real-world data sets demonstrate that fcLSH is comparable (and often superior) to traditional hashing-based approaches for search radius up to 20 in high-dimensional Hamming space.

Variable selection based on locally linear embedding mapping for near-infrared spectral analysis

LSH (Locality Sensitive Hashing) is one of the best known methods for solving the c-approximate nearest neighbor problem in high dimensional spaces. This paper presents a variant of the LSH algorithm, focusing on the special case of where all points in the dataset lie on the surface of the unit hypersphere in a d-dimensional Euclidean space. The LSH scheme is based on a family of hash functions that preserves locality of points. This paper points out that when all points are constrained to lie on the surface of the unit hypersphere, there exist hash functions that partition the space more efficiently than the previously proposed methods. The design of these hash functions uses randomly rotated regular polytopes and it partitions the surface of the unit hypersphere like a Voronoi diagram. Our new scheme improves the exponent ρ, the main indicator of the performance of the LSH algorithm.

This paper proposes a unified framework for outlier detection in high dimensional spaces from an ensemble-learning viewpoint. Moreover, to demonstrate the usefulness of our framework, we developed a very simple and fast algorithm, namely SOE1, in which only subspaces with one dimension is used for mining outliers from large categorical datasets. Experimental results demonstrate the superiority of SOE1 algorithm.

When the traditional algorithm is used to calculate the feature classification of high-dimensional non-equilibrium and large data set, it is easy to appear the problem of low accuracy and recall rate of feature selection. Therefore, a feature selection algorithm based on granular fusion is designed. By using the regularization feature of the data, the original big data aggregate is transformed into a small-scale data subset. On the basis of this, the feature selection function of the data particle is obtained. Finally, the weight fusion calculation of each feature subset is carried out. The feature classification of high-dimensional non-equilibrium big data set is realized. The experimental results show that the feature selection algorithm based on granular fusion can realize the feature selection and recall of high dimensional unbalanced data sets. The accuracy of the method is higher than that of the traditional method, which shows that the method is feasible and effective.KeywordsHigh dimensional dataNon-equilibrium featureGranulation fusionFeature selection

The paper proposes a new approach in data censoring, which allows correcting diagnostic errors in the data sets in case when these samples are described in high-dimensional feature spaces. Considering this case as a separate task is explained by the fact that in high-dimensional spaces most of the methods of outliers detection and data filtering, both statistical and metric, stop working. At the same time, for the tasks of medical diagnostics, given the complexity of the objects and phenomena studied, a large number of descriptive characteristics are the norm rather than the exception. To solve this problem, an approach that focuses on local similarity between objects belonging to the same class and uses the function of rival similarity (FRiS function) as a measure of similarity has been proposed. In this approach for efficient data cleaning from misclassified objects, the most informative and relevant low-dimensional feature subspace is selected, in which the separability of classes after their correction will be maximal. Class separability here means the similarity of objects of one class to each other and their dissimilarity to objects of another class. Cleaning data from class errors can consist both in their correction and removing the objects-outliers from the data set. The described method was implemented as a FRiS-LCFS algorithm (FRiS Local Censoring with Feature Selection) and tested on model and real biomedical problems, including the problem of diagnosing prostate cancer based on DNA microarray analysis. The developed algorithm showed its competitiveness in comparison with the standard methods for filtering data in high-dimensional spaces.

COMBINATORIAL AND DISCRETE GEOMETRY Finite Point Configurations, J. Pach Packing and Covering, G. Fejes Toth Tilings, D. Schattschneider and M. Senechal Helly-Type Theorems and Geometric Transversals, R. Wenger Pseudoline Arrangements, J.E. Goodman Oriented Matroids, J. Richter-Gebert and G.M. Ziegler Lattice Points and Lattice Polytopes, A. Barvinok New! Low-Distortion Embeddings of Finite Metric Spaces, P. Indyk and J. Matousek New! Geometry and Topology of Polygonal Linkages, R. Connelly and E.D. Demaine New! Geometric Graph Theory, J. Pach Euclidean Ramsey Theory, R.L. Graham Discrete Aspects of Stochastic Geometry, R. Schneider Geometric Discrepancy Theory and Uniform Distribution, J.R. Alexander, J. Beck, and W.W.L. Chen Topological Methods, R.T. Zivaljevic Polyominoes, S.W. Golomb and D.A. Klarner POLYTOPES AND POLYHEDRA Basic Properties of Convex Polytopes, M. Henk, J. Richter-Gebert, and G.M. Ziegler Subdivisions and Triangulations of Polytopes, C.W. Lee Face Numbers of Polytopes and Complexes, L.J. Billera and A. Bjoerner Symmetry of Polytopes and Polyhedra, E. Schulte Polytope Skeletons and Paths, G. Kalai Polyhedral Maps, U. Brehm and E. Schulte ALGORITHMS AND COMPLEXITY OF FUNDAMENTAL GEOMETRIC OBJECTS Convex Hull Computations, R. Seidel Voronoi Diagrams and Delaunay Triangulations, S. Fortune Arrangements, D. Halperin Triangulations and Mesh Generation, M. Bern Polygons, J. O'Rourke and S. Suri Shortest Paths and Networks, J.S.B. Mitchell Visibility, J. O'Rourke Geometric Reconstruction Problems, S.S. Skiena New! Curve and Surface Reconstruction, T.K. Dey Computational Convexity, P. Gritzmann and V. Klee Computational Topology, G. Vegter Computational Real Algebraic Geometry, B. Mishra GEOMETRIC DATA STRUCTURES AND SEARCHING Point Location, J. Snoeyink New! Collision and Proximity Queries, M.C. Lin and D. Manocha Range Searching, P.K. Agarwal Ray Shooting and Lines in Space, M. Pellegrini Geometric Intersection, D.M. Mount New! Nearest Neighbors in High-Dimensional Spaces, P. Indyk COMPUTATIONAL TECHNIQUES Randomization and Derandomization, O. Cheong, K. Mulmuley, and E. Ramos Robust Geometric Computation, C.K. Yap Parallel Algorithms in Geometry, M.T. Goodrich Parametric Search, J.S. Salowe New! The Discrepancy Method in Computational Geometry, B. Chazelle APPLICATIONS OF DISCRETE AND COMPUTATIONAL GEOMETRY Linear Programming, M. Dyer, N. Megiddo, and E. Welzl Mathematical Programming, M.H. Todd Algorithmic Motion Planning, M. Sharir Robotics, D. Halperin, L.E. Kavraki, and J.-C. Latombe Computer Graphics, D. Dobkin and S. Teller New! Modeling Motion, L.J. Guibas Pattern Recognition, J. O'Rourke and G.T. Toussaint Graph Drawing, R. Tamassia and G. Liotta Splines and Geometric Modeling, C.L. Bajaj New! Surface Simplification and 3D Geometry Compression, J. Rossignac Manufacturing Processes, R. Janardan and T.C. Woo Solid Modeling, C.M. Hoffmann New! Computation of Robust Statistics: Depth, Median, and Related Measures, P.J. Rousseeuw and A. Struyf New! Geographic Information Systems, M. van Kreveld Geometric Application of the Grassmann-Cayley Algebra, N.L. White Rigidity and Scene Analysis, W. Whiteley Sphere Packing and Coding Theory, G.A. Kabatiansky and J.A. Rush Crystals and Quasicrystals, M. Senechal New! Biological Applications of Computational Topology, H. Edelsbrunner New! GEOMETRIC SOFTWARE Software, J. Joswig Two Computation Geometry Libraries: LEDA and CGAL, L. Kettner and S. Naher Index of Defined Terms New! Index of Cited Authors

A dataset can be clustered by merging the bucket indices that come from the random projection of locality sensitive hashing functions. It should be noted that for this to work the merging interval must be calculated first. To improve the feasibility of large scale data clustering in high dimensional space we propose an enhanced Locality Sensitive Hashing Clustering Method. Firstly, multiple hashing functions are generated. Secondly, data points are projected to bucket indices. Thirdly, bucket indices are clustered to get class labels. Experimental results showed that on synthetic datasets this method achieves high accuracy at much improved cluster speeds. These attributes make it well suited to clustering data in high dimensional space.

Many real applications such as businesses and health generate large categorical datasets with uncertainty. A fundamental task is to efficiently discover hidden and non-trivial patterns from such large uncertain categorical datasets. Since the exact value of an attribute is often unknown in uncertain categorical datasets, conventional clustering analysis algorithms do not provide a suitable means for dealing with categorical data, uncertainty, and stability. The ability of decision making in the presence of vagueness and uncertainty in data can be handled using Rough Set Theory. Though, recent categorical clustering techniques based on Rough Set Theory help but they suffer from low accuracy, high computational complexity, and generalizability especially on data sets where they sometimes fail or hardly select their best clustering attribute. The main objective of this research is to propose a new information theoretic based Rough Purity Approach (RPA). Another objective of this work is to handle the problems of traditional Rough Set Theory based categorical clustering techniques. Hence, the ultimate goal is to cluster uncertain categorical datasets efficiently in terms of the performance, generalizability and computational complexity. The RPA takes into consideration information-theoretic attribute purity of the categorical-valued information systems. Several extensive experiments are conducted to evaluate the efficiency of RPA using a real Supplier Base Management (SBM) and six benchmark UCI datasets. The proposed RPA is also compared with several recent categorical data clustering techniques. The experimental results show that RPA outperforms the baseline algorithms. The significant percentage improvement with respect to time (66.70%), iterations (83.13%), purity (10.53%), entropy (14%), and accuracy (12.15%) as well as Rough Accuracy of clusters show that RPA is suitable for practical usage. We conclude that as compared to other techniques, the attribute purity of categorical-valued information systems can better cluster the data. Hence, RPA technique can be recommended for large scale clustering in multiple domains and its performance can be enhanced for further research.