Clustering in Feature Space for Interesting Pattern Identification of Categorical Data

In this paper, a kernel-based nonlinear version of the oblique subspace projection (OBSP) operator is defined in terms of kernel functions. Input data are implicitly mapped into a high-dimensional kernel feature space by a nonlinear mapping, which is associated with a kernel function. The OBSP expression is then derived in the feature space, which is kernelized in terms of the kernel functions in order to avoid explicit computation in the high-dimensional feature space. The resulting kernelized OBSP algorithm is equivalent to a nonlinear OBSP in the original input space. Experimental results based on simulated hyperspectral data and real hyperspectral imagery shows that the kernel oblique subspace projection (KOBSP) outperforms the conventional OBSP.

Social networking sites such as Facebook or Twitter attract millions of users, who everyday post an enormous amount of content in the form of tweets, comments and posts. Since social network texts are usually short, learning tasks have to deal with a very high dimensional and sparse feature space, in which most features have low frequencies. As a result, extracting useful knowledge from such noisy data is a challenging task, that converts large-scale short-text learning tasks in social environments into one of the most relevant problems in machine learning and data mining. Feature selection is one of the most known and commonly used techniques for reducing the impact of the high dimensional feature space in text learning. A wide variety of feature selection techniques can be found in the literature applied to traditional, long-texts and document collections. However, short-texts coming from the social Web pose new challenges to this well-studied problem as texts’ shortness offers a limited context to extract enough statistical evidence about words relations (e.g. correlation), and instances usually arrive in continuous streams (e.g. Twitter timeline), so that the number of features and instances is unknown, among other problems. This paper surveys feature selection techniques for dealing with short texts in both offline and online settings. Then, open issues and research opportunities for performing online feature selection over social media data are discussed.

Orthogonal discriminant analysis algorithms have recently been proposed. However, these methods don't address the singularity problem in the high dimensional feature space. In this paper, we present a new method called Direct Orthogonal Discriminant Analysis (DODA), which is able to extract all the orthogonal discriminant vectors simultaneously in the high-dimensional feature space and does not suffer the singularity problem. This method is very simple and easy to be implemented. Experimental results show that the proposed method is very competitive in comparison with some existing dimensionality reduction algorithms.

We propose a kernel-based fuzzy clustering algorithm to cluster data in the feature space. Our method uses kernel functions to project data from the original space into a high dimensional feature space, and then divides them into groups through their similarities in the feature space with an incremental clustering approach. After clustering, data patterns of the same cluster in the feature space are then grouped with an arbitrarily shaped boundary in the original space. The effectiveness of our method is demonstrated in the experiments.

Support vector machines are known for their high capability of generalization and have been successfully applied to various classification and regression problems by employing kernel techniques to define nonlinear feature maps from a low dimensional input space into a very high dimensional feature space. Kernel techniques have an advantage in making possible to work in the implicitly introduced feature spaces without cost of computations. However, kernel functions are exploited without specific insight into problems. Given a feature map explicitly, a kernel function can naturally be defined by the inner product between data pairs in the feature space. This paper proposes an approach to acquire optimal feature maps which realize both the linear separability and the maximization of margin by adaptive learning on training data

Clustering high dimensional data with sparse features is challenging because pairwise distances between data items are not informative in high dimensional space. To address this challenge, we propose two novel semi-supervised clustering methods that incorporate prior knowledge in the form of pairwise cluster membership constraints. In particular, we project high-dimensional data onto a much reduced-dimension subspace, where rough clustering structure defined by the prior knowledge is strengthened. Metric learning is then performed on the subspace to construct more informative pairwise distances. We also propose to propagate constraints locally to improve the informativeness of pairwise distances. When the new methods are evaluated using two real benchmark data sets, they show substantial improvement using only limited prior knowledge.

The massive photometric data collected from multiple large-scale sky surveys offer significant opportunities for measuring distances of celestial objects by photometric redshifts. However, catastrophic failure is an unsolved problem with a long history and it still exists in the current photometric redshift estimation approaches (such as the k-nearest neighbor (KNN) algorithm). In this paper, we propose a novel two-stage approach by integration of KNN and support vector machine (SVM) methods together. In the first stage, we apply the KNN algorithm to photometric data and estimate their corresponding zphot. Our analysis has found two dense regions with catastrophic failure, one in the range of zphot ϵ [0.3, 1.2] and the other in the range of zphot ϵ [1.2, 2.1]. In the second stage, we map the photometric input pattern of points falling into the two ranges from their original attribute space into a high dimensional feature space by using a Gaussian kernel function from an SVM. In the high dimensional feature space, many outliers resulting from catastrophic failure by simple Euclidean distance computation in KNN can be identified by a classification hyperplane of SVM and can be further corrected. Experimental results based on the Sloan Digital Sky Survey (SDSS) quasar data show that the two-stage fusion approach can significantly mitigate catastrophic failure and improve the estimation accuracy of photometric redshifts of quasars. The percents in different |δz| ranges and root mean square (rms) error by the integrated method are 83.47%, 89.83%, 90.90% and 0.192, respectively, compared to the results by KNN (71.96%, 83.78%, 89.73% and 0.204).

Kernel principal component analysis (KPCA) is a method for performing a nonlinear form of principal component analysis (PCA). It involves nonlinear mapping of the input data onto a high dimensional feature space and performing PCA there. Through KPCA, nonlinear relations in the input space are detected as linear relations in the feature space. Furthermore, explicit computation of the mapping is not necessary in KPCA. Only dot products of the mapped data items in the feature space are required which are obtained directly from the input data using a kernel function. Moreover, control limits of traditional statistical process monitoring indices are usually determined under assumption of normal (Gaussian) distribution. Although, it is recognized that this is inappropriate in nonlinear processes, the impact of using kernel density estimation (KDE), which is a nonparametric method, to determine control limits in KPCA-based process monitoring has not been reported. This paper seeks to bridge this gap. The KPCA with KDE approach is applied to the Tennessee Eastman process for the detection of faults. The results confirm that associating KPCA with kernel density estimated control limits provides better monitoring performance than using control limits based on the normal probability density function.

A new theoretical model of Pattern Recognition principles was proposed, which is based on matter cognition instead of matter classification in traditional statistical Pattern Recognition. This new model is closer to the function of human being, rather than traditional statistical Pattern Recognition using optimal separating as its main principle. So the new model of Pattern Recognition is called the Biomimetic Pattern Recognition (BPR). Its mathematical basis is placed on topological analysis of the sample set in the high dimensional feature space. Therefore, it is also called the Topological Pattern Recognition (TPR). The fundamental idea of this model is based on the fact of the continuity in the feature space of any one of the certain kinds of samples. We experimented with the Biomimetic Pattern Recognition (BPR) by using artificial neural networks, which act through covering the high dimensional geometrical distribution of the sample set in the feature space. Omnidirectionally cognitive tests were done on various kinds of animal and vehicle models of rather similar shapes. For the total 8800 tests, the correct recognition rate is 99.87%. The rejection rate is 0.13% and on the condition of zero error rates, the correct rate of BPR was much better than that of RBF-SVM.

Based on general findings from the field of neuroscience and their algorithmic implementations using signal processing, information theory and machine learning techniques, this paper highlights the advantages of modelling a signal in a sparse and high-dimensional feature space. The emphasis is put on the hierarchical organisation, very high dimensionality and sparseness aspects of auditory information, that allow unsupervised learning of meaningful auditory objects from simple linear projections. When the dictionaries are learned using independent component analysis (ICA), it is shown that specific spectro-temporal modulation patterns are learned to optimally represent speech, noise and tonal components. In a noisy isolated-word speech recognition task, sparse and high-dimensional features have shown greater robustness to noise compared to a standard system based on a dense low-dimensional feature space. This brings new ways of thinking in the field of recognition and classification of acoustic signals.

Quantitative evaluation of similarity between feature densities of images is an important step in several computer vision and data‐mining applications such as registration of two or more images and retrieval and clustering of images. Previously we had introduced a new class of similarity measures based on entropic graphs to estimate Rènyi's α‐entropy, α‐Jensen difference divergence, α‐mutual information, and other divergence measures for image registration. Entropic graphs such as the minimum spanning tree (MST) and k‐Nearest neighbor (kNN) graph allow the estimation of such similarity measures in higher dimensional feature spaces. A major drawback of histogram‐based estimates of such measures is that they cannot be reliably constructed in higher dimensional feature spaces.In this article, we shall briefly extrapolate upon the use of entropic graph based divergence measures mentioned above. Additionally, we shall present estimates of other divergence viz the Geometric‐Arithmetic mean divergence and Henze–Penrose affinity. We shall present the application of these measures for pairwise image registration using features derived from independent component analysis of the images. An extension of pairwise image registration is to simultaneously register multiple images, a challenging problem that arises while constructing atlases of organs in medical imaging. Using entropic graph methods we show the feasibility of such simultaneous registration using graph based higher dimensional estimates of entropy measures. Finally we present a new nonlinear correlation measure that is invariant to nonlinear transformations of the underlying feature space and can be reliably constructed in higher dimensions. We present an image clustering experiment to demonstrate the robustness of this measure to nonlinear transformations and contrast it with the clustering performance of the linear correlation coefficient. © 2007 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 16, 130–145, 2006

Highlights A low time complexity and high accuracy recognition method of tomato’s disease stages was proposed.Small number of disease image samples can support a stable and high accuracy method with less computation time and less memory cost.Provided a solution to establish a real-time and low energy consumption disease recognize system.Abstract. Disease stages recognition of tomato is important for the timely diagnosis and prevention of tomato diseases. In this article, the color and texture features were extracted, and the kernel mutual subspace method (KMSM) was introduced to establish a rapid Tomato’s disease stage recognition method. Firstly, the color and textural features were extracted from tomato leaf and mapped to a high-dimensional space using a Gaussian kernel function. Secondly, a nonlinear disease feature subspace was established by applying principal component analysis (PCA) based on the mapped high-dimensional space. Finally, disease stages are recognized by calculating the canonical angles between the testing subspace and reference sub-spaces. To validate recognition rate of the proposed method, we conducted 10-fold cross-validation for experiments using the tomato disease sets contained in PlantVillage and artificial intelligence (AI) Challenger 2018 datasets. Also compared with the support vector machine (SVM) and VGG 16 methods, the results show that, accuracy of those three evaluated methods on the PlantVillage datasets are 99.34%, 89.2%, and 97%, respectively; and accuracy on the AI Challenger 2018 datasets are 98.66%, 71.14%, and 73.93%, respectively. Moreover, average training and recognition time of the proposed method is 0.1496 and 0.008 s, respectively, which is faster than SVM and VGG16 methods. In conclusion, the proposed method can be carried out in real-time recognition intelligent equipment which requires less memory space, less computation time, and with higher recognition rate, and low energy consumption. Keywords: Color and textural feature, High dimensional feature space, Kernel mutual subspace method, Tomato’s disease stages recognition,.

We develop a face recognition algorithm which is insensitive to gross variation in lighting direction and facial expression. Taking a pattern classification approach, we consider each pixel in an image as a coordinate in a high-dimensional space. We take advantage of the observation that the images of a particular face under varying illumination direction lie in a 3-D linear subspace of the high dimensional feature space — if the face is a Lambertian surface without self-shadowing. However, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather than explicitly modeling this deviation, we project the image into a subspace in a manner which discounts those regions of the face with large deviation. Our projection method is based on Fisher's Linear Discriminant and produces well separated classes in a low-dimensional subspace even under severe variation in lighting and facial expressions. The Eigenface technique, another method based on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensive experimental results demonstrate that the proposed “Fisherface” method has error rates that are significantly lower than those of the Eigenface technique when tested on the same database.

Sensorimotor information provided by mossy fibers (MF) is mapped to high-dimensional space by a huge number of granule cells (GrC) in the cerebellar cortex's input layer. Significant studies have demonstrated the computational advantages and primary contributor of this expansion recoding. Here, we propose a novel perspective on the expansion recoding where each GrC serve as a kernel basis function, thereby the cerebellum can operate like a kernel machine that implicitly use high dimensional (even infinite) feature spaces. We highlight that the generation of kernel basis function is indeed biologically plausible scenario, considering that the key idea of kernel machine is to memorize important input patterns. We present potential regimes for developing kernels under constrained resources and discuss the advantages and disadvantages of each regime using various simulation settings.

Beamforming is a powerful processing tool for direction-of-arrival (DOA) estimation. The performance of a beamformer is limited by physical characteristics of the array in use, and searching space is often crowded in the form expressed in Euclid space. When the input space is transformed into a feature space with high dimension, e.g. infinite dimension in Hilbert space, it exhibits highly sparse characteristics. On that basis, the beamforming problem can be included in the framework of the compressive sensing (CS). In this paper, we map the sampled data into a high dimensional feature space with sparse representation induced by a kernel function. The DOA estimation is solved by using a kernel-based matching pursuit algorithm. When compared with the traditional CS, the new method improves the feature separability, and produces a better performance distinguishing coherent sources with fewer snapshots.