Sparse Subspace Clustering: Algorithm, Theory, and Applications

The problem of clustering noisy and incompletely observed high-dimensional data points into a union of low-dimensional subspaces and a set of outliers is considered. The number of subspaces, their dimensions, and their orientations are assumed unknown. We propose a simple low-complexity subspace clustering algorithm, which applies spectral clustering to an adjacency matrix obtained by thresholding the correlations between data points. In other words, the adjacency matrix is constructed from the nearest neighbors of each data point in spherical distance. A statistical performance analysis shows that the algorithm exhibits robustness to additive noise and succeeds even when the subspaces intersect. Specifically, our results reveal an explicit tradeoff between the affinity of the subspaces and the tolerable noise level. We furthermore prove that the algorithm succeeds even when the data points are incompletely observed with the number of missing entries allowed to be (up to a log-factor) linear in the ambient dimension. We also propose a simple scheme that provably detects outliers, and we present numerical results on real and synthetic data.

Similarity search plays an important role in many applications involving high-dimensional data. Due to the known dimensionality curse, the performance of most existing indexing structures degrades quickly as the feature dimensionality increases. Hashing methods, such as locality sensitive hashing (LSH) and its variants, have been widely used to achieve fast approximate similarity search by trading search quality for efficiency. However, most existing hashing methods make use of randomized algorithms to generate hash codes without considering the specific structural information in the data. In this paper, we propose a novel hashing method, namely, robust hashing with local models (RHLM), which learns a set of robust hash functions to map the high-dimensional data points into binary hash codes by effectively utilizing local structural information. In RHLM, for each individual data point in the training dataset, a local hashing model is learned and used to predict the hash codes of its neighboring data points. The local models from all the data points are globally aligned so that an optimal hash code can be assigned to each data point. After obtaining the hash codes of all the training data points, we design a robust method by employing l2,1 -norm minimization on the loss function to learn effective hash functions, which are then used to map each database point into its hash code. Given a query data point, the search process first maps it into the query hash code by the hash functions and then explores the buckets, which have similar hash codes to the query hash code. Extensive experimental results conducted on real-life datasets show that the proposed RHLM outperforms the state-of-the-art methods in terms of search quality and efficiency.

Finding a new drug which is both safe and efficient is an expensive and time-consuming endeavour. In particular, establishing the ‘dose-effect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them. To improve dose-effect estimation, Zhou, Liu, Fang et al. developed a new general-purpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments. To facilitate implementation, Zhou, Liu, Fang et al. set up a user-friendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform dose-effect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.

High-dimensional data in many applications can be considered as samples drawn from a union of multiple lowdimensional manifolds. Assigning data points into their own manifolds is referred to manifold clustering. Inspired by recent advances in subspace clustering, in this paper, we present an efficient approach for manifold clustering, called Local Convex Representation (LCR), in which each data point is represented as a convex combination of other points in the local neighborhood and under some mild conditions the nonzero coefficients are guaranteed to correspond to the data points lying on the same manifold. Moreover, we incorporate the estimated intrinsic dimension of the manifold to prune the minor nonzero coefficients and validate that the pruning step helps LCR yield remarkable improvements. Experiments on synthetic data as well as real world data demonstrate promising performance.

Surface display techniques, including contour mapping, usually use an interpolant when data values are on a rectangular grid; but when values are irregularly located some form of assumed or estimated distance-weighting function is often used to estimate the influence of data points at other locations. If moving average (or some other) techniques are then applied, slopes may be zero at data points; extrema may fall at data points exclusively; or surfaces may not pass through data points.An alternative method is proposed whereby a map is automatically divided into suitable triangular domains with a data point at each vertex. Surface estimation and contour plotting are then performed independently for each triangle using a measured or best-fit plane associated with each data point. The approach requires the utilization of three non-original aspects: 1) the derivation of a local homogeneous area coordinate system for any arbitrary triangle; 2) the construction of a data-structure linking each triangular domain with its three neighbouring triangles and three associated data points, and 3) the use of a conforming triangular finite-element interpolating function.Use of the first two of these concepts permits the economic generation and optimization of a triangular mesh from a set of data points. Optimization criteria used to define the best triangular partition are described in some detail, along with computer timing for this step. Use of the first and third concepts permits the interpolation of a smooth surface over the whole map even though each triangular element is estimated and plotted independently. The requirements for a suitable interpolating function are discussed, and an interpolant is suggested that preserves elevation and slope at each data point as well as elevation and slope continuity between domains. Extrema need not be located at data points but are constrained by their associated planes. Contour line segments are produced by division of each triangle into N2 sub-triangles, elevation estimation at each resulting node, and linear interpolation and plotting withing each sub-triangle. Resolution is then a function of N and local data point density. Extension of the method to higher dimensions is briefly discussed.

All data presented pertain to 11 advertised brands of same low-priced, high-consumption food product. The data span an eight-year period with nine data points for each brand. Two measurements were taken during first year with about a four-month interval between readings; remaining seven measurements were obtained at approximate yearly intervals. The 11 brands came from three separate markets, and each reported value was based on no fewer than 200 principal female shoppers in a family who used product category. Sampling was of two types: (1) replicated probability samples of blocks with clusters split proportionally between central city and surrounding suburban towns or (2) random samples from metropolitan telephone directory with specification of individual telephone numbers. The share-of-families data were obtained from question: of do you buy? If respondents had difficulty with usually buy, interviewers were instructed to add: the you buy most often. Share of families, for study's food category, is judged to be a reasonable estimate of market share based on an additional analysis. The surveys also included a question on units purchased per week. These data, in recent surveys, have been multiplied by users to develop individual volume shares. The resulting percentages have been close to share-offamily percentages. The reliability of brand bought as an indicator of share of families was also examined. Data from two additional questions were analyzed: What of do you have in house right now? and particular have you purchased most often in past three weeks? Although there are interesting differences, high similarity in percentages exists across three questions. The top-of-mind awareness data were obtained by asking, When you think of (the type of product) what first comes to mind? This was first question asked a respondent, after she was qualified as principal female shopper and a user of pro uct category. The top-of-mind awareness and share-of-families data for 11 brands are shown in figure. All data points are on a percentage basis. The order is based on market performance accumulated over nine data points. For each change in share-of-families between successive data points was determined algebraically by subtracting share of families for each data point from preceding data point. These values were accumulated and 11 brands were rank-ordered by algebraic total as shown in Column 1 of Tables 1 and 2. Thus, Brand A has largest total data point increase in share of families, and Brand K has largest total decrease in data point share changes.

An Efficient Workflow for Collecting, Entering, and Proofing Field Data: Harnessing Voice Recording and Dictation Software

Much research has been conducted recently relating to ubiquitous intelligent computing. Ubiquitous intelligence-enabled techniques, such as clustering and image segmentation, have focused on the development of intelligence methodologies. In this paper, a simultaneous mode-seeking and clustering algorithm called the Generalized Transport Mean Shift (GTMS) was introduced. The data points were designated as the ‘transporter–trailer’ characteristic. The important concept of transportation was used to solve the problem of redundant computations of mode-seeking algorithms. The time complexity of the GTMS algorithm is much lower than that of the Mean Shift (MS) algorithm. This means it is able to be used in a problem that has a very high data point, in particular, the segmentation of images containing the green vegetation. The proposed algorithm was tested on clustering and image-segmentation problems. The experimental results showed that the GTMS algorithm improves upon the existing algorithms in terms of both accuracy and time consumption. The GTMS algorithm’s highest speed is also 333.98 times faster than that of the standard MS algorithm. The redundancy computation can be reduced by omitting more than 90% of the data points at the third iteration of the mode-seeking process. This is because GTMS algorithm mainly reduces the data in the mode-seeking process. Thus, use of the GTMS algorithm would allow for the building of an intelligent portable device for surveying green vegetables in a ubiquitous environment.

Subspace clustering has important and wide applications in computer vision and pattern recognition. It is a challenging task to learn low-dimensional subspace structures due to the possible errors (e.g., noise and corruptions) existing in high-dimensional data. Recent subspace clustering methods usually assume a sparse representation of corrupted errors and correct the errors iteratively. However large corruptions in real-world applications can not be well addressed by these methods. A novel optimization model for robust subspace clustering is proposed in this paper. The objective function of our model mainly includes two parts. The first part aims to achieve a sparse representation of each high-dimensional data point with other data points. The second part aims to maximize the correntropy between a given data point and its low-dimensional representation with other points. Correntropy is a robust measure so that the influence of large corruptions on subspace clustering can be greatly suppressed. An extension of our method with explicit introduction of representation error terms into the model is also proposed. Half-quadratic minimization is provided as an efficient solution to the proposed robust subspace clustering formulations. Experimental results on Hopkins 155 dataset and Extended Yale Database B demonstrate that our method outperforms state-of-the-art subspace clustering methods.

The objective of the study summarized in this paper was to establish a robust method to detect outliers in rate and pressure data used for decline analysis and forecasting, pressure and rate transient analysis, and similar workflows. Presence of outliers contributes significantly to the uncertainty and non-uniqueness associated with the rate/pressure transient analysis workflow. In time-series data usually encountered in rate/pressure transient analysis workflows, we observed that the neighboring data points of an inlying data point will be closer to the point as compared to closeness of an outlier data point. The outlier detection method we developed exploits this observation. It relies on the concept of local density around a data point, and locality is quantified by a function of distance of the data point from its k-nearest neighbors. This concept further maps into a local outlier factor, which signifies the quantitative extent of a data point being an outlier. Outliers will have a higher value of local outlier factor than will inlying data points. A threshold value of local outlier factor is used to label the data points as outliers, and this threshold is set using the histogram of local outlier factor values. Most commonly used outlier detection methods in the industry assume that rate/pressure data follow a known model (e.g., a Gaussian probability distribution model), but outliers deviate strongly from this model. In the usual parametric-model-based approach, an a priori assumption of the correct model needs to be verified, which can limit the validity of the approach. The newly proposed outlier detection method, being non-parametric, is more robust as it is independent of the assumption that data follows a known model. We present a synthetic case study demonstrating the importance of outlier detection and outlining the limitations of currently used outlier detection methods. We then validate our method through synthetic examples generated using numerical models of multi-stage hydraulically fractured wells in unconventional reservoirs. Upon validation we demonstrate application of our method using field examples from four major shale plays.

We investigate graph-based Laplacian semi-supervised learning at low labeling rates (ratios of labeled to total number of data points) and establish a threshold for the learning to be well posed. Laplacian learning uses harmonic extension on a graph to propagate labels. It is known that when the number of labeled data points is finite while the number of unlabeled data points tends to infinity, the Laplacian learning becomes degenerate and the solutions become roughly constant with a spike at each labeled data point. In this work, we allow the number of labeled data points to grow to infinity as the total number of data points grows. We show that for a random geometric graph with length scale $$\varepsilon >0$$ , if the labeling rate $$\beta \ll \varepsilon ^2$$ , then the solution becomes degenerate and spikes form. On the other hand, if $$\beta \gg \varepsilon ^2$$ , then Laplacian learning is well-posed and consistent with a continuum Laplace equation. Furthermore, in the well-posed setting we prove quantitative error estimates of $$O(\varepsilon \beta ^{-1/2})$$ for the difference between the solutions of the discrete problem and continuum PDE, up to logarithmic factors. We also study p-Laplacian regularization and show the same degeneracy result when $$\beta \ll \varepsilon ^p$$ . The proofs of our well-posedness results use the random walk interpretation of Laplacian learning and PDE arguments, while the proofs of the ill-posedness results use $$\Gamma $$ -convergence tools from the calculus of variations. We also present numerical results on synthetic and real data to illustrate our results.

One of the stages in drying the nutmeg is drying the nutmeg in the afternoon which is directly illuminated by the sun. The nutmeg owners in South Aceh is very difficult to get information about rain falling during the day if they are outside or are not in the nutmeg drying area. The process of monitoring the occurrence of rain during the drying of the nutmeg has been done manually by monitoring or seeing the state of the sky when it is cloudy. The purpose of this research is to create a rain monitoring system based on the “Internet of things” internet to make it easier for nutmeg owners to get information about rain falling from a distance. This research uses nodeMCU which functions as a controller of a rain monitoring system, a raindrop sensor as a rain detection sensor. The working system of the rain monitoring tool is by connecting the nodeMCU to the led light and if the raindrop sensor is exposed to rain then the led light will turn on or light up and the raindrop sensor reads the data point. If the data point is read 1024, it indicates that the condition is not raining, while if the data point value is 700 -1023, it indicates the condition of the drizzle, and the data point value from 0-600 indicates the heavy rain condition. The raindrop sensor sends data points to the ThingsIO.AI website as a condition notification that the nutmeg drying area is drizzling, heavy rain, or no rain. For monitoring of nutmeg drying area, it can be accessed remotely via the “website ThingsIO.AI” using smartphones and computers that are connected by the internet. Based on the results of testing the average data point delivery time is 1 minute and the value of the data point obtained is 488-1009

In machine learning,active learningrefers to algorithms that autonomously select the data points from which they will learn. There are many data mining applications in which large amounts of unlabeled data are readily available, but labels (e.g., human annotations or results coming from complex experiments) are costly to obtain. In such scenarios, an active learning algorithm aims at identifying data points that, if labeled and used for training, would most improve the learned model. Labels are then obtained only for the most promising data points. This speeds up learning and reduces labeling costs. Support vector machine (SVM) classifiers are particularly well‐suited for active learning due to their convenient mathematical properties. They perform linear classification, typically in a kernel‐induced feature space, which makes expressing the distance of a data point from the decision boundary straightforward. Furthermore, heuristics can efficiently help estimate how strongly learning from a data point influences the current model. This information can be used to actively select training samples. After a brief introduction to the active learning problem, we discuss different query strategies for selecting informative data points and review how these strategies give rise to different variants of active learning withSVMs.This article is categorized under:Technologies > Machine Learning

MapReduce skyline query processing with partitioning and distributed dominance tests

Stochastic Gradient Descent (SGD) is widely used to train a machine learning model over large datasets, yet its slow convergence rate can be a bottleneck. As a remarkable family of variance reduction techniques, memory algorithms such as SAG and SAGA have been proposed to accelerate the convergence rate of SGD. However, these algorithms need to store per training data point corrections in memory. The unlimited space usage feature is impractical for modern large-scale applications, especially over data points that arrive over time (referred to as streaming data in this paper). To overcome this weakness, this paper investigates the methods that bound the space usage in the state-of-the-art family of variance-reduced stochastic gradient descent over streaming data, and presents CHEAPS2AGA. At each step of updating the model, the key idea of CHEAPS2AGA is always reserving N random data points as samples, while re-using information about past stochastic gradients across all the observed data points with limited space usage. In addition, training an accurate model over streaming data requires the algorithm to be time-efficient. To accelerate the model training phase, CHEAPS2AGA embraces a lock-free data structure to insert new data points and remove unused data points in parallel, and updates the model parameters without using any locking. We conduct comprehensive experiments to compare CHEAPS2AGA to prior related algorithms suited for streaming data. The experimental results demonstrate the practical competitiveness of CHEAPS2AGA in terms of scalability and accuracy.