Improving neighborhood construction with Apollonius region algorithm based on density for clustering

This paper provides a new similarity detection algorithm. Given an input set of multi-dimensional data points, where each data point is assumed to be multi-dimensional, and an additional reference data point for similarity finding, the algorithm uses kernel method that embeds the data points into a low dimensional manifold. Unlike other kernel methods, which consider the entire data for the embedding, our method selects a specific set of kernel eigenvectors. The eigenvectors are chosen to separate between the data points and the reference data point so that similar data points can be easily identified as being distinct from most of the members in the dataset.

Finding a new drug which is both safe and efficient is an expensive and time-consuming endeavour. In particular, establishing the ‘dose-effect relationship’ – how beneficial a drug is at different dosages – can be challenging. Predicting this curve requires gathering experimental data by exposing and recording how cells respond to various levels of the drug. However, extreme values are often observed at low and high dosages, potentially introducing errors that are hard to correct in the prediction process. Yet, these extreme observations are sometimes genuine so researchers cannot just ignore them. To improve dose-effect estimation, Zhou, Liu, Fang et al. developed a new general-purpose approach. It uses advanced statistical modelling to account for extremes in lab data. This strategy outperformed other methods when dealing with these observations while also providing higher efficiency in data analysis with more uniform data in experiments. To facilitate implementation, Zhou, Liu, Fang et al. set up a user-friendly tool baptized ‘REAP’; this free online resource allows scientists without advanced statistical experience to harness the new approach and to perform dose-effect analysis more easily and accurately. This could boost research across many different disciplines that examine the effects of chemicals on cells.

In many fields of machine learning such as classifying and clustering, neighborhood construction algorithms are used to model local relationships between data samples and to build global structure from local information. In finding connection among the data points, neighborhood is undeniably useful for data processing. Therefore, a very major issue is to find a novel approach to locating neighborhood among data points. If the geometric relationships existing between the data points in the neighborhood area are accurately explored, it will be feasible to observe the behavioral rules as well as the similarities among the data. This will also help identify indirect and direct neighborhood ranges. This study aims to find neighborhood accurately by means of the Apollonius circle zones. The experimental validation against well-known k-nearest neighbor and e-neighborhood is also an indication of the robustness of the method in real data sets.

The goal of clustering is to partition the data points into clusters, such that the data points in the same cluster are similar. Therefore, similarity measure is one of the most critical issues for clustering. In this paper, we present a novel similarity measure based on intrinsic dimension, where the local intrinsic dimension of each data point is considered as a new feature to describe the data points, leading to a new type of similarity measure combining the new feature and original features. The main idea is that the data points in the same cluster are expected to have the same intrinsic dimension while they have similar values of the traditional features. The proposed method is evaluated on some artificial data sets and the experiment results illustrate the effectiveness of the proposed similarity measure. Moreover, the segmentation results of natural images based on the proposed similarity measure show that the intrinsic dimension is worthy of being considered as a new feature of the data points in more applications.

Stochastic Gradient Descent (SGD) is widely used to train a machine learning model over large datasets, yet its slow convergence rate can be a bottleneck. As a remarkable family of variance reduction techniques, memory algorithms such as SAG and SAGA have been proposed to accelerate the convergence rate of SGD. However, these algorithms need to store per training data point corrections in memory. The unlimited space usage feature is impractical for modern large-scale applications, especially over data points that arrive over time (referred to as streaming data in this paper). To overcome this weakness, this paper investigates the methods that bound the space usage in the state-of-the-art family of variance-reduced stochastic gradient descent over streaming data, and presents CHEAPS2AGA. At each step of updating the model, the key idea of CHEAPS2AGA is always reserving N random data points as samples, while re-using information about past stochastic gradients across all the observed data points with limited space usage. In addition, training an accurate model over streaming data requires the algorithm to be time-efficient. To accelerate the model training phase, CHEAPS2AGA embraces a lock-free data structure to insert new data points and remove unused data points in parallel, and updates the model parameters without using any locking. We conduct comprehensive experiments to compare CHEAPS2AGA to prior related algorithms suited for streaming data. The experimental results demonstrate the practical competitiveness of CHEAPS2AGA in terms of scalability and accuracy.

Surface display techniques, including contour mapping, usually use an interpolant when data values are on a rectangular grid; but when values are irregularly located some form of assumed or estimated distance-weighting function is often used to estimate the influence of data points at other locations. If moving average (or some other) techniques are then applied, slopes may be zero at data points; extrema may fall at data points exclusively; or surfaces may not pass through data points.An alternative method is proposed whereby a map is automatically divided into suitable triangular domains with a data point at each vertex. Surface estimation and contour plotting are then performed independently for each triangle using a measured or best-fit plane associated with each data point. The approach requires the utilization of three non-original aspects: 1) the derivation of a local homogeneous area coordinate system for any arbitrary triangle; 2) the construction of a data-structure linking each triangular domain with its three neighbouring triangles and three associated data points, and 3) the use of a conforming triangular finite-element interpolating function.Use of the first two of these concepts permits the economic generation and optimization of a triangular mesh from a set of data points. Optimization criteria used to define the best triangular partition are described in some detail, along with computer timing for this step. Use of the first and third concepts permits the interpolation of a smooth surface over the whole map even though each triangular element is estimated and plotted independently. The requirements for a suitable interpolating function are discussed, and an interpolant is suggested that preserves elevation and slope at each data point as well as elevation and slope continuity between domains. Extrema need not be located at data points but are constrained by their associated planes. Contour line segments are produced by division of each triangle into N2 sub-triangles, elevation estimation at each resulting node, and linear interpolation and plotting withing each sub-triangle. Resolution is then a function of N and local data point density. Extension of the method to higher dimensions is briefly discussed.

Similarity search plays an important role in many applications involving high-dimensional data. Due to the known dimensionality curse, the performance of most existing indexing structures degrades quickly as the feature dimensionality increases. Hashing methods, such as locality sensitive hashing (LSH) and its variants, have been widely used to achieve fast approximate similarity search by trading search quality for efficiency. However, most existing hashing methods make use of randomized algorithms to generate hash codes without considering the specific structural information in the data. In this paper, we propose a novel hashing method, namely, robust hashing with local models (RHLM), which learns a set of robust hash functions to map the high-dimensional data points into binary hash codes by effectively utilizing local structural information. In RHLM, for each individual data point in the training dataset, a local hashing model is learned and used to predict the hash codes of its neighboring data points. The local models from all the data points are globally aligned so that an optimal hash code can be assigned to each data point. After obtaining the hash codes of all the training data points, we design a robust method by employing l2,1 -norm minimization on the loss function to learn effective hash functions, which are then used to map each database point into its hash code. Given a query data point, the search process first maps it into the query hash code by the hash functions and then explores the buckets, which have similar hash codes to the query hash code. Extensive experimental results conducted on real-life datasets show that the proposed RHLM outperforms the state-of-the-art methods in terms of search quality and efficiency.

All data presented pertain to 11 advertised brands of same low-priced, high-consumption food product. The data span an eight-year period with nine data points for each brand. Two measurements were taken during first year with about a four-month interval between readings; remaining seven measurements were obtained at approximate yearly intervals. The 11 brands came from three separate markets, and each reported value was based on no fewer than 200 principal female shoppers in a family who used product category. Sampling was of two types: (1) replicated probability samples of blocks with clusters split proportionally between central city and surrounding suburban towns or (2) random samples from metropolitan telephone directory with specification of individual telephone numbers. The share-of-families data were obtained from question: of do you buy? If respondents had difficulty with usually buy, interviewers were instructed to add: the you buy most often. Share of families, for study's food category, is judged to be a reasonable estimate of market share based on an additional analysis. The surveys also included a question on units purchased per week. These data, in recent surveys, have been multiplied by users to develop individual volume shares. The resulting percentages have been close to share-offamily percentages. The reliability of brand bought as an indicator of share of families was also examined. Data from two additional questions were analyzed: What of do you have in house right now? and particular have you purchased most often in past three weeks? Although there are interesting differences, high similarity in percentages exists across three questions. The top-of-mind awareness data were obtained by asking, When you think of (the type of product) what first comes to mind? This was first question asked a respondent, after she was qualified as principal female shopper and a user of pro uct category. The top-of-mind awareness and share-of-families data for 11 brands are shown in figure. All data points are on a percentage basis. The order is based on market performance accumulated over nine data points. For each change in share-of-families between successive data points was determined algebraically by subtracting share of families for each data point from preceding data point. These values were accumulated and 11 brands were rank-ordered by algebraic total as shown in Column 1 of Tables 1 and 2. Thus, Brand A has largest total data point increase in share of families, and Brand K has largest total decrease in data point share changes.

Affinity propagation (AP) algorithm, as a novel clustering method, does not require the users to specify the initial cluster centers in advance, which regards all data points as potential exemplars (cluster centers) equally and groups the clusters totally by the similar degree among the data points. But in many cases there exist some different intensive areas within the same data set, which means that the data set does not distribute homogeneously. In such situation the AP algorithm cannot group the data points into ideal clusters. In this paper, we proposed an extended AP clustering algorithm to deal with such a problem. There are two steps in our method: firstly the data set is partitioned into several data density types according to the nearest distances of each data point; and then the AP clustering method is, respectively, used to group the data points into clusters in each data density type. Two experiments are carried out to evaluate the performance of our algorithm: one utilizes an artificial data set and the other uses a real seismic data set. The experiment results show that groups are obtained more accurately by our algorithm than OPTICS and AP clustering algorithm itself.

To solve the shortcoming of the fuzzy c-means clustering algorithm(FCM) that fails to cluster the non-convex datasets effectively, we propose an improved fuzzy c-means clustering algorithm(IFCM). IFCM first constructs the nearest neighbor network graph by finding the r neighbors of each data point through the 3N algorithm, if a connected relationship exists on two data points in the graph, the Euclidean Distance is employed to measure the distance between them; Otherwise, it performs a more reasonable similarity calculation strategy, which merges the distance value mentioned above with the Euclidean Distance between the other two data points which have the furthest distance in the dataset. Meanwhile, it updates the membership degree of each data point in each cluster and the center of each cluster. IFCM will repeat the process mentioned above until all the data points in the dataset have been classified properly. Further analysis on artificial and UCI datasets verifies the importance of similarity strategy improvement for clustering, justifying the rationality and effectiveness of IFCM.

A self-adaptive local metric learning method for classification

Clustering is an extensive research area in data science. The aim of clustering is to discover groups and to identify interesting patterns in datasets. Crisp (hard) clustering considers that each data point belongs to one and only one cluster. However, it is inadequate as some data points may belong to several clusters, as is the case in text categorization. Thus, we need more flexible clustering. Fuzzy clustering methods, where each data point can belong to several clusters, are an interesting alternative. Yet, seeding iterative fuzzy algorithms to achieve high quality clustering is an issue. In this paper, we propose a new linear and efficient initialization algorithm MaxMin Linear to deal with this problem. Then, we validate our theoretical results through extensive experiments on a variety of numerical real-world and artificial datasets. We also test several validity indices, including a new validity index that we propose, Transformed Standardized Fuzzy Difference (TSFD).

Dominant sets are a new graph partition method for pairwise data clustering proposed by Pavan and Pelillo. We address the problem of dominant sets with a coalitional game model, in which each data point is treated as a player and similar data points are encouraged to group together for cooperation. We propose betrayal and hermit rules to describe the cooperative behaviors among the players. After applying the betrayal and hermit rules, an optimal and stable graph partition emerges, and all the players in the partition will not change their groups. For computational feasibility, we design an approximate algorithm for finding a dominant set of mutually similar players and then apply the algorithm to an application such as image denoising. In image denoising, every pixel is treated as a player who seeks similar partners according to its patch appearance in its local neighborhood. By averaging the noisy effects with the similar pixels in the dominant sets, we improve nonlocal means image denoising to restore the intrinsic structure of the original images and achieve competitive denoising results with the state-of-the-art methods in visual and quantitative qualities.

The objective of the study summarized in this paper was to establish a robust method to detect outliers in rate and pressure data used for decline analysis and forecasting, pressure and rate transient analysis, and similar workflows. Presence of outliers contributes significantly to the uncertainty and non-uniqueness associated with the rate/pressure transient analysis workflow. In time-series data usually encountered in rate/pressure transient analysis workflows, we observed that the neighboring data points of an inlying data point will be closer to the point as compared to closeness of an outlier data point. The outlier detection method we developed exploits this observation. It relies on the concept of local density around a data point, and locality is quantified by a function of distance of the data point from its k-nearest neighbors. This concept further maps into a local outlier factor, which signifies the quantitative extent of a data point being an outlier. Outliers will have a higher value of local outlier factor than will inlying data points. A threshold value of local outlier factor is used to label the data points as outliers, and this threshold is set using the histogram of local outlier factor values. Most commonly used outlier detection methods in the industry assume that rate/pressure data follow a known model (e.g., a Gaussian probability distribution model), but outliers deviate strongly from this model. In the usual parametric-model-based approach, an a priori assumption of the correct model needs to be verified, which can limit the validity of the approach. The newly proposed outlier detection method, being non-parametric, is more robust as it is independent of the assumption that data follows a known model. We present a synthetic case study demonstrating the importance of outlier detection and outlining the limitations of currently used outlier detection methods. We then validate our method through synthetic examples generated using numerical models of multi-stage hydraulically fractured wells in unconventional reservoirs. Upon validation we demonstrate application of our method using field examples from four major shale plays.

One of the stages in drying the nutmeg is drying the nutmeg in the afternoon which is directly illuminated by the sun. The nutmeg owners in South Aceh is very difficult to get information about rain falling during the day if they are outside or are not in the nutmeg drying area. The process of monitoring the occurrence of rain during the drying of the nutmeg has been done manually by monitoring or seeing the state of the sky when it is cloudy. The purpose of this research is to create a rain monitoring system based on the “Internet of things” internet to make it easier for nutmeg owners to get information about rain falling from a distance. This research uses nodeMCU which functions as a controller of a rain monitoring system, a raindrop sensor as a rain detection sensor. The working system of the rain monitoring tool is by connecting the nodeMCU to the led light and if the raindrop sensor is exposed to rain then the led light will turn on or light up and the raindrop sensor reads the data point. If the data point is read 1024, it indicates that the condition is not raining, while if the data point value is 700 -1023, it indicates the condition of the drizzle, and the data point value from 0-600 indicates the heavy rain condition. The raindrop sensor sends data points to the ThingsIO.AI website as a condition notification that the nutmeg drying area is drizzling, heavy rain, or no rain. For monitoring of nutmeg drying area, it can be accessed remotely via the “website ThingsIO.AI” using smartphones and computers that are connected by the internet. Based on the results of testing the average data point delivery time is 1 minute and the value of the data point obtained is 488-1009