Feature weighted multi-view possibilistic c-means with feature reduction

Possibilistic fuzzy c-means clustering (PFCM) algorithm improves the noise sensitivity of the fuzzy c-means clustering (FCM) algorithm and the serious coincident clustering phenomenon of the possibilistic c-means clustering (PCM) algorithm. However, the Euclidean distance of the PFCM treats all features equally in the clustering process and doesn't consider the imbalance between sample features, resulting in low clustering accuracy. Furthermore, as the number of clusters increases, the PFCM still has a partially coincident clustering problem due to the lack of between-class relationships of possibilistic memberships. Therefore, a feature weighted cutset-type possibilistic fuzzy c-means clustering (FW-C-PFCM) algorithm is proposed by introducing the cutset theory and feature weights into the PFCM in this paper. Firstly, the proposed FW-C-PFCM introduces a feature weighted parameter in the objective function, and different features should take different weight values according to the distribution of samples. Secondly, the cutset theory is utilized to divide the data into inner core and outer core regions, and some possibilistic memberships inside each cluster core are modified in the iterative process. Finally, the experiments on synthetic multi-class datasets and the Iris dataset show that the FW-C-PFCM improves the partially coincident clustering phenomenon and overcomes the imbalance between sample features. The comparative experiments on a color image also indicate that the FW-C-PFCM enhances the segmentation accuracy and generates better clustering results than PFCM.

In a real-world data set there is always the possibility, rather high in our opinion, that different features may have different degrees of relevance. Most machine learning algorithms deal with this fact by either selecting or deselecting features in the data preprocessing phase. However, we maintain that even among relevant features there may be different degrees of relevance, and this should be taken into account during the clustering process. With over 50 years of history, K-Means is arguably the most popular partitional clustering algorithm there is. The first K-Means based clustering algorithm to compute feature weights was designed just over 30 years ago. Various such algorithms have been designed since but there has not been, to our knowledge, a survey integrating empirical evidence of cluster recovery ability, common flaws, and possible directions for future research. This paper elaborates on the concept of feature weighting and addresses these issues by critically analysing some of the most popular, or innovative, feature weighting mechanisms based in K-Means

DBSCAN is arguably the most popular density-based clustering algorithm, and it is capable of recovering non-spherical clusters. One of its main weaknesses is that it treats all features equally. In this paper, we propose a density-based clustering algorithm capable of calculating feature weights representing the degree of relevance of each feature, which takes the density structure of the data into account. First, we improve DBSCAN and introduce a new algorithm called DBSCANR. DBSCANR reduces the number of parameters of DBSCAN to one. Then, a new step is introduced to the clustering process of DBSCANR to iteratively update feature weights based on the current partition of data. The feature weights produced by the weighted version of the new clustering algorithm, W-DBSCANR, measure the relevance of variables in a clustering and can be used in feature selection in data mining applications where large and complex real-world data are often involved. Experimental results on both artificial and real-world data have shown that the new algorithms outperformed various DBSCAN type algorithms in recovering clusters in data.

TFIDF algorithm was used for feature weighting in text classification. But the result of classification was not very well because of lack of class information in feature weighting. The known class information in the training set was used to improve the traditional TFIDF feature weight algorithm. Class distinction ability and class description ability were introduced, respectively expressed by inverse class frequency and term frequency in class, document frequency in class. A new feature weight algorithm based on class information, TF_IDT, was proposed. Naive Bayes classifier was used to test the algorithm. The precision, recall and F1 measure were significantly increased. Macro F1 measure raise by 6.46%. It was proved to be useful for improving text classification to use class information in feature weighting. In addition, the computational complexity of the proposed algorithm was lower and more suitable for use in fields of limited computing capability. Keywords-text classification; feature weight; inverse class frequency; term frequency in class; document frequency in class

Despite being a well‐known problem, feature weighting and feature selection are a major predicament for clustering. Most of the algorithms, which provide weighting or selection of features, require the number of clusters to be known in advance. On the other hand, the existing automatic clustering procedures that can determine the number of clusters are computationally expensive and often do not make a room for feature weighting or selection. In this paper, we propose a Gibbs sampling‐based algorithm for the Dirichlet process mixture model, which can determine the number of clusters and can also incorporate a near‐optimal feature weighting. We show that in the limiting case, the algorithm approaches a hard clustering procedure, which resembles minimization of an underlying clustering objective similar to weighted k‐means with an additional forfeit for the number of clusters and hence retains the simplicity of the Llyod's heuristics. To avoid the trivial solution of the resulting linear program, we include an additional entropic penalty on the feature weights. The proposed algorithm is tested on several synthetic and real‐life datasets. Through a detailed experimental analysis, we demonstrate the competitiveness of our proposal against the baseline as well as state‐of‐the‐art procedures for centre‐based high‐dimensional clustering.

Feature weighting is considered as an important machine learning approach to deal with the problem of estimating the quality of attributes for pattern classification applications. Local Hyperlinear Learning based Relief (LH-Relief) was shown to be very efficient in estimating attributions in high-dimensional data involving irrelevant noises. However, the convergence of LH-Relief can not be guaranteed. In this paper, we propose an innovative feature weighting algorithm to solve the problem of LH-Relief, called Iterative Local Hyperlinear Learning based Relief (ILH-Relief). ILH-Relief is based on LH-Relief using classical margin maximization. The key idea is to estimate the feature weights through local approximation and gradient descent. To demonstrate the viability and the effectiveness of our formulation for feature selection in supervised learning, we perform extensive experiments on both UCI and Microarray datasets. The proposed algorithm can save at least half of feature ranking time with a better classification performance compared with other feature weighting methods.

Image retrieval is the process of finding images in the database that are similar to the query image by measuring how close the feature values of the query image are to other images. Image retrieval is currently dominated by approaches that combine several different representations or features. The optimal weight of each feature is needed in combining the image features such as color features, texture features, and shape features. In this study, we use a multi-layer perceptron artificial neural network (MLP) method to obtain feature weights automatically and simultaneously look for optimal weights. The color moment is used to find nine color features, Gray Level Co-occurrence Matrix (GLCM) to find four texture features, and Hu Moment to find seven shape features totaling 20 neurons and all of these features become the input layer in our MLP network. Three neurons in output layers become the automatic weight of each feature. These weights are used to combine each feature's role in obtaining the relevant image. Euclidean Distance is used to measure similarity. The average precision values obtained using automatic feature weights are 93.94% for the synthetic dataset, 91.19% for the Coil-100 dataset, and 54.31% for the Wang dataset. These results have an average difference of 5.06% with the target so automatic feature weighting works well. This value is obtained at a hidden layer size of 11 and a learning rate of 0.1. In addition, the use of automatic feature weighting gives more accurate results compared to manual feature weighting.

TFIDF algorithm was used for feature weighting in text classification. But the result of classification was not very well because of lack of class information in feature weighting. The known class information in the training set was used to improve the traditional TFIDF feature weight algorithm. Class distinction ability and class description ability were introduced, respectively expressed by inverse class frequency and term frequency in class, document frequency in class. A new feature weight algorithm based on class information, TF_IDT, was proposed. Naïve Bayes classifier was used to test the algorithm. The precision, recall and F1 measure were significantly increased. Macro F1 measure raise by 6.46%. It was proved to be useful for improving text classification to use class information in feature weighting. In addition, the computational complexity of the proposed algorithm was lower and more suitable for use in fields of limited computing capability.

Minkowski metric, feature weighting and anomalous cluster initializing in K-Means clustering

In this paper, we propose an algorithm that performs fuzzy clustering and feature weighting simultaneously and in an unsupervised manner. The feature set is divided into logical subsets of features, and a degree of relevance is dynamically assigned to each subset based on its partial degree of dissimilarity. The proposed algorithm is computationally and implementationally simple, and learns a different set of feature weights for each cluster. The cluster dependent feature weights have two advantages. First, they help in partitioning the data set into more meaningful clusters. Second, they can be used as part of a more complex learning system to enhance its learning behavior. The performance of the proposed algorithm is illustrated by using it to segment color images.

Naive Bayes (NB) is a widely used classifier in the field of machine learning. However, its conditional independence assumption does not hold true in real-world applications. In literature, various feature weighting approaches have attempted to alleviate this assumption. Almost all of these approaches consider the relationship between feature-class (relevancy) and feature-feature (redundancy) independently, to determine the weights of features. We argue that these two relationships are mutually dependent and both cannot be improved simultaneously, i.e., form a trade-off. This paper proposes a new paradigm to determine the feature weight by formulating it as a multi-objective optimisation problem to balance the trade-off between relevancy and redundancy. Multi-objective artificial bee colony-based feature weighting technique for naive Bayes (MOABC-FWNB) is proposed. An extensive experimental study was conducted on 20 benchmark UCI datasets. Experimental results show that MOABC-FWNB outperforms NB and other existing state-of-the-art feature weighting techniques.

Personalization is the adaptation of the services to fit the user’s interests, characteristics and needs. The key to effective personalization is user profiling. Apart from traditional collaborative and content-based approaches, a number of classification and clustering algorithms have been used to classify user related information to create user profiles. However, they are not able to achieve accurate user profiles. In this paper, we present a new clustering algorithm, namely Multi-Dimensional Clustering (MDC), to determine user profiling. The MDC is a version of the Instance-Based Learner (IBL) algorithm that assigns weights to feature values and considers these weights for the clustering. Three feature weight methods are proposed for the MDC and, all three, have been tested and evaluated. Simulations were conducted with using two sets of user profile datasets, which are the training (includes 10,000 instances) and test (includes 1000 instances) datasets. These datasets reflect each user’s personal information, preferences and interests. Additional simulations and comparisons with existing weighted and non-weighted instance-based algorithms were carried out in order to demonstrate the performance of proposed algorithm. Experimental results using the user profile datasets demonstrate that the proposed algorithm has better clustering accuracy performance compared to other algorithms. This work is based on the doctoral thesis of the corresponding author.

Clustering with multiview data is becoming a hot topic in data mining, pattern recognition, and machine learning. In order to realize an effective multiview clustering, two issues must be addressed, namely, how to combine the clustering result from each view and how to identify the importance of each view. In this paper, based on a newly proposed objective function which explicitly incorporates two penalty terms, a basic multiview fuzzy clustering algorithm, called collaborative fuzzy c-means (Co-FCM), is firstly proposed. It is then extended into its weighted view version, called weighted view collaborative fuzzy c-means (WV-Co-FCM), by identifying the importance of each view. The WV-Co-FCM algorithm indeed tackles the above two issues simultaneously. Its relationship with the latest multiview fuzzy clustering algorithm Collaborative Fuzzy K-Means (Co-FKM) is also revealed. Extensive experimental results on various multiview datasets indicate that the proposed WV-Co-FCM algorithm outperforms or is at least comparable to the existing state-of-the-art multitask and multiview clustering algorithms and the importance of different views of the datasets can be effectively identified.

Supervised pattern recognition for the prediction of contrast-enhancement appearance in brain tumors from multivariate magnetic resonance imaging and spectroscopy.

Tensorized multi-dimensional multi-view clustering based on nonnegative matrix factorization.