An Efficient Feature Selection Method for Activity Classification

<p>Classification aims to identify a class label of an instance according to the information from its characteristics or features. Unfortunately, many classification problems have a large feature set containing irrelevant and redundant features, which reduce the classification performance. In order to address the above problem, feature selection is proposed to select a small subset of relevant features. There are three main types of feature selection methods, i.e. wrapper, embedded and filter approaches. Wrappers use a classification algorithm to evaluate candidate feature subsets. In embedded approaches, the selection process is embedded in the training process of a classification algorithm. Different from the other two approaches, filters do not involve any classification algorithm during the selection process. Feature selection is an important process but it is not an easy task due to its large search space and complex feature interactions. Because of the potential global search ability, Evolutionary Computation (EC), especially Particle Swarm Optimization (PSO), has been widely and successfully applied to feature selection. However, there is potential to improve the effectiveness and efficiency of EC-based feature selection. The overall goal of this thesis is to investigate and improve the capability of EC for feature selection to select small feature subsets while maintaining or even improving the classification performance compared to using all features. Different aspects of feature selection are considered in this thesis such as the number of objectives (single-objective/multi-objective), the fitness function (filter/wrapper), and the searching mechanism. This thesis introduces a new fitness function based on mutual information which is calculated by an estimation approach instead of the traditional counting approach. Results show that the estimation approach works well on both continuous and discrete data. More importantly, mutual information calculated by the estimation approach can capture feature interactions better than the traditional counting approach. This thesis develops a novel binary PSO algorithm, which is the first work to redefine some core concepts of PSO such as velocity and momentum to suit the characteristics of binary search spaces. Experimental results show that the proposed binary PSO algorithm evolve better solutions than other binary EC algorithms when the search spaces are large and complex. Specifically, on feature selection, the proposed binary PSO algorithm can select smaller feature subsets with similar or better classification accuracies, especially when there are a large number of features. This thesis proposes surrogate models for wrapper-based feature selection. The surrogate models use surrogate training sets which are subsets of informative instances selected from the training set. Experimental results show that the proposed surrogate models assist PSO to reduce the computational cost while maintaining or even improving the classification performance compared to using only the original training set. The thesis develops the first wrapper-based multi-objective feature selection algorithm using MOEA/D. A new decomposition strategy using multiple reference points for MOEA/D is designed, which can deal with different characteristics of multi-objective feature selection such as highly discontinuous Pareto fronts and complex relationships between objectives. The experimental results show that the proposed algorithm can evolve more diverse non-dominated sets than other multi-objective algorithms. This thesis introduces the first PSO-based feature selection algorithm for transfer learning. In the proposed algorithm, the fitness function uses classification performance to reduce the differences between domains while maintaining the discriminative ability on the target domain. The experimental results show that the proposed algorithm can select feature subsets which achieve better classification performance than four state-of-the-art feature-based transfer learning algorithms.</p>

Network traffic in cloud computing is characterized by a large volume of data, with potential high dimensionality and high levels of redundancy. “Big data” not only retard the execution process of intrusion detection systems (IDS) but they can also lead to unsatisfactory classification accuracy. The efficient correlation-based feature selection (ECOFS) approach proposed in this paper, can handle linearly and nonlinearly dependent data and it can eliminate redundant and irrelevant features. Its effectiveness has been evaluated through its employment in an intrusion detection system. A Libsvm-IDS has been built to operate using the features selected by the proposed ECOFS algorithm. The performance of the hybrid Libsvm-IDS + ECOFS approach has been evaluated using two well-known intrusion detection evaluation datasets, namely the KDD Cup 99 and the NSL-KDD. The evaluation results show that our ECOFS algorithm selects the smallest number of features, resulting in the lowest computational cost for the Libsvm-IDS, without any performance compromise. In fact, our algorithm has achieved higher accuracy compared with two well established methods.

Interactive visual clustering allows the user to be involved into the clustering through visualizing process via interactive visualization. In order to perform effective interaction in the visual clustering process, the efficient feature selection methods are required to identify the most dominating features. Hence, in this paper an improved visual clustering system is proposed using an efficient feature selection method. The relevant features for visual clustering are identified based on their contribution to the entropy. Experimental results show that the proposed method works well in finding the best cluster.

In this paper, PCA based modeling of geometric structure of the face for automatic face detection is presented. The method improves the face detection rate and limits the search space. Skin Color Modeling (SCM) is one of the best face detection techniques for image and video. However, feature selection is very important for even better template matching performance in terms of detection rate and time. This paper presents an efficient feature extraction and selection method based on geometric structure of the facial image boundary and interior. To model the geometric structure of face, Principle Component Analysis (PCA) and canny edge detection are used. Fusion of PCA based geometric modeling and SCM method provides higher face detection accuracy and improves time complexity. Both models provide filtering of image in term of pixel values to get the face location that are very fast and efficient for large image databases.

A simple and efficient filter feature selection method via document-term matrix unitization

This paper proposes an SVM ensemble approach for improving textural classification performance. Finding informative patterns in an image texture is an important issue for image classification and remains a challenge. Local spatial pattern mapping (LSPM) method has been proposed for texture classification based on a complex network model. The purpose of the method was manipulating the spatial distribution in an image texture with multi-radial distance analysis. Although the classification performance was improved, there is a limitation by using a single support vector machine (SVM) as a classifier. Accordingly, we propose an SVM ensemble method with Bagging technique to overcome the limitation by showing improved textural classification performance. In experiments, the SVM ensemble classification performance is compared to the single SVM, SVM with cross-validation and k-NN classifiers by using the Brodatz, UIUC and Outex texture databases. As results, the SVM ensemble is shown to be effective for improving textural classification performance as compared to the other classifiers.

Recent advances and emerging challenges of feature selection in the context of big data

Most feature selection methods only perform well on datasets with relatively small set of features. In the case of large feature sets and small number of data points, almost none of the existing feature selection methods help in achieving high accuracy. This paper proposes a novel approach to optimize the feature selection process through Frequent Pattern Growth algorithm to find sets of features that appear frequently among the top features selected by the main feature selection methods. Our experimental evaluation on two datasets containing a small and very large number of features shows that our approach significantly improves the accuracy results of the dataset with a very large number of features.

<p>Classification problems often have a large number of features, but not all of them are useful for classification. Irrelevant and redundant features may even reduce the classification accuracy. Feature selection is a process of selecting a subset of relevant features, which can decrease the dimensionality, shorten the running time, and/or improve the classification accuracy. There are two types of feature selection approaches, i.e. wrapper and filter approaches. Their main difference is that wrappers use a classification algorithm to evaluate the goodness of the features during the feature selection process while filters are independent of any classification algorithm. Feature selection is a difficult task because of feature interactions and the large search space. Existing feature selection methods suffer from different problems, such as stagnation in local optima and high computational cost. Evolutionary computation (EC) techniques are well-known global search algorithms. Particle swarm optimisation (PSO) is an EC technique that is computationally less expensive and can converge faster than other methods. PSO has been successfully applied to many areas, but its potential for feature selection has not been fully investigated. The overall goal of this thesis is to investigate and improve the capability of PSO for feature selection to select a smaller number of features and achieve similar or better classification performance than using all features. This thesis investigates the use of PSO for both wrapper and filter, and for both single objective and multi-objective feature selection, and also investigates the differences between wrappers and filters. This thesis proposes a new PSO based wrapper, single objective feature selection approach by developing new initialisation and updating mechanisms. The results show that by considering the number of features in the initialisation and updating procedures, the new algorithm can improve the classification performance, reduce the number of features and decrease computational time. This thesis develops the first PSO based wrapper multi-objective feature selection approach, which aims to maximise the classification accuracy and simultaneously minimise the number of features. The results show that the proposed multi-objective algorithm can obtain more and better feature subsets than single objective algorithms, and outperform other well-known EC based multi-objective feature selection algorithms. This thesis develops a filter, single objective feature selection approach based on PSO and information theory. Two measures are proposed to evaluate the relevance of the selected features based on each pair of features and a group of features, respectively. The results show that PSO and information based algorithms can successfully address feature selection tasks. The group based method achieves higher classification accuracies, but the pair based method is faster and selects smaller feature subsets. This thesis proposes the first PSO based multi-objective filter feature selection approach using information based measures. This work is also the first work using other two well-known multi-objective EC algorithms in filter feature selection, which are also used to compare the performance of the PSO based approach. The results show that the PSO based multiobjective filter approach can successfully address feature selection problems, outperform single objective filter algorithms and achieve better classification performance than other multi-objective algorithms. This thesis investigates the difference between wrapper and filter approaches in terms of the classification performance and computational time, and also examines the generality of wrappers. The results show that wrappers generally achieve better or similar classification performance than filters, but do not always need longer computational time than filters. The results also show that wrappers built with simple classification algorithms can be general to other classification algorithms.</p>

Feature selection is an essential step in the preprocessing of data in pattern recognition and data mining. Nowadays, the feature selection problem as an optimization problem can be solved with nature-inspired algorithm. In this paper, we propose an efficient feature selection method based on the cuckoo search algorithm called CBCSEM. The proposed method avoids the premature convergence of traditional methods and the tendency to fall into local optima, and this efficient method is attributed to three aspects. Firstly, the chaotic map increases the diversity of the initialization of the algorithm and lays the foundation for its convergence. Then, the proposed two-population elite preservation strategy can find the attractive one of each generation and preserve it. Finally, Lévy flight is developed to update the position of a cuckoo, and the proposed uniform mutation strategy avoids the trouble that the search space is too large for the convergence of the algorithm due to Lévy flight and improves the algorithm exploitation ability. The experimental results on several real UCI datasets show that the proposed method is competitive in comparison with other feature selection algorithms.

An important problem in quantitative medical image analysis is a large number of features (often highly correlated) to instance ratio. To handle this, we developed a feature selector and an ensemble classifier based on a modified version of random subspace method. We propose using a fusion of feature selection concepts: ranking based, correlation based and random subspaces, to develop a concordance correlation coefficient based random subspace method (CCC RSM) feature selector. It forms random feature subsets with weakly correlated yet relevant features while the ensemble classification is achieved by training the base classifier with these feature subsets. Axillary lymph node (ALNs) metastases is one of the most important prognostic factors in breast cancer. We applied CCC RSM for four binary class classifications based on the number of metastatic ALNs: (i) = 1 vs 0 (ii) 1-3 vs 0 (iii) 1-3 vs = 4, and (iv) 4 vs 0. We extracted textural kinetics from habitats of fifty eight dynamic contrast enhanced magnetic resonance imaging breast tumors. We used three classifiers to compare the accuracies achieved by CCC RSM with random subspaces (RS), wrappers and correlation based feature selector (CFS). For each binary classification we achieved the best accuracy (= 78%) using CCC RSM.

PSOLDA: A particle swarm optimization approach for enhancing classification accuracy rate of linear discriminant analysis

Aggregate stability is a useful soil physical dynamic index of soil resistivity to surface wind and water erosion in all ecosystems, especially, in arid and semi-arid regions. Two machine learning techniques including support vector machines (SVMs) and artificial neural networks (ANNs) were used to develop predictive models for the estimation of geometric mean diameter (GMD) of soil aggregates. An empirical multiple linear regression (MLR) model was also constructed as the benchmark to compare their performances. Furthermore, the influence of feature space dimension reduction using parallel genetic algorithm (PGA) on the prediction accuracy of all investigated techniques was evaluated. The ANN model achieved greater accuracy in GMD prediction as compared to the MLR and SVM models. The obtained ERROR% value in GMD prediction using the ANN model was 6.9%, while it was 15.7 and 10.6% for the MLR and SVM models, respectively. Feature selection using PGA improved the prediction accuracy of all investigated techniques. The coefficient of determination (R2) values between the measured and the predicted GMD values using PGA-based MLR, SVM, and ANN models increased by 20.0, 12.2, and 8.8% in comparison with the proposed MLR, SVM, and ANN models. In conclusion, it appears that the PGA-based ANN model could be considered as an alternative to conventional regression models for the GMD prediction.

Garlic prices fluctuate dramatically in recent years and it is very difficult to predict garlic prices. The autoregressive integrated moving average (ARIMA) model is currently the most important method for predicting garlic prices. However, the ARIMA model can only predict the linear part of the garlic prices, and cannot predict its nonlinear part. Therefore, it is urgent to adopt a method to analyze the nonlinear characteristics of garlic prices. After comparing the advantages and disadvantages of several major prediction models which used to forecast nonlinear time series, using support vector machine (SVM) model to predict the nonlinear part of garlic prices and establish ARIMA-SVM hybrid forecast model to predict garlic prices. The monthly average price data of garlic in 2010-2017 was used to test the effect of ARIMA model, SVM model and ARIMA-SVM model. The experimental results show that: (1) Garlic price is affected by many factors but the most is the supply and demand relationship; (2) The SVM model has a good effect in dealing with the nonlinear relationship of garlic prices; (3) The ARIMA-SVM hybrid model is better than the single ARIMA model and SVM model on the accuracy of garlic price prediction, it can be used as an effective method to predict the short-term price of garlic.

Identifying important biomarkers to improve disease diagnosis and treatment is a significant topic of research in bioinformatics. However, bioinformatics datasets frequently have a large number of features per sample or instance. This problem, known as “high dimensionality,” can be alleviated through the use of dimension reducing techniques such as feature (gene) selection which remove unnecessary features. There are many versions of feature selection, with varying biases and predictive abilities. However, predictive power is but one factor to consider when choosing a feature selection technique: one must also consider the technique's stability, that is, its ability to create feature subsets which remain valid in the face of changes to the data. While there has been work in determining the relative stability of different feature selection techniques, this does not always help determine whether a chosen feature selection technique will give stable feature subsets for a specific dataset. Factors such as difficulty of learning (e.g., dataset difficulty) may also influence feature selection stability, making generally-true facts about different techniques not applicable to a given dataset. In this work, we study how dataset difficulty can affect the stability of feature selection techniques, leading to good performance from bad techniques and vice versa. We use a set of twenty-six DNA microarray datasets with varying levels of difficulty of learning, along with four levels of dataset perturbation, six feature selection techniques with various levels of stability, and twelve feature subset sizes. The results show that as the dataset difficulty increases, the stability decreases. However, the relative stability between the techniques remains the same. Additionally, the more difficult the dataset, the more the stability is affected by changes to the data. We also found that unstable rankers are more affected by the transition between Easy and Moderate datasets, whereas the stable techniques are more affected by the change between Moderate and Hard datasets. Lastly, as the feature subset size increases, the stability increases and the difference between the levels of dataset difficulty decreases. Overall, we conclude that difficulty of learning must be taken into account before interpreting stability results.