Fuzzy Data Mining by Heuristic Rule Extraction and Multiobjective Genetic Rule Selection

We examine the effect of genetic rule selection as a postprocessing procedure in fuzzy data mining. Usually a large number of fuzzy rules are extracted in a heuristic manner from numerical data using a rule evaluation criterion in fuzzy data mining. It is, however, very difficult for human users to understand thousands of fuzzy rules. Thus it is necessary to decrease the number of extracted fuzzy rules when our task is to present understandable knowledge to human users. In this paper, we use genetic rule selection to decrease the number of extracted fuzzy rules. Through computational experiments, we examine the effect of genetic rule selection. First we extract fuzzy rules that satisfy minimum support and confidence levels. Thousands of fuzzy rules are extracted from numerical data in a heuristic manner. Then we apply genetic rule selection to extracted fuzzy rules. Experimental results show that genetic rule selection significantly decreases the number of extracted fuzzy rules without degrading their classification accuracy.

In this chapter, we explain how multi-objective genetic algorithms can be applied to the design of fuzzy rule-based systems for pattern classification problems. For a multi-class pattern classification problem with many continuous attributes (e.g., wine classification with 13 continuous attributes [1]), a fuzzy rule-based classification system is designed by a multi-objective genetic algorithm [2, 3] with two objectives: to minimize the size of the fuzzy rule-based classification system and to maximize its performance [4, 5]. The size of the fuzzy rule-based classification system is expressed by the number of fuzzy if-then rules, and its performance is measured by the number of correctly classified training patterns. In our approach to the design of the fuzzy rule-based classification system, first a large number of fuzzy if-then rules are generated from training patterns as candidate rules for the rule selection. Then a small number of fuzzy if-then rules are selected from the candidate rules by the two-objective genetic algorithm. The two-objective genetic algorithm tries to find all the non-dominated solutions (i.e., non-dominated rule sets) of the rule selection problem with the above-mentioned objectives. One difficulty of the rule selection method for high-dimensional pattern classification problems with many continuous attributes is that the number of candidate fuzzy if-then rules becomes intractably large. For example, the number of candidate rules is 613 ≅ 1.3 × 1010 for the wine classification problem with 13 continuous attributes if we have six fuzzy sets (terms) on each axis of the 13-dimensional pattern space.

One advantage of evolutionary multiobjective optimization (EMO) algorithms over classical approaches is that many non-dominated solutions can be simultaneously obtained by their single run. This paper shows how this advantage can be utilized in genetic rule selection for the design of fuzzy rule-based classification systems. Our genetic rule selection is a two-stage approach. In the first stage, a pre-specified number of candidate rules are extracted from numerical data using a data mining technique. In the second stage, an EMO algorithm is used for finding non-dominated rule sets with respect to three objectives: to maximize the number of correctly classified training patterns, to minimize the number of rules, and to minimize the total rule length. Since the first objective is measured on training patterns, the evolution of rule sets tends to overfit to training patterns. The question is whether the other two objectives work as a safeguard against the overfitting. In this paper, we examine the effect of the three-objective formulation on the generalization ability (i.e., classification rates on test patterns) of obtained rule sets through computer simulations where many non-dominated rule sets are generated using an EMO algorithm for a number of high-dimensional pattern classification problems.

This paper proposes a hybrid approach to the design of a compact fuzzy rule-based classification system with a small number of linguistic rules. The proposed approach consists of two procedures: rule extraction from a trained neural network and rule selection by a genetic algorithm. We first describe how linguistic rules can be extracted from a multilayer feedforward neural network that has been already trained for a classification problem with many continuous attributes. In our rule extraction procedure, a linguistic input vector corresponding to the antecedent part of a linguistic rule is presented to the trained neural network, and the fuzzy output vector front the trained neural network is examined for determining the consequent part and the grade of certainty of that linguistic rule. Next we explain how a genetic algorithm can be utilized for selecting a small number of significant linguistic rules from a large number of extracted rules. Our rule selection problem has two objectives: to minimize the number of selected linguistic rules and to maximize the number of correctly classified patterns by the selected linguistic rules. A multi-objective genetic algorithm is employed for finding a set of non-dominated solutions with respect to these two objectives. Finally we illustrate our hybrid approach by computer simulations on real-world test problems.

We propose a hybrid algorithm of two fuzzy genetics-based machine learning approaches (i.e., Michigan and Pittsburgh) for designing fuzzy rule-based classification systems. First, we examine the search ability of each approach to efficiently find fuzzy rule-based systems with high classification accuracy. It is clearly demonstrated that each approach has its own advantages and disadvantages. Next, we combine these two approaches into a single hybrid algorithm. Our hybrid algorithm is based on the Pittsburgh approach where a set of fuzzy rules is handled as an individual. Genetic operations for generating new fuzzy rules in the Michigan approach are utilized as a kind of heuristic mutation for partially modifying each rule set. Then, we compare our hybrid algorithm with the Michigan and Pittsburgh approaches. Experimental results show that our hybrid algorithm has higher search ability. The necessity of a heuristic specification method of antecedent fuzzy sets is also demonstrated by computational experiments on high-dimensional problems. Finally, we examine the generalization ability of fuzzy rule-based classification systems designed by our hybrid algorithm.

We examine the performance of compact fuzzy rule-based classification systems that consist of a small number of simple fuzzy rules with high comprehensibility. Those fuzzy systems are designed in a heuristic manner using rule selection criteria. We first describe fuzzy rule-based classification. Next we describe heuristic rule selection criteria using the terminology in data mining: confidence and support. A small number of fuzzy rules are extracted from numerical data based on each rule selection criterion. Then we examine the classification performance of extracted fuzzy rules through computational experiments on a number of benchmark data sets from the UCI ML Repository. Our results should be viewed as the lowest benchmark performance of fuzzy rule-based classification systems because fuzzy rules are extracted using a simple heuristic method with no optimization or tuning procedures. Nevertheless our results on some data sets are comparable to reported results by the C4.5 algorithm in the literature.

Rule extraction from data is one of the key technologies for solving the bottlenecks in artificial intelligence. Artificial neural networks are well suited for representing any knowledge in given data. Extraction of logical/fuzzy rules from the trained artificial neural network is of great importance to researchers in the fields of artificial intelligence and soft computing. Fuzzy rule sets are capable of approximating any nonlinear mapping relationships. Extraction of rules from data has been discussed in terms of fuzzy modeling, fuzzy clustering, and classification with fuzzy rule sets. This special issue entitled"Rule Extraction from Data" is aimed at providing the readers with good insights into the advanced studies in the field of rule extraction from data using neural networks/fuzzy rule sets. I invited seven research papers best suited for the theme of this special issue. All the papers were reviewed rigorously by two reviewers each. The first paper proposes an interesting rule extraction method from data using neural networks. Ishikawa presents a combination of learning with an immediate critic and a structural learning with forgetting. This method is capable of generating skeletal networks for logical rule extraction from data with correct and wrong answers. The proposed method is applied to rule extraction from lense data. The second paper presents a new methodology for logical rule extraction based on transformation of MLP (multilayered perceptron) to a logical network. Duck et al. applied their C-MLP2LN to the Iris benchmark classification problem as well as real-world medical data with very good results. In the third paper, Geczy and Usui propose fuzzy rule extraction from trained artificial neural networks. The proposed algorithm is implied from their theoretical study, not from heuristics. Their study enables to initially consider derivation of crisp rules from trained artificial neural network, and in case of conflict, application of fuzzy rules. The proposed algorithm is experimentally demonstrated with the Iris benchmark classification problem. The fourth paper presents a new framework for fuzzy modeling using genetic algorithm. The authors have broken new ground of fuzzy rule extraction from neural networks. For the fuzzy modeling, they have proposed a particular type of neural networks containing nodes representing membership functions. In this fourth paper, the authors discuss input variable selection for the fuzzy modeling under multiple criteria with different importance. A target system with a strong nonlinearity is used for demonstrating the proposed method. Kasabov, et al. present, in the fifth paper, a method for extraction of fuzzy rules that have different level of abstraction depending on several modifiable thresholds. Explanation quality becomes better with higher threshold values. They apply the proposed method to the Iris benchmark classification problem and to a real world problem. J. Yen and W. Gillespie address interpretability issue of Takagi-Sugeno-Kang model, one of the most popular fuzzy mdoels, in the fifth paper. They propose a new approach of fuzzy modeling that ensures not only a high approximation of the input-output relationship in the data, but also good insights about the local behavior of the model. The proposed method is applied to fuzzy modeling of sinc function and Mackey-Glass chaotic time series data. The last paper discusses fuzzy rule extraction from numerical data for high-dimensional classification problems. H.Ishibuchi, et al. have been pioneering methods for classification of data using fuzzy rules and genetic algorithm. In this last paper, they introduced a new criterion, simplicity of each rule, together with the conventional ones, compactness of rule base and classification ability, for high-dimensional problem. The Iris data is used for demonstrating their new classification method. They applied it also to wine data and credit data. I hope that the readers will be encouraged to explore the frontier to establish a new paradigm in the field of knowledge representation and rule extraction.

In this paper, we examine the effectiveness of genetic rule selection with a multi-classifier coding scheme for ensemble classifier design. Genetic rule selection is a two-stage method. The first stage is rule extraction from numerical data using a d

Rule extraction based on PROMETHEE-assisted multi-objective genetic algorithm for generating interpretable neural networks

Rule extraction using Recursive-Rule extraction algorithm with J48graft combined with sampling selection techniques for the diagnosis of type 2 diabetes mellitus in the Pima Indian dataset

Learning fuzzy rules from iterative execution of games

We have been using fuzzy data mining techniques to extract patterns that represent behavior for intrusion detection. We describe a variety of modifications that we have made to the data mining algorithms in order to improve accuracy and efficiency. We use sets of fuzzy association rules that are mined from network audit data as models of behavior. To detect anomalous behavior, we generate fuzzy association rules from new audit data and compute the similarity with sets mined from normal data. If the similarity values are below a threshold value, an alarm is issued. We describe an algorithm for computing fuzzy association rules based on Borgelt's (2001) prefix trees, modifications to the computation of support and confidence of fuzzy rules, a new method for computing the similarity of two fuzzy rule sets, and feature selection and optimization with genetic algorithms. Experimental results demonstrate that we can achieve better running time and accuracy with these modifications.

In this paper, we present a recursive algorithm for extracting classification rules from feedforward neural networks (NNs) that have been trained on data sets having both discrete and continuous attributes. The novelty of this algorithm lies in the conditions of the extracted rules: the rule conditions involving discrete attributes are disjoint from those involving continuous attributes. The algorithm starts by first generating rules with discrete attributes only to explain the classification process of the NN. If the accuracy of a rule with only discrete attributes is not satisfactory, the algorithm refines this rule by recursively generating more rules with discrete attributes not already present in the rule condition, or by generating a hyperplane involving only the continuous attributes. We show that for three real-life credit scoring data sets, the algorithm generates rules that are not only more accurate but also more comprehensible than those generated by other NN rule extraction methods.

Association Rule Mining is one of the important data mining activities and has received substantial attention in the literature. Association rule mining is a computationally and I/O intensive task. In this paper, we propose a solution approach for mining optimized fuzzy association rules of different orders. We also propose an approach to define membership functions for all the continuous attributes in a database by using clustering techniques. Although single objective genetic algorithms are used extensively, they degenerate the solution. In our approach, extraction and optimization of fuzzy association rules are done together using multi-objective genetic algorithm by considering the objectives such as fuzzy support, fuzzy confidence and rule length. The effectiveness of the proposed approach is tested using computer activity dataset to analyze the performance of a multi processor system and network audit data to detect anomaly based intrusions. Experiments show that the proposed method is efficient in many scenarios.

A genetic algorithm for generating fuzzy classification rules