A BOOLEAN ALGEBRA BASED RULE EXTRACTION ALGORITHM FOR NEURAL NETWORKS WITH BINARY OR BIPOLAR INPUTS

When performing predictive modeling, the key criterion is always accuracy. With this in mind, complex techniques like neural networks or ensembles are normally used, resulting in opaque models impossible to interpret. When models need to be comprehensible, accuracy is often sacrificed by using simpler techniques directly producing transparent models; a tradeoff termed the accuracy vs. comprehensibility tradeoff. In order to reduce this tradeoff, the opaque model can be transformed into another, interpretable, model; an activity termed rule extraction. In this paper, it is argued that rule extraction algorithms should gain from using oracle data; i.e. test set instances, together with corresponding predictions from the opaque model. The experiments, using 17 publicly available data sets, clearly show that rules extracted using only oracle data were significantly more accurate than both rules extracted by the same algorithm, using training data, and standard decision tree algorithms. In addition, the same rules were also significantly more compact; thus providing better comprehensibility. The overall implication is that rules extracted in this fashion explain the predictions made on novel data better than rules extracted in the standard way; i.e. using training data only

This paper proposes a hybrid approach to the design of a compact fuzzy rule-based classification system with a small number of linguistic rules. The proposed approach consists of two procedures: rule extraction from a trained neural network and rule selection by a genetic algorithm. We first describe how linguistic rules can be extracted from a multilayer feedforward neural network that has been already trained for a classification problem with many continuous attributes. In our rule extraction procedure, a linguistic input vector corresponding to the antecedent part of a linguistic rule is presented to the trained neural network, and the fuzzy output vector front the trained neural network is examined for determining the consequent part and the grade of certainty of that linguistic rule. Next we explain how a genetic algorithm can be utilized for selecting a small number of significant linguistic rules from a large number of extracted rules. Our rule selection problem has two objectives: to minimize the number of selected linguistic rules and to maximize the number of correctly classified patterns by the selected linguistic rules. A multi-objective genetic algorithm is employed for finding a set of non-dominated solutions with respect to these two objectives. Finally we illustrate our hybrid approach by computer simulations on real-world test problems.

We investigate the expressive power of neural networks from the point of view of descriptive complexity. We study neural networks that use floating-point numbers and piecewise polynomial activation functions from two perspectives: (i) the general scenario where neural networks run for an unlimited number of computation steps and have unrestricted topologies, and (ii) classical feedforward neural networks that have the topology of layered acyclic graphs and run for only a constant number of computation steps. We characterize these neural networks via Boolean networks formalized via a recursive rule-based logic. In particular, we show that the sizes of the neural networks and the corresponding Boolean rule formulae are polynomially related. In fact, in the translation from Boolean rules to neural networks, the blow-up is only linear. Our translations result in a time delay, defined as the number of computation steps that the output-object of the translation (e.g. a neural network or Boolean rule formula) uses to simulate a single computation step of the input-object. In the translation from neural networks to Boolean rules, the time delay of the resulting formula is polylogarithmic in the size of the neural network. In the converse translation, the time delay of the neural network is linear in the formula size. Ultimately, we obtain translations between neural networks, Boolean networks, the diamond-free fragment of modal substitution calculus and a class of recursive Boolean circuits. Our translations offer a method, for almost any activation function F, of translating any neural network in our setting into an equivalent neural network that uses F at each node. This even includes linear activation functions, which is possible due to using floats rather than actual reals!

The main challenge in using supervised neural networks in data mining applications is to get explicit knowledge from these models. For this purpose, a study on knowledge acquisition from supervised neural networks employed for classification problems is presented. An algorithm for rule extraction from neural networks, based on the RX algorithm is developed. This algorithm, named modified RX, is experimentally evaluated in two different domains: Iris Plants Database and Pima Indians Diabetes Database. The results are compared to those obtained by classification trees. As far as the efficacy is concerned, one observes that the successful application of the algorithm mainly depends on the knowledge representation acquired by the connectionist model, whereas the efficiency only depends on the neural network training time.

Mortality rate due to fatal heart disease (HD) or cardiovascular disease (CVD) has increased drastically over the world in recent decades. HD is a very hazardous problem prevailing among people which is treatable if detected early. But in most of the cases, the disease is not diagnosed until it becomes severe. Hence, it is requisite to develop an effective system which can accurately diagnosis HD and provide a concise description for the underlying causes [risk factors (RFs)] of the disease, so that in future HD can be controlled only by managing the primary RFs. Recently, researchers are using various machine learning algorithms for HD diagnosis, and neural network (NN) is one among them which has attracted tons of people because of its high performance. But the main obstacle with a NN is its black-box nature, i.e., its incapability in explaining the decisions. So, as a solution to this pitfall, the rule extraction algorithms can be very effective as they can extract explainable decision rules from NNs with high prediction accuracies. Many neural-based rule extraction algorithms have been applied successfully in various medical diagnosis problems. This study assesses the performance of rule extraction algorithms for HD diagnosis, particularly those that construct rules recursively from NNs. Because they subdivide a rule’s subspace until the accuracy improves, recursive algorithms are known for delivering interpretable decisions with high accuracy. The recursive rule extraction algorithms’ efficacy in HD diagnosis is demonstrated by the results. Along with the significant data ranges for the primary RFs, a maximum accuracy of 82.59% is attained.

Rule extraction based on PROMETHEE-assisted multi-objective genetic algorithm for generating interpretable neural networks

A modified fuzzy min–max neural network with rule extraction and its application to fault detection and classification

Rule extraction from data is one of the key technologies for solving the bottlenecks in artificial intelligence. Artificial neural networks are well suited for representing any knowledge in given data. Extraction of logical/fuzzy rules from the trained artificial neural network is of great importance to researchers in the fields of artificial intelligence and soft computing. Fuzzy rule sets are capable of approximating any nonlinear mapping relationships. Extraction of rules from data has been discussed in terms of fuzzy modeling, fuzzy clustering, and classification with fuzzy rule sets. This special issue entitled"Rule Extraction from Data" is aimed at providing the readers with good insights into the advanced studies in the field of rule extraction from data using neural networks/fuzzy rule sets. I invited seven research papers best suited for the theme of this special issue. All the papers were reviewed rigorously by two reviewers each. The first paper proposes an interesting rule extraction method from data using neural networks. Ishikawa presents a combination of learning with an immediate critic and a structural learning with forgetting. This method is capable of generating skeletal networks for logical rule extraction from data with correct and wrong answers. The proposed method is applied to rule extraction from lense data. The second paper presents a new methodology for logical rule extraction based on transformation of MLP (multilayered perceptron) to a logical network. Duck et al. applied their C-MLP2LN to the Iris benchmark classification problem as well as real-world medical data with very good results. In the third paper, Geczy and Usui propose fuzzy rule extraction from trained artificial neural networks. The proposed algorithm is implied from their theoretical study, not from heuristics. Their study enables to initially consider derivation of crisp rules from trained artificial neural network, and in case of conflict, application of fuzzy rules. The proposed algorithm is experimentally demonstrated with the Iris benchmark classification problem. The fourth paper presents a new framework for fuzzy modeling using genetic algorithm. The authors have broken new ground of fuzzy rule extraction from neural networks. For the fuzzy modeling, they have proposed a particular type of neural networks containing nodes representing membership functions. In this fourth paper, the authors discuss input variable selection for the fuzzy modeling under multiple criteria with different importance. A target system with a strong nonlinearity is used for demonstrating the proposed method. Kasabov, et al. present, in the fifth paper, a method for extraction of fuzzy rules that have different level of abstraction depending on several modifiable thresholds. Explanation quality becomes better with higher threshold values. They apply the proposed method to the Iris benchmark classification problem and to a real world problem. J. Yen and W. Gillespie address interpretability issue of Takagi-Sugeno-Kang model, one of the most popular fuzzy mdoels, in the fifth paper. They propose a new approach of fuzzy modeling that ensures not only a high approximation of the input-output relationship in the data, but also good insights about the local behavior of the model. The proposed method is applied to fuzzy modeling of sinc function and Mackey-Glass chaotic time series data. The last paper discusses fuzzy rule extraction from numerical data for high-dimensional classification problems. H.Ishibuchi, et al. have been pioneering methods for classification of data using fuzzy rules and genetic algorithm. In this last paper, they introduced a new criterion, simplicity of each rule, together with the conventional ones, compactness of rule base and classification ability, for high-dimensional problem. The Iris data is used for demonstrating their new classification method. They applied it also to wine data and credit data. I hope that the readers will be encouraged to explore the frontier to establish a new paradigm in the field of knowledge representation and rule extraction.

Representing the knowledge learned by neural networks in the form of interpretable rules is a prudent technique to justify the decisions made by neural networks. Heretofore many algorithms exist to extract symbolic rules from neural networks, but among them, a few extract rules from deep neural networks trained using deep learning techniques. So, this paper proposes an algorithm to extract rules from a multi-hidden layer neural network, pre-trained using deep belief network and fine-tuned using back propagation. The algorithm analyzes each node of a layer and extracts knowledge from each layer separately. The process of knowledge extraction from the first hidden layer is different from the other layers. Consecutively, the algorithm combines all the knowledge extracted and refines them to construct a final ruleset consisting of symbolic rules. The algorithm further subdivides the subspace of a rule in the ruleset if it satisfies certain conditions. Results show that the algorithm extracted rules with higher accuracy compared to some existing rule extraction algorithms. Other than accuracy, the efficacy of the extracted rules is also validated with fidelity and various other performance measures.

Artificial Neural Network (ANN) is one of the powerful techniques of machine learning. It has shown its effectiveness in both prediction and classification problems. However, in some fields there is still some reticence towards their use mainly the fact that they do not justify their answers. The lack of transparency on how ANN makes decisions motivated us to develop our rule extraction algorithm that extracts comprehensible rules with high accuracy and high fidelity. The aim is to generate a set of rules that mimic the decision of ANN and cover a larger set of patterns. The obtained rule sets should satisfy a well-balanced trade-off between the fidelity, the accuracy and the comprehensibility. The proposed algorithm consists of a three steps: ANN learning phase, rule extraction phase and rule simplification phase. The rule extraction phase is based on the extraction of the association rules while the rules simplification procedure is based on the laws of Boolean algebra. To evaluate the performance of our algorithm, the system has been studied using four datasets, and then compared with other rule extraction methods. The results show that our proposal offers a small set of rules having the highest accuracy and fidelity values.

This paper presents a hybrid data mining mechanism to extract expert knowledge from historical data and extend expert systems' reasoning capabilities by using fuzzy neural network (FNN)-based learning & rule extraction algorithm. Our hybrid data mining mechanism is based on association rule extraction mechanism, FNN learning and fuzzy rule extraction algorithm. Most of traditional data mining mechanisms are depended ()n association rule extraction algorithm. However, the basic association rule-based data mining systems has not the learning ability. Therefore, there is a problem to extend the knowledge base adaptively. In addition, sequential patterns of association rules can`t represent the complicate fuzzy logic in real-world. To resolve these problems, we suggest the hybrid data mining mechanism based on association rule-based data mining, FNN learning and fuzzy rule extraction algorithm. Our hybrid data mining mechanism is consisted of four phases. First, we use general association rule mining mechanism to develop an initial rule base. Then, in the second phase, we adopt the FNN learning algorithm to extract the hidden relationships or patterns embedded in the historical data. Third, after the learning of FNN, the fuzzy rule extraction algorithm will be used to extract the implicit knowledge from the FNN. Fourth, we will combine the association rules (initial rule base) and fuzzy rules. Implementation results show that the hybrid data mining mechanism can reflect both association rule-based knowledge extraction and FNN-based knowledge extension.

This article addresses the topic of extracting logical rules from data by means of artificial neural networks. The approach based on piecewise linear neural networks is revisited, which has already been used for the extraction of Boolean rules in the past, and it is shown that this approach can be important also for the extraction of fuzzy rules. Two important theoretical properties of piecewise-linear neural networks are proved, allowing an elaboration of the basic ideas of the approach into several variants of an algorithm for the extraction of Boolean rules. That algorithm has already been used in two real-world applications. Finally, a connection to the extraction of rules of the Łukasiewicz logic is established, relying on recent results about rational McNaughton functions. Based on one of the constructive proofs of the McNaughton theorem, an algorithm is formulated that in principle allows extracting a particular kind of formulas of the Łukasiewicz predicate logic from piecewise-linear neural networks trained with rational data.

As was seen in Chapter 12 of BFE, several Boolean-based algebraic structures have been studied, that is, algebras having as support a Boolean algebra and whose operations are Boolean functions; homomorphisms connecting such algebras and expressed by Boolean functions have also been investigated. Chapter 13 of BFE deals with Boolean arithmetic and Boolean geometry. The former means the study of divisibility between Boolean functions. In Boolean geometry the role of the space is played by a Boolean algebra and one looks for analogues of the basic concepts of geometry; for instance, a “good” analogue of the distance function is the symmeric difference d(x, y) = x + y. As was seen in BFE, Chapter 14, Boolean analysis replaces the real line by a Boolean algebra, while the functions dealt with are Boolean functions.

Boolean algebra is the primary mathematical tool used in designing the computing machines that have come into widespread use in the latter part of this century. Switching functions, described using two-valued Boolean algebra, are implemented using transistor circuits [4]. These transistors act as on-off switches, which is a natural implementation for representing the primitive operations in a Boolean algebra. Once the primitive operations of a Boolean algebra are implemented using transistor circuits, any Boolean function can be implemented using these primitive operations. Operations AND, OR and NOT form the primitive operations of Boolean algebra. Other variations of these primitive operations such and NAND and NOT can also be used as the set of primitive operations.

In order to make full use of the driver’s long-term driving experience in the process of perception, interaction and vehicle control of road traffic information, a driving behavior rule extraction algorithm based on artificial neural network interface (ANNI) and its integration is proposed. Firstly, based on the cognitive learning theory, the cognitive driving behavior model is established, and then the cognitive driving behavior is described and analyzed. Next, based on ANNI, the model and the rule extraction algorithm (ANNI-REA) are designed to explain not only the driving behavior but also the non-sequence. Rules have high fidelity and safety during driving without discretizing continuous input variables. The experimental results on the UCI standard data set and on the self-built driving behavior data set, show that the method is about 0.4% more accurate and about 10% less complex than the common C4.5-REA, Neuro-Rule and REFNE. Further, simulation experiments verify the correctness of the extracted driving rules and the effectiveness of the extraction based on cognitive driving behavior rules. In general, the several driving rules extracted fully reflect the execution mechanism of sequential activity of driving comprehensive cognition, which is of great significance for the traffic of mixed traffic flow under the network of vehicles and future research on unmanned driving.