Generalized kernel function Fisher discriminant for pattern recognition

The decision tree-based classification is a popular approach for pattern recognition and data mining. Most decision tree induction methods assume training data being present at one central location. Given the growth in distributed databases at geographically dispersed locations, the methods for decision tree induction in distributed settings are gaining importance. This paper extends two well-known decision tree methods for centralized data to distributed data settings. The first method is an extension of CHAID algorithm and generates single feature based multi-way split decision trees. The second method is based on Fisher's linear discriminant (FLO) function and generates multifeature binary trees. Both methods aim to generate compact trees and are able to handle multiple classes. The suggested extensions for distributed environment are compared to their centralized counterparts and also to each other. Theoretical analysis and experimental tests demonstrate the effectiveness of the extensions. In addition, the side-by-side comparison highlights the advantages and deficiencies of these methods under different settings of the distribution environments.

Fisher Discriminant Analysis (FDA) is a widely used technique for signal classification. Its application varies from face recognition to speaker recognition. FDA aims to project a given feature onto a projected space, where the features coming from the same class are moved closer, while those coming from different classes are moved farther. However, in the original formulation of FDA, the number of orthogonal projection directions is limited by the number of classes, which may hinder the effectiveness of FDA as a projection technique. In this paper, we propose to use new between-class scatter matrices to replace the original between-class scatter matrix, in order to increase the number of orthogonal projection directions. We call FDA with these new between-class scatter matrices the Modified FDA (MFDA). The effectiveness of MFDA and FDA as a projection technique is compared through doing two audio signal classification tasks. Both linear version and kernel version of MFDA and FDA are evaluated, and experimental results show that MFDA can outperform FDA in both classification tasks.

Fisher discriminant analysis (FDA) is a classic supervised dimensionality reduction method in statistical pattern recognition. FDA can maximize the scatter between different classes, while minimizing the scatter within each class. As it only utilizes the labeled data and ignores the unlabeled data in the analysis process of FDA, it cannot be used to solve the unsupervised learning problems. Its performance is also very poor in dealing with semi-supervised learning problems in some cases. Recently, several semi-supervised learning methods as an extension of FDA have proposed. Most of these methods solve the semi-supervised problem by using a tradeoff parameter that evaluates the ratio of the supervised and unsupervised methods. In this paper, we propose a general semi-supervised dimensionality learning idea for the partially labeled data, namely the reconstruction probability class of labeled and unlabeled data. Based on the probability class optimizes Fisher criterion function, we propose a novel Semi-Supervised Local Fisher Discriminant Analysis (S2LFDA) method. Experimental results on real-world datasets demonstrate its effectiveness compared to the existing similar correlation methods.

To generalize the Fisher Discriminant Analysis (FDA) algorithm to the case of discriminant functions belonging to a nonlinear, finite dimensional function space F (Nonlinear FDA or NFDA), it is sufficient to expand the input data by computing the output of a basis of F when applied to it [1,2,3,4]. The solution to NFDA can then be found like in the linear case by solving a generalized eigenvalue problem on the between- and within-classes covariance matrices (see e.g. [5]). The goal of NFDA is to find linear projections of the expanded data (i.e., nonlinear transformations of the original data) that minimize the variance within a class and maximize the variance between different classes. Such a representation is of course ideal to perform classification. The application of NFDA to pattern recognition is particularly appealing, because for a given input signal and a fixed function space it has no parameters and it is easy to implement and apply. Moreover, given C classes only C - 1 projections are relevant [5]. As a consequence, the feature space is very small and the algorithm has low memory requirements and high speed during recognition. Here we apply NFDA to a handwritten digit recognition problem using the MNIST database, a standard and freely available set of 70,000 handwritten digits (28 × 28 pixels large), divided into a training set (60,000 digits) and a test set (10,000 digits). Several established pattern recognition methods have been applied to this database by Le Cun et al. [6]. Their paper provides a standard reference work to benchmark new algorithms. We perform NFDA on spaces of polynomials of a given degree d, whose corresponding basis functions include all monomials up to order d in all input variables. It is clear that the problem quickly becomes intractable because of the high memory requirements. For this reason, the input dimensionality is first reduced by principal component analysis. On the preprocessed data we then apply NFDA by expanding the training patterns in the polynomial space and solving the linear FDA eigenvalue problem. As mentioned above, since we have 10 classes we only need to compute the first 9 eigenvectors. Since the within-class variance is minimized, the patterns belonging to different classes tend to cluster in the feature space when projected on the eigenvectors. For this reason we classify the digits with a simple method such as Gaussian classifiers. We perform simulations with polynomials of degree 2 to 5. With polynomials of degree 2 the explosion in the dimensionality of the expanded space with increasing number of input dimensions is relatively restricted, so that it is possible to use up to 140 dimensions. With higher order polynomials one has to rely on a smaller number of input dimensions, but since the function space gets larger and includes new nonlinearities, one obtains a remarkable improvement in performance. The best performance is achieved with polynomials of degree 3 and 35 input dimensions, with an error rate of 1.5% on test data. This error rate is comparable to but does not outperform that of the most elaborate algorithms (Table 1). The performance of NFDA is however remarkable considering the simplicity of the method and the fact that it has no a priori knowledge on the problem, in contrast for example to the LeNet-5 algorithm [6] which has been designed specifically for handwritten character recognition. In addition, for recognition, NFDA has to store and compute only 9 functions and has thus small memory requirements and a high recognition speed. It is also possible to formulate NFDA using the kernel trick, in which case one can in principle use function spaces of infinite dimensionality [1,2,3,4]. However, the limiting factor in that formulation is the number of training patterns, which makes it not realistic for this application. The performance of NFDA could be further improved using for example a more problem-specific preprocessing of the patterns (e.g., by increasing the size of the training set with new patterns generated by artificial distortion of the original one), boosting techniques, or mixture of experts with other algorithms [5,6].

A fast Markov chain based algorithm for MIML learning

Summary form only given. In the paper, the comparison of discriminative capabilities of conventional dimensionality reduction methods and the integration of a dimensionality reduction module and recognizer design with minimum classification error rate are discussed. Conventionally, principal component analysis (PCA) and Fisher's linear discriminant (FLD) are two most popular and widely used dimensionality reduction methods for pattern recognition. However, the objectives of these two methods are quite different. PCA basically tries to faithfully keep the original data representation but FLD tries to separate data distribution of different classes. It therefore seems that FLD can provide more discriminative characteristics to pattern recognition than PCA. However, a completely optimal feature extractor can never be anything but an optimal recognizer. It is only when constraints are placed on the classifier that one can formulate nontrivial problems. We apply a minimum error formulation (MEF) to integrate the design of dimensionality reduction module and pattern recognizer. The experimental results show that such an integration provides very good recognition performance on a data set of hand-written Chinese characters even when the feature number has been significantly reduced.

Correlation synthetic discriminant functions (SDFs) represent a practical and novel extension of matched spatial filter (MSF) correlators for distortion-invariant multi-object and multi-class pattern recognition. This paper reviews the off-line synthesis of such filters and the advantageous features of correlation shape control that they provide. We then concentrate on extensive tests performed with these filters to assess their performance in the identification of ship images, subjected to 3-D distortions. The pattern recognition problem addressed involves multi-object, multi-class recognition with aspect distortion-invariance in the presence of clutter. An adaptive threshold is shown to allow recognition of objects in the presence of spatially-varying modulation. The noise performance of these filters is also found to be most excellent. Correct classification rates approaching 98% can be obtained with these correlation SDFs.

Performance analysis of support vector machine, Gaussian Process Regression, sequential quadratic programming algorithms in modeling hydrogen-rich syngas production from catalyzed co-gasification of biomass wastes from oil palm

With the improvement of security requirements of individuals, businesses and countries, the identity authentication system based on face recognition technology is applied widely. The identity methods based on single theory, due to a variety of factors, have their own inherent limitations. In order to improve the efficiency and accuracy of face recognition, based on fusion of wavelet packet sub-images, a face recognition algorithm named FW-FLD is proposed in this paper by combining PCA (Principal Component Analysis) and FLD(Fisher Linear Discriminant). Firstly, the training samples are decomposed using wavelet packet. The fusion weights of the decomposed sub-images are calculated according to their energy distribution and the weighted fusion images are obtained, which reserve the images characteristic in the frequency. Then the features of the fused images are extracted using PCA, and the optimized Fisher space is constructed using FLD. Finally, face images are classified by measuring the projection coefficients of optimized training samples and testing samples in Fisher space. Experimental results on CMU PIE, JAFFE and AR face databases show that the proposed algorithm is robust, and can adapt to face recognition with various illumination, facial expressions and gestures. Compared with other algorithms, it not only improves the face recognition rate, but also has a higher time efficiency.

At present, problems related to data mining are aris-ing in various poorly formalizable areas of research(genetics, archaeology, medicine, etc.). Such problemsare characterized by insufficient knowledge about theobjects under study, which hampers the development ofmathematical models for them; by a large number ofmultitype (quantitative and qualitative) factors and arelatively small amount of data; and by the requirementthat analysis results be represented in a form clear toexperts in applied areas.A promising approach to solving such problems isbased on the class of logical decision functions. Theyare frequently represented in the form of a decisiontree. A detailed description of this class and referencescan be found in [1].The construction of decision functions with a mini-mal risk of erroneous predictions is an important task inmethods based on logical decision functions and inother data analysis techniques. It is well known that thecomplexity of the class of decision functions (thecapacity characteristic of the class [2]) is a importantfactor affecting the quality of decisions. To achieve thebest quality, we need a certain tradeoff between thecomplexity of the class and the accuracy of decisionsobtained on the training sample. Thus, the problemarises of choosing an optimal complexity of the class.In numerous applications, empirical data (trainingsample) are supplemented with expert knowledge,which is not tightly related to a distribution model(expert knowledge can be specified as upper bounds forthe risk; they can express constraints on the class of dis-tributions or the class of decision functions, specifypreference rules used in decision making, etc.). Whenan optimal complexity of the class is chosen, availableempirical data and expert knowledge have to be jointlytaken into account.This problem can be solved within the framework ofthe Bayesian learning theory. The idea behind thisapproach is to use a priori knowledge about the prob-lem in question so that each possible distribution (strat-egy, the state of nature) is assigned a certain weight.This weight reflects the expert’s intuitive confidence inwhether the unknown true distribution coincides withthat under consideration.In this paper, we propose a Bayesian recognitionmodel based on a finite set of events. The model is usedto develop and analyze methods for constructing logi-cal decision functions. While choosing an optimal com-plexity of the class, the model takes into account empir-ical data and expert knowledge.In pattern recognition, the task is to predict the pat-tern’s index for an arbitrary object of the populationdescribed by a set of variables. The prediction is madeby analyzing a training sample consisting of the valuesof these variables, together with the index of the corre-sponding pattern for each object. The variables can beof different types; i.e., some of them can be quantita-tive, while the others can be qualitative. As a rule, theproblem is solved in a class of decision functions inwhich an optimal function is sought according to agiven criterion. The class of logical decision functionsis defined on a set of partitions of the feature space intoa finite number of subdomains described by conjunc-tions of simple predicates. The number of subdomainsdetermines the degree of complexity of a logical func-tion.A Bayesian pattern recognition model on a finite setof events is defined by formulating certain statementsthat avoid the local metric properties of the featurespace (the transition from points of the space to events,where an event is understood as that of the original vari-ables taking values from a certain subdomain of thepartition). Moreover, the pattern recognition problem isconsidered on the values of a discrete unordered vari-able, the concept of a learning method (a mapping from

Background: Screening indexes of tumor serum markers for benign and malignant solitary pulmonary nodules (SPNs) were analyzed to find the optimum method for diagnosis. Methods: Enzyme-linked immunosorbent assays, an automatic immune analyzer and radioimmunoassay methods were used to examine the levels of 8 serum markers in 164 SPN patients, and the sensitivity for differential diagnosis of malignant or benign SPN was compared for detection using a single plasma marker or a combination of markers. The results for serological indicators that closely relate to benign and malignant SPNs were screened using the Fisher discriminant analysis and a non-conditional logistic regression analysis method, respectively. The results were then verified by the k-means clustering analysis method. Results: The sensitivity when using a combination of serum markers to detect SPN was higher than that using a single marker. By Fisher discriminant analysis, cytokeratin 19 fragments (CYFRA21-1), carbohydrate antigen 125 (CA125), squamous cell carcinoma antigen (SCC) and breast cancer antigen (CA153), which relate to the benign and malignant SPNs, were screened. Through non-conditional logistic regression analysis, CYFRA21-1, SCC and CA153 were obtained. Using the k-means clustering analysis, the cophenetic correlation coefficient (0.940) obtained by the Fisher discriminant analysis was higher than that obtained with logistic regression analysis (0.875). Conclusion: This study indicated that the Fisher discriminant analysis functioned better in screening out serum markers to recognize the benign and malignant SPN. The combined detection of CYFRA21-1, CA125, SCC and CA153 is an effective way to distinguish benign and malignant SPN, and will find an important clinical application in the early diagnosis of SPN.

Currently, data mining based on machine learning plays a key role in decision support in various industries. An important practical problem of machine learning is the implementation of object classification in real time, which can be achieved by parallelizing data processing algorithms both for input data and for decision function data. To improve the efficiency of parallelizing machine learning methods, a unified decision function model has been developed. The Relevance of this research is to present a unified decision function model in the framework of machine learning algorithms and functions for its parallelization both in terms of input data and decision function data.The essence of the presented approach is that the features of the operation of metric methods of machine learning are analyzed, independent data for processing are presented using different categories of the analyzed property, developed decision function model describes the object features for input data and decision function data using standardized elements and including functions for their parallel processing based on group parallelization of objects. The proposed approach is based on the use of methods for analyzing algorithms and computational complexity, mathematical statistics and the methodology of designing parallel algorithms. Experiments have shown that parallelization of the proposed decision function model for the potential function method allows increasing the classification efficiency for one object using additional computing resources, and for a group of objects within the limits of the computer's memory size or planning horizon.The novelty of the proposed approach is that the model differs from existing ones in a method of formalizing objects and their features using unified elements for training and classification objects and has a structure and functions oriented towards its parallel processing by pattern recognition methods based on decision functions within the framework of group parallelization of objects. Theoretical significance: the model is unified and can be used to parallelize other pattern recognition methods that can be described by similar parameters, architecture, and classification features. The practical significance of the proposed approach is that the model allows decomposing the pattern classification problem into separate subtasks of finding regularities between input data and decision function data.

The Kovats retention index is the most robust retention parameter used to identify analytes in gas chromatography. The Kovats retention index’s value in similar compounds tends to have a similar pattern. This pattern allows a close relationship between chemical structure and the retention index value. The Fuzzy Support Vector Regression (FSVR) method is a robust algorithm that has been used to solve various regression problems. This study aimed to evaluate the FSVR algorithm performance to predict the Kovats retention index of flavor and fragrance compounds. The model was built using the Python programming language. The retention index value calculation begins with inputting the molecular descriptor data into the sigmoid fuzzy membership function. The dataset consists of 51 compounds then partitioned into two sets, namely 80% in the training set and 20% in the testing set. The best parameters to build the model were obtained using hyperparameter tuning. The kernels used are linear kernel and Radial Basis Function (RBF) kernel. The model with a linear kernel with parameter values of C = 1000 and epsilon = 10 managed to get the best results compared to the model with an RBF kernel, with R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.977 and Root Mean Square Error (RMSE) = 37.21 on the testing set. This model also can predict the Kovats retention index on the testing set with an average difference of 2.8%. These results showed that the linear kernel is better than the RBF kernel in predicting the value of the Kovats retention index. It can be concluded that the FSVR method can be used to predict the retention index of flavor and fragrance compounds.

The mechanical actions involved in recycling cashmere fibers, notably from woven fabrics, cause fiber damage. There are three distinct forms of damage: partial cleavages, axial fractures, and short fibers. Fisher's discriminant analysis techniques are applied to two groups of cashmere fibers, new and recycled. The discriminant function is tested on additional samples from the two groups, all of which are correctly allocated. Thus, the model is apparently capable of distinguishing between new and recycled cashmere fibers.

In this paper Fisher's discriminant analysis (FDA) is used for detecting and diagnosing faults in a real plant. FDA provides an optimal lower dimensional representation in terms of discriminating between classes of data, where, in this context of fault diagnosis, each class corresponds to data collected during a specific, known fault. A discriminant function is applied to detect and diagnose faults using both simulated and real data collected from a plant: a two-tank system, showing good results.