Generic multimodal biometric fusion

Multimodal biometrics is the frontier to unimodal biometrics as it integrates the information obtained from multiple biometric sources at various fusion levels i.e. sensor level, feature extraction level, match score level, or decision level. In this article, fingerprint, palmprint, and iris are used for verification of an individual. The wavelet transformation is used to extract features from fingerprint, palmprint, and iris. Further the PCA is used for dimensionality reduction. The fusion of traits is employed at three levels: feature level; feature level combined with match score level; and feature level combined with decision level. The main objective of this research is to observe effect of combined fusion levels on verification of an individual. The performance of three cases of fusion is measured in terms of EER and represented with ROC. The experiments performed on 100 different subjects from publicly available databases demonstrate that combining feature level with match score level and feature level with decision level fusion both outperforms fusion at only a feature level.

Face recognition system contains lots of challenges due to various environmental factors, background variations, poor quality of camera, different illumination and others. Since twins are involved with criminal activities, twin identification becomes an essential task. The proposed system is focused on identifying the identical twins for the still images. The fusion based approach has been implemented in the proposed system. It combines the features extracted by using Principal Component Analysis (PCA), Histogram Oriented of Gradients (HOG), Local Binary Pattern (LBP), Gabor and distance between the facial components. Three types of fusion such as Decision Level Fusion, Feature Level Fusion and Score Level Fusion are used in the proposed approach. Based on these fusion generated scores, the twin has been identified. In the proposed system, Particle Swarm Optimization is used for the best feature selection and SVM classifier is used for training and testing the image. The proposed system provides better results when compared with the other twin detection techniques.

Biometric systems based solely on one-modal biometrics are often not able to meet the desired performance requirements for large user population applications, due to problems such as noisy data, intra-class variations, restricted degrees of freedom, nonuniversity, spoof attacks, and unacceptable error rates. Multimodal biometrics refers to the use of a combination of two or more biometric modalities in a single identification system. The most compelling reason to combine different modalities is to improve the recognition accuracy. This can be done when features of different biometrics are statistically independent. This paper overviews and discusses the various scenarios that are possible in multimodal biometric systems using fingerprint, face and iris recognition, the levels of fusion that are possible and the integration strategies that can be adopted to fuse information and improve overall system accuracy. This paper also discusses how the image quality of fingerprint, face and iris used in the multimodal biometric systems affects the overall identification accuracy and the need of staffing for the secondary human validation. For a large user population identification system, which often has more than tens or hundreds of millions of subject images already enrolled in the matcher databases and has to process more than hundreds of thousands of identification requests, the system's identification accuracy and the need of staffing levels to properly operate the system are two of the most important factors in determining whether a system is properly designed and integrated

Kernel methods for classification is a well-studied area in which data are implicitly mapped from a lower-dimensional space to a higher dimensional space to improve classification accuracy. However, for most kernel methods, one must still choose a kernel to use for the problem. Since there is, in general, no way of knowing which kernel is the best, multiple kernel learning (MKL) is a technique used to learn the aggregation of a set of valid kernels into a single (ideally) superior kernel. The aggregation can be done using weighted sums of the precomputed kernels, but determining the summation weights is not a trivial task. Furthermore, MKL does not work well with large datasets because of limited storage space and prediction speed. In this paper, we address all three of these multiple kernel challenges. First, we introduce a new linear feature level fusion technique and learning algorithm, GAMKLp. Second, we put forth three new algorithms, DeFIMKL, DeGAMKL, and DeLSMKL, for nonlinear fusion of kernels at the decision level. To address MKL's storage and speed drawbacks, we apply the Nystrom approximation to the kernel matrices. We compare our methods to a successful and state-of-the-art technique called MKL-group lasso (MKLGL), and experiments on several benchmark datasets show that some of our proposed algorithms outperform MKLGL when applied to support vector machine (SVM)-based classification. However, to no surprise, there does not seem to be a global winner but instead different strategies that a user can employ. Experiments with our kernel approximation method show that we can routinely discard most of the training data and at least double prediction speed without sacrificing classification accuracy. These results suggest that MKL-based classification techniques can be applied to big data efficiently, which is confirmed by an experiment using a large dataset.

In a biometric authentication system using protected templates, a pseudonymous identifier is the part of a protected template that can be directly compared. Each compared pair of pseudonymous identifiers results in a decision testing whether both identifiers are derived from the same biometric characteristic. Compared to an unprotected system, most existing biometric template protection methods cause to a certain extent degradation in biometric performance. Fusion is therefore a promising way to enhance the biometric performance in template-protected biometric systems. Compared to feature level fusion and score level fusion, decision level fusion has not only the least fusion complexity, but also the maximum interoperability across different biometric features, template protection and recognition algorithms, templates formats, and comparison score rules. However, performance improvement via decision level fusion is not obvious. It is influenced by both the dependency and the performance gap among the conducted tests for fusion. We investigate in this paper several fusion scenarios (multi-sample, multi-instance, multi-sensor, multi-algorithm, and their combinations) on the binary decision level, and evaluate their biometric performance and fusion efficiency on a multi-sensor fingerprint database with 71,994 samples.

Fault location for power grid based on transient travelling wave data fusion via asynchronous voltage measurements

Recently, development in intelligent transportation systems (ITS) requires the input of various kinds of data in real-time and from multiple sources, which imposes additional research and application challenges. Ongoing studies on Data Fusion (DF) have produced significant improvement in ITS and manifested an enormous impact on its growth. This paper reviews the implementation of DF methods in ITS to facilitate traffic flow analysis (TFA) and solutions that entail the prediction of various traffic variables such as driving behavior, travel time, speed, density, incident, and traffic flow. It attempts to identify and discuss real-time and multi-sensor data sources that are used for various traffic domains, including road/highway management, traffic states estimation, and traffic controller optimization. Moreover, it attempts to associate abstractions of data level fusion, feature level fusion, and decision level fusion on DF methods to better understand the role of DF in TFA and ITS. Consequently, the main objective of this paper is to review DF methods used for real-time and multi-sensor (heterogeneous) TFA studies. The review outcomes are (i) a guideline of constructing DF methods which involve preprocessing, filtering, decision, and evaluation as core steps, (ii) a description of the recent DF algorithms or methods that adopt real-time and multi-sensor sources data and the impact of these data sources on the improvement of TFA, (iii) an examination of the testing and evaluation methodologies and the popular datasets and (iv) an identification of several research gaps, some current challenges, and new research trends.

Non-stationary feature fusion of face and palmprint multimodal biometrics

This paper reviews several information fusion techniques and strategies in the application of multimodal biometrics system using face and palmprint images. Multimodal biometric is able to overcome several limitations in single modal biometric such as intra-class variations, less discriminative power, noise data and redundant features. By consolidating two kinds of modality a better performance can be achieved. Information fusion in multimodal biometrics can be carried out at three possible levels, i.e. feature, matching score and decision levels. Fusions at these three levels have their own attributes, thus this paper is aimed to compare their effectiveness. A specific fusion rule is necessary to combine the information at each level. Several numbers of analyses on verification and identification shows matching score fusion is able to achieve the best performance which is 98% recognition rates and 98.5% GAR at 0.1% FAR when tested using AR face and PolyU palmprint datasets.

Multimodal biometrics-based systems aim to improve the recognition accuracy of human beings using more than one physical and/or behavioral characteristics of a person. In this paper, different fusion schemes at matching score level and feature level are employed to obtain a robust recognition system using several standard feature extractors. The proposed method involves the consideration of a face–iris multimodal biometric system using score level and feature level fusion. Principal Component Analysis (PCA), subspace Linear Discriminant Analysis (LDA), subpattern-based PCA, modular PCA and Local Binary Patterns (LBP) are global and local feature extraction methods applied on face and iris images. In fact, different feature sets obtained from five local and global feature extraction methods for unimodal iris biometric system are concatenated at feature level fusion called iris feature vector fusion (iris-FVF), while for unimodal face biometric system, LBP is used to achieve efficient texture descriptors. Feature selection is performed using Particle Swarm Optimization (PSO) at feature level fusion step to reduce the dimension of feature vectors for improving the recognition performance. Our proposed method is validated by forming three datasets using ORL, BANCA, FERET face databases and CASIA, UBIRIS iris databases. The results based on recognition performance and ROC analysis demonstrate that the proposed matching score level fusion scheme using Weighted Sum rule, tanh normalization, iris-FVF and facial features extracted by LBP achieves a significant improvement over unimodal and multimodal methods. Support Vector Machine (SVM) and t-norm normalization are also used to improve the recognition performance of the proposed method.

The work collected in this dissertation addresses the problem of data fusion. In other words, this is the problem of making decisions (also known as the problem of classification in the machine learning and statistics communities) when data from multiple sources are available, or when decisions/confidence levels from a panel of decision-makers are accessible. This problem has become increasingly important in recent years, especially with the ever-increasing popularity of autonomous systems outfitted with suites of sensors and the dawn of the ``age of big data.'' While data fusion is a very broad topic, the work in this dissertation considers two very specific techniques: feature-level fusion and decision-level fusion. In general, the fusion methods proposed throughout this dissertation rely on kernel methods and fuzzy integrals. Both are very powerful tools, however, they also come with challenges, some of which are summarized below. I address these challenges in this dissertation. Kernel methods for classification is a well-studied area in which data are implicitly mapped from a lower-dimensional space to a higher-dimensional space to improve classification accuracy. However, for most kernel methods, one must still choose a kernel to use for the problem. Since there is, in general, no way of knowing which kernel is the best, multiple kernel learning (MKL) is a technique used to learn the aggregation of a set of valid kernels into a single (ideally) superior kernel. The aggregation can be done using weighted sums of the pre-computed kernels, but determining the summation weights is not a trivial task. Furthermore, MKL does not work well with large datasets because of limited storage space and prediction speed. These challenges are tackled by the introduction of many new algorithms in the following chapters. I also address MKL's storage and speed drawbacks, allowing MKL-based techniques to be applied to big data efficiently. Some algorithms in this work are based on the Choquet fuzzy integral, a powerful nonlinear aggregation operator parameterized by the fuzzy measure (FM). These decision-level fusion algorithms learn a fuzzy measure by minimizing a sum of squared error (SSE) criterion based on a set of training data. The flexibility of the Choquet integral comes with a cost, however---given a set of N decision makers, the size of the FM the algorithm must learn is 2N. This means that the training data must be diverse enough to include 2N independent observations, though this is rarely encountered in practice. I address this

Affective analysis of video content has greatly increased the possibilities of the way we perceive and deal with media. Different kinds of strategies have been tried, but results are still opened to improvements. Most of the problems come from the lack of standardized test set and real affective models. In order to cope with these issues, in this paper we describe the results of our work on the determination of affective models for evaluation of video clips using audiovisual low-level features. The affective models were developed following two classes of psychological theories of affect: categorial and dimensional. The affective models were created from real data, acquired through a series of user experiments. They reflect the affective state of a viewer after watching a certain scene from a movie. We evaluate the detection of Pleasure, Arousal and Dominance coefficients as well as the detection rate of six affective categories. For this end, two Bayesian network topologies are used, a Hidden Markov Model and an Autoregressive Hidden Markov Model. The measurements were done using audio-only models, video-only models and fused models. Fusion is done using two different methods, a Decision Level Fusion and Feature Level Fusion. All tests were conducted using localized affective models, both categorial and dimensional. Results are presented in terms of detection rate and accuracy for affective families, affective dimensions and probabilistic networks. Arousal was the best detected dimension, followed by dominance and pleasure.

Considering the use of a multi-modal framework to enhance emotion recognition, we propose to combine electroencephalography (EEG) and electrooculogram (EOG) through decision level fusion(DLF) and feature level fusion(FLF) for emotion recognition. By using different temporal window sizes to segment the signal, we explore the duration of the emotion of the EOG signal and the EEG signal. Then, some temporal window sizes that are friendly to both EOG signal and EEG signal are selected for segmentation and emotion recognition. According to the different degree of dependence of subjects, the accuracy of the proposed algorithm on subject-dependent and subject-independent is verified on the DEAP dataset. For subject-dependent, using feature level fusion strategy with a window size of 6 seconds, the accuracy is 0.9562 in terms of arousal, and 0.9558 in terms of valence. For subject-independent, using feature level fusion strategy with a window size of 5 seconds, the accuracy is 0.8638 in terms of arousal, and 0.8542 in terms of valence. The experimental results show that the proposed algorithm can better enhance emotion recognition.

In this advance era of digitalization, the trend towards the automation of recognition of individual's identity based upon physiological or behavioural characteristics has become the necessity & need of the time. Various, research have been conducted on multimodal identification biometric system with three traits like face, IRIS & finger vein traits and the model performance was evaluated on score level fusion, feature level fusion and at the accuracy level. However, for better results on identity recognition, it is required to fuse more traits and hence to study the improvements achieved, while recognizing any identity in person. The multimodal biometric fusion of more features supports to create robust system for identity identification. The performed work intends to explore the effect of using deep learning algorithms on identity identification by using a wider range of recognition traits like iris, face, and fingerprint & handwritten signature. The results of the performed work shows that fusion of handwritten signature along with the three traits like face, IRIS & finger vein, leads to a much robust multimodal biometric feature fusion which leads to considerable improvement in the physical identification of any person.

multimodal biometric systems have been widely used to overcome the limitation of unimodal biometric systems and to achieve high recognition accuracy. However, users feel inconvenience because most of the multimodal systems require several steps in order to acquire multimodal biometric data, which also requires the specific behaviors of users. In this research, we propose a new multimodal biometric recognition of touched fingerprint and finger-vein. This paper is novel in the following four ways. First, we can get a fingerprint and a finger-vein image at the same time by the proposed device, which acquires the fingerprint and finger-vein images from the first and second knuckles of finger, respectively. Second, the device's size is so small that we can adopt it on a mobile device, easily. Third, fingerprint recognition is done based on the minutia points of ridge area and finger-vein recognition is performed based on local binary pattern (LBP) with appearance information of finger area. Fourth, based on decision level fusion, we combined two results of fingerprint and finger-vein recognition. Experimental results confirmed the efficiency and usefulness of the proposed method.