Multi-modal optimization techniques for improving qualitative features of biophysical neural models

Biometrics are increasingly used for access control, fraud detection, and authentication systems. Nevertheless, attackers can deceive such systems using forged biometrics. This research proposes a novel method that makes biometric security systems more resilient to such attacks. The proposed method transforms the user’s biometric data into an irreversible code to protect the original data. This code combines data from multiple biometric modalities, making fabricating a false biometric harder. Additionally, the proposed method does not depend on any secret keys, which helps avoid cases of stolen tokens. The proposed method utilizes the generative adversarial network (GAN) to generate synthetic biometric templates from multiple modalities, which is considered a transformation function for biometric data. Three fusion levels are presented; features from multiple biometric modalities are extracted first in each fusion level. Subsequently, the features train a generative adversarial network to produce synthesized biometric templates. These synthesized templates serve as secure substitutes for the original biometrics during authentication, preventing direct exposure of raw biometric data. We evaluated our methods on the CASIA-V3-Internal and MMU1 iris datasets and the AT&T (ORL) and FERET face datasets. The results showed that our proposed methods can achieve higher accuracy, usability, and improved security compared to a single biometric modality. The proposed feature-level, GAN-based, and decision-level fusion schemes achieved 2.03%, 0.82%, and 0.0297% error rates, respectively, for CASIA and ORL datasets and 1.53%, 0.80%, and 0.0313% error rates, respectively, for MMU1 and FERET datasets. Moreover, we have demonstrated that our method resists pre-image and correlation attacks.

Multimodal sentiment analysis (MSA) aims to use a variety of sensors to obtain and process information to predict the intensity and polarity of human emotions. The main challenges faced by current multi-modal sentiment analysis include: how the model extracts emotional information in a single modality and realizes the complementary transmission of multimodal information; how to output relatively stable predictions even when the sentiment embodied in a single modality is inconsistent with the multi-modal label; how can the model ensure high accuracy when a single modal information is incomplete or the feature extraction performance not good. Traditional methods do not take into account the interaction of unimodal contextual information and multi-modal information. They also ignore the independence and correlation of different modalities, which perform poorly when multimodal sentiment representations are asymmetric. To address these issues, this paper first proposes unimodal feature extraction network (UFEN) to extract unimodal features with stronger representation capabilities; then introduces multi-task fusion network (MTFN) to improve the correlation and fusion effect between multiple modalities. Multilayer feature extraction, attention mechanisms and Transformer are used in the model to mine potential relationships between features. Experimental results on MOSI, MOSEI and SIMS datasets show that the proposed method achieves better performance on multimodal sentiment analysis tasks compared with state-of-the-art baselines.

In this paper, we address the problem of conditional modality learning, whereby one is interested in generating one modality given the other. While it is straightforward to learn a joint distribution over multiple modalities using a deep multi-modal architecture, we observe that such models are not very effective at conditional generation. Hence, we address the problem by learning conditional distributions between the modalities. We use variational methods for maximizing the corresponding conditional log-likelihood. The resultant deep model, which we refer to as conditional multimodal autoencoder (CMMA), forces the latent representation obtained from a single modality alone to be `close' to the joint representation obtained from multiple modalities. We use the proposed model to generate faces from attributes. We show that the faces generated from attributes using the proposed model are qualitatively and quantitatively more representative of the attributes from which they were generated, than those obtained by other deep generative models. We also propose a secondary task, whereby the existing faces are modified by modifying the corresponding attributes. We observe that the modifications in face introduced by the proposed model are representative of the corresponding modifications in attributes.

Active learning has been demonstrated to be an effective approach to reducing human labeling effort in multimedia annotation tasks. However, most of the existing active learning methods for video annotation are studied in a relatively simple context where concepts are sequentially annotated with fixed effort and only a single modality is applied. However, we usually have to deal with multiple modalities, and sequentially annotating concepts without preference cannot suitably assign annotation effort. To address these two issues, in this paper we propose a multi-concept multi-modality active learning method for video annotation in which multiple concepts and multiple modalities can be simultaneously taken into consideration. In each round of active learning, this method selects the concept that is expected to get the highest performance gain and a batch of suitable samples to be annotated for this concept. Then, a graph-based semi-supervised learning is conducted on each modality for the selected concept. The proposed method is able to sufficiently explore the human effort by considering both the learnabilities of different concepts and the potentials of different modalities. Experimental results on TRECVID 2005 benchmark have demonstrated its effectiveness and efficiency.

Focus on Authors

Introduction to the special issue of “Advances in Molecular Imaging”

Multimodal data, especially imaging and non-imaging data, is being routinely acquired in the context of disease diagnostics; however computational challenges have limited the ability to quantitatively integrate imaging and non-imaging data channels with different dimensionalities for making diagnostic and prognostic predictions. The objective of this work is to create a common subspace to simultaneously accommodate both the imaging and non-imaging data, called a metaspace. This metaspace can be used to build a meta-classifier that produces better classification results than a classifier that is based on a single modality alone. In this paper, we present a novel Supervised Regularized Canonical Correlation Analysis (SRCCA) algorithm that (1) enables the quantitative integration of data from multiple modalities using a feature selection scheme, (2) is regularized, and (3) is computationally cheap. We leverage this SRCCA framework towards the fusion of proteomic and histologic image signatures for identifying prostate cancer patients at risk for biochemical recurrence following radical prostatectomy. For a cohort of 19 prostate cancer patients, SRCCA was able to yield a lower fused dimensional metaspace comprising both the histological and proteomic attributes. In conjunction with SRCCA, a random forest classifier was able to identify patients at risk for biochemical failure with a maximum accuracy of 93%. The classifier performance in the SRCCA space was statistically significantly higher compared to the fused data representations obtained either with Canonical Correlation Analysis (CCA) or Regularized CCA.

A lot of research has been done on the detection of mental workload (MWL) using various bio-signals. Recently, deep learning has allowed for novel methods and results. A plethora of measurement modalities have proven to be valuable in this task, yet studies currently often only use a single modality to classify MWL. The goal of this research was to classify perceived mental workload (PMWL) using a deep neural network (DNN) that flexibly makes use of multiple modalities, in order to allow for feature sharing between modalities. To achieve this goal, an experiment was conducted in which MWL was simulated with the help of verbal logic puzzles. The puzzles came in five levels of difficulty and were presented in a random order. Participants had 1 h to solve as many puzzles as they could. Between puzzles, they gave a difficulty rating between 1 and 7, seven being the highest difficulty. Galvanic skin response, photoplethysmograms, functional near-infrared spectrograms and eye movements were collected simultaneously using LabStreamingLayer (LSL). Marker information from the puzzles was also streamed on LSL. We designed and evaluated a novel intermediate fusion multimodal DNN for the classification of PMWL using the aforementioned four modalities. Two main criteria that guided the design and implementation of our DNN are modularity and generalisability. We were able to classify PMWL within-level accurate (0.985 levels) on a seven-level workload scale using the aforementioned modalities. The model architecture allows for easy addition and removal of modalities without major structural implications because of the modular nature of the design. Furthermore, we showed that our neural network performed better when using multiple modalities, as opposed to a single modality. The dataset and code used in this paper are openly available.

This chapter examines children's overlapping gesture and speech uses. On the basis of caregiver response to child gestures it investigates the semantic relatedness of information being conveyed. In view of prior claims that gestures are often redundant with words, the study examines an Information theory account of redundancy and an Interactive account and shows that when children communicate through a single modality adults provide fewer responses than when children combine modalities, and these responses differ qualitatively from those found in response to multiple modalities. The paper suggests that this provides a motivation for development from using a single modality to using multiple modalities and that neither communicative channel can be considered redundant. Keywords: Gesture; multimodality; caregiver response; redundancy

<p indent="0mm">With the rapid development of artificial intelligence technology, deep learning techniques have begun to be applied in geological researchs. Compared to traditional lithofacies identification methods, the application of deep learning not only resulted in the implementation of efficient automatic lithofacies identification but also resulted in the accuracy improvement and uncertainty reduction of interpretation. Although excellent achievements in the field of single-well log identification have been obtained by using the powerful learning capability of deep learning, it is found that the feature representation of a single modality can hardly encompass the complete information of relevant geological bodies. In order to overcome the limitations of feature representation in a single modality and to optimally utilize the values contained in multiple modalities, some scholars have proposed the application of multimodal fusion method to enhance the model learning performance. Based on the previous research status, in this paper, aiming to elaborate on the application and development prospects of deep learning-related technologies in the field of lithofacies identification, we have introduced representative traditional identification techniques and those techniques integrated with the machine learning and have further highlighted the necessity and huge potential of machine learning applied in the lithofacies identification. Additionally, we proposed a fusion strategy for the combination of multimodal fusion technology with deep learning in order to provide prior knowledge and methods for related researchers. Finally, we have briefly summarized the current fusion strategies and have elaborated the existing challenges in relevant geological researches.

Considering the difficulty to obtain complete multi-modality MRI scans in some real-world data acquisition situations, synthesizing MRI data is a highly relevant and important topic to complement diagnosis information in clinical practice. In this study, we present a novel MRI synthesizer, called AutoGAN-Synthesizer, which automatically discovers generative networks for cross-modality MRI synthesis. Our AutoGAN-Synthesizer adopts gradient-based search strategies to explore the generator architecture by determining how to fuse multi-resolution features and utilizes GAN-based perceptual searching losses to handle the trade-off between model complexity and performance. Our AutoGAN-Synthesizer can search for a remarkable and light-weight architecture with 6.31 Mb parameters only occupying 12 GPU hours. Moreover, to incorporate richer prior knowledge for MRI synthesis, we derive K-space features containing the low- and high-spatial frequency information and incorporate such features into our model. To our best knowledge, this is the first work to explore AutoML for cross-modality MRI synthesis, and our approach is also capable of tailoring networks given either different multiple modalities or just a single modality as input. Extensive experiments show that our AutoGAN-Synthesizer outperforms the state-of-the-art MRI synthesis methods both quantitatively and qualitatively. The code are available at https://github.com/HUuxiaobin/AutoGAN-Synthesizer.KeywordsCross-modality MRI synthesisGenerative adversarial networksNeural architecture search

In multimodal biometric systems, the fusion of information is an important step. Information can be fused at the different levels, i.e., at the feature, matching or decision level, of the recognition system. In this research feature fusion of three biometric features viz. face, profile face, and ear has fused using discriminant correlation analysis. Discriminant correlation analysis performs an effective functional fusion through maximizing the pair correlations of the two feature sets and simultaneously eliminating correlations between classes and limiting correlations to the classes. In pattern recognition applications, the proposed method can be used to fuse features extracted from multiple modalities or combine different vectors extracted from a single modality. Many sets of experiments carried out on different biometric databases and using various techniques for extraction of the features demonstrate the effectiveness of the proposed method, which results in more advanced approaches.

Vehicle re-identification (Re-ID) task has far-reaching practical application value and research significance in busy traffic monitoring scenes. Most existing works are based on the single RGB modality or RGB-IR (Infrared) cross-modality. Still, it is difficult to be widely applied in multi-scene real life because of the inadequacy of a single modality at the limited daytime. Recently, multi-spectral Re-ID has been proposed to simultaneously use the RGB and IR spectral data to build a data cell that can complement each other to deal with this condition. For multi-spectral vehicle Re-ID, existing methods simply fuse features extracted from a data cell with multiple modalities, which ignores the large gap as well as the relationship between different modalities. To address this problem, in this work, we propose the Generative and Attentive Fusion Network (GAFNet) to fuse the multiple data sources for the vehicle Re-ID task. First, at the input-level, we enrich the data cell by adding the generated transitional modality images, which could involve different data distribution from the original modality images. Note that the newly generated images have high similarity with both the target modality and the source modality, therefore they would help reduce the gap between modalities and indirectly improve the consistency inside a data cell, and further enhance the weight of key information. Second, at the feature-level, we concatenate and fuse the features of the original data and the transitional modalities from different dimensions with the proposed attentive feature fusion module to further align the features and enhance their focused areas in the feature maps. Extensive experimental results on two standard benchmarks demonstrate that our method is effective and outperforms the state-of-the-art methods.

Human communication is inherently tied to emotions, which play a critical role in guiding and enhancing social interactions. For neurodivergent individuals, particularly children, challenges often arise in expression and interpretation of emotions. Emotion detection technologies can therefore serve as powerful tools to aid in communication and to improve social interaction. However, emotional changes among neurodivergent individuals span a wider spectrum and exhibit greater subtle differences. Existing emotion detection models have been predominantly trained with data in single modality. Integrating data from multiple modalities provides a more comprehensive approach to understanding emotions, mirroring the way humans naturally perceive the world through all five senses. This study presents a Multimodal Emotion Detection System that leverages publicly available datasets to enhance recognition accuracy. By fusing diverse data sources, the proposed model captures subtle emotional cues more effectively than traditional methods. Experimental results confirm its robustness and suitability for real-world applications.

Biometrics identification using multiple modalities has attracted the attention of many researchers as it produces more robust and trustworthy results than single modality biometrics. In this paper, we present a novel multimodal recognition system that trains a deep learning network to automatically learn features after extracting multiple biometric modalities from a single data source, i.e., facial video clips. Utilizing different modalities, i.e., left ear, left profile face, frontal face, right profile face, and right ear, present in the facial video clips, we train supervised denoising auto-encoders to automatically extract robust and non-redundant features. The automatically learned features are then used to train modality specific sparse classifiers to perform the multimodal recognition. Moreover, the proposed technique has proven robust when some of the above modalities were missing during the testing. The proposed system has three main components that are responsible for detection, which consists of modality specific detectors to automatically detect images of different modalities present in facial video clips; feature selection, which uses supervised denoising sparse auto-encoders network to capture discriminative representations that are robust to the illumination and pose variations; and classification, which consists of a set of modality specific sparse representation classifiers for unimodal recognition, followed by score level fusion of the recognition results of the available modalities. Experiments conducted on the constrained facial video dataset (WVU) and the unconstrained facial video dataset (HONDA/UCSD), resulted in a 99.17% and 97.14% Rank-1 recognition rates, respectively. The multimodal recognition accuracy demonstrates the superiority and robustness of the proposed approach irrespective of the illumination, nonplanar movement, and pose variations present in the video clips even in the situation of missing modalities.