Method of multimodal biometric data analysis for optimal efficiency evaluation of recognition algorithms and systems

Development of biometric human identification algorithms and systems is reviewed. A crisis that prevents them from being used in large computing systems is demonstrated. Comprehensive reliable tests are considered as a way to overcome the crisis. Ways to take into account the influence various factors may have on the reliability of evaluating the practical efficiency of remote human identification algorithms and systems are addressed. A method to determine and calculate quantitative quality criteria of multimodal biometric data is proposed; the possibility of extrapolating the results of testing biometric algorithms to practical application is discussed. Quality functions and the artificiality of biometric data given as the measure of proximity to biometric data registered from an unaware recognition subject are studied.

The value of artificial intelligence combined with multimodal data analysis in tumor immunotherapy and targeted therapy.

Multimodal single-cell omics technologies enable multiple molecular programs to be simultaneously profiled at a global scale in individual cells, creating opportunities to study biological systems at a resolution that was previously inaccessible. However, the analysis of multimodal single-cell omics data is challenging due to the lack of methods that can integrate across multiple data modalities generated from such technologies. Here, we present Matilda, a multi-task learning method for integrative analysis of multimodal single-cell omics data. By leveraging the interrelationship among tasks, Matilda learns to perform data simulation, dimension reduction, cell type classification, and feature selection in a single unified framework. We compare Matilda with other state-of-the-art methods on datasets generated from some of the most popular multimodal single-cell omics technologies. Our results demonstrate the utility of Matilda for addressing multiple key tasks on integrative multimodal single-cell omics data analysis. Matilda is implemented in Pytorch and is freely available from https://github.com/PYangLab/Matilda.

Template-Protection-Techniken stellen eine wichtige Ergänzung zu biometrischen Systemen, um das gewünschte Sicherheitsniveau zu erreichen und die Anforderungen an den Schutz der Privatsphäre zu erfüllen. Diese Techniken ermöglichen Erneuerung und Widerruf der Referenzendaten in biometrischen Systemen. In dieser Arbeit wird ein systematisches Evaluierungsframework entwickelt, das sich an die wesentlichen Kriterien und Anforderungen an biometrische Template-Protection-Verfahren festhält. Seine Anwendbarkeit wird durch die Analyse der verschiedenen Algorithmen demonstriert. Die in dieser Arbeit präsentierte Evaluierung ist grundlegend für eine vollständige Analyse. Darüber hinaus ermöglicht es einen Nachweis der Sicherheit und Einhaltung der Privatsphäre. Es schafft eine Basis für Zertifizierung und Benchmark der Template-Protection-Verfahren.

Multimedia data analysis is at the heart of multimedia research, where the patterns, information, and knowledge are mined and recognized from the multimedia data for various applications. We have witnessed significant performance improvements in multimedia data analysis driven by the recent advances in artificial intelligence and deep learning. While most of the existing methods focus on analyzing unimodal data, many novel multimodal multimedia data analysis techniques have also been developed and achieved better performance than those using unimodal data. Multimodal multimedia data has proven to be effective to improve the performance and reliability of data analysis systems and applications. It is also important to match and retrieve data with similar contents across modalities. However, due to the highly complex nature of multimodal multimedia data, further research in multimodal multimedia data analysis is needed to address the challenges including missing modalities, lack of benchmark datasets, and so on.

In general, the ‘small sample (n)-large feature ( Open image in new window )’ problem of bioinformatics, image analysis, high throughput molecular screening, astronomy, and other high dimensional applications makes the features highly collinear. In this context, the paper presents a new feature extraction algorithm to address this ‘large Open image in new window small n’ issue associated with multimodal data sets. The proposed algorithm judiciously integrates the concept of both regularization and shrinkage with canonical correlation analysis to extract important features. To deal with the singularity problem, the proposed method increases the diagonal elements of covariance matrices by using regularization parameters, while the off-diagonal elements are decreased by shrinkage coefficients. The concept of hypercuboid equivalence partition matrix of rough hypercuboid approach is used to compute both significance and relevance measures of a feature. The importance of the proposed algorithm over other existing methods is established extensively on real life multimodal omics data set.

Research on multimodal data analysis such as annotated image analysis is becoming more important than ever due to the increase in the amount of data. One of the approaches to this problem is multimodal topic models as an extension of latent Dirichlet allocation (LDA). Symmetric correspondence topic models (SymCorrLDA) are state-of-the-art multimodal topic models that can appropriately model multimodal data considering inter-modal dependencies. Incidentally, hierarchically structured categories can help users find relevant data from a large amount of data collection. Hierarchical topic models such as hierarchical latent Dirichlet allocation (hLDA) can discover a tree-structured hierarchy of latent topics from a given unimodal data collection; however, no hierarchical topic models can appropriately handle multimodal data considering intermodal mutual dependencies. In this paper, we propose h-SymCorrLDA to discover latent topic hierarchies from multimodal data by combining the ideas of the two previously mentioned models: multimodal topic models and hierarchical topic models. We demonstrate the effectiveness of our model compared with several baseline models through experiments with two datasets of annotated images.

Multi-modal big data analysis has emerged as a vital tool within the Industrial Internet of Things (IIoT). Despite its significance, challenges persist, particularly the absence of an efficient analytical architecture. In this study, we introduce a novel multi-modal big data analysis framework tailored for IIoT. Our approach features a cloud-edge collaborative training solution designed to optimize the training of deep learning models, along with an edge-based rapid reasoning solution for deploying models and facilitating swift inference on resource-constrained devices in industrial settings.

Single-cell omics enable the dissection of cellular heterogeneity, yet the high dimensionality, inherent noise, and sparsity present significant challenges. These challenges are amplified for rare cell populations, which are often difficult to annotate reliably but can be central to development and disease. As single-cell assays increasingly capture multiple molecular layers, the integrative analysis of such multimodal data further increases complexity. Here, we propose Hydra, a deep generative framework based on an ensemble of variational autoencoders for effective learning of unimodal and multimodal single-cell omics data. Hydra implements interpretable modules for capturing cell-type-specific molecular signatures. The ensemble of such interpretable modules enables reproducible feature selection and robust cell-type annotation, with particular effectiveness for rare populations. We benchmarked Hydra on a repertoire of 21 datasets, including unimodal and multimodal single-cell omics data. Our results demonstrate that Hydra offers comparable to superior performance to several state-of-the-art methods. Finally, we highlight the utility of Hydra in robustly annotating brain cellular subtypes and preserving disease-relevant signatures using our previously published dataset that profiles Alzheimer's disease.

Multimedia systems, such as social media platforms, play a crucial role in disseminating vital information during calamities. This information is shared in various formats such as images, text, videos, audio, etc. Therefore, it becomes important to have a system that can identify multimodal data to classify relevant information. This paper proposes a new age classification method for multimodal data using advanced and improved transformer models, such as Vision Transformer and Generative Pre-trained Transformer 2, for image and text classification, respectively. These models were combined using an ensemble model (Random Forest Classifier), achieving an accuracy of 84.66% on the multimodal data. Furthermore, the proposed model demonstrates higher prediction accuracy compared to traditional Convolutional Neural Network (CNN) models which have an accuracy of 71.43%, exceeding it by 13.23%. A comparison with convolutional models is conducted to underscore the advantages of transformer models and to substantiate the necessity of the experiment. Our proposed classification model using Vision Transformer and GPT-2, along with an ensemble model, can be replicated by researchers in disaster management, humanitarian aid organizations, and social media platforms looking to filter and prioritize information during emergencies.

In the era of big data, a large number of multimodal data are constantly generated and accumulated. In the practical application analysis of multimodal big data, the modal incompleteness, real-time processing, modal imbalance and high-dimensional attributes pose severe challenges to the multimodal fusion method. In view of the above characteristics of multimodal data and the shortcomings of existing multimodal fusion algorithms, this chapter studies the multimodal data fusion algorithms from four aspects: incomplete modal analysis fusion, incremental modal clustering fusion, heterogeneous modal migration fusion and low dimensional modal sharing fusion. A series of multimodal data fusion models and algorithms are designed.KeywordsMultimodal dataData fusionUnsupervised learning

As an important branch of wireless sensor network (WSN) in biomedical field, body sensor network (BSN) could remotely monitor a variety of human health data in real time by means of energy-saving and high-accuracy sensor technology. We proposed a multi-modal health data fusion and analysis method based on the data collected from BSN. Single-modal Holter monitoring and multi-modal health monitoring were performed on 60 patients with confirmed heart disease and proved that the proposed method could effectively improve the detection rate of asymptomatic myocardial ischemia and provide a new auxiliary judgment method for clinical application. Aiming at the requirement of privacy preserving in the data fusion process of wireless body sensor network, a new SMART-RR algorithm was proposed. Simulation results showed that MART-RR was an energy-saving privacy preserving data fusion algorithm with small data communications, high privacy protection and accuracy.

A kernel machine method for detecting higher order interactions in multimodal datasets: Application to schizophrenia

Characterizing cancer presents a delicate challenge as it involves deciphering complex biological interactions within the tumor's microenvironment. Clinical trials often provide histology images and molecular profiling of tumors, which can help understand these interactions. Despite recent advances in representing multimodal data for weakly supervised tasks in the medical domain, achieving a coherent and interpretable fusion of whole slide images and multi-omics data is still a challenge. Each modality operates at distinct biological levels, introducing substantial correlations between and within data sources. In response to these challenges, we propose a novel deep-learning-based approach designed to represent multi-omics & histopathology data for precision medicine in a readily interpretable manner. While our approach demonstrates superior performance compared to state-of-the-art methods across multiple test cases, it also deals with incomplete and missing data in a robust manner. It extracts various scores characterizing the activity of each modality and their interactions at the pathway and gene levels. The strength of our method lies in its capacity to unravel pathway activation through multimodal relationships and to extend enrichment analysis to spatial data for supervised tasks. We showcase its predictive capacity and interpretation scores by extensively exploring multiple TCGA datasets and validation cohorts. The method opens new perspectives in understanding the complex relationships between multimodal pathological genomic data in different cancer types and is publicly available on Github.

By focusing on immersive interaction among users, the burgeoning Metaverse can be viewed as a natural extension of existing social media. Similar to traditional online social networks, there are numerous security and privacy issues in the Metaverse (e.g., attacks on user authentication and impersonation). In this paper, we develop a holistic research agenda for zero-trust user authentication in social virtual reality (VR), an early prototype of the Metaverse. Our proposed research includes four concrete steps: investigating biometrics-based authentication that is suitable for continuously authenticating VR users, leveraging federated learning (FL) for protecting user privacy in biometric data, improving the accuracy of continuous VR authentication with multimodal data, and boosting the usability of zero-trust security with adaptive VR authentication. Our preliminary study demonstrates that conventional FL algorithms are not well suited for biometrics-based authentication of VR users, leading to an accuracy of less than 10%. We discuss the root cause of this problem, the associated open challenges, and several future directions for realizing our research vision.