A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding

Multisource and multimodal data fusion plays a pivotal role in large-scale artificial intelligence applications involving big data. However, the choice of fusion strategies for different scenarios is often based on experimental comparisons, which leads to increased computational costs during model training and suboptimal performance during testing. In this paper, we present a theoretical analysis of early fusion, late fusion, and gradual fusion methods. We derive equivalence conditions between early and late fusions within the framework of generalized linear models. Moreover, we analyze the failure conditions of early fusion in the presence of nonlinear feature-label relationships. Furthermore, we propose an approximate equation for evaluating the accuracy of early and late fusion methods as a function of sample size, feature quantity, and modality number. We also propose a critical sample size threshold at which the performance dominance of early fusion and late fusion models undergoes a reversal. Finally, we introduce a fusion method selection paradigm for selecting the most appropriate fusion method prior to task execution and demonstrate its effectiveness through extensive numerical experiments. Our theoretical framework is expected to solve the problems of computational and resource costs in model construction, improving the scalability and efficiency of data fusion methods.

The integration of Artificial Intelligence (AI) for fault detection and diagnosis (FDD) in renewable energy systems has become essential for handling complex, multimodal data from wind turbine installations. This study introduces a multimodal AI framework that evaluates feature-level (early) and decision-level (late) fusion strategies. Sensor data are organized into electrical (currents, voltages, power) and mechanical (torque, speed) modalities, enabling both modality-specific learning and cross-modal analysis via early and late fusion. This paper makes several key contributions. First, it introduces the proposed Adaptive Polynomial Kolmogorov–Arnold Network (AdaptPolyKAN) model for multi-class fault classification in wind energy conversion systems (WECS). Second, it develops a robust preprocessing pipeline that integrates wavelet denoising and Z-score normalization to enhance the quality and consistency of multimodal sensor data. Third, it provides a comprehensive comparison between AdaptPolyKAN and six other machine learning models Neural Networks (NN), Support Vector Machine (SVM), Naive Bayes (NB), Quadratic Discriminant Analysis (QDA), Generalized Regression Neural Network (GRNN), and Recurrent Neural Network (RNN) highlighting the benefits brought by the proposed preprocessing and fusion procedures. Fourth, it presents an in-depth analysis of early and late fusion strategies, offering practical insights into their effectiveness for WECS fault diagnosis. Experimental results demonstrate that early and late fusion significantly enhance classification accuracy and robustness compared to single-modality or single-classifier approaches. The proposed AdaptPolyKAN framework achieves near-perfect accuracy (99.99% with early fusion) while remaining computationally efficient, making it suitable for real-time condition monitoring and predictive maintenance.

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

In advertising, identifying the content safety of web pages is a significant concern since advertisers do not want brands to be associated with threatening content. At the same time, publishers would like to maximize the number of web pages on which they can place ads. Thus, a fine balance must be achieved while classifying content safety in order to satisfy both advertisers and publishers. In this paper, we propose a multimodal machine learning framework that fuses visual and textual information from web pages to improve current predictions of content safety. The primary focus is on late fusion, which involves combining final model outputs of separate modalities, such as images and text, to arrive at a single decision. This paper presents a fully automated machine learning framework that performs binary and multilabel classification using late fusion techniques. We also introduce additional work in early fusion, which involves extracting and fusing intermediate features from the two separate models. Our algorithms are applied to data extracted from relevant web pages in the advertising industry. Both of our late and early fusion methods obtain significant improvements over algorithms currently in use.

As natural language interfaces enable users to express increasingly complex natural language queries, there is a parallel explosion of user review content that can allow users to better find items such as restaurants, books, or movies that match these expressive queries. While Neural Information Retrieval (IR) methods have provided state-of-the-art results for matching queries to documents, they have not been extended to the task of Reviewed-Item Retrieval (RIR), where query-review scores must be aggregated (or fused) into item-level scores for ranking. In the absence of labeled RIR datasets, we extend Neural IR methodology to RIR by leveraging self-supervised methods for contrastive learning of BERT embeddings for both queries and reviews. Specifically, contrastive learning requires a choice of positive and negative samples, where the unique two-level structure of our item-review data combined with meta-data affords us a rich structure for the selection of these samples. For contrastive learning in a Late Fusion scenario (where we aggregate query-review scores into item-level scores), we investigate the use of positive review samples from the same item and/or with the same rating, selection of hard positive samples by choosing the least similar reviews from the same anchor item, and selection of hard negative samples by choosing the most similar reviews from different items. We also explore anchor sub-sampling and augmenting with meta-data. For a more end-to-end Early Fusion approach, we introduce contrastive item embedding learning to fuse reviews into single item embeddings. Experimental results show that Late Fusion contrastive learning for Neural RIR outperforms all other contrastive IR configurations, Neural IR, and sparse retrieval baselines, thus demonstrating the power of exploiting the two-level structure in Neural RIR approaches as well as the importance of preserving the nuance of individual review content via Late Fusion methods.

This paper presents a detailed framework for Gaussian mixture model (GMM)-based articulatory inversion equipped with special postprocessing smoothers, and with the capability to perform audio-visual information fusion. The effects of different acoustic features on the GMM inversion performance are investigated and it is shown that the integration of various types of acoustic (and visual) features improves the performance of the articulatory inversion process. Dynamic Kalman smoothers are proposed to adapt the cutoff frequency of the smoother to data and noise characteristics; Kalman smoothers also enable the incorporation of auxiliary information such as phonetic transcriptions to improve articulatory estimation. Two types of dynamic Kalman smoothers are introduced: global Kalman (GK) and phoneme-based Kalman (PBK). The same dynamic model is used for all phonemes in the GK smoother; it is shown that GK improves the performance of articulatory inversion better than the conventional low-pass (LP) smoother. However, the PBK smoother, which uses one dynamic model for each phoneme, gives significantly better results than the GK smoother. Different methodologies to fuse the audio and visual information are examined. A novel modified late fusion algorithm, designed to consider the observability degree of the articulators, is shown to give better results than either the early or the late fusion methods. Extensive experimental studies are conducted with the MOCHA database to illustrate the performance gains obtained by the proposed algorithms. The average RMS error and correlation coefficient between the true (measured) and the estimated articulatory trajectories are 1.227 mm and 0.868 using audiovisual information fusion and GK smoothing, and 1.199 mm and 0.876 using audiovisual information fusion together with PBK smoothing based on a phonetic transcription of the utterance.

In social networks, users can easily share information and express their opinions. Given the huge amount of data posted by many users, it is difficult to search for relevant information. In addition to individual posts, it would be useful if we can recommend groups of people with similar interests. Past studies on user preference learning focused on single-modal features such as review contents or demographic information of users. However, such information is usually not easy to obtain in most social media without explicit user feedback. In this paper, we propose a multimodal feature fusion approach to implicit user preference prediction which combines text and image features from user posts for recommending similar users in social media. First, we use the convolutional neural network (CNN) and TextCNN models to extract image and text features, respectively. Then, these features are combined using early and late fusion methods as a representation of user preferences. Lastly, a list of users with the most similar preferences are recommended. The experimental results on real-world Instagram data show that the best performance can be achieved when we apply late fusion of individual classification results for images and texts, with the best average top-k accuracy of 0.491. This validates the effectiveness of utilizing deep learning methods for fusing multimodal features to represent social user preferences. Further investigation is needed to verify the performance in different types of social media.

Human experts evaluate singing quality based on many perceptual parameters such as intonation, rhythm, and vibrato, with reference to music theory. We proposed previously the Perceptual Evaluation of Singing Quality (PESnQ) framework that incorporated acoustic features related to these perceptual parameters in combination with the cognitive modeling concept of the telecommunication standard Perceptual Evaluation of Speech Quality to evaluate singing quality. In this study, we present further the study of the PESnQ framework to approximate the human judgments. First, we find that a linear combination of the individual perceptual parameter human scores can predict their overall singing quality judgment. This provides us with a human parametric judgment equation. Next, the prediction of the individual perceptual parameter scores from the PESnQ acoustic features show a high correlation with the respective human scores, which means more meaningful feedback to learners. Finally, we compare the performance of early fusion and late fusion of the acoustic features in predicting the overall human scores. We find that the late fusion method is superior to that of the early fusion method. This work underlines the importance of modeling human perception in automatic singing quality assessment.

Non-invasive brain-computer interfaces (BCIs) have been widely used for neural decoding, linking neural signals to control devices. Hybrid BCI systems using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have received significant attention for overcoming the limitations of EEG- and fNIRS-standalone BCI systems. However, most hybrid EEG-fNIRS BCI studies have focused on late fusion because of discrepancies in their temporal resolutions and recording locations. Despite the enhanced performance of hybrid BCIs, late fusion methods have difficulty in extracting correlated features in both EEG and fNIRS signals. Therefore, in this study, we proposed a deep learning-based early fusion structure, which combines two signals before the fully-connected layer, called the fNIRS-guided attention network (FGANet). First, 1D EEG and fNIRS signals were converted into 3D EEG and fNIRS tensors to spatially align EEG and fNIRS signals at the same time point. The proposed fNIRS-guided attention layer extracted a joint representation of EEG and fNIRS tensors based on neurovascular coupling, in which the spatially important regions were identified from fNIRS signals, and detailed neural patterns were extracted from EEG signals. Finally, the final prediction was obtained by weighting the sum of the prediction scores of the EEG and fNIRS-guided attention features to alleviate performance degradation owing to delayed fNIRS response. In the experimental results, the FGANet significantly outperformed the EEG-standalone network. Furthermore, the FGANet has 4.0% and 2.7% higher accuracy than the state-of-the-art algorithms in mental arithmetic and motor imagery tasks, respectively.

In this manuscript, multimodal emotion recognition using a decision-level fusion and feature-level fusion approach is proposed. In the first approach, decision-level fusion approach is proposed, which is considered a late fusion, where fine-tuned models are developed for each modality. For that, the input is taken from IEMOCAP Database, initially, it is tokenized to a length of 128 tokens and given to a transformer-based BERT model. In the second approach, a feature-level fusion approach is proposed which is considered an early fusion where features from each modality are combined and then fed to the attention-based LSTM. For that, the input is taken from IEMOCAP Database, which contains three modalities: text, speech and Video. Here, text features are extracted with the CNN model, speech features are extracted using the OPENSMILE toolkit and Video features are extracted using a 3D-CNN architecture. Then the proposed approaches are simulated with python. The performance metrics, such as accuracy, sensitivity, specificity, precision, and recall, are evaluated. Then the performance of the proposed first approach is compared with the second approach. The simulation results of the second approach provide a higher accuracy of 0.98%; a higher sensitivity of 0.96%, and a higher sensitivity of 0.75% than the first approach.

Handcrafted features and late fusion with deep learning for bird sound classification

This paper focuses on the comparison between two fusion methods, namely early fusion and late fusion. The former fusion is carried out at kernel level, also known as multiple kernel learning, and in the latter, the modalities are fused through logistic regression at classifier score level. Two kinds of multilayer fusion structures, differing in the quantities of feature/kernel groups in a lower fusion layer, are constructed for early and late fusion systems, respectively. The goal of these fusion methods is to put each of various features into effect and mine redundant information of the combination of them, and then to develop a generic and robust semantic indexing system to bridge semantic gap between human concepts and these low-level visual features. Performance evaluated on both TRECVID2009 and TRECVID2010 datasets demonstrates that the systems with our proposed multilayer fusion methods at kernel level perform more stably to reach the goal than the classification-score-level fusion; the most effective and robust one with highest MAP score is constructed by early fusion with two-layer equally weighted composite kernel learning.

Fusing visual and clinical information for lung tissue classification in high-resolution computed tomography

Biometric technique becomes essential to identify individuals in different applications. Face sketch is one of the biometric methods, which are used to identify criminals. In this paper, a face sketch synthesis and recognition model is proposed. In this model, the photo images are transformed to pseudo-sketch images using linear regression technique. Moreover, Gabor filters are used to extract the features from three scales of the images. For each scale, a face sketch image is matched with face pseudo-sketches instead of the original photos to identify an unknown individual. Minimum distance classifier is used to match the sketches with pseudo-sketches in each scale. Further, a classification level fusion is used to combine the outputs of the classifiers at three scales namely, decision, rank, and score level fusion. CHUK database images is used in our experiments. The experimental results show that the proposed model is superior to other existed models in terms of identification accuracy. Moreover, matching sketch images with pseudo-sketches achieved accuracy better than matching sketch images with the original photo images. The proposed model achieved a recognition rate ranged from 82.95 to 94.32 % using single scale, while the accuracy increased to 94.32, 96.6 and 97.7 % when the decision, rank, and score level fusion, respectively, are used.

Recognition algorithms that use data obtained by imaging faces in the thermal spectrum are promising in achieving invariance to extreme illumination changes that are often present in practice. In this paper we analyze the performance of a recently proposed face recognition algorithm that combines visual and thermal modalities by decision level fusion. We examine (i) the effects of the proposed data preprocessing in each domain, (ii) the contribution to improved recognition of different types of features, (iii) the importance of prescription glasses detection, in the context of both 1-to-N and 1-to-1 matching (recognition vs. verification performance). Finally, we discuss the significance of our results and, in particular, identify a number of limitations of the current state-of-the-art and propose promising directions for future research.