Fusing Visual and Textual Information to Determine Content Safety

In this paper we explore automatic genre labelling of motion picture previews using audio-visual features present in movie trailers and the focus is on fusion techniques (early fusion and late fusion) and the resultant improvement on classification accuracy. This paper proposes a novel combination of deep learned features (from a pretrained VGG-16 model) obtained using a state-of-the-art shot detector and hand-crafted audio features. This combination of features and an associated comparison of early and late fusion with these features has not been attempted in the literature before. Furthermore, two popular fusion techniques and three distinct classification algorithms are investigated to determine the optimal fusion technique and classifier combination. The study uses a subset of the LMTD-9 movie trailer dataset with selected genres (action, comedy, drama and horror). The best performing low-level audio features are comprised of timbre features extracted using the MIRtoolbox followed by standalone mel-frequency cepstral coefficients. The best performing high-level audio feature is tonality. Audio features are augmented by visual features extracted using a pre-trained convolutional neural network (VGG-16). Feature fusion (early and late fusion) methods are investigated together with classification methods such as extreme gradient boosting, support vector machine and a neural network. Evaluation metrics such as precision, recall, confusion matrices and F1 score are used to measure classification accuracy. Early fusion methods outperform late fusion methods with a classification performance gain of approximately 10% for a four class classification problem. The best classification performance for early fusion obtained with a support vector machine is (73.12% accuracy), followed by the extreme gradient boosting classifier (69.37% accuracy) and neural network classifier (67.50% accuracy), whereas chance is 25%. It is shown that superior classification performance can be achieved by employing early feature fusion of low-level audio descriptors, high-level audio descriptors and high-level visual feature descriptors together with suitable classifiers.

Clinical and health-related quality-of-life outcomes after early and late spinal fusion in pediatric patients with congenital scoliosis at 10-year follow-up.

The integration of Artificial Intelligence (AI) for fault detection and diagnosis (FDD) in renewable energy systems has become essential for handling complex, multimodal data from wind turbine installations. This study introduces a multimodal AI framework that evaluates feature-level (early) and decision-level (late) fusion strategies. Sensor data are organized into electrical (currents, voltages, power) and mechanical (torque, speed) modalities, enabling both modality-specific learning and cross-modal analysis via early and late fusion. This paper makes several key contributions. First, it introduces the proposed Adaptive Polynomial Kolmogorov–Arnold Network (AdaptPolyKAN) model for multi-class fault classification in wind energy conversion systems (WECS). Second, it develops a robust preprocessing pipeline that integrates wavelet denoising and Z-score normalization to enhance the quality and consistency of multimodal sensor data. Third, it provides a comprehensive comparison between AdaptPolyKAN and six other machine learning models Neural Networks (NN), Support Vector Machine (SVM), Naive Bayes (NB), Quadratic Discriminant Analysis (QDA), Generalized Regression Neural Network (GRNN), and Recurrent Neural Network (RNN) highlighting the benefits brought by the proposed preprocessing and fusion procedures. Fourth, it presents an in-depth analysis of early and late fusion strategies, offering practical insights into their effectiveness for WECS fault diagnosis. Experimental results demonstrate that early and late fusion significantly enhance classification accuracy and robustness compared to single-modality or single-classifier approaches. The proposed AdaptPolyKAN framework achieves near-perfect accuracy (99.99% with early fusion) while remaining computationally efficient, making it suitable for real-time condition monitoring and predictive maintenance.

Multisource and multimodal data fusion plays a pivotal role in large-scale artificial intelligence applications involving big data. However, the choice of fusion strategies for different scenarios is often based on experimental comparisons, which leads to increased computational costs during model training and suboptimal performance during testing. In this paper, we present a theoretical analysis of early fusion, late fusion, and gradual fusion methods. We derive equivalence conditions between early and late fusions within the framework of generalized linear models. Moreover, we analyze the failure conditions of early fusion in the presence of nonlinear feature-label relationships. Furthermore, we propose an approximate equation for evaluating the accuracy of early and late fusion methods as a function of sample size, feature quantity, and modality number. We also propose a critical sample size threshold at which the performance dominance of early fusion and late fusion models undergoes a reversal. Finally, we introduce a fusion method selection paradigm for selecting the most appropriate fusion method prior to task execution and demonstrate its effectiveness through extensive numerical experiments. Our theoretical framework is expected to solve the problems of computational and resource costs in model construction, improving the scalability and efficiency of data fusion methods.

This paper focuses on the comparison between two fusion methods, namely early fusion and late fusion. The former fusion is carried out at kernel level, also known as multiple kernel learning, and in the latter, the modalities are fused through logistic regression at classifier score level. Two kinds of multilayer fusion structures, differing in the quantities of feature/kernel groups in a lower fusion layer, are constructed for early and late fusion systems, respectively. The goal of these fusion methods is to put each of various features into effect and mine redundant information of the combination of them, and then to develop a generic and robust semantic indexing system to bridge semantic gap between human concepts and these low-level visual features. Performance evaluated on both TRECVID2009 and TRECVID2010 datasets demonstrates that the systems with our proposed multilayer fusion methods at kernel level perform more stably to reach the goal than the classification-score-level fusion; the most effective and robust one with highest MAP score is constructed by early fusion with two-layer equally weighted composite kernel learning.

Deep learning (DL) tends to be the integral part of Autonomous Vehicles (AVs). Therefore the development of scene analysis modules that are robust to various vulnerabilities such as adversarial inputs or cyber-attacks is becoming an imperative need for the future AV perception systems. In this paper, we deal with this issue by exploring the recent progress in Artificial Intelligence (AI) and Machine Learning (ML) to provide holistic situational awareness and eliminate the effect of the previous attacks on the scene analysis modules. We propose novel multi-modal approaches against which achieve robustness to adversarial attacks, by appropriately modifying the analysis Neural networks and by utilizing late fusion methods. More specifically, we propose a holistic approach by adding new layers to a 2D segmentation DL model enhancing its robustness to adversarial noise. Then, a novel late fusion technique has been applied, by extracting direct features from the 3D space and project them into the 2D segmented space for identifying inconsistencies. Extensive evaluation studies using the KITTI odometry dataset provide promising performance results under various types of noise.

COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios

Hip fractures in the elderly are a major health care problem in the society. In the clinic, it is important for medical specialists to identify high-risk patients to assist them in the decision-making process in choosing the right treatment. In this paper, we propose a multimodal machine learning model for prediction of 30-days mortality of elderly hip fracture patients. The paper addresses both the clinical problem of identifying high-risk patients and the specific risks involved, as well as the technical problem of how to fuse information from different modalities as input to one predictive model. Our model uses features from three modalities: hip X-ray images, chest X-ray images and structured data and fuses them based on early fusion and late fusion techniques for the prediction task. Our model uses a convolutional neural network to extract features from the chest and hip images before combining them with the structured data. The fused features are further passed through a fusion classifier for the final prediction. The proposed model outperforms a replicated version of Almelo Hip Fracture Score (AHFS-a) with an AUC score of 0.786 vs 0.717. Finally, by the analysis of feature importance, we found that chest X-ray images contain important signs to predict the 30-days mortality of elderly hip fracture patients. We also found that structured and chest X-ray modalities were more important in predicting high-risk patients as compared to hip X-ray modality, though the final results on the test set show that structured, hip and chest X-ray modalities together are needed to get the best performance for predicting 30-days mortality. Further, we achieved the best performance using early fusion with random forest technique, though late fusion achieved a competitive performance.

The precise diagnosis of Alzheimer’s disease is critical for patient treatment, especially at the early stage, because awareness of the severity and progression risks lets patients take preventative actions before irreversible brain damage occurs. It is possible to gain a holistic view of Alzheimer’s disease staging by combining multiple data modalities, known as image fusion. In this paper, the study proposes the early detection of Alzheimer’s disease using different modalities of Alzheimer’s disease brain images. First, the preprocessing was performed on the data. Then, the data augmentation techniques are used to handle overfitting. Also, the skull is removed to lead to good classification. In the second phase, two fusion stages are used: pixel level (early fusion) and feature level (late fusion). We fused magnetic resonance imaging and positron emission tomography images using early fusion (Laplacian Re-Decomposition) and late fusion (Canonical Correlation Analysis). The proposed system used magnetic resonance imaging and positron emission tomography to take advantage of each. Magnetic resonance imaging system’s primary benefits are providing images with excellent spatial resolution and structural information for specific organs. Positron emission tomography images can provide functional information and the metabolisms of particular tissues. This characteristic helps clinicians detect diseases and tumor progression at an early stage. Third, the feature extraction of fused images is extracted using a convolutional neural network. In the case of late fusion, the features are extracted first and then fused. Finally, the proposed system performs XGB to classify Alzheimer’s disease. The system’s performance was evaluated using accuracy, specificity, and sensitivity. All medical data were retrieved in the 2D format of 256 × 256 pixels. The classifiers were optimized to achieve the final results: for the decision tree, the maximum depth of a tree was 2. The best number of trees for the random forest was 60; for the support vector machine, the maximum depth was 4, and the kernel gamma was 0.01. The system achieved an accuracy of 98.06%, specificity of 94.32%, and sensitivity of 97.02% in the case of early fusion. Also, if the system achieved late fusion, accuracy was 99.22%, specificity was 96.54%, and sensitivity was 99.54%.

Fusing visual and clinical information for lung tissue classification in high-resolution computed tomography

This paper proposes a comprehensive deep-learning framework, SentiStack, for Bitcoin price forecasting and trading strategy evaluation by integrating multimodal data sources, including market indicators, macroeconomic variables, and sentiment information extracted from financial news and social media. The model architecture is based on a Stacking-LSTM ensemble, which captures complex temporal dependencies and non-linear patterns in high-dimensional financial time series. To enhance predictive power, sentiment embeddings derived from full-text analysis using the DeepSeek language model are fused with traditional numerical features through early and late data fusion techniques. Empirical results demonstrate that the proposed model significantly outperforms baseline strategies, including Buy & Hold and Random Trading, in cumulative return and risk-adjusted performances. Feature ablation experiments further reveal the critical role of sentiment and macroeconomic inputs in improving forecasting accuracy. The sentiment-enhanced model also exhibits strong performance in identifying high-return market movements, suggesting its practical value for data-driven investment decision-making. Overall, this study highlights the importance of incorporating soft information, such as investor sentiment, alongside traditional quantitative features in financial forecasting models.

Obtaining a compact and discriminative representation of facial and body expressions is a difficult problem in emotion recognition. Part of the difficulty is capturing microexpressions, i.e., short, involuntary expressions that last for only a fraction of a second: at a micro-temporal scale, there are so many other subtle face and body movements that do not convey semantically meaningful information. We present a novel approach to this problem by exploiting the sparsity of the frequent micro-temporal motion patterns. Local space-time features are extracted over the face and body region for a very short time period, e.g., few milliseconds. A codebook of microexpressions is learned from the data and used to encode the features in a sparse manner. This allows us to obtain a representation that captures the most salient motion patterns of the face and body at a micro-temporal scale. Experiments performed on the AVEC 2012 dataset show our approach achieving the best published performance on the arousal dimension based solely on visual features. We also report experimental results on audio-visual emotion recognition, comparing early and late data fusion techniques.

The definition of parameters is a crucial step in the development of a system for identifying emotions in speech. Although there is no agreement on which are the best features for this task, it is generally accepted that prosody carries most of the emotional information. Most works in the field use some kind of prosodic features, often in combination with spectral and voice quality parametrizations. Nevertheless, no systematic study has been done comparing these features. This paper presents the analysis of the characteristics of features derived from prosody, spectral envelope, and voice quality as well as their capability to discriminate emotions. In addition, early fusion and late fusion techniques for combining different information sources are evaluated. The results of this analysis are validated with experimental automatic emotion identification tests. Results suggest that spectral envelope features outperform the prosodic ones. Even when different parametrizations are combined, the late fusion of long-term spectral statistics with short-term spectral envelope parameters provides an accuracy comparable to that obtained when all parametrizations are combined.

As natural language interfaces enable users to express increasingly complex natural language queries, there is a parallel explosion of user review content that can allow users to better find items such as restaurants, books, or movies that match these expressive queries. While Neural Information Retrieval (IR) methods have provided state-of-the-art results for matching queries to documents, they have not been extended to the task of Reviewed-Item Retrieval (RIR), where query-review scores must be aggregated (or fused) into item-level scores for ranking. In the absence of labeled RIR datasets, we extend Neural IR methodology to RIR by leveraging self-supervised methods for contrastive learning of BERT embeddings for both queries and reviews. Specifically, contrastive learning requires a choice of positive and negative samples, where the unique two-level structure of our item-review data combined with meta-data affords us a rich structure for the selection of these samples. For contrastive learning in a Late Fusion scenario (where we aggregate query-review scores into item-level scores), we investigate the use of positive review samples from the same item and/or with the same rating, selection of hard positive samples by choosing the least similar reviews from the same anchor item, and selection of hard negative samples by choosing the most similar reviews from different items. We also explore anchor sub-sampling and augmenting with meta-data. For a more end-to-end Early Fusion approach, we introduce contrastive item embedding learning to fuse reviews into single item embeddings. Experimental results show that Late Fusion contrastive learning for Neural RIR outperforms all other contrastive IR configurations, Neural IR, and sparse retrieval baselines, thus demonstrating the power of exploiting the two-level structure in Neural RIR approaches as well as the importance of preserving the nuance of individual review content via Late Fusion methods.

In social networks, users can easily share information and express their opinions. Given the huge amount of data posted by many users, it is difficult to search for relevant information. In addition to individual posts, it would be useful if we can recommend groups of people with similar interests. Past studies on user preference learning focused on single-modal features such as review contents or demographic information of users. However, such information is usually not easy to obtain in most social media without explicit user feedback. In this paper, we propose a multimodal feature fusion approach to implicit user preference prediction which combines text and image features from user posts for recommending similar users in social media. First, we use the convolutional neural network (CNN) and TextCNN models to extract image and text features, respectively. Then, these features are combined using early and late fusion methods as a representation of user preferences. Lastly, a list of users with the most similar preferences are recommended. The experimental results on real-world Instagram data show that the best performance can be achieved when we apply late fusion of individual classification results for images and texts, with the best average top-k accuracy of 0.491. This validates the effectiveness of utilizing deep learning methods for fusing multimodal features to represent social user preferences. Further investigation is needed to verify the performance in different types of social media.