Multimodal CNN Fusion Architecture with Multi-Features for Heart Sound Classification

The origin of tobacco is the most important factor in determining the style characteristics and intrinsic quality of tobacco. There are many applications for the identification of tobacco origin by near-infrared spectroscopy. In order to improve the accuracy of the tobacco origin classification, a near-infrared spectrum (NIRS) identification method based on multimodal convolutional neural networks (CNN) was proposed, taking advantage of the strong feature extraction ability of the CNN. Firstly, the one-dimensional convolutional neural network (1-D CNN) is used to extract and combine the pattern features of one-dimensional NIRS data, and then the extracted features are used for classification. Secondly, the one-dimensional NIRS data are converted into two-dimensional spectral images, and the structure features are extracted from two-dimensional spectral images by the two-dimensional convolutional neural network (2-D CNN) method. The classification is performed by the combination of global and local training features. Finally, the influences of different network structure parameters on model identification performance are studied, and the optimal CNN models are selected and compared. The multimodal NIR-CNN identification models of tobacco origin were established by using NIRS of 5,200 tobacco samples from 10 major tobacco producing provinces in China and 3 foreign countries. The classification accuracy of 1-D CNN and 2-D CNN models was 93.15% and 93.05%, respectively, which was better than the traditional PLS-DA method. The experimental results show that the application of 1-D CNN and 2-D CNN can accurately and reliably distinguish the NIRS data, and it can be developed into a new rapid identification method of tobacco origin, which has an important promotion value.

CardioPHON: Quality assessment and self-supervised pretraining for screening of cardiac function based on phonocardiogram recordings

The heart sound signal is one of the signals that reflect the health of the heart. Research on the heart sound signal contributes to the early diagnosis and prevention of cardiovascular diseases. As a commonly used deep learning network, convolutional neural network (CNN) has been widely used in images. In this paper, the method of analyzing heart sound through using CNN has been studied. Firstly, the original data set was preprocessed, and then the heart sounds were segmented on U-net, based on the deep CNN. Finally, the classification of heart sounds was completed through CNN. The data from 2016 PhysioNet/CinC Challenge was utilized for algorithm validation, and the following results were obtained. When the heart sound segmented, the overall accuracy rate was 0.991, the accuracy of the first heart sound was 0.991, the accuracy of the systolic period was 0.996, the accuracy of the second heart sound was 0.996, and the accuracy of the diastolic period was 0.997, and the average accuracy rate was 0.995; While in classification, the accuracy was 0.964, the sensitivity was 0.781, and the specificity was 0.873. These results show that deep learning based on CNN shows good performance in the segmentation and classification of the heart sound signal.

This paper presents an automatic normal and abnormal heart sound classification model developed based on deep learning algorithm. MITHSDB heart sounds datasets obtained from the 2016 PhysioNet/Computing in Cardiology Challenge database were used in this research with the assumption that the electrocardiograms (ECG) were recorded simultaneously with the heart sounds (phonocardiogram, PCG). The PCG time series are segmented per heart beat, and each sub-segment is converted to form a square intensity matrix, and classified using convolutional neural network (CNN) models. This approach removes the need to provide classification features for the supervised machine learning algorithm. Instead, the features are determined automatically through training, from the time series provided. The result proves that the prediction model is able to provide reasonable and comparable classification accuracy despite simple implementation. This approach can be used for real-time classification of heart sounds in Internet of Medical Things (IoMT), e.g. remote monitoring applications of PCG signal.

Most of the causes of death are related to cardiovascular disease. Heart sound classification plays a key role in the early detection of cardiovascular disease. The distinction between normal and abnormal heart sounds is not obvious, making heart sound classification a challenging task. This paper proposes a method based on statistical analysis and feature engineering to analyze the features extracted from heart sound signals. The discriminative features are selected and input into the classifier to classify the heart sound. While ensuring accuracy, the method reduces the scale of the input data and computational time for classification. The selected distinguishable features achieved an accuracy of 96.5 % for the classification of abnormal and normal heart sounds. The experimental results show that the selected features can achieve a high accuracy rate in the classification of heart sound, which is suitable for real-time applications.

Heart sounds classification plays an important role in cardiovascular disease detection. Currently, deep learning methods for heart sound classification with heavy parameters consumption cannot be deployed in environments with limited memory and computational budgets. Besides, de-noising of heart sound signals (HSSs) can affect accuracy of heart sound classification, because erroneous removal of meaningful components may lead to heart sound distortion. In this paper, an automated heart sound classification method using adaptive wavelet threshold and 1D LDCNN (One-dimensional Lightweight Deep Convolutional Neural Network) is proposed. In this method, we exploit WT (Wavelet Transform) with an adaptive threshold to de-noise heart sound signals (HSSs). Furthermore, we utilize 1D LDCNN to realize automatic feature extraction and classification for de-noised heart sounds. Experiments on PhysioNet/CinC 2016 show that our proposed method achieves the superior classification results and excels in consumption of parameter comparing to state-of-the-art methods.

Analysis of PCG signals using quality assessment and homomorphic filters for localization and classification of heart sounds

Combining machine learning in neural networks with multimodal fusion strategies offers an interesting potential for classification tasks but the optimum fusion strategies for many applications have yet to be determined. Here we address this issue in the context of human activity recognition, making use of a state-of-the-art convolutional network architecture (Inception I3D) and a huge dataset (NTU RGB+D). As modalities we consider RGB video, optical flow, and skeleton data. We determine whether the fusion of different modalities can provide an advantage as compared to uni-modal approaches, and whether a more complex early fusion strategy can outperform the simpler late-fusion strategy by making use of statistical correlations between the different modalities. Our results show a clear performance improvement by multi-modal fusion and a substantial advantage of an early fusion strategy,

Heart sounds convey important information regarding potential heart diseases. Currently, heart sound classification attracts many researchers from the fields of telemedicine, digital signal processing, and machine learning—among others—mainly to identify cardiac pathology as quickly as possible. This article proposes chaogram as a new transform to convert heart sound signals to colour images. In the proposed approach, the output image is, therefore, the projection of the reconstructed phase space representation of the phonocardiogram (PCG) signal on three coordinate planes. This has two major benefits: (1) it makes possible to apply deep convolutional neural networks to heart sounds and (2) it is also possible to employ a transfer learning scheme by converting a heart sound signal to an image. The performance of the proposed approach was verified on the PhysioNet dataset. Due to the imbalanced data on this dataset, it is common to assess the results quality using the average of sensitivity and specificity, which is known as score, instead of accuracy. In this study, the best results were achieved using the model, which achieved a score of 88.06%.

Heart sounds are critical for early detection of cardiovascular diseases, yet existing studies mostly focus on traditional signal segmentation, feature extraction, and shallow classifiers, which often fail to sufficiently capture the dynamic and nonlinear characteristics of heart sounds, limit recognition of complex heart sound patterns, and are sensitive to data imbalance, resulting in poor classification performance. To address these limitations, this study proposes a novel heart sound classification method that integrates improved Mel-frequency cepstral coefficients (MFCC) for feature extraction with a convolutional neural network (CNN) and a deep Transformer model. In the preprocessing stage, a Butterworth filter is applied for denoising, and continuous heart sound signals are directly processed without segmenting the cardiac cycles, allowing the improved MFCC features to better capture dynamic characteristics. These features are then fed into a CNN for feature learning, followed by global average pooling (GAP) to reduce model complexity and mitigate overfitting. Lastly, a deep Transformer module is employed to further extract and fuse features, completing the heart sound classification. To handle data imbalance, the model uses focal loss as the objective function. Experiments on two public datasets demonstrate that the proposed method performs effectively in both binary and multi-class classification tasks. This approach enables efficient classification of continuous heart sound signals, provides a reference methodology for future heart sound research for disease classification, and supports the development of wearable devices and home monitoring systems.

The classification of clinically significant prostate cancer (csPCa) lesions remains one of the most important challenges in prostate cancer diagnosis. For this, multimodal convolutional neural networks (CNNs) have achieved outstanding results. Nevertheless, the data used in these studies may only partially represent the total burden of csPCa cases. Hence, it is necessary to design reliable models that perform well in limited data scenarios and involving information from different centers (multicentric). A deep Riemannian geometric learning architecture was introduced to capture the intermediate relationships between bi-parametric MRI (bp-MRI) deep representations coded from a 3D multimodal convolutional backbone and considering their geometry. For this, several multimodal bp-MRI fusion strategies were explored to assess their ability to classify csPCa lesions in scenarios where the percentage of available training data was progressively reduced and multicentric data were involved. The proposed method outperformed baseline CNN techniques with an AUC-ROC of 0.96. More remarkably, the method remained stable even only using 10% of the available training data. Additionally, considering multicentric information, this approach also demonstrates generalization ability by losing only 5.4% of the AUC testing data from different acquisition centers, compared to the 10.4% loss of the baseline method. A new deep learning-based method that improves generalization under scenarios with limited data translates to better support for clinicians in accurately classifying csPCa lesions on unseen data.

Mental stress can lead to traffic accidents by reducing a driver’s concentration or increasing fatigue while driving. In recent years, demand for methods to detect drivers’ stress in advance to prevent dangerous situations increased. Thus, we propose a novel method for detecting driving stress using nonlinear representations of short-term (30 s or less) physiological signals for multimodal convolutional neural networks (CNNs). Specifically, from hand/foot galvanic skin response (HGSR, FGSR) and heart rate (HR) short-term input signals, first, we generate corresponding two-dimensional nonlinear representations called continuous recurrence plots (Cont-RPs). Second, from the Cont-RPs, we use multimodal CNNs to automatically extract FGSR, HGSR, and HR signal representative features that can effectively differentiate between stressed and relaxed states. Lastly, we concatenate the three extracted features into one integrated representation vector, which we feed to a fully connected layer to perform classification. For the evaluation, we use a public stress dataset collected from actual driving environments. Experimental results show that the proposed method demonstrates superior performance for 30-s signals, with an overall accuracy of 95.67%, an approximately 2.5–3% improvement compared with that of previous works. Additionally, for 10-s signals, the proposed method achieves 92.33% classification accuracy, which is similar to or better than the performance of other methods using long-term signals (over 100 s).

Time series classification using local distance-based features in multi-modal fusion networks

The fusion of convolutional neural networks (CNNs) and vision transformers (ViTs) is a powerful strategy for remote sensing scene classification (RSSC), leveraging their complementary strengths in local texture and global context modeling. To fully exploit these complementary strengths for addressing the inter-class similarity and high intra-class variability caused by the complex spatial arrangements of RS scenes, this work introduces the hierarchically aligned bidirectional CNN-Transformer network (HAB-Net), which fosters deeper, two-way integration through two novel components. First, Hierarchical Attention Gating (HAG) distills multi-level attention from the ViT into spatial gates, compelling the CNN to focus on semantically critical regions. In turn, CNN-driven Feedback Attention (CFA) completes the loop by feeding these enhanced, detail-rich CNN features back into the ViT, refining its tokens via cross-attention. Ultimately, a dual-stream head merges these mutually reinforced features for scene classification. Extensive experiments on the UCM, AID, and NWPU-RESISC45 benchmarks demonstrate that HAB-Net achieves state-of-the-art performance. Ablation studies confirm the indispensable contributions of both HAG and CFA, validating the effectiveness of the proposed deep bidirectional integration strategy.

An early detection of heart disease can save lives, caution individuals and also help to determine the type of treatment to be given to the patients. The first test of diagnosing a heart disease is through auscultation - listening to the heart sounds. The interpretation of heart sounds is subjective and requires a professional skill to identify the abnormalities in these sounds. A medical practitioner uses a stethoscope to perform an initial screening by listening for irregular sounds from the patient's chest. Later, echocardiography and electrocardiography tests are taken for further diagnosis. However, these tests are expensive and require specialized technicians to operate. A simple and economical way is vital for monitoring in homecare or rural hospitals and urban clinics. This dissertation is focused on developing a patient-centered device for initial screening of the heart sounds that is both low cost and can be used by the users on themselves, and later share the readings with the healthcare providers. An innovative mobile health service platform is created for analyzing and classifying heart sounds. Certain properties of heart sounds have to be evaluated to identify the irregularities such as the number of heart beats and gallops, intensity, frequency, and duration. Since heart sounds are generated in low frequencies, human ears tend to miss certain sounds as the high frequency sounds mask the lower ones. Therefore, this dissertation provides a solution to process the heart sounds using several signal processing techniques, identifies the features in the heart sounds and finally classifies them. This dissertation enables remote patient monitoring through the integration of advanced wireless communications and a customized low-cost stethoscope. It also permits remote management of patients' cardiac status while maximizing patient mobility. The smartphone application facilities recording, processing, visualizing, listening, and classifying heart sounds. The application also generates an electronic medical record, which is encrypted using the efficient elliptic curve cryptography and sent to the cloud, facilitating access to physicians for further analysis. Thus, this dissertation results in a patient-centered device that is essential for initial screening of the heart sounds, and could be shared for further diagnosis with the medical care practitioners.