Multi-modal advanced deep learning architectures for breast cancer survival prediction

Ovarian cancer (OC) is a common reason for mortality among women. Deep learning has recently proven better performance in predicting OC stages and subtypes. However, most of the state-of-the-art deep learning models employ single modality data, which may afford low-level performance due to insufficient representation of important OC characteristics. Furthermore, these deep learning models still lack to the optimization of the model construction, which requires high computational cost to train and deploy them. In this work, a hybrid evolutionary deep learning model, using multi-modal data, is proposed. The established multi-modal fusion framework amalgamates gene modality alongside with histopathological image modality. Based on the different states and forms of each modality, we set up deep feature extraction network, respectively. This includes a predictive antlion-optimized long-short-term-memory model to process gene longitudinal data. Another predictive antlion-optimized convolutional neural network model is included to process histopathology images. The topology of each customized feature network is automatically set by the antlion optimization algorithm to make it realize better performance. After that the output from the two improved networks is fused based upon weighted linear aggregation. The deep fused features are finally used to predict OC stage. A number of assessment indicators was used to compare the proposed model to other nine multi-modal fusion models constructed using distinct evolutionary algorithms. This was conducted using a benchmark for OC and two benchmarks for breast and lung cancers. The results reveal that the proposed model is more precise and accurate in diagnosing OC and the other cancers.

Presented here are the results of an investigation conducted to determine the effectiveness of deep learning (DL)‐based systems utilizing the power of transfer learning for detecting breast cancer in histopathological images. It is shown that DL models that are not specifically developed for breast cancer detection can be trained using transfer learning to effectively detect breast cancer in histopathological images. The outcome of the analysis enables the selection of the best DL architecture for detecting cancer with high accuracy. This should facilitate pathologists to achieve early diagnoses of breast cancer and administer appropriate treatment to the patient. The experimental work here used the BreaKHis database consisting of 7909 histopathological pictures from 82 clinical breast cancer patients. The strategy presented for DL training uses various image processing techniques for extracting various feature patterns. This is followed by applying transfer learning techniques in the deep convolutional networks like ResNet, ResNeXt, SENet, Dual Path Net, DenseNet, NASNet, and Wide ResNet. Comparison with recent literature shows that ResNext‐50, ResNext‐101, DPN131, DenseNet‐169 and NASNet‐A provide an accuracy of 99.8%, 99.5%, 99.675%, 99.725%, and 99.4%, respectively, and outperform previous studies.

Automatic depression classification from multimodal input data is a challenging task. Modern methods use paralinguistic information such as audio and video signals. Using linguistic information such as speech signals and text data for depression classification is a complicated task in deep learning models. Best audio and video features are built to produce a dependable depression classification system. Textual signals related to depression classification are analyzed using text-based content data. Moreover, to increase the achievements of the depression classification system, audio, visual, and text descriptors are used. So, a deep learning-based depression classification model is developed to detect the person with depression from multimodal data. The EEG signal, Speech signal, video, and text are gathered from standard databases. Four stages of feature extraction take place. In the first stage, the features from the decomposed EEG signals are attained by the empirical mode decomposition (EMD) method, and features are extracted by means of linear and nonlinear feature extraction. In the second stage, the spectral features of the speech signals from the Mel-frequency cepstral coefficients (MFCC) are extracted. In the third stage, the facial texture features from the input video are extracted. In the fourth stage of feature extraction, the input text data are pre-processed, and from the pre-processed data, the textual features are extracted by using the Transformer Net. All four sets of features are optimally selected and combined with the optimal weights to get the weighted fused features using the enhanced mountaineering team-based optimization algorithm (EMTOA). The optimal weighted fused features are finally given to the hybrid attention-based dilated network (HADN). The HDAN is developed by combining temporal convolutional network (TCN) with bidirectional long short-term memory (Bi-LSTM). The parameters in the HDAN are optimized with the assistance of the developed EMTOA algorithm. At last, the classified output of depression is obtained from the HDAN. The efficiency of the developed deep learning HDAN is validated by comparing it with various traditional classification models.

Breast cancer remains one of the most prevalent and deadly cancers affecting women worldwide. Early detection is crucial, particularly for younger women, as traditional screening methods like mammography often struggle with accuracy in cases of dense breast tissue. Infrared thermography offers a non-invasive imaging alternative that enhances early detection by capturing subtle thermal variations indicative of breast abnormalities. This study investigates and compares the performance of various deep learning and machine learning models in analyzing thermographic data to classify breast tissue as healthy, benign, or malignant. To maximize detection accuracy, data preprocessing, feature extraction, and dimensionality reduction were implemented to isolate distinguishing characteristics across tissue types. Leveraging advanced feature extraction and visualization techniques inspired by geospatial data methodologies, we evaluated several deep learning architectures and classical classifiers using the DRM-IR and Breast Thermography Mendeley thermal datasets. Among the tested models, the ResNet152 architecture combined with a Support Vector Machine (SVM) classifier delivered the highest performance, achieving 97.62% accuracy, 95.79% precision, 98.53% recall, 94.52% specificity, an F1 score of 97.16%, an area under the curve (AUC) of 99%, a latency of 0.06 s, and CPU utilization of 88.66%. These findings underscore the potential of integrating infrared thermography with advanced deep learning and machine learning approaches to significantly improve the accuracy and efficiency of breast cancer detection, supporting its role as a valuable tool for early diagnosis.

Breast cancer is the most prevalent cancer globally and presents a significant threat to human health. The low survival rate for advanced stages highlights the importance of early and accurate diagnosis. Pathology images, providing insights into cancer's cellular aggressiveness, are crucial for breast cancer diagnosis. Advances in deep learning have transformed automated medical image diagnosis. However, traditional deep learning methods, reliant on large labeled datasets, face challenges with the typically small size of medical image datasets. To overcome this, our study combines deep transfer learning (DTL) with convolutional neural networks(CNN). We utilize and enhance a ResNet50 model pretrained on a large dataset. Additionally, we apply four different deep transfer learning methods to feature extraction and classification of the BreakHis dataset, aiming to determine if breast cancer is benign or malignant. Ultimately, our system achieved the best average accuracy rates under four different magnifications: 98.57%, 96.83%, 97.76%, and 97.16%. These results demonstrate the accuracy and potential of deep transfer learning in classifying breast cancer histopathological images and its promising application in tumor prediction.

In this paper, we leverage multimodal data to classify minerals using a multi-stream neural network. In a previous study on the Tinto dataset, which consisted of a 3D hyperspectral point cloud from the open-pit mine Corta Atalaya in Spain, we successfully identified mineral classes by employing various deep learning models. However, this prior work solely relied on hyperspectral data as input for the deep learning models. In this study, we aim to enhance accuracy by incorporating multimodal data, which includes hyperspectral images, RGB images, and a 3D point cloud. To achieve this, we have adopted a graph-based neural network, known for its efficiency in aggregating local information, based on our past observations where it consistently performed well across different hyperspectral sensors. Subsequently, we constructed a multi-stream neural network tailored to handle multimodality. Additionally, we employed a channel attention module on the hyperspectral stream to fully exploit the spectral information within the hyperspectral data. Through the integration of multimodal data and a multi-stream neural network, we achieved a notable improvement in mineral classification accuracy: 19.2%, 4.4%, and 5.6% on the LWIR, SWIR, and VNIR datasets, respectively.

Recent advances in data collecting methods have enabled life scientists to collect multimodal data from a variety of biological application fields. Multimodal deep learning is becoming more widespread as the capacity of multimodal data streams and deep learning algorithms grows. This leads to the creation of models capable of consistently processing and interpreting multimodal data. The fusion of multimodal data, such as genetic data and histopathological images, is important to better understand cancer heterogeneity and complexity for specific therapies, as well as improving predictions for cancer research. In this brief overview, we present several pioneering deep learning models where we focus on different architectures that has seen use in the medical field, shedding some light on the importance of deep learning in cancer diagnosis considering its immense results in recent years. We particularly set our eyes on multimodal deep learning approaches fusing histopathological images and genomic data to achieve better results than a single modality.

Multimedia systems, such as social media platforms, play a crucial role in disseminating vital information during calamities. This information is shared in various formats such as images, text, videos, audio, etc. Therefore, it becomes important to have a system that can identify multimodal data to classify relevant information. This paper proposes a new age classification method for multimodal data using advanced and improved transformer models, such as Vision Transformer and Generative Pre-trained Transformer 2, for image and text classification, respectively. These models were combined using an ensemble model (Random Forest Classifier), achieving an accuracy of 84.66% on the multimodal data. Furthermore, the proposed model demonstrates higher prediction accuracy compared to traditional Convolutional Neural Network (CNN) models which have an accuracy of 71.43%, exceeding it by 13.23%. A comparison with convolutional models is conducted to underscore the advantages of transformer models and to substantiate the necessity of the experiment. Our proposed classification model using Vision Transformer and GPT-2, along with an ensemble model, can be replicated by researchers in disaster management, humanitarian aid organizations, and social media platforms looking to filter and prioritize information during emergencies.

Multimedia data analysis is at the heart of multimedia research, where the patterns, information, and knowledge are mined and recognized from the multimedia data for various applications. We have witnessed significant performance improvements in multimedia data analysis driven by the recent advances in artificial intelligence and deep learning. While most of the existing methods focus on analyzing unimodal data, many novel multimodal multimedia data analysis techniques have also been developed and achieved better performance than those using unimodal data. Multimodal multimedia data has proven to be effective to improve the performance and reliability of data analysis systems and applications. It is also important to match and retrieve data with similar contents across modalities. However, due to the highly complex nature of multimodal multimedia data, further research in multimodal multimedia data analysis is needed to address the challenges including missing modalities, lack of benchmark datasets, and so on.

In smart homes, data generated from real-time sensors for human activity recognition is complex, noisy and imbalanced. It is a significant challenge to create machine learning models that can classify activities which are not as commonly occurring as other activities. Machine learning models designed to classify imbalanced data are biased towards learning the more commonly occurring classes. Such learning bias occurs naturally, since the models better learn classes which contain more records. This paper examines whether fusing real-world imbalanced multi-modal sensor data improves classification results as opposed to using unimodal data; and compares deep learning approaches to dealing with imbalanced multi-modal sensor data when using various resampling methods and deep learning models. Experiments were carried out using a large multi-modal sensor dataset generated from the Sensor Platform for HEalthcare in a Residential Environment (SPHERE). The data comprises 16104 samples, where each sample comprises 5608 features and belongs to one of 20 activities (classes). Experimental results using SPHERE demonstrate the challenges of dealing with imbalanced multi-modal data and highlight the importance of having a suitable number of samples within each class for sufficiently training and testing deep learning models. Furthermore, the results revealed that when fusing the data and using the Synthetic Minority Oversampling Technique (SMOTE) to correct class imbalance, CNN-LSTM achieved the highest classification accuracy of 93.67% followed by CNN, 93.55%, and LSTM, i.e. 92.98%.

Recent advancements in deep learning have shown that multimodal inference can be particularly useful in tasks like autonomous driving, human health, and production line monitoring. However, deploying state-of-the-art multimodal models in distributed IoT systems poses unique challenges since the sensor data from low-cost edge devices can get corrupted, lost, or delayed before reaching the cloud. These problems are magnified in the presence of asymmetric data generation rates from different sensor modalities, wireless network dynamics, or unpredictable sensor behavior, leading to either increased latency or degradation in inference accuracy, which could affect the normal operation of the system with severe consequences like human injury or car accident. In this paper, we propose PATCH, a framework of speculative inference to adapt to these complex scenarios. PATCH serves as a plug-in module in the existing multimodal models, and it enables speculative inference of these off-the-shelf deep learning models. PATCH consists of 1) a Masked-AutoEncoder-based cross-modality imputation module to impute missing data using partially-available sensor data, 2) a lightweight feature pair ranking module that effectively limits the searching space for the optimal imputation configuration with low computation overhead, and 3) a data alignment module that aligns multimodal heterogeneous data streams without using accurate timestamp or external synchronization mechanisms. We implement PATCH in nine popular multimodal models using five public datasets and one self-collected dataset. The experimental results show that PATCH achieves up to 13% mean accuracy improvement over the state-of-art method while only using 10% of training data and reducing the training overhead by 73% compared to the original cost of retraining the model.

Deep learning models in genomics; are we there yet?

Breast cancer encompasses various subtypes with distinct prognoses, necessitating accurate stratification methods. Current techniques rely on quantifying gene expression in limited subsets. Given the complexity of breast tissues, effective detection and classification of breast cancer is crucial in medical imaging. This study introduces a novel method, MPa-DCAE, which uses a multi-patch-based deep convolutional auto-encoder (DCAE) framework combined with VGG19 to detect and classify breast cancer in histopathology images. The proposed MPa-DCAE model leverages the hierarchical feature extraction capabilities of VGG19 within a DCAE framework, designed to capture intricate patterns in histopathology images. By using a multi-patch approach, regions of interest are extracted from pathology images to facilitate localized feature learning, enhancing the model's discriminatory power. The auto-encoder component enables unsupervised feature learning, increasing resilience and adaptability to variations in image features. Experiments were conducted at various magnifications on the CBIS-DDSM and MIAS datasets to validate model performance. Experimental results demonstrated that the MPa-DCAE model outperformed existing methods. For the CBIS-DDSM dataset, the model achieved a precision of 97.96%, a recall of 94.85%, and an accuracy of 98.36%. For the MIAS dataset, it achieved a precision of 97.99%, a recall of 97.2%, and an accuracy of 98.95%. These results highlight the model's robustness and potential for clinical application in computer-assisted diagnosis. The MPa-DCAE model, integrating VGG19 and DCAE, proves to be an effective, automated approach for diagnosing breast cancer. Its high accuracy and generalizability make it a promising tool for clinical practice, potentially improving patient care in histopathology-based breast cancer diagnosis.

This study aims to develop a deep learning (DL) model based on whole-slide images (WSIs) to predict the pathological stage of clear cell renal cell carcinoma (ccRCC). The histopathological images of 513 ccRCC patients were downloaded from The Cancer Genome Atlas (TCGA) database and randomly divided into training set and validation set according to the ratio of 8∶2. The CLAM algorithm was used to establish the DL model, and the stability of the model was evaluated in the external validation set. DL features were extracted from the model to construct a prognostic risk model, which was validated in an external dataset. The results showed that the DL model showed excellent prediction ability with an area under the curve (AUC) of 0.875 and an average accuracy score of 0.809, indicating that the model could reliably distinguish ccRCC patients at different stages from histopathological images. In addition, the prognostic risk model constructed by DL characteristics showed that the overall survival rate of patients in the high-risk group was significantly lower than that in the low-risk group (P = 0.003), and AUC values for predicting 1-, 3- and 5-year overall survival rates were 0.68, 0.69 and 0.69, respectively, indicating that the prediction model had high sensitivity and specificity. The results of the validation set are consistent with the above results. Therefore, DL model can accurately predict the pathological stage and prognosis of ccRCC patients, and provide certain reference value for clinical diagnosis.

Convolutional Neural Network (CNN) models are a type of deep learning architecture introduced to achieve the correct classification of breast cancer. This paper has a two-fold purpose. The first aim is to investigate the various deep learning models in classifying breast cancer histopathology images. This study identified the most accurate models in terms of the binary, four, and eight classifications of breast cancer histopathology image databases. The different accuracy scores obtained for the deep learning models on the same database showed that other factors such as pre-processing, data augmentation, and transfer learning methods can impact the ability of the models to achieve higher accuracy. The second purpose of our manuscript is to investigate the latest models that have no or limited examination done in previous studies. The models like ResNeXt, Dual Path Net, SENet, and NASNet had been identified with the most cutting-edge results for the ImageNet database. These models were examined for the binary, and eight classifications on BreakHis, a breast cancer histopathology image database. Furthermore, the BACH database was used to investigate these models for four classifications. Then, these models were compared with the previous studies to find and propose the most state-of-the-art models for each classification. Since the Inception-ResNet-V2 architecture achieved the best results for binary and eight classifications, we have examined this model in our study as well to provide a better comparison result. In short, this paper provides an extensive evaluation and discussion about the experimental settings for each study that had been conducted on the breast cancer histopathology images.