Text Feature Extraction and Representation of Chinese Mental Verbs Based on Deep Learning

Comparison of deep learning models and a typical process-based model in glacio-hydrology simulation

In recent years, deep learning models have demonstrated remarkable success in various image processing tasks, ranging from object recognition to medical image analysis. However, their performance often degrades in the presence of unseen data or adversarial attacks, highlighting the need for enhancing robustness and generalization. This paper explores innovative approaches to address these challenges, aiming to improve the reliability and applicability of deep learning models in real-world scenarios. The first section of the paper delves into the importance of robustness and generalization in deep learning models for image processing tasks. It discusses the implications of model vulnerabilities, such as overfitting to training data and susceptibility to adversarial perturbations, on the reliability of model predictions. [1] Through a comprehensive review of existing literature, various factors influencing robustness and generalization are identified, including dataset diversity, model architecture, regularization techniques, and adversarial training methods. The paper proposes novel methodologies to enhance the robustness and generalization capabilities of deep learning models. One key approach involves the integration of diverse training data sources, including synthetic data augmentation and domain adaptation techniques, to expose the model to a wider range of scenarios and improve its ability to generalize to unseen data. Additionally, advanced regularization techniques, such as dropout and batch normalization, are explored to mitigate overfitting and improve model generalization. The paper investigates the effectiveness of adversarial training strategies in enhancing model robustness against adversarial attacks. By incorporating adversarially generated examples during training, deep learning models can learn to better resist perturbations and maintain performance under adversarial conditions. Moreover, the paper explores the potential of incorporating uncertainty estimation methods, such as Bayesian neural networks and Monte Carlo dropout, to quantify model uncertainty and improve robustness in uncertain environments. This paper presents a comprehensive investigation into enhancing robustness and generalization in deep learning models for image processing tasks. By addressing key challenges such as overfitting, dataset bias, and adversarial vulnerabilities, the proposed methodologies offer promising avenues for improving the reliability and applicability of deep learning models in real-world scenarios. DOI: https://doi.org/10.52783/pst.193

Uncertainty quantification-based robust deep learning for building energy systems considering distribution shift problem

Detecting programmed death ligand 1 (PD-L1) expression based on immunohistochemical (IHC) staining is an important guide for the treatment of lung cancer with immune checkpoint inhibitors. However, this method has problems such as high staining costs, tumor heterogeneity, and subjective differences among pathologists. Therefore, the application of deep learning models to segment and quantitatively predict PD-L1 expression in digital sections of Hematoxylin and eosin (H&E) stained lung squamous cell carcinoma is of great significance. We constructed a dataset comprising H&E-stained digital sections of lung squamous cell carcinoma and used a Transformer Unet (TransUnet) deep learning network with an encoder-decoder design to segment PD-L1 negative and positive regions and quantitatively predict the tumor cell positive score (TPS). The results showed that the dice similarity coefficient (DSC) and intersection overunion (IoU) of deep learning for PD-L1 expression segmentation of H&E-stained digital slides of lung squamous cell carcinoma were 80 and 72%, respectively, which were better than the other seven cutting-edge segmentation models. The root mean square error (RMSE) of quantitative prediction TPS was 26.8, and the intra-group correlation coefficients with the gold standard was 0.92 (95% CI: 0.90-0.93), which was better than the consistency between the results of five pathologists and the gold standard. The deep learning model is capable of segmenting and quantitatively predicting PD-L1 expression in H&E-stained digital sections of lung squamous cell carcinoma, which has significant implications for the application and guidance of immune checkpoint inhibitor treatments. And the link to the code is https://github.com/Baron-Huang/PD-L1-prediction-via-HE-image.

Brain tumour detection is a critical medical procedure where early and accurate diagnosis significantly improves patient outcomes. This study explores the application of deep learning models, specifically VGG19 and InceptionV3, for detecting brain tumours in MRI images. We fine-tuned both models using transfer learning and evaluated them on a dataset of MRI scans. InceptionV3 achieved 100% validation accuracy, while VGG19 achieved 95%, demonstrating their high efficacy in medical image classification. In addition to deep learning, the study integrates insights from traditional image processing techniques, such as edge detection and probabilistic neural networks (PNNs), highlighting how combining deep learning with traditional methods can enhance image preprocessing and quality. The results underscore that while deep learning models are highly effective in brain tumour classification, traditional techniques still play a vital role in optimizing the overall detection process.

Bone deterioration from osteoporosis creates fractures that primarily affect females who have reached menopause and older adults. Early detection of osteoporosis requires affordable methods because current diagnostic systems are both expensive and challenging to use. The current application of deep learning models for bone radiography analysis faces three major limitations: shallow architecture, insufficient dataset adaptation, and inadequate feature extraction methods. Talk-based machine learning techniques rely heavily on human-generated feature construction; however, they fail to recognize the delicate image patterns that are present in medical contexts. VGG-16 provides strong power, but businesses must invest in advanced and expensive hardware platforms while maintaining challenges with data quantity. The detection of osteoporosis in knee X-rays through deep learning and the ResNet-50 model using transfer learning remains our primary proposal for solving existing issues. An overview of 372 X-ray images with verified medical information was constructed for T-score ratings determined using a quantitative ultrasound system. Following fine-tuning of our model based on ResNet-50, we achieved 90% accuracy, which exceeded all alternative models tested, including VGG-16 with 88% accuracy using fine-tuning, 80% without fine-tuning, a 3-layer CNN at 66%, ResNet-18 at 79%, and non-fine-tuned ResNet-50 at 83%. The clinical value of the ResNet-50 model stems from its high accuracy, strong sensitivity, and specificity rates. The detection reliability of osteoporosis by the deep transfer learning CNN ResNet-50 has been proven through these findings, which provide healthcare practitioners with an effective diagnostic tool for early detection and prevention of fractures. Our research indicates that deep learning using transfer learning strategies produces a crucial enhancement in medical imaging systems, leading to better patient results.

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

PurposeTo conduct a head-to-head comparison between deep learning (DL) and radiomics models across institutions for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC) and to investigate the model robustness and generalizability through rigorous internal and external validation.MethodsThis retrospective study included 2304 preoperative images of 576 HCC lesions from two centers, with MVI status determined by postoperative histopathology. We developed DL and radiomics models for predicting the presence of MVI using B-mode ultrasound, contrast-enhanced ultrasound (CEUS) at the arterial, portal, and delayed phases, and a combined modality (B + CEUS). For radiomics, we constructed models with enlarged vs. original regions of interest (ROIs). A cross-validation approach was performed by training models on one center’s dataset and validating the other, and vice versa. This allowed assessment of the validity of different ultrasound modalities and the cross-center robustness of the models. The optimal model combined with alpha-fetoprotein (AFP) was also validated. The head-to-head comparison was based on the area under the receiver operating characteristic curve (AUC).ResultsThirteen DL models and 25 radiomics models using different ultrasound modalities were constructed and compared. B + CEUS was the optimal modality for both DL and radiomics models. The DL model achieved AUCs of 0.802–0.818 internally and 0.667–0.688 externally across the two centers, whereas radiomics achieved AUCs of 0.749–0.869 internally and 0.646–0.697 externally. The radiomics models showed overall improvement with enlarged ROIs (P < 0.05 for both CEUS and B + CEUS modalities). The DL models showed good cross-institutional robustness (P > 0.05 for all modalities, 1.6–2.1% differences in AUC for the optimal modality), whereas the radiomics models had relatively limited robustness across the two centers (12% drop-off in AUC for the optimal modality). Adding AFP improved the DL models (P < 0.05 externally) and well maintained the robustness, but did not benefit the radiomics model (P > 0.05).ConclusionCross-institutional validation indicated that DL demonstrated better robustness than radiomics for preoperative MVI prediction in patients with HCC, representing a promising solution to non-standardized ultrasound examination procedures.

In environmental monitoring, deep learning models are used where we can either use past observations or extrapolated values with high uncertainty as input. The lag scheme is commonly applied during the modeling and construction process, in the application of deep learning models in multivariate time series prediction. For an adaptive feature engineering, an automated lag scheme is essential for improving the training efficiency. In multivariate time series (MTS) models, the predictive accuracy of artificial neural network ANN-type models can be improved by including more features. It is assumed that when processing a certain number of multivariate features, the timeliness and lag time of the inter-influencing between any pair of elements are different. This research aims to adopt an adaptive approach to solve it, namely, multi-level lag scheme. The research methods include literature review, searching for relevant technology frontiers, feasibility studies, selection and design solutions, modeling, data collection and pre-processing, experiments, evaluation, comprehensive analysis and conclusions. In proof of concept, we demonstrated a practical case of seasonal ANN type MTS model and public service on air quality. In terms of models, ANN type models were attempted with ARIMA as the comparing baseline. We used public data set of more than two base stations with pollution varying from low to high and including southern to northern district of a small city. Conclusions can be drawn from the analysis of multiple experimental results, proving that the proposed solution can effectively improve the training efficiency of the model. This is of great significance, so that most such models can be implemented to adaptively use lagged past measured data as input, instead of synchronously inputting future prediction values, which can greatly improve the practical application of the model in predictive ability.

Disease prediction, a central problem in clinical care and management, has gained much significance over the last decade. Nursing notes documented by caregivers contain valuable information concerning a patient's state, which can aid in the development of intelligent clinical prediction systems. Moreover, due to the limited adaptation of structured electronic health records in developing countries, the need for disease prediction from such clinical text has garnered substantial interest from the research community. The availability of large, publicly available databases such as MIMIC-III, and advancements in machine and deep learning models with high predictive capabilities have further facilitated research in this direction. In this work, we model the latent knowledge embedded in the unstructured clinical nursing notes, to address the clinical task of disease prediction as a multi-label classification of ICD-9 code groups. We present EnTAGS, which facilitates aggregation of the data in the clinical nursing notes of a patient, by modeling them independent of one another. To handle the sparsity and high dimensionality of clinical nursing notes effectively, our proposed EnTAGS is built on the topics extracted using Non-negative matrix factorization. Furthermore, we explore the applicability of deep learning models for the clinical task of disease prediction, and assess the reliability of the proposed models using standard evaluation metrics. Our experimental evaluation revealed that the proposed approach consistently exceeded the state-of-the-art prediction model by 1.87% in accuracy, 12.68% in AUPRC, and 11.64% in MCC score.

Plant disease detection plays a critical role in ensuring agricultural productivity and food security. Accurate and efficient identification of plant diseases can aid in timely interventions, reducing potential crop losses. This paper presents a comprehensive study involving the application of deep learning models, namely CNN, ResNet, VGG, and Inception, to grayscale plant disease image datasets for apple, grape, and corn. The datasets consist of grayscale images divided into multiple disease classes. Initial experiments using standard deep learning models demonstrated reasonable accuracy. To further enhance performance, manual feature extraction techniques, including Histogram of Oriented Gradients (HOG), Oriented FAST and Rotated BRIEF (ORB), and Scale-Invariant Feature Transform (SIFT), were applied alongside deep features extracted using pretrained VGG and ResNet models. The extracted deep learning and handcrafted features were concatenated to form a comprehensive feature set. A Random Forest Classifier (RFC) was then applied to this combined feature set, resulting in improved accuracy compared to individual models.

Double negative T (DNT) cells are a unique subset of CD3 + TCRαβ + T lymphocytes that lack CD4, CD8, or NK1.1 expression and constitute 3–5% of the total T cell population in C57BL/6 mice. They have increasingly gained recognition for their novel roles in the immune system, especially under autoimmune conditions. Conventional machine learning approaches such as principal component analysis have been employed in single-cell RNA sequencing (scRNA-seq) analysis to characterize DNT cells. However, advanced deep learning models such as Single Cell Variational Inference (scVI) have the capability to capture nonlinear gene expression patterns in the sequencing data. In this study, employing the deep learning methodology, we have revealed novel markers for splenic DNT cells in C57BL/6 mice which were validated with flow cytometry analysis. We classified DNT cells into two subgroups, naïve DNT (nDNT) cells differentiated by the expression of Ly6C and activated DNT (aDNT) cells differentiated by the expression of MHC-II. A prior study had predicted elevated expression of CD137/4-1BB encoded by Tnfrsf9 in nDNT cells; however, our analysis predicted and validated that CD137 was a marker for aDNT cells instead of nDNT cells. Innovatively, our data also identified CD30 encoded by Tnfrsf8 and CD153/CD30L encoded by Tnfsf8 as additional markers for aDNT cells. In addition, we classified three subgroups in nDNT cells and two subgroups in aDNT cells. Our scVI analysis suggested, and flow cytometry analysis confirmed, that Ly49G2 encoded by Klra7 was a marker for the nDNT0 subgroup. Importantly, we validated that MHC-II was indeed expressed by a subset of human DNT cells suggesting the presence of a human aDNT population. Furthermore, we found increased expression of CD30, CD153, and CD137 on aDNT cells in MRL/lpr mice compared to those in C57BL/6 mice suggesting potential pathogenic roles of these molecules in autoimmunity. Together, our comprehensive analysis has uncovered and validated novel markers for different subpopulations of DNT cells that can be used in the phenotypic and/or functional characterization of these relatively rare cells in health and disease.

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

We developed a multimodal ultrasound (US) deep learning (DL) fusion model to automatically classify early fibrosis in patients with chronic kidney disease (CKD). This prospective study included patients with CKD who underwent continuous gray-scale US, superb microvascular imaging, and strain elastography from May to November 2022. According to the pathological tubular atrophy and interstitial fibrosis score, patients were divided into minimal and mild groups (affected area ≤10% and 11 − 25% of the total cortical volume, respectively). The dataset was divided into training (70%) and test (30%) sets. A DL model combining the features of the three US modes was developed to predict early fibrosis in patients with CKD. We compared these findings with the area under the receiver operating characteristic curve (AUC) of the clinical model by analyzing the receiver operating characteristic curve in the test set. The AUC of single-mode DL based on gray-scale US, superb microvascular imaging, and strain elastography was 0.682, 0.745, and 0.648, respectively, while that of the multimodal US DL model was 0.86. The accuracy, specificity, and sensitivity of the multimodal US DL model were 0.779, 0.767, and 0.796, respectively, and the negative and positive predictive values were 0.842 and 0.706, respectively. The AUC of the multimodal US DL model was significantly better than that of the single-mode DL and clinical models. The DL algorithm developed using multimodal US images can effectively predict early fibrosis in patients with CKD with significantly greater accuracy than single-mode DL or clinical models.

We aimed at comparing intratumoral and peritumoral deep learning, radiomics, and fusion models in predicting KRAS mutations in rectal cancer using endorectal ultrasound imaging. This study included 304 patients with rectal cancer from Fujian Medical University Union Hospital. The patients were randomly divided into a training group (213 patients) and a test group (91 patients) at a 7:3 ratio. Radiomics and deep learning models were established using primary tumor and peritumoral images. In the optimally performing regions-of-interest, two fusion strategies, a feature-based and a decision-based model, were employed to build the fusion models. The Shapley additive explanation (SHAP) method was used to evaluate the significance of features in the optimal radiomics, deep learning, and fusion models. The performance of each model was assessed using the area under the receiver operating characteristic curve (AUC) and decision curve analysis (DCA). In the test cohort, both the radiomics and deep learning models exhibited optimal performance with a 10-pixel patch extension, yielding AUC values of 0.824 and 0.856, respectively. The feature-based DLRexpand10_FB model attained the highest AUC (0.896) across all study sets. In addition, the DLRexpand10_FB model demonstrated excellent sensitivity, specificity, and DCA. SHAP analysis underscored the deep learning feature (DL_1) as the most significant factor in the hybrid model. The feature-based fusion model DLRexpand10_FB can be employed to predict KRAS gene mutations based on pretreatment endorectal ultrasound images of rectal cancer. The integration of peritumoral regions enhanced the predictive performance of both the radiomics and deep learning models.