Crowd Density Estimation using Imperfect Labels

A cross-modal crowd counting method combining CNN and cross-modal transformer

Counting the number of person has received much attention in recent years. Most of the existing crowd counting methods adopted density map regression pipeline, which formulates the crowd counting problem to two fragmented part: density map regression and integration of the overall counting. To solve this problem, this paper presents a multi-task deep learning scheme to enhance the counting performance. More specifically, we firstly build a multi-scale deep convolutional neural network, based on combining the feature maps of conv layers with different filters, to solve the multi-scale problem in crowd counting. Secondly, we develop the multi-task structure that can simultaneously learn the density map and the global counting. Experiments on large scale crowd counting datasets, Shanghaitech and WorldExpo10, demonstrate that the proposed method achieves much reduction in counting error respectively

We developed a multimodal ultrasound (US) deep learning (DL) fusion model to automatically classify early fibrosis in patients with chronic kidney disease (CKD). This prospective study included patients with CKD who underwent continuous gray-scale US, superb microvascular imaging, and strain elastography from May to November 2022. According to the pathological tubular atrophy and interstitial fibrosis score, patients were divided into minimal and mild groups (affected area ≤10% and 11 − 25% of the total cortical volume, respectively). The dataset was divided into training (70%) and test (30%) sets. A DL model combining the features of the three US modes was developed to predict early fibrosis in patients with CKD. We compared these findings with the area under the receiver operating characteristic curve (AUC) of the clinical model by analyzing the receiver operating characteristic curve in the test set. The AUC of single-mode DL based on gray-scale US, superb microvascular imaging, and strain elastography was 0.682, 0.745, and 0.648, respectively, while that of the multimodal US DL model was 0.86. The accuracy, specificity, and sensitivity of the multimodal US DL model were 0.779, 0.767, and 0.796, respectively, and the negative and positive predictive values were 0.842 and 0.706, respectively. The AUC of the multimodal US DL model was significantly better than that of the single-mode DL and clinical models. The DL algorithm developed using multimodal US images can effectively predict early fibrosis in patients with CKD with significantly greater accuracy than single-mode DL or clinical models.

Adversarial scale-adaptive neural network for crowd counting

Crowd counting plays an important role in urban management and public security. Recently, deep learning has shown a great advantage in making the quality of crowd counting more accurate. However, how to apply deep learning models to embedded terminals is still a challenging issue. The main contradiction of the problem lies in high demand for computing resources for deep learning and strict limitation of power consumption from embedded devices. In order to achieve the crowd counting by edge computing (in embedded terminals), we propose a tiny model based on convolutional neural networks. The model can be switched into other two forms to adapt tradeoff between the accuracy and efficiency. Especially, different forms share main parameters so as to save storage and avoid retraining. What is more, this model supports variant sizes of input images, which benefits the applications to a variety of embedded devices. The experiments on two different embedded terminals have shown the effectiveness, efficiency, and flexibility of our proposed model.

PurposeTo conduct a head-to-head comparison between deep learning (DL) and radiomics models across institutions for predicting microvascular invasion (MVI) in hepatocellular carcinoma (HCC) and to investigate the model robustness and generalizability through rigorous internal and external validation.MethodsThis retrospective study included 2304 preoperative images of 576 HCC lesions from two centers, with MVI status determined by postoperative histopathology. We developed DL and radiomics models for predicting the presence of MVI using B-mode ultrasound, contrast-enhanced ultrasound (CEUS) at the arterial, portal, and delayed phases, and a combined modality (B + CEUS). For radiomics, we constructed models with enlarged vs. original regions of interest (ROIs). A cross-validation approach was performed by training models on one center’s dataset and validating the other, and vice versa. This allowed assessment of the validity of different ultrasound modalities and the cross-center robustness of the models. The optimal model combined with alpha-fetoprotein (AFP) was also validated. The head-to-head comparison was based on the area under the receiver operating characteristic curve (AUC).ResultsThirteen DL models and 25 radiomics models using different ultrasound modalities were constructed and compared. B + CEUS was the optimal modality for both DL and radiomics models. The DL model achieved AUCs of 0.802–0.818 internally and 0.667–0.688 externally across the two centers, whereas radiomics achieved AUCs of 0.749–0.869 internally and 0.646–0.697 externally. The radiomics models showed overall improvement with enlarged ROIs (P < 0.05 for both CEUS and B + CEUS modalities). The DL models showed good cross-institutional robustness (P > 0.05 for all modalities, 1.6–2.1% differences in AUC for the optimal modality), whereas the radiomics models had relatively limited robustness across the two centers (12% drop-off in AUC for the optimal modality). Adding AFP improved the DL models (P < 0.05 externally) and well maintained the robustness, but did not benefit the radiomics model (P > 0.05).ConclusionCross-institutional validation indicated that DL demonstrated better robustness than radiomics for preoperative MVI prediction in patients with HCC, representing a promising solution to non-standardized ultrasound examination procedures.

Crowd counting is of great importance to many applications. Conventional vision-based approaches require line of sight and pose privacy concerns, while most radio-based approaches involve high deployment cost. In this paper, we propose to utilize WiFi channel state information (CSI) to infer crowd count in a device-free way, with only one pair of WiFi transmitter and receiver. The proposed method establishes the statistical relationship between the variation of CSI and the number of people with deep neural networks (DNN) and thereafter estimates the people count according to the real-time CSI through the trained DNN model. Evaluations demonstrate the effectiveness of the method. For the crowd size of 6, the counting error was within 1 person for 100% of the cases. For the crowd size of 34, the counting error was within 1 person for 97.7% of the cases and within 2 persons for 99.3% of the cases.

Endoscopic endonasal transsphenoidal surgery (EETS) is a minimally invasive procedure that accesses the sellar and parasellar regions. Various anatomical structures must be identified during the operation, particularly the sella turcica and internal carotid artery (ICA) bilaterally. In the present retrospective cohort study, authors aimed to evaluate the performance of a deep learning (DL) model in detecting the sella turcica and ICA bilaterally in EETS video footage, with the goal of recognizing crucial landmarks and preventing potentially fatal injury. The authors collected images from the endoscopic video footage of 98 patients who had undergone EETS from January 2015 to June 2024. The ICAs and sella turcica were labeled by neurosurgeons, and the entire dataset was divided into training, validation, and test datasets at a ratio of 7:2:1. The model for ICA and sella turcica detection was trained using the YOLOv5s object detection architecture, and precision, recall, mean average precision (mAP)@0.5, and mAP@0.5:0.95 were reported during the validation process. Moreover, the confusion matrix and area under the receiver operating characteristic curve (AUC) were assessed from the model using unseen images from the test dataset. The DL model had precision, recall, mAP@0.5, and mAP@0.5:0.95 of 0.942, 0.955, 0.969, and 0.617, respectively, for all objects in the training processes with validation. For testing the model with unseen images, the AUC was 0.97 (95% CI 0.95-0.98), whereas average precision was 0.99 (95% CI 0.99-1.00). For ICA detection with a multiclass approach, the AUCs were 0.98 (95% CI 0.97-0.99) for the absence of any ICA, 0.93 (95% CI 0.91-0.95) for 1 ICA in the images, and 0.95 (95% CI 0.93-0.96) for both ICAs in the image. Additionally, accuracy for the ICA and sella turcica was 0.958 and 0.965, respectively. Complex anatomical landmarks should be recognized during EETS. The computer vision model was effective in detecting the sella turcica and ICA bilaterally, as well as in identifying and avoiding fatal complications. For the model to generalize with reliability, it requires novel, unseen data from various settings to refine it and facilitate transfer learning.

COVID-19 is highly infectious and causes acute respiratory disease. Machine learning (ML) and deep learning (DL) models are vital in detecting disease from computerized chest tomography (CT) scans. The DL models outperformed the ML models. For COVID-19 detection from CT scan images, DL models are used as end-to-end models. Thus, the performance of the model is evaluated for the quality of the extracted feature and classification accuracy. There are four contributions included in this work. First, this research is motivated by studying the quality of the extracted feature from the DL by feeding these extracted to an ML model. In other words, we proposed comparing the end-to-end DL model performance against the approach of using DL for feature extraction and ML for the classification of COVID-19 CT scan images. Second, we proposed studying the effect of fusing extracted features from image descriptors, e.g., Scale-Invariant Feature Transform (SIFT), with extracted features from DL models. Third, we proposed a new Convolutional Neural Network (CNN) to be trained from scratch and then compared to the deep transfer learning on the same classification problem. Finally, we studied the performance gap between classic ML models against ensemble learning models. The proposed framework is evaluated using a CT dataset, where the obtained results are evaluated using five different metrics The obtained results revealed that using the proposed CNN model is better than using the well-known DL model for the purpose of feature extraction. Moreover, using a DL model for feature extraction and an ML model for the classification task achieved better results in comparison to using an end-to-end DL model for detecting COVID-19 CT scan images. Of note, the accuracy rate of the former method improved by using ensemble learning models instead of the classic ML models. The proposed method achieved the best accuracy rate of 99.39%.

Manufacturing process of Lithium-ion battery electrode has a direct impact on the resulting practical properties of the cell such as durability, safety and overall performance. In this scenario, together with experimental efforts to understand the correlation between manufacturing process parameters and final overall electrode performance, computational tools have shown the potential to produce insights on the manufacturing properties interdependencies. The ARTISTIC initiative [1] pioneered the development of a series of computational 3D-resolved physics-based models describing each step of the manufacturing process following a sequential pattern, going from the slurry phase, through the drying phase, to produce a calendered electrode. Such physics-based models have the capability to predict how manufacturing parameters impact the electrode microstructure. At the end of each phase the output microstructure is carefully validated with experimental data, assuring that the model is in good agreement with real properties. For the calendering phase, the Discrete Element Method (DEM) is used for the model simulation.One of the main limitations of the DEM simulations is their high computational cost, preventing their direct application in electrode optimization loops. In this matter, and given the great amount of already validated data that was produced in our previous works by using the ARTISTIC model, we take one step forward in the development of a new generation model employing machine learning (ML) techniques to accelerate the physics-based simulations but keeping their accuracy. In this presentation we report a novel time-dependent deep learning (DL) model of the battery electrodes manufacturing process. The DL model is demonstrated for calendering of nickel manganese cobalt (NMC111) electrodes, and trained with time-series data arising from physics-based DEM simulations coming from the ARTISTIC physics-based modeling pipeline [2]. The innovative DL model presented in this work can predict an electrode microstructure evolution over time at a given compression degree during calendering and provide the associated final relaxed electrode microstructure. Here we used the formulation of 96% AM and 4% CBD as a proof of concept.The DL model predictions are validated by comparing data evaluation metrics (e.g., mean square error (MSE) and R2 score) and electrode functional metrics (contact surface area, porosity, diffusivity, and tortuosity factor), showing very good accuracy with respect to the DEM simulations (all the properties are above 90% accurate when compared to target data). Regarding transport properties and given the high accuracy of the DL model to predict tortuosity factor and porosity values we can conclude that the model correctly predicts the effect of the calendering on the transport properties since tortuosity factor and porosity has a high impact on the effective transport properties of Li-ions in the electrolyte. The DL model can remarkably capture the elastic recovery of the electrode upon compression (spring-back phenomenon) and the main 3D electrode microstructure features without using the functional descriptors during its training. In terms of computational cost, the DL model performs a timestep in 15 seconds (wall time) while the DEM model performs an equivalent time step in ∼47 minutes (wall time). Given the DL model significant lower computational cost than the DEM simulations, it paves the way toward quasi-real-time optimization loops of the 3D electrode architecture predicting the calendering conditions to adopt in order to obtain the desired electrode performance. Moreover, we consider that our model has tremendous potential to grow in several aspects such as accuracy and predictability. Therefore, our incoming efforts will be focused on continuing the development of the model and testing it for other formulations and in other manufacturing stages.[1] ARTISTIC Project website. https://www.erc-artistic.eu.[2] D. E. Galvez-Aranda, T. L. Dinh, U. Vijay, F. M. Zanotto, A. A. Franco, Time-Dependent Deep Learning Manufacturing Process Model for Battery Electrode Microstructure Prediction. Adv. Energy Mater. 2024, 2400376. https://doi.org/10.1002/aenm.202400376 Figure 1

Due to domain shift, a large performance drop is usually observed when a trained crowd counting model is deployed in the wild. While existing domain-adaptive crowd counting methods achieve promising results, they typically regard each crowd image as a whole and reduce domain discrepancies in a holistic manner, thus limiting further improvement of domain adaptation performance. To this end, we propose to untangle domain-invariant crowd and domain-specific background from crowd images and design a fine-grained domain adaption method for crowd counting. Specifically, to disentangle crowd from background, we propose to learn crowd segmentation from point-level crowd counting annotations in a weakly-supervised manner. Based on the derived segmentation, we design a crowd-aware domain adaptation mechanism consisting of two crowd-aware adaptation modules, i.e., Crowd Region Transfer (CRT) and Crowd Density Alignment (CDA). The CRT module is designed to guide crowd features transfer across domains beyond background distractions. The CDA module dedicates to regularising target-domain crowd density generation by its own crowd density distribution. Our method outperforms previous approaches consistently in the widely-used adaptation scenarios.

Electricity load forecasting is a crucial undertaking within all the deregulated markets globally. Among the research challenges on a global scale, the investigation of deep transfer learning (DTL) in the field of electricity load forecasting represents a fundamental effort that can inform artificial intelligence applications in general. In this paper, a comprehensive study is reported regarding day-ahead electricity load forecasting. For this purpose, three sequence-to-sequence (Seq2seq) deep learning (DL) models are used, namely the multilayer perceptron (MLP), the convolutional neural network (CNN) and the ensemble learning model (ELM), which consists of the weighted combination of the outputs of MLP and CNN models. Also, the study focuses on the development of different forecasting strategies based on DTL, emphasizing the way the datasets are trained and fine-tuned for higher forecasting accuracy. In order to implement the forecasting strategies using deep learning models, load datasets from three Greek islands, Rhodes, Lesvos, and Chios, are used. The main purpose is to apply DTL for day-ahead predictions (1–24 h) for each month of the year for the Chios dataset after training and fine-tuning the models using the datasets of the three islands in various combinations. Four DTL strategies are illustrated. In the first strategy (DTL Case 1), each of the three DL models is trained using only the Lesvos dataset, while fine-tuning is performed on the dataset of Chios island, in order to create day-ahead predictions for the Chios load. In the second strategy (DTL Case 2), data from both Lesvos and Rhodes concurrently are used for the DL model training period, and fine-tuning is performed on the data from Chios. The third DTL strategy (DTL Case 3) involves the training of the DL models using the Lesvos dataset, and the testing period is performed directly on the Chios dataset without fine-tuning. The fourth strategy is a multi-task deep learning (MTDL) approach, which has been extensively studied in recent years. In MTDL, the three DL models are trained simultaneously on all three datasets and the final predictions are made on the unknown part of the dataset of Chios. The results obtained demonstrate that DTL can be applied with high efficiency for day-ahead load forecasting. Specifically, DTL Case 1 and 2 outperformed MTDL in terms of load prediction accuracy. Regarding the DL models, all three exhibit very high prediction accuracy, especially in the two cases with fine-tuning. The ELM excels compared to the single models. More specifically, for conducting day-ahead predictions, it is concluded that the MLP model presents the best monthly forecasts with MAPE values of 6.24% and 6.01% for the first two cases, the CNN model presents the best monthly forecasts with MAPE values of 5.57% and 5.60%, respectively, and the ELM model achieves the best monthly forecasts with MAPE values of 5.29% and 5.31%, respectively, indicating the very high accuracy it can achieve.

Predicting craniofacial fibrous dysplasia growth status: an exploratory study of a hybrid radiomics and deep learning model based on computed tomography images

We aimed at comparing intratumoral and peritumoral deep learning, radiomics, and fusion models in predicting KRAS mutations in rectal cancer using endorectal ultrasound imaging. This study included 304 patients with rectal cancer from Fujian Medical University Union Hospital. The patients were randomly divided into a training group (213 patients) and a test group (91 patients) at a 7:3 ratio. Radiomics and deep learning models were established using primary tumor and peritumoral images. In the optimally performing regions-of-interest, two fusion strategies, a feature-based and a decision-based model, were employed to build the fusion models. The Shapley additive explanation (SHAP) method was used to evaluate the significance of features in the optimal radiomics, deep learning, and fusion models. The performance of each model was assessed using the area under the receiver operating characteristic curve (AUC) and decision curve analysis (DCA). In the test cohort, both the radiomics and deep learning models exhibited optimal performance with a 10-pixel patch extension, yielding AUC values of 0.824 and 0.856, respectively. The feature-based DLRexpand10_FB model attained the highest AUC (0.896) across all study sets. In addition, the DLRexpand10_FB model demonstrated excellent sensitivity, specificity, and DCA. SHAP analysis underscored the deep learning feature (DL_1) as the most significant factor in the hybrid model. The feature-based fusion model DLRexpand10_FB can be employed to predict KRAS gene mutations based on pretreatment endorectal ultrasound images of rectal cancer. The integration of peritumoral regions enhanced the predictive performance of both the radiomics and deep learning models.

Deep learning breast cancer risk models demonstrate improved accuracy compared with traditional risk models but have not been prospectively tested. We compared the accuracy of a deep learning risk score derived from the patient's prior mammogram to traditional risk scores to prospectively identify patients with cancer in a cohort due for screening. We collected data on 119 139 bilateral screening mammograms in 57 617 consecutive patients screened at 5 facilities between September 18, 2017, and February 1, 2021. Patient demographics were retrieved from electronic medical records, cancer outcomes determined through regional tumor registry linkage, and comparisons made across risk models using Wilcoxon and Pearson χ2 2-sided tests. Deep learning, Tyrer-Cuzick, and National Cancer Institute Breast Cancer Risk Assessment Tool (NCI BCRAT) risk models were compared with respect to performance metrics and area under the receiver operating characteristic curves. Cancers detected per thousand patients screened were higher in patients at increased risk by the deep learning model (8.6, 95% confidence interval [CI] = 7.9 to 9.4) compared with Tyrer-Cuzick (4.4, 95% CI = 3.9 to 4.9) and NCI BCRAT (3.8, 95% CI = 3.3 to 4.3) models (P &lt; .001). Area under the receiver operating characteristic curves of the deep learning model (0.68, 95% CI = 0.66 to 0.70) was higher compared with Tyrer-Cuzick (0.57, 95% CI = 0.54 to 0.60) and NCI BCRAT (0.57, 95% CI = 0.54 to 0.60) models. Simulated screening of the top 50th percentile risk by the deep learning model captured statistically significantly more patients with cancer compared with Tyrer-Cuzick and NCI BCRAT models (P &lt; .001). A deep learning model to assess breast cancer risk can support feasible and effective risk-based screening and is superior to traditional models to identify patients destined to develop cancer in large screening cohorts.