Concrete crack classification by a lightweight Bayesian CNN: a comparative evaluation with UNet on performance improvement through a Bayesian approach

Probabilistic SAR-based water segmentation with adapted Bayesian convolutional neural network

El Niño Southern Oscillation (ENSO) significantly influences Earth's system and regional climate patterns. Despite dedicated efforts spanning decades, the precise prediction of ENSO events through numerical modelling beyond  one-year lead time remains a formidable challenge. The advent of deep learning-based approaches marks a transformative era in climate and weather prediction. However, many machine learning-based studies attempting ENSO prediction are confined to singular estimates, lacking adequate quantification of uncertainty in learned parameters and overlooking the crucial need for a nuanced understanding of ENSO prediction confidence. Here, we introduce a deep learning-based Bayesian Convolutional Neural Network (BCNN) model that provides probabilistic predictions for ENSO  with a lead time of up to 24 months. The Bayesian layers within the CNN maintain the capability to predict a distribution of learned parameters. The inherent capacity for uncertainty modeling enhances the reliability of BNNs, making them particularly valuable in operational services. The model was initially trained using globally gridded monthly sea surface temperature and upper 300 m integrated ocean potential temperature predictors from 25 Climate Model Intercomparison Project (CMIP) models, including 11 CMIP5 and 14 CMIP6. To address systematic errors in the BCNN, reflecting those of the CMIP samples, a learning transfer technique was applied using Simple Ocean Data Assimilation (SODA) reanalysis predictors spanning from 1871 to 1980. Validation of the all-season correlation skill of the Nino3.4 index in the BCNN model demonstrates significantly higher accuracy compared to current state-of-the-art dynamical forecast systems. This research holds substantial socio-economic implications as it enhances our forecasting capabilities and rigorously quantifies forecast uncertainties, providing valuable insights for planning and policy-making.

Ultrasound (US) imaging is a complex imaging modality, where the tissues are typically characterised by an inhomogeneous image intensity and by a variable image definition at the boundaries that depends on the direction of the incident sound wave. For this reason, conventional image segmentation approaches where the regions of interest are represented by exact masks are inherently inefficient for US images. To solve this issue, we present the first application of a Bayesian convolutional neural network (CNN) based on Monte Carlo dropout on US imaging. This approach is particularly relevant for quantitative applications since differently from traditional CNNs, it enables to infer for each image pixel not only the probability of being part of the target but also the algorithm confidence (i.e. uncertainty) in assigning that probability. In this work, this technique has been applied on US images of the femoral cartilage in the framework of a new application, where high-refresh-rate volumetric US is used for guidance in minimally invasive robotic surgery for the knee. Two options were explored, where the Bayesian CNN was trained with the femoral cartilage contoured either on US, or on magnetic resonance imaging (MRI) and then projected onto the corresponding US volume. To evaluate the segmentation performance, we propose a novel approach where a probabilistic ground-truth annotation was generated combining the femoral cartilage contours from registered US and MRI volumes. Both cases produced a significantly better segmentation performance when compared against traditional CNNs, achieving a dice score coefficient increase of about 6% and 8%, respectively.

Recent studies have been shown the vulnerability of Deep neural networks (DNNs) by a small adversarial perturbation in images that humans cannot distinguish and a well-trained neural network misclassifies. There are many defense methods, e.g. randomness, ensemble, and adversarial training that can improve the robustness of neural networks with respect to adversaries. Among them, adversarial training has been being the most outstanding defense. However, the training cost becomes a big challenge to adversarial learning. In this paper, we propose a new defense algorithm called Bayes without Bayesian Learning, which does not add the training phase in resisting adversarial attacks. Our method has based on the stochastic components of Bayesian Convolutional Neural Network (BCNN) to prevent the forceful gradient based attacks and generate the ensemble model to enhance the performance of models. In order to avoid adversarial training, we utilize pretrained CNN models on both natural and perturbed data. Therefore, this approach can be applied to any model with learned parameters to accelerate the computation of the defensive method. Our defensive algorithm has significantly improved the accuracy of pretrained CNN models under the high level of attacks. For instance, on ImageNet with Resnet50 network, we achieve 57% prediction accuracy in the top5, under strong PGD attack. We also confirm 3% improvement when combining our method with adversarial training. Experimental results prove the efficiency of our BCNN model that does not need Bayesian learning but still resists adversarial perturbation on both small and large scale datasets as CIFAR10, CIFAR100, and ImageNet.

Where to model the epistemic uncertainty of Bayesian convolutional neural networks for classification

From a medical standpoint, detecting the size and shape of hard inclusions hidden in soft three-dimensional objects is of great significance for early detection of cancer through palpation. Soft robots, especially soft grippers, substantially broaden robots' palpation capabilities from soft to hard materials without the assistance of a camera. We have recently introduced a CNN-Bayes approach which added a Naïve Bayes classifier to a convolutional neural network (CNN) architecture called SoftTactNet for variable stiffness object recognition on a three-finger FinRay soft gripper. SoftTactNet itself lacks uncertainty estimations though it can reach a certain level of recognition accuracy. In this paper, we further improve the framework by merging Bayes method directly into CNN architectures and build a new Bayes-SoftTactNet for object recognition. The new approach, using a prior distribution instead of point estimation, allows the network to present results with uncertainty estimates. We conduct new experiments using the same soft gripper with tactile sensor arrays to grasp different variable stiffness objects surrounded by non-different soft material and generate tactile images as dataset. The results show that our new algorithm is more efficient than the previous approach and still able to achieve higher recognition accuracy than general deterministic CNNs.

Fault diagnosis is a vital technique to pinpoint the machine malfunctions in manufacturing systems. In recent years, the deep learning techniques greatly improve the fault detection accuracy, but there still remain some problems. If one fault is absent in the training data or the fault signal is disturbed by severe noise interference, the fault classifier may misjudge the health state. This problem limits the reliability of the fault diagnosis in real applications. In this paper, we enhance the fault diagnosis method by using Bayesian Convolutional Neural Network (BCNN). A Shannon entropy-based method is presented to quantify the prediction uncertainty. The BCNN turns the deterministic predictions to probabilistic distributions and enhances the robustness of the fault diagnosis. The uncertainty quantification method helps to indicate the wrong predictions, detect unknown faults, and discover the strong disturbances. Then, a fine-tuning strategy is applied to enhance the model performance further. The potential usability of the proposed method in monitoring the motors of 3D printers is studied. And the experiment is conducted on a motor bearing dataset provided by Case Western Reserve University. The proposed BCNN achieves 99.82% fault classification accuracy over nine health conditions. Its robustness is verified by comparing the testing accuracy with three other methods on the noisy datasets. And the uncertainty quantification method successfully detects the outlier inputs.

This study was aimed to explore the adoption value of magnetic resonance imaging (MRI) under convolutional neural networks (CNN) in the therapeutic effect of nasopharyngeal carcinoma (NPC) radiotherapy. A total of 54 NPC patients were recruited. CNN was employed to perform 3D visualization processing on magnetic resonance (MR) images of NPC patients. MRI changes were analyzed before and after the patient received radiotherapy. The image segmentation and radiotherapy effects of CNN were evaluated by the Recall, intersection over union (IOU), postoperative apparent diffusion coefficient (ADC), and diagnostic coincidence rate. Moreover, gradient vector flow (GVF) algorithm, fuzzy c-means (FCM), and SegNet were adopted for comparative evaluation. Recall of CNN was 94.89% and the IOU was 84.16%, which was remarkably different from other algorithms (P < 0.05). After analysis of the MRI images of patients receiving radiotherapy, ADC of local residual patients was 1.108 ± 0.097 measured by CNN, the ADC was 1.826 ± 0.115, and the missed diagnosis rate was only 7.14%. In summary, CNN had a good effect on the localization and segmentation of NPC patients, and can accurately evaluate the effect of patients receiving radiotherapy, which can assist clinical diagnosis and treatment of NPC.

Vision transformer-based autonomous crack detection on asphalt and concrete surfaces

BackgroundThe study of plant photosynthesis is essential for productivity and yield. Thanks to the development of high-throughput phenotyping (HTP) facilities, based on chlorophyll fluorescence imaging, photosynthetic traits can be measured in a reliable, reproducible and efficient manner. In most state-of-the-art HTP platforms, these traits are automatedly analyzed at individual plant level, but information at leaf level is often restricted by the use of manual annotation. Automated leaf tracking over time is therefore highly desired. Methods for tracking individual leaves are still uncommon, convoluted, or require large datasets. Hence, applications and libraries with different techniques are required. New phenotyping platforms are initiated now more frequently than ever; however, the application of advanced computer vision techniques, such as convolutional neural networks, is still growing at a slow pace. Here, we provide a method for leaf segmentation and tracking through the fine-tuning of Mask R-CNN and intersection over union as a solution for leaf tracking on top-down images of plants. We also provide datasets and code for training and testing on both detection and tracking of individual leaves, aiming to stimulate the community to expand the current methodologies on this topic.ResultsWe tested the results for detection and segmentation on 523 Arabidopsis thaliana leaves at three different stages of development from which we obtained a mean F-score of 0.956 on detection and 0.844 on segmentation overlap through the intersection over union (IoU). On the tracking side, we tested nine different plants with 191 leaves. A total of 161 leaves were tracked without issues, accounting to a total of 84.29% correct tracking, and a Higher Order Tracking Accuracy (HOTA) of 0.846. In our case study, leaf age and leaf order influenced photosynthetic capacity and photosynthetic response to light treatments. Leaf-dependent photosynthesis varies according to the genetic background.ConclusionThe method provided is robust for leaf tracking on top-down images. Although one of the strong components of the method is the low requirement in training data to achieve a good base result (based on fine-tuning), most of the tracking issues found could be solved by expanding the training dataset for the Mask R-CNN model.

Chronic wounds cause a significant burden to affected patients and to society. Effective and objective diagnostic and monitoring methods are needed in wound care, and artificial intelligence offers one promising alternative. In this study, real-world wound images were used to train a convolutional neural network to automatically segment wound area and wound tissues on an image. The study included altogether 362 images of venous, arterial, vasculitis and pyoderma gangrenosum wounds. The model was based on a convolutional neural network architecture U-Net, and fully supervised learning was utilised during the training phase. Wound area reached a Dice Similarity Coefficient (DSC) of 0.927 and Intersection over Union (IoU) of 0.868 using an augmented dataset with pretraining. Fibrinous exudate and granulation performed fairly well with DSC 0.750 and 0.696, and with IoU 0.659 and 0.601, respectively. Necrosis present in only 56 images achieved lower performance with DSC 0.503 and IoU 0.502. In conclusion, this study suggested that it is possible to train a neural network to perform well with images taken for purely clinical purposes. Besides wound area, several wound structures can be identified, but wound structure identification performance is dependent on the number of images featuring the structure.

Funding Acknowledgements Type of funding sources: Private grant(s) and/or Sponsorship. Main funding source(s): Japan Organization of Occupational Health and Safety onbehalf J3D Introduction Improvements in cardiovascular imaging have contributed to the accurate diagnosis and treatment of cardiovascular diseases. In recent years, we can find a new demand from surgical departments, i.e., 3D heart models of patients for preoperative simulations. However, a 3D surgical model needs a detailed segmentation of heart structures from clinical images because these details are essential for achieving accurate polygon data of an exact 3D model. Thus, in most cases, manual segmentation is required for complicated heart shapes such as the chamber wall, valves, and papillary muscles, which causes a prolonged duration time. Purpose We aim to achieve an automated heart segmentation using a convolutional neural network (CNN) trained using deep learning techniques for the rapid creation of 3D surgical heart models of individual patients. Methods We constructed our original CNN program based on the latest artificial intelligence techniques and trained it to extract shapes of the heart from cardiac computed tomography (CT) images. The training data was 361 slices selected from CT scans of 10 patients. We used data augmentation to increase the amount and diversity of the data into 24,052 slices. The training result with the best Intersection over Union (IoU), one of the evaluation metrics used in deep learning, was saved. Finally, we used the best-trained CNN to construct 3D polygon data of two surgical cases of hypertrophic obstructive cardiomyopathy (HOCM) for preoperative assessments. Results The IoU attained 0.85 after deep learning. The time required to complete the 3D polygon data for the first HOCM case was 5 minutes for segmentation by the trained CNN and 3 hours for data correction by a human operator. Similarly, the time required for the second case was 5 minutes for segmentation without manual correction. We had to correct the segmentation for the first case because we needed an exact 3D model for the preoperative assessment (Fig. 1). According to our records of the other eight 3D heart models in the lab, the work for a 3D polygon shape from CT images needs a median 30 hours (quartiles 23-50 hours) when the procedure is fully manual and non-continuous with breaks in between. Conclusion The CNN-based segmentation aided the constructing heart shapes from cardiac CT images of preoperative patients. Although the performance, reaching IoU of 0.85, was insufficient for fully automatic segmentation, the methodology can shorten the process duration from several hours to several minutes for detailed segmentation of heart structures. We previously applied the CNN-based segmentation to the aorta, aortic stenosis valves, and atheromatous plaque in clinical images, demonstrating adequate segmentation performance. The proposed methodology can be applied as a fundamental technology of cardiovascular imaging for obtaining the actual structures of a target object as 3D coordinate values or a 3D model within a reasonable duration time. Abstract Figure 1

Accurate lesion segmentation is critical for cancer diagnosis and treatment. Convolutional neural networks (CNNs) are widely used for medical image segmentation but struggle to capture long-range dependencies. Transformers mitigate this limitation but come with high computational costs. Mamba, a state-space model (SSM), efficiently models long-range dependencies but lacks precision in fine details. To address these challenges, this study aimed to develop a novel segmentation approach that combines the strengths of CNNs, Transformers, and Mamba, enhancing both global context understanding and local feature extraction in medical image segmentation. We propose AttmNet, a U-shaped network designed for medical image segmentation, which incorporates a novel structure called MAM (Multiscale-Convolution, Self-Attention, and Mamba). The MAM block integrates multi-layer convolution for multi-scale feature learning with an Att-Mamba component that combines self-attention and Mamba to effectively capture global context while preserving fine details. We evaluated AttmNet on four public datasets for breast, skin, and lung lesion segmentation. AttmNet outperformed state-of-the-art methods in terms of intersection over union (IoU) and Dice similarity coefficients. On the breast ultrasound (BUS) dataset, AttmNet achieved a 3.38% improvement in IoU and a 4.54% increase in Dice over the next best method. On the breast ultrasound images (BUSI) dataset, AttmNet's IoU and Dice coefficients were 1.17% and 3.21% higher than the closest competitor, respectively. In the PH2 Dermoscopy Image dataset, AttmNet surpassed the next best model by 0.25% in both IoU and Dice. On the larger coronavirus disease 2019 (COVID-19) Lung dataset, AttmNet maintained strong performance, achieving higher IoU and Dice scores than the next best models, SegMamba and TransUNet. AttmNet is a powerful and efficient tool for medical image segmentation, addressing the limitations of existing methods through its advanced design. The MAM block significantly enhances segmentation accuracy while maintaining computational efficiency, making AttmNet highly suitable for clinical applications. The code is available at https://github.com/hyb2840/AttmNet.

A genetic algorithm (GA), is used to optimize the many parameters of a convolutional neural network (CNN) that control the structure of the network. CNNs are used in image classification problems where it is necessary to generate feature descriptors to discern between image classes. Because of the deep representation of image data that CNNs are capable of generating, they are increasingly popular in research and industry applications. With the increasing number of use cases for CNNs, more and more time is being spent to come up with optimal CNN structures for different applications. Where one CNN might succeed at classification, another can fail. As a result, it is desirable to more easily find an optimal CNN structure to increase classification accuracy. In the proposed method, a GA is used to evolve the parameters that influence the structure of a CNN. The GA compares CNNs by training them on images of concrete containing cracks. The best CNN after several generations of the GA is then compared to the state-of-the-art CNN for crack detection. This work shows that it is possible to generalize the process of optimizing a CNN for image classification through the use of a GA.

Understanding the spatial distribution of irrigated croplands is crucial for food security and water use. To map land cover classes with high-spatial-resolution images, it is necessary to analyze the semantic information of target objects in addition to the spectral or spatial–spectral information of local pixels. Deep convolutional neural networks (DCNNs) can characterize the semantic features of objects adaptively. This study uses DCNNs to extract irrigated croplands from Sentinel-2 images in the states of Washington and California in the United States. We integrated the DCNNs of 101 layers, discarded pooling layers, and employed dilation convolution to preserve location information; these are models which were used based on fully convolutional network (FCN) architectures. The findings indicated that irrigated croplands may be effectively detected at various phases of crop growth in the fields. A quantitative analysis of the trained models revealed that the three models in the two states had the lowest values of Intersection over Union (IoU) and Kappa, i.e., 0.88 and 0.91, respectively. The deep models’ temporal portability across different years was acceptable. The lowest values of recall and OA (overall accuracy) from 2018 to 2021 were 0.91 and 0.87, respectively. In Washington, the lowest OA value from 10 to 300 m resolution was 0.76. This study demonstrates the potential of FCNs + DCNNs approaches for mapping irrigated croplands across large regions, providing a solution for irrigation mapping. The spatial resolution portability of deep models could be improved further by designing model architectures.