Fully Residual Convolutional Neural Networks for Aerial Image Segmentation

In this paper, we study the influence of both long and short skip connections on Fully Convolutional Networks (FCN) for biomedical image segmentation. In standard FCNs, only long skip connections are used to skip features from the contracting path to the expanding path in order to recover spatial information lost during downsampling. We extend FCNs by adding short skip connections, that are similar to the ones introduced in residual networks, in order to build very deep FCNs (of hundreds of layers). A review of the gradient flow confirms that for a very deep FCN it is beneficial to have both long and short skip connections. Finally, we show that a very deep FCN can achieve near-to-state-of-the-art results on the EM dataset without any further post-processing.

Automatic real-time crack detection using lightweight deep learning models

Nuclei segmentation is a fundamental step to measure cell motility and morphology in High-throughput microscopy images. One prerequisite of automatic nuclei segmentation is to manually label a huge amount of cell images, which is a labor-intensive and time-consuming process. This paper develops a semi-supervised learning approach to reduce the dependence on the amount of labeled images, and it consists of three main steps. First, cell regions are automatically extracted from the background through a novel convolutional neural network (CNN). Comparing with the state-of-art segmentation network structure, our designed network structure has fewer parameters and combines layers of the feature hierarchy that refines the spatial precision of the output probability map. Second, discriminator networks are explored to produce confidence maps for unlabeled microscopy images by applying a fully convolution network (FCN) as the discriminator network. It also can be used to improve the performance of the segmentation network by coupling the adversarial loss with standard cross-entropy loss. Finally, combining both labeled images and unlabeled images with confidence maps to train the segmentation network can identify individual nuclei on the extracted cell regions. Experimental results on only 84 images showed that SEG score could achieve 77.9% and F1 score could achieve 76.0%, which outperformed existing using 670 images segmentation methods. Such promising result suggested that the proposed semi-supervised nuclei segmentation approach could achieve high segmentation accuracy with a small set of labeled images, which is a desirable property to analyze microscopy images in the real clinical practice.

This study proposes an automatic building footprint extraction framework that consists of a convolutional neural network (CNN)-based segmentation and an empirical polygon regularization that transforms segmentation maps into structured individual building polygons. The framework attempts to replace part of the manual delineation of building footprints that are involved in surveying and mapping field with algorithms. First, we develop a scale robust fully convolutional network (FCN) by introducing multiple scale aggregation of feature pyramids from convolutional layers. Two postprocessing strategies are introduced to refine the segmentation maps from the FCN. The refined segmentation maps are vectorized and polygonized. Then, we propose a polygon regularization algorithm consisting of a coarse and fine adjustment, to translate the initial polygons into structured footprints. Experiments on a large open building data set including 181 000 buildings showed that our algorithm reached a high automation level where at least 50% of individual buildings in the test area could be delineated to replace manual work. Experiments on different data sets demonstrated that our FCN-based segmentation method outperformed several most recent segmentation methods, and our polygon regularization algorithm is robust in challenging situations with different building styles, image resolutions, and even low-quality segmentation.

To reduce the cost of production and the pollution of the environment that is due to the overapplication of herbicide in paddy fields, the location information of rice seedlings and weeds must be detected in site-specific weed management (SSWM). With the development of deep learning, a semantic segmentation method with the SegNet that is based on fully convolutional network (FCN) was proposed. In this paper, RGB color images of seedling rice were captured in paddy field, and ground truth (GT) images were obtained by manually labeled the pixels in the RGB images with three separate categories, namely, rice seedlings, background, and weeds. The class weight coefficients were calculated to solve the problem of the unbalance of the number of the classification category. GT images and RGB images were used for data training and data testing. Eighty percent of the samples were randomly selected as the training dataset and 20% of samples were used as the test dataset. The proposed method was compared with a classical semantic segmentation model, namely, FCN, and U-Net models. The average accuracy rate of the SegNet method was 92.7%, whereas the average accuracy rates of the FCN and U-Net methods were 89.5% and 70.8%, respectively. The proposed SegNet method realized higher classification accuracy and could effectively classify the pixels of rice seedlings, background, and weeds in the paddy field images and acquire the positions of their regions.

Hyper-fusion network for semi-automatic segmentation of skin lesions.

Bridging 2D and 3D segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5D solutions.

Fatigue fracture is one of the most common metallic component failure cases in manufacturing industries. The observation on fractography can provide direct evidence for failure analysis. In this study, an image semantic segmentation method based on fully convolutional networks (FCNs) was proposed to figure out the boundary between fatigue crack propagation and fast fracture regions from optical microscope (OM) fractography images. Furthermore, a novel morphing‐based data augmentation method was adopted to enable few‐shot learning of sample images. The proposed framework can successfully segment two categories, namely, the crack propagation and fast fracture regions, thus differentiating the boundary of two regions in one image. This artificial intelligence (AI)‐assisted fatigue analysis architecture can complete the failure analysis procedure in 0.5 s and prove the feasibility of fatigue failure analysis. The segmentation accuracy of self‐developed network achieves 95.4% for the fatigue crack propagation region, as well as 97.2% for the fast fracture region. Not only for semantic segmentation DNN, we also prove that our novel data augmentation method can applied at the instance segmentation DNN, such as mask regional convolutional neural network (mask R‐CNN), one state‐of‐the‐art deep learning network for instance segmentation, to achieve similar accuracy.

Hybrid CNN-transformer network for interactive learning of challenging musculoskeletal images

Manual approaches of fiber recognition are often time-consuming, laborious, and subjective. In this paper, a novel vision-based measurement (VBM) framework of fiber recognition based on image segmentation and deep convolutional neural networks (DCNNs) is proposed. The proposed segmentation method can be applied to the segmentation of overlapping and adhering translucent fibers and has better segmentation performance. At the same time, the segmentation method provides a large number of single-fiber training samples and test samples for DCNN and provides a basis for multifiber map recognition using voting strategies. Finally, four frequently used kinds of fibers, i.e., cotton, camel hair, viscose, and yak cashmere, are selected for experiments. In order to decrease the chance of overfitting, the data augmentation methods are utilized to enlarge the data sets formed by the segmented single-fiber images. In the experimental phase, performance differences are evaluated between the four network architectures, namely, AlexNet, cashmere and wool classification-Net, VGG-Net-16, and GoogLeNet. The GoogLeNet’s single-fiber classification accuracy rate reaches 96.6%, and the average accuracy rate of multifiber identification strategy reaches 99.5%. The results show that the proposed VBM framework of fiber recognition is effective.

Farmland is the cornerstone of agriculture and is important for food security and social production. Farmland assessment is essential but traditional methods are usually expensive and slow. Deep learning methods have been developed and widely applied recently in image recognition, semantic understanding, and many other application domains. In this research, we used fully convolutional networks (FCN) as the deep learning model to evaluate farmland grades. Normalized difference vegetation index (NDVI) derived from Landsat images was used as the input data, and the China National Cultivated Land Grade Database within Jiangsu Province was used to train the model on cloud computing. We also applied an image segmentation method to improve the original results from the FCN and compared the results with classical machine learning (ML) methods. Our research found that the FCN can predict farmland grades with an overall F1 score (the harmonic mean of precision and recall) of 0.719 and F1 score of 0.909, 0.590, 0.740, 0.642, and 0.023 for non-farmland, level I, II, III, and IV farmland, respectively. Combining the FCN and image segmentation method can further improve prediction accuracy with results of fewer noise pixels and more realistic edges. Compared with conventional ML, at least in farmland evaluation, FCN provides better results with higher precision, recall, and F1 score. Our research indicates that by using remote sensing NDVI data, the deep learning method can provide acceptable farmland assessment without fieldwork and can be used as a novel supplement to traditional methods. The method used in this research will save a lot of time and cost compared with traditional means.

Semantic segmentation on high-resolution aerial images plays a significant role in many remote sensing applications. Although the Deep Convolutional Neural Network (DCNN) has shown great performance in this task, it still faces the following two challenges: intra-class heterogeneity and inter-class homogeneity. To overcome these two problems, a novel dual-path DCNN, which contains a spatial path and an edge path, is proposed for high-resolution aerial image segmentation. The spatial path, which combines the multi-level and global context features to encode the local and global information, is used to address the intra-class heterogeneity challenge. For inter-class homogeneity problem, a Holistically-nested Edge Detection (HED)-like edge path is employed to detect the semantic boundaries for the guidance of feature learning. Furthermore, we improve the computational efficiency of the network by employing the backbone of MobileNetV2. We enhance the performance of MobileNetV2 with two modifications: (1) replacing the standard convolution in the last four Bottleneck Residual Blocks (BRBs) with atrous convolution; and (2) removing the convolution stride of 2 in the first layer of BRBs 4 and 6. Experimental results on the ISPRS Vaihingen and Potsdam 2D labeling dataset show that the proposed DCNN achieved real-time inference speed on a single GPU card with better performance, compared with the state-of-the-art baselines.

To evaluate the performance, agreement, and efficiency of a fully convolutional network (FCN) for liver lesion detection and segmentation at CT examinations in patients with colorectal liver metastases (CLMs). This retrospective study evaluated an automated method using an FCN that was trained, validated, and tested with 115, 15, and 26 contrast material-enhanced CT examinations containing 261, 22, and 105 lesions, respectively. Manual detection and segmentation by a radiologist was the reference standard. Performance of fully automated and user-corrected segmentations was compared with that of manual segmentations. The interuser agreement and interaction time of manual and user-corrected segmentations were assessed. Analyses included sensitivity and positive predictive value of detection, segmentation accuracy, Cohen κ, Bland-Altman analyses, and analysis of variance. In the test cohort, for lesion size smaller than 10 mm (n = 30), 10-20 mm (n = 35), and larger than 20 mm (n = 40), the detection sensitivity of the automated method was 10%, 71%, and 85%; positive predictive value was 25%, 83%, and 94%; Dice similarity coefficient was 0.14, 0.53, and 0.68; maximum symmetric surface distance was 5.2, 6.0, and 10.4 mm; and average symmetric surface distance was 2.7, 1.7, and 2.8 mm, respectively. For manual and user-corrected segmentation, κ values were 0.42 (95% confidence interval: 0.24, 0.63) and 0.52 (95% confidence interval: 0.36, 0.72); normalized interreader agreement for lesion volume was -0.10 ± 0.07 (95% confidence interval) and -0.10 ± 0.08; and mean interaction time was 7.7 minutes ± 2.4 (standard deviation) and 4.8 minutes ± 2.1 (P < .001), respectively. Automated detection and segmentation of CLM by using deep learning with convolutional neural networks, when manually corrected, improved efficiency but did not substantially change agreement on volumetric measurements.© RSNA, 2019Supplemental material is available for this article.

A data augmentation approach to train fully convolutional networks for left ventricle segmentation

Explainable multi-module semantic guided attention based network for medical image segmentation