Toward Automatic Building Footprint Delineation From Aerial Images Using CNN and Regularization

Hyper-fusion network for semi-automatic segmentation of skin lesions.

Farmland is the cornerstone of agriculture and is important for food security and social production. Farmland assessment is essential but traditional methods are usually expensive and slow. Deep learning methods have been developed and widely applied recently in image recognition, semantic understanding, and many other application domains. In this research, we used fully convolutional networks (FCN) as the deep learning model to evaluate farmland grades. Normalized difference vegetation index (NDVI) derived from Landsat images was used as the input data, and the China National Cultivated Land Grade Database within Jiangsu Province was used to train the model on cloud computing. We also applied an image segmentation method to improve the original results from the FCN and compared the results with classical machine learning (ML) methods. Our research found that the FCN can predict farmland grades with an overall F1 score (the harmonic mean of precision and recall) of 0.719 and F1 score of 0.909, 0.590, 0.740, 0.642, and 0.023 for non-farmland, level I, II, III, and IV farmland, respectively. Combining the FCN and image segmentation method can further improve prediction accuracy with results of fewer noise pixels and more realistic edges. Compared with conventional ML, at least in farmland evaluation, FCN provides better results with higher precision, recall, and F1 score. Our research indicates that by using remote sensing NDVI data, the deep learning method can provide acceptable farmland assessment without fieldwork and can be used as a novel supplement to traditional methods. The method used in this research will save a lot of time and cost compared with traditional means.

Fully Convolutional Network (FCN), which can adopt various Convolutional Neural Networks (CNN), are now increasingly being used in remote sensing communities. CNN are improved constantly either in accuracy or by reducing parameters for a given equivalent accuracy. This paper investigates five widely used CNNs (AlexNet, VGG16, ResNet, SqueezeNet, and a pruned VGG16) in the context of FCN for coastal beach classification of imagery acquired by Unmanned Aerial Vehicles (UAV). Our experiments show that (1) not every CNN is suitable to FCN for semantic segmentations of images though each CNN approximately achieved an equivalent accuracy for image labeling; (2) band reduced pruning of existing CNN has the least impact on implementation and accuracy. To examine the capability of convolutional layers capturing semantic features, this paper also carries out beach classification experiments using hypercolumn methods with VGG16.

Early detection and treatment are crucial for recovery from cancer. Due to rapid advancement in medical imaging techniques, today doctors can scan various body parts from different perspectives to diagnose cancer or see the progress during a course of treatment. Such techniques are less painful and require less preparation and recovery. The recent evolution of AI, particularly deep learning algorithms have shown potential in computer-aided diagnosis of cancer. These include analysis of cancer images for detection of tumor region, prognosis, skin tissue inspection, etc. Convolutional Neural Network (CNN) based image processing techniques are showing promising results in the diagnosis of breast cancer, skin cancer, prostate cancer, lung cancer, etc. Literature shows many other CNN-based models such as feed-forward neural network, Deep CNN, Multi-Crop CNN, fully convolutional networks (FCNs), Deep Fully Convolutional Network (DFCNet), Recurrent neural networks, etc. The CNN focus solely on the image-specific features and hence require a lesser number of input parameters. A CNN model reduces the size of the input image vector without losing the features critical for making an accurate prediction. This makes it easier to process and reduces computational complexity. CNNs are also not sensitive to the object positioning in an image since the approximate position of image motifs relative to others is known. CNN deals with the variations in images through augmentation and other pre-processing techniques. This makes them robust to various straining conditions in images. This chapter presents an overview of a generic CNN architecture followed by a discussion on major advances in CNN models and their applications for cancer image analysis.

Automatic fetal brain extraction from 2D in utero fetal MRI slices using deep neural network

Land cover is an important variable of the terrestrial ecosystem that provides information for natural resources management, urban sprawl detection, and environment research. To classify land cover with high-spatial-resolution multispectral remote sensing imagery is a difficult problem due to heterogeneous spectral values of the same object on the ground. Fully convolutional networks (FCNs) are a state-of-the-art method that has been increasingly used in image segmentation and classification. However, a systematic quantitative comparison of FCNs on high-spatial-multispectral remote imagery was not yet performed. In this paper, we adopted the three FCNs (FCN-8s, Segnet, and Unet) for Gaofen-2 (GF2) satellite imagery classification. Two scenes of GF2 with a total of 3329 polygon samples were used in the study area and a systematic quantitative comparison of FCNs was conducted with red, green, blue (RGB) and RGB+near infrared (NIR) inputs for GF2 satellite imagery. The results showed that: (1) The FCN methods perform well in land cover classification with GF2 imagery, and yet, different FCNs architectures exhibited different results in mapping accuracy. The FCN-8s model performed best among the Segnet and Unet architectures due to the multiscale feature channels in the upsampling stage. Averaged across the models, the overall accuracy (OA) and Kappa coefficient (Kappa) were 5% and 0.06 higher, respectively, in FCN-8s when compared with the other two models. (2) High-spatial-resolution remote sensing imagery with RGB+NIR bands performed better than RGB input at mapping land cover, and yet the advantage was limited; the OA and Kappa only increased an average of 0.4% and 0.01 in the RGB+NIR bands. (3) The GF2 imagery provided an encouraging result in estimating land cover based on the FCN-8s method, which can be exploited for large-scale land cover mapping in the future.

Most current semantic segmentation approaches rely on deep convolutional neural networks (CNNs). However, their use of convolution operations with local receptive fields causes failures in modeling contextual spatial relations. Prior works have tried to address this issue by using graphical models or spatial propagation modules in networks. But such models often fail to capture long-range spatial relationships between entities, which leads to spatially fragmented predictions. In this work, we introduce a simple yet effective network unit, the spatial relation module, to learn and reason about global relationships between any two spatial positions, and then produce relation-enhanced feature representations. The spatial relation module is general and extensible, and can be used in a plug-and-play fashion with the existing fully convolutional network (FCN) framework. We evaluate spatial relation module-equipped networks on semantic segmentation tasks using two aerial image datasets. The networks achieve very competitive results, bringing significant improvements over baselines.

Automatic real-time crack detection using lightweight deep learning models

To facilitate the management of detonators, an automatic spotting system is proposed for detonator coded characters based on convolutional neural networks. The system contains a multi-scale detection network, a multi-label recognition network and a post-processing layer. An improved fully convolution network (FCN) is designed as the multi-scale detection network to obtain the accurate response map of the detonator image. Two subnetworks are parallel integrated into the FCN to perform a coarse-to-fine detection. An improved Jaccard loss function with a regularization term is defined to train the FCN. The region of interest (ROI) of the detonator image is achieved when the response map is post-processed by the post-processing layer. Finally, a modified multi-label network is used to recognize the detonator coded characters in the ROI. The experimental results indicate that the proposed system achieves a better spotting performance for detonator coded characters than the state-of-the-art text spotting methods in terms of accuracy and efficiency.

Computed tomography (CT) is commonly used as a diagnostic and treatment planning imaging modality in craniomaxillofacial (CMF) surgery to correct patient's bony defects. A major disadvantage of CT is that it emits harmful ionizing radiation to patients during the exam. Magnetic resonance imaging (MRI) is considered to be much safer and noninvasive, and often used to study CMF soft tissues (e.g., temporomandibular joint and brain). However, it is extremely difficult to accurately segment CMF bony structures from MRI since both bone and air appear to be black in MRI, along with low signal-to-noise ratio and partial volume effect. To this end, we proposed a 3D deep-learning based cascade framework to solve these issues. Specifically, a 3D fully convolutional network (FCN) architecture is first adopted to coarsely segment the bony structures. As the coarsely segmented bony structures by FCN tend to be thicker, convolutional neural network (CNN) is further utilized for fine-grained segmentation. To enhance the discriminative ability of the CNN, we particularly concatenate the predicted probability maps from FCN and the original MRI, and feed them together into the CNN to provide more context information for segmentation. Experimental results demonstrate a good performance and also the clinical feasibility of our proposed 3D deep-learning based cascade framework.

Semantic segmentation from aerial imagery is one of the most essential tasks in the field of remote sensing with various potential applications ranging from map creation to intelligence service. One of the most challenging factors of these tasks is the very heterogeneous appearance of artificial objects like buildings, cars and natural entities such as trees, low vegetation in very high-resolution digital images. In this paper, we propose an efficient deep learning approach to aerial image segmentation. Our approach utilizes the architecture of fully convolutional network (FCN) based on the backbone ResNet101 with additional upsampling skip connections. Besides typical color channels, we also use DSM and normalized DSM (nDSM) as the input data of our models. We achieve overall accuracy of 91%, which is in top 4 among 140 submissions from all over the world on the well-known Vaihingen dataset from ISPRS 2D Semantic Labeling Contest. Especially, our approach yields better results then all state-of-the-art methods in segmentation of car objects.

Glaucoma, the leading cause of irreversible blindness, must be diagnosed early and thus treated in time. However, it has no noticeable symptoms in its early stages and may not be detected easily. This paper aims to compare two well-known convolutional neural network (CNN) structures, namely Fully Convolutional Networks (FCNs) and U-Net for the segmentation of the optic disc (OD) and optic cup (OC) from retinal fundus images which play an important role in glaucoma diagnosis. The performance of both models is assessed using qualitative parameters such as the Dice coefficient, Jaccard index, and cup-to-disc ratio (CDR) error. In our experiment, the U-Net model yields more accurate segmentation results with 0.9601 average pixel accuracy and 0.9255 dice score for OD segmentation, outperforming the FCNs model with 0.9560 average pixel accuracy and 0.9132 dice score for OD segmentation. However, FCNs have a shorter inference time of 0. 0043 seconds against U-net’s 0. 0062 seconds making FCNs more suitable for real-time applications. The restrictions related to this study include biases from using only one dataset acquired from particular imaging devices, dependency on mask-based cropping techniques, and comparison being restricted to two fundamental architectures. This work presents the contribution of the deep learning models in improving glaucoma screening and therefore helping in avoiding blindness.

In technologically underdeveloped areas, water pollution threatens the living environment of local residents, so remote sensing monitoring of the features around reservoirs is necessary. Fully convolutional networks (FCNs) offer great potential for extracting high-resolution features due to their unlimited input image size and higher accuracy compared to convolutional neural networks. Therefore, a proposal to classify WorldView-2 images is implemented with a sixty-eight thousand iterations of fine-tuning and fully trained combined training method based on a fully convolutional network (SEFCN). The chosen images depict the urban area of Yingde, which is located to the northeast of the Feilaixia Reservoir, Qingyuan, Guangdong Province, China. The SEFCN combines an FCN-32s and FCN-16s to better integrate the deep features and shallow features and effectively improve the classification accuracy. Additionally, the loss value fully converges with enough iterations, and the overfitting caused by the superposition of two full trainings is avoided. The SEFCN model achieves the highest accuracy among all compared classification models and obtains the best classified results on the WorldView-2 images, as demonstrated by attaining the highest F1 scores in each category. After the classified images are optimized using a conditional random field, the current status of the study area is analyzed, and several suggestions for land use in the urban area of Yingde are made. The experiments still have deficiencies in the application of high-resolution remote sensing image classification with different sensors, and the classification results can be optimized and improved in other aspects.

Network traffic flow can be represented as time series. Outliers of these time series is important for the computer network security. The existing methods for the time series anomaly detection mainly operate on the entire time series to detect the abnormal series, seldom find out the position of the abnormal point. To solve this defect, we propose a method to discover the location of the outliers in time series based on a full convolutional network. We first map the time series into two-dimensional grayscale images by sliding window approach, and then feed them into the full convolutional network. In the classified stage, the feature maps of the last convolution layer are restored to the same size as the input image by up-sampling. Finally, we predict the class for each pixel and calculate the loss on a pixel-by-pixel basis. We tested the model on Yahoo Webscope S5 anomaly benchmark dataset, and the results show that our proposed method can achieve a mean pixel accuracy of 99.4% and mean IOU of 97% on the test dataset.

Understanding the spatial distribution of irrigated croplands is crucial for food security and water use. To map land cover classes with high-spatial-resolution images, it is necessary to analyze the semantic information of target objects in addition to the spectral or spatial–spectral information of local pixels. Deep convolutional neural networks (DCNNs) can characterize the semantic features of objects adaptively. This study uses DCNNs to extract irrigated croplands from Sentinel-2 images in the states of Washington and California in the United States. We integrated the DCNNs of 101 layers, discarded pooling layers, and employed dilation convolution to preserve location information; these are models which were used based on fully convolutional network (FCN) architectures. The findings indicated that irrigated croplands may be effectively detected at various phases of crop growth in the fields. A quantitative analysis of the trained models revealed that the three models in the two states had the lowest values of Intersection over Union (IoU) and Kappa, i.e., 0.88 and 0.91, respectively. The deep models’ temporal portability across different years was acceptable. The lowest values of recall and OA (overall accuracy) from 2018 to 2021 were 0.91 and 0.87, respectively. In Washington, the lowest OA value from 10 to 300 m resolution was 0.76. This study demonstrates the potential of FCNs + DCNNs approaches for mapping irrigated croplands across large regions, providing a solution for irrigation mapping. The spatial resolution portability of deep models could be improved further by designing model architectures.