Cloud Detection for Remote Sensing Images Based on Difference Features and Semantic Segmentation Network

Semantic segmentation has been proposed as a tool to accelerate the processing of natural history collection images. However, developing a flexible and resilient segmentation network requires an approach for adaptation which allows processing different datasets with minimal training and validation. This paper presents a cross-validation approach designed to determine whether a semantic segmentation network possesses the flexibility required for application across different collections and institutions. Consequently, the specific objectives of cross-validating the semantic segmentation network are to (a) evaluate the effectiveness of the network for segmenting image sets derived from collections different from the one in which the network was initially trained on; and (b) test the adaptability of the segmentation network for use in other types of collections. The resilience to data variations from different institutions and the portability of the network across different types of collections are required to confirm its general applicability. The proposed validation method is tested on the Natural History Museum semantic segmentation network, designed to process entomological microscope slides. The proposed semantic segmentation network is evaluated through a series of cross-validation experiments designed to test using data from two types of collections: microscope slides (from three institutions) and herbarium sheets (from seven institutions). The main contribution of this work is the method, software and ground truth sets created for this cross-validation as they can be reused in testing similar segmentation proposals in the context of digitization of natural history collections. The cross-validation of segmentation methods should be a required step in the integration of such methods into image processing workflows for natural history collections.

A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation

Moderate resolution imaging spectrometer (MODIS) images are widely used in land, ocean, and atmospheric monitoring, due to their wide spectral coverage, high temporal resolution, and convenient data acquisition. Accurate cloud detection is critical to the fine processing and application of MODIS images. Owing to spatial resolution limitations and the influence of mixed pixels, most MODIS cloud detection algorithms struggle to effectively recognize of clouds and ground objects. Here, we propose a novel cloud detection method based on land surface reflectance and a multiscale feature convolutional neural network to achieve high-precision cloud detection, particularly for thin clouds and clouds over bright surface. A monthly surface reflectance dataset was constructed by MODIS products (MOD09A1) and employed to provide background information for cloud detection. Difference-based samples were obtained using surface reflectance as well MODIS images of different phases based on difference operations. The multiscale feature network (MFCD-Net) using an atrous spatial pyramid pooling and a channel and spatial attention module integrated low-level spatial features and high-level semantic information to capture multiscale features and generate a high-precision cloud mask. For cloud detection experiments and quantitative analysis, 61 MODIS images acquired at different times on various underlying surface types were used. Cloud detection results were compared to those of UNet, Deeplabv3+, UNet++, PSPNet, and top of atmosphere-based (MFCD-TOA) methods. The proposed method performed well, with the highest overall accuracy (96.55%), precision (92.13%), and recall (88.90%). It improved cloud detection accuracy in various scenarios, reducing thin cloud omission and bright surface misidentification.

Linear feature extraction is crucial for special objects in semantic segmentation networks, such as slot marking and lanes. The objects with linear characteristics have global contextual information dependency. It is very difficult to capture the complete information of these objects in semantic segmentation tasks. To improve the linear feature extraction ability of the semantic segmentation network, we propose introducing the dilated convolution with vertical and horizontal kernels (DVH) into the task of feature extraction in semantic segmentation networks. Meanwhile, we figure out the outcome if we put the different vertical and horizontal kernels on different places in the semantic segmentation networks. Our networks are trained on the basis of the SS dataset, the TuSimple lane dataset and the Massachusetts Roads dataset. These datasets consist of slot marking, lanes, and road images. The research results show that our method improves the accuracy of the slot marking segmentation of the SS dataset by 2%. Compared with other state-of-the-art methods, our UnetDVH-Linear (v1) obtains better accuracy on the TuSimple Benchmark Lane Detection Challenge with a value of 97.53%. To prove the generalization of our models, road segmentation experiments were performed on aerial images. Without data argumentation, the segmentation accuracy of our model on the Massachusetts roads dataset is 95.3%. Moreover, our models perform better than other models when training with the same loss function and experimental settings. The experiment result shows that the dilated convolution with vertical and horizontal kernels will enhance the neural network on linear feature extraction.

Quantitative identification of sandstone microscopic images is an essential task for sandstone reservoir characterization. The widely-used classical Gazzi-Dickinson point-counting method can be subjective, inconsistent and time-consuming. Furthermore, by directly putting labeled microscopic images of all rock types into image recognition models for training, most previous studies did not address the petrographic principle of artificial identification. In this study, U-Net and U-Net++ semantic segmentation networks that incorporated the sandstone petrographic principle in quantitative identification of sandstone was introduced. Automatic identification of sandstone microscopic images requires prior knowledge learned from the identified sandstones with similar compositions. First, hundreds of thin-sections of typical sandstone reservoirs were selected from several key petroleum basins in China. Second, one-to-one single and orthogonal polarized images were taken for them. Third, the annotation software was used to label the type of each skeleton grain, including quartz, feldspar, lithic fragment and pore. Finally, 480 sets of data, each of which includes single and orthogonal polarized images and their ".json" format annotation results, were obtained for training and testing of the U-Net model to quantitatively analyze sandstone microscopic images. Within the 480 sets of data, 6798 sandstone skeleton grains, including 4542 quartzes, 796 feldspars, 1248 lithic fragments and 212 pores were labeled. The sandstone thin-section quantitative identification model trained by 392 data sets achieved a training accuracy of 96% with the intersection over union at 78% for quartz, and a training accuracy of 88% with the intersection over union at 56% for lithic fragments. The remaining 88 data sets were used for testing, and the accuracy was 87% with its intersection over union at 74% for quartz and a training accuracy of 77% with the intersection over union at 54% for lithic fragments. As a classic fully convolutional network that excels in processing medical images, the U-Net or U-Net++ semantic segmentation network has also performed very well in quantitative identification of sandstone microscopic images. After the proportion of each sandstone skeleton grain has been identified, the simple subdivision descriptive petrographic classification of the sandstone was determined according to the classic Dickinson sandstone taxonomic criteria. In other words, most current deep learning algorithms classify sandstones at the bulk rock level, but this U-Net model has been extended to the mineral level for comprehensive identification. Our vision-based sandstone lithology identification model has not only improved the accuracy of artificial identification but also reduced the instability and subjectivity of the traditional manual processing and expert decision-making approach. In the future, we plan to increase the number and coverage of labeled thin-section images to evaluate the impact on the accuracy and consistency of the U-Net or U-Net++ model, and to expand the approach to identify other terrigenous clastic rock. Furthermore, we hope to improve the capability of the model to identify grains, such as monocrystalline and polycrystalline quartz from "quartz", K-feldspar and plagioclase from "feldspar", and igneous, metamorphic and sedimentary lithic fragments from "lithic fragments".

In remote-sensing image (RSI) semantic segmentation, the dependence on large-scale and pixel-level annotated data has been a critical factor restricting its development. In this letter, we propose an unsupervised semantic segmentation network embedded with geometry consistency (UGCNet) for RSIs, which imports the adversarial-generative learning strategy into a semantic segmentation network. The proposed UGCNet can be trained on a source-domain dataset and achieve accurate segmentation results on a different target-domain dataset. Furthermore, for refining the remote-sensing target geometric representation such as densely distributed buildings, we propose a geometry-consistency (GC) constraint that can be embedded in both image-domain adaptation process and semantic segmentation network. Therefore, our model could achieve cross-domain semantic segmentation with target geometric property preservation. The experimental results on Massachusetts and Inria buildings datasets prove that the proposed unsupervised UGCNet could achieve a very comparable segmentation accuracy with the fully supervised model, which validates the effectiveness of the proposed method.

Semantic segmentation network is able to detect multi-scale traffic sign effectively. However, probabilities that different scale features participate in decision are equal in the network. In this work, we propose a local multi-scale attention module based on channel-attention, which extracts local features generated by dilated convolution and weights scale features. We demonstrate that adopting local multi-scale attention module on a semantic segmentation network achieves a better performance for locating multi-scale traffic signs than state-of-art convolutional networks by the experimental results on a Chinese traffic sign dataset.

Vision-based perception is an import link in driverless technology, and semantic segmentation is one of the mostly uses atrous convolution with a large amount of computation and high memory consumption to extract high-resolution feature maps. As a result, the current mainstream semantic segmentation network lacks the segmentation speed and cannot be effectively applied in driverless technology. To solve this problem, a semantic segmentation network with better real-time performance is proposed. Firstly, a lightweight convolutional neural network is used as the encoder, and stride convolution and regular convolution are used to replace the time-consuming and memory-consuming atrous convolution. Secondly, in order to obtain the feature map similar to Deeplab v3+, a new joint upsampling module, Multi-scale Feature Map Joint Pyramid Upsamping(MJPU), is proposed. By fusing multiple feature maps in the encoder, a high resolution feature map with richer semantic information is generated. Experments on the Cityscapes dataset show that, compared with the popular semantic segmentation network Deeplab V3+, the proposed network can improve the segmentation speed to 31.4 FPS/s without losing a lot of performance(mIoU of 44.03%). Therefore, the network proposed in this paper has better real-time performance and is more suitable for driverless scenes.

The lane line detection methods based on semantic segmentation networks have achieved remarkable results in recent years. However, the semantic segmentation networks are aimed at the pixel level. These methods have a large amount of computation, which can reduce the real-time performance. To reduce the calculation, we propose a Light-weight Lane line detection Network (LLNet). A new sub-layer is established. We also adopt a jumping structure between two sub-layers to enhance the supervisory role of ground truth. Furthermore, we adopt two branches, including instance segmentation and embeddable branch. The combination of two branches can filter out the wrongly detected pixels and further improve the accuracy of detection. Experimental results in the Tusimple dataset show that the detection accuracy of the proposed network is comparable with LaneNet. Meanwhile, it has good real-time performance, and the processing time of a single image is 10.285 ms, which is about one-half of LaneNet.

Semantic segmentation of high-resolution optical remote sensing images is an important but challenging task. To solve the problem that many semantic segmentation networks fail to efficiently utilize global and local context information to improve the segmentation performance, this paper proposes a semantic segmentation network based on sparse self-attention (SDANet) to model the global context dependencies. Specifically, the feature maps are first divided into four regions in spatial and channel dimensions, respectively, and the divided feature maps are rearranged to form new regions. Second, the position and channel self-attention operations are performed on the rearranged regions. Third, the feature maps are restored to the original combination and the position together with channel self-attention operations are performed again to obtain the output feature maps. Finally, semantic segmentation is completed based on the output feature maps. Extensive experiments conducted on the ISPRS Vaihingen dataset demonstrate that the proposed method is superior to self-attention-based DANet, CCNet, and other general semantic segmentation networks, such as FCN, Deeplabv3+, HRNet, etc.

Semantic segmentation plays an increasingly important role in the field of computer vision. The current semantic segmentation algorithms are mainly based on full convolutional network downsampling for feature extraction, however, the reduction of image resolution due to downsampling will inevitably lead to the loss of image information and the process is irreversible. Although features can be reused to compensate for the loss, the problem of lost pixel correlation and inaccurate segmentation results still occur. In order to solve these problems, the attention mechanism and the generative adversarial network are fused into the semantic segmentation network, and a semantic segmentation network is proposed. The attention mechanism is added to the generator and discriminator modules and trained successively, then the generator module with rich information and global information is extracted, and its output is fused with the feature graph sampled above, in order to reduce the loss of local information and complete the task of image segmentation, an interpolation method based on image edge is introduced to sample the image, which improves the bilinear interpolation edge blur. Experiments on Cityscapes dataset show that the proposed network model is effective and reliable.

In most parts of the electromagnetic spectrum, solar radiation cannot penetrate clouds. Therefore, cloud detection and masking are essential in image preprocessing for observing the Earth and analyzing its properties. Because clouds vary in size, shape, and structure, an accurate algorithm is required for removing them from the area of interest. This task is usually more challenging over bright surfaces such as exposed sunny deserts or snow than over water bodies or vegetated surfaces. The overarching goal of the current study is to explore and compare the performance of three Convolutional Neural Network architectures (U-Net, SegNet, and DeepLab) for detecting clouds in the VENμS satellite images. To fulfil this goal, three VENμS tiles in Israel were selected. The tiles represent different land-use and cover categories, including vegetated, urban, agricultural, and arid areas, as well as water bodies, with a special focus on bright desert surfaces. Additionally, the study examines the effect of various channel inputs, exploring possibilities of broader usage of these architectures for different data sources. It was found that among the tested architectures, U-Net performs the best in most settings. Its results on a simple RGB-based dataset indicate its potential value for any satellite system screening, at least in the visible spectrum. It is concluded that all of the tested architectures outperform the current VENμS cloud-masking algorithm by lowering the false positive detection ratio by tens of percents, and should be considered an alternative by any user dealing with cloud-corrupted scenes.

Semantic segmentation is a fundamental task in remote sensing image processing. It provides pixel-level classification, which is important for many applications, such as building extraction and land use mapping. The development of convolutional neural network has considerably improved the performance of semantic segmentation. Most semantic segmentation networks are the encoder–decoder structure. Bilinear interpolation is an ordinary upsampling method in the decoder, but bilinear interpolation only considers its own features and inserts three times its own features. This over-simple and data-independent bilinear upsampling may lead to suboptimal results. In this work, we propose an upsampling method based on local relations to replace bilinear interpolation. Upsampling is performed by correlating the local relationship of feature maps of adjacent stages, which can better integrate local and global information. We also design a fusion module based on local similarity. Our proposed method with ResNet101 as the backbone of the segmentation network can improve the average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F_{1}$ </tex-math></inline-formula> score and overall accuracy of the Vaihingen data set by 2.69% and 1.31%, respectively. Our proposed method also has fewer parameters and less inference time.

PROBA-V (PRoject for On-Board Autonomy–Vegetation) was launched in May-2013 as an operational continuation to the vegetation (VGT) instruments on-board the Système Pour l’Observation de la Terre (SPOT)-4 and -5 satellites. The first reprocessing campaign of the PROBA-V archive from Collection 0 (C0) to Collection 1 (C1) aims at harmonizing the time series, thanks to improved radiometric and geometric calibration and cloud detection. The evaluation of PROBA-V C1 focuses on (i) qualitative and quantitative assessment of the new cloud detection scheme; (ii) quantification of the effect of the reprocessing by comparing C1 to C0; and (iii) evaluation of the spatio-temporal stability of the combined SPOT/VGT and PROBA-V archive through comparison to METOP/advanced very high resolution radiometer (AVHRR). The PROBA-V C1 cloud detection algorithm yields an overall accuracy of 89.0%. Clouds are detected with very few omission errors, but there is an overdetection of clouds over bright surfaces. Stepwise updates to the visible and near infrared (VNIR) absolute calibration in C0 and the application of degradation models to the SWIR calibration in C1 result in sudden changes between C0 and C1 Blue, Red, and NIR TOC reflectance in the first year, and more gradual differences for short-wave infrared (SWIR). Other changes result in some bias between C0 and C1, although the root mean squared difference (RMSD) remains well below 1% for top-of-canopy (TOC) reflectance and below 0.02 for the normalized difference vegetation index (NDVI). Comparison to METOP/AVHRR shows that the recent reprocessing campaigns on SPOT/VGT and PROBA-V have resulted in a more stable combined time series.

Accurate and rapid cloud detection is exceedingly significant for improving the downlink efficiency of on-orbit data, especially for the microsatellites with limited power and computational ability. However, the inference speed and large model limit the potential of on-orbit implementation of deep-learning-based cloud detection method. In view of the above problems, this paper proposes a lightweight network based on depthwise separable convolutions to reduce the size of model and computational cost of pixel-wise cloud detection methods. The network achieves lightweight end-to-end cloud detection through extracting feature maps from the images to generate the mask with the obtained maps. For the visible and thermal infrared bands of the Landsat 8 cloud cover assessment validation dataset, the experimental results show that the pixel accuracy of the proposed method for cloud detection is higher than 90%, the inference speed is about 5 times faster than that of U-Net, and the model parameters and floating-point operations are reduced to 12.4% and 12.8% of U-Net, respectively.