Unsupervised Deep Joint Segmentation of Multitemporal High-Resolution Images

Change detection is usually treated as a problem of explicitly detecting land cover transitions in satellite images obtained at different times, and helps with emergency response and government management. This study presents an unsupervised change detection method based on the image fusion of multi-temporal images. The main objective of this study is to improve the accuracy of unsupervised change detection from high-resolution multi-temporal images. Our method effectively reduces change detection errors, since spatial displacement and spectral differences between multi-temporal images are evaluated. To this end, a total of four cross-fused images are generated with multi-temporal images, and the iteratively reweighted multivariate alteration detection (IR-MAD) method—a measure for the spectral distortion of change information—is applied to the fused images. In this experiment, the land cover change maps were extracted using multi-temporal IKONOS-2, WorldView-3, and GF-1 satellite images. The effectiveness of the proposed method compared with other unsupervised change detection methods is demonstrated through experimentation. The proposed method achieved an overall accuracy of 80.51% and 97.87% for cases 1 and 2, respectively. Moreover, the proposed method performed better when differentiating the water area from the vegetation area compared to the existing change detection methods. Although the water area beneath moderate and sparse vegetation canopy was captured, vegetation cover and paved regions of the water body were the main sources of omission error, and commission errors occurred primarily in pixels of mixed land use and along the water body edge. Nevertheless, the proposed method, in conjunction with high-resolution satellite imagery, offers a robust and flexible approach to land cover change mapping that requires no ancillary data for rapid implementation.

A new method is proposed for change analysis with weight of significance between two multi-temporal multi-spectral images.This method gives us areas which indicate the assigned temporal change, for example, from vegetation to bare soil. Image dataare projected onto a feature space in which the assigned change is emphasized, and temporal changes between two images aredetected with suppression ofirrelevant changes. The validity ofthe method is confirmed by numerical simulation. The methodis successfully applied to actual multi-temporal and multi-spectral images. 1. INTRODUCTION Change analysis is one of the most important processings for monitoring environment. It provides us with information aboutchange, for example, of natural environment to urban area. In change analysis, it is required to analyze the particular changerather than to detect all types of change. There have been a lot of methods reported for change detection[1]. Most of them,however, detect all types of change without specifying the change[2], or analyze the changes categorically using supervisedclassification of two multi-temporal images[3][4]. In the former, the change to be analyzed is detected indirectly using cluster-ing, and in the latter, classification error is serious. A direct method has been required.We propose a method PACE (Particular Change Extractor) for analyzing temporal change with weight of significance. Themethod PACE enables us to select a particular type of change such as from vegetation to bare soil, and extracts areas in whichspectral pattern has the particular change between two multi-temporal and multi-spectral images. The method consists of threeprocedures; data projection, change detection and suppression of irrelevant changes. In the first two, we produce a feature spacederived from an assigned change so that the change is emphasized. Canonical correlation analysis[5] is used for it. We projecta set of image data onto the feature space. The difference between two transformed image data indicates temporal changes. As

This paper presents a novel method for segmentation and change detection of multispectral images using proximal splitting-based clustering and multiclass support vector machine (MSVM). Initially, the multitemporal satellite images are preprocessed and then textures are extracted using Difference of Offset Gaussian filter. In general, the traditional clustering method uses Euclidean distance as a prime factor for segmentation process. For multitextured images such as remotely sensed images, this metric provides inconsistent output. To achieve better segmented results, proximal splitting algorithm has been proposed. This method has been considered as a solution for iterative minimization problem, which is required to find exact changes between the multitemporal images. The MSVM is chosen to group the segmented clusters into a fixed number of classes, since the clusters obtained from the proximal splitting algorithm are not independent with each other. Then, the classified images are subjected to image differencing method to detect the changes. Experimentation is performed with two real data sets of Landsat7 images, which illustrates that the mean of difference in area obtained by the proposed method is reduced by an average of 35.24% compared to the conventional system. The validity index obtained for data set 1 using proposed algorithm is lower than the existing methods.

In this paper a change detection study was conducted using multi-temporal images from two commonly used sensors, MODIS and TM, between June and October, 2003 over Southern Ontario, Canada to evaluate the sensitivity of MODIS images for seasonal land cover changes. Post-classification change detection was used to determine the type of change that had occurred and allow for from-to types of changes to be evaluated. NDVI image differencing was also performed on the MODIS and TM images to compare the vegetation index changes at different spatial resolutions. It was found that MODIS classifications approximated those produced with TM data only when incorporating the thermal band in the classification procedure which takes advantage of the urban heat island effect. Results demonstrate that MODIS post-classification change detection can approximate the levels of change/no-change compared to TM post-classification however the type of change was not accurate due to the spectral mixing that occurs at the coarser 250 meter spatial resolution of MODIS data. The more change at TM level for a MODIS pixel, the higher the likelihood of this corresponding to change at the MODIS level. This study demonstrates that MODIS data would be best suited for detecting changes in large agricultural areas with large field size of homogeneous crop type and growth stage or large areas of forest stands with similar characteristics.

SAR data allow regular monitoring of target areas since their acquisition is not affected by weather or light condition problems. This characteristic makes them optimal for civil protection tasks, such as earthquake damage assessment, where quick responses in any weather and light conditions are needed. Change detection (CD) methods address this application by identifying changes over an analyzed area using bi-temporal or multi-temporal SAR images. These methods detect the changes using features that provide useful information about the changes. In the last years, CD methods exploiting deep learning (DL) models have become popular because of their capability to learn features from the input data. Many DL CD methods exploit supervised models to identify changes using labeled bi-temporal or multi-temporal data to train the model. However, the gathering of labeled multi-temporal data is challenging. Transfer learning methods avoid the supervised training and the labeled data gathering by exploiting pre-trained models as feature extractors for CD tasks. They achieve good results when the target images are similar to the images used to pre-train the model. We focus this chapter on the unsupervised DL CD methods that exploit unsupervised DL models to extract feature maps. Unsupervised DL models automatically learn features from unlabeled samples during the training. The feature maps retrieved from these models are used to detect changed areas in multi-temporal images and handle the speckle noise.

Time series analysis of multispectral satellite images using game theory classifier

Segmentation is a low-level processing aimed at the partition of an image in disjoint regions, each one homogeneous with respect to some properties like intensity, texture, shape, etc. Such a task is needed in many high-level processing and applications in such diverse fields as remote-sensing, medical imaging, video coding, and industrial automation, just to name a few. In this work of thesis, we first deal with the problem of color based image segmentation, that is based on some form of homogeneity in the color/spectral properties of the image, and then move to the more complex task of texture based segmentation, where the aim is to recognize complex structures in the image which are typically non homogeneous in terms of spectral properties. Concerning color based segmentation, we focus on a specific class of Markov Random Field (MRF) probability models, namely the Tree-Structured Markov Random Fields (TS-MRF), recently proposed to provide a hierarchical multiscale description of images. Based on such a model, the unsupervised image segmentation is carried out by means of a sequence of nested class splits, where each class is modeled as a local binary MRF. As a first contribution, we first extend the model to generic tree structures removing the unnecessary binary constraint, and then devise a new TS-MRF unsupervised segmentation technique which improves upon the original algorithm by selecting a better tree structure and eliminating spurious classes. Such results are obtained by using the Mean-Shift procedure to estimate the number of pdf modes at each node, and to obtain a more reliable initial clustering for subsequent MRF optimization. To this end, we devise a new reliable and fast clustering algorithm based on the Mean-Shift technique, which makes use of a variable-bandwidth strategy based on the k-Nearest Neighbors (k-NN) technique, and is implemented with a speed-up strategy that cuts significantly the computational complexity. Experimental results on synthetic data and real life images from the remote sensing domain prove the potential of the proposed method. Turning to texture based segmentation, we present a novel multiscale texture model, and a related algorithm for the unsupervised segmentation of color images. Elementary textures are characterized by their spatial interactions with neighboring regions along selected directions. Such interactions are modeled in turn by means of a set of Markov chains, one for each direction, whose parameters are collected in a feature vector that synthetically describes the texture. Based on the feature vectors, the texture are then recursively merged, giving rise to larger and more complex textures, which appear at different scales of observation: accordingly, the model is named Hierarchical Multiple Markov Chain (H-MMC). The Texture Fragmentation and Reconstruction (TFR) algorithm, addresses the unsupervised segmentation problem based on the H-MMC model. The ``fragmentation'' step allows one to find the elementary textures of the model, while the ``reconstruction'' step defines the hierarchical image segmentation based on a probabilistic measure (texture score) which takes into account both region scale and inter-region interactions. The performance of the proposed method was assessed through the Prague segmentation benchmark, based on mosaics of real natural textures, and also tested on real-world natural images from the Berkeley dataset. Finally, an application of the TFR algorithm for the unsupervised segmentation of multiresolution remotely sensed images is proposed, based on the significant presence of textural content in last generation high-resolution satellite images. The obtained results further assess the potential of the proposed method for such real life applications.

Rapid urbanization took place in the Pearl River Delta of south China since 1980. Although drastic land use change took place in very short interval within this area, hardly any research has been done on this phenomenon for lacking of available data. Remote sensing is presently the most favorable observation method for land use and land cover change (LUCC) researches. While located in the south of China, the Pearl River Delta suffers from heavy cloud cover for more than half of the year. This makes real-time LUCC monitoring and change detection almost impossible with commonly used optical remote sensing data. In this paper, the orbital highest resolution SAR (Synthetic Aperture Radar) data - Fine Mode Radarsat data was used for trail of short-term land use change detection. Three scenes of repeat-pass Radarsat data was collected over the study area. Although repeat-pass Radarsat enable continuous land use monitoring under all weather condition, images acquired during different time are inevitably affected by seasonal land cover change and variable environmental status such as air humidity and raining. Besides, some significant observation bias might be induced because of the platform and sensor instability. All these variations and instability made short-term land use change detection quite a perplex problem. In this paper, short-term land use change caused by human activity was considered as abnormal phenomena in both spatial and temporal domain in time series images. And a Density-based Anomaly Detection (DBAD) algorithm was designed to detect abnormally changed land parcels in time series Radarsat images. Firstly, totally 3 scenes of fine mode Radarsat images were collected in the study area from January 1st to May 3rd, 2006. Simply stacked temporal images reveal apparent backscattering variation between the three scenes of images, which mainly owes to the fast vegetable growth during the observation period. Then image segmentation was done on the multi-temporal Radarsat images and object features including mean value of backscattering coefficient (Mean), minimal value of backscattering (Min), homogeneity of gray level co-occurrence matrix (GLCMhomo) and dissimilarity of gray level co-occurrence matrix (GLCMdis) were extracted basing on segmented image objects. After that change- vector was constructed for each land objects. In the third step DBAD algorithm was applied to the change vector dataset to detect anomaly change in the 3 scenes of images. Finally field surveying data plus manual interpretation were used for validation. Comparing with object-based image regression method, DBAD results in better accuracy. Besides, data validation also shows that DBAD have better accuracy in both under-constructed area and newly built up area (error lower than 12%). While for built up area and some mixed used area, it gains relatively lower accuracy than other land types (from 10% to 28.57%). To conclude, short-term land use change in time series images could be defined as spatial and temporal anomaly in remote sensing images. By extending traditional anomaly detection to spatial-temporal anomaly detection, land use change caused by human activity could be effectively detected during short time intervals. The algorithm DBAD focus only on the density of change vectors in feature space, which is independent of the amplitude and direction of change vectors. This enable DBAD effectively discriminate temporal image variation caused by observation system, environment or seasonal land cover change, especially in vegetation and cultivated area which changed remarkably during the observation period, from land use change caused by human activities. This helps to decrease the false alarming in short-term change detection. © 2008 SPIE.

The accurate land cover change detection is critical to improve the landscape dynamics analysis and mitigate desertification problems efficiently. Desertification detection is a challenging problem because of the high degree of similarity between some desertification cases and like-desertification phenomena, such as deforestation. This article provides an effective approach to detect deserted regions based on Landsat imagery and variational autoencoder (VAE). The VAE model, as a deep learning-based model, has gained special attention in features extraction and modeling due to its distribution-free assumptions and superior nonlinear approximation. Here, a VAE approach is applied to spectral signatures for detecting pixels affected by the land cover change. The considered features are extracted from multitemporal images and include multispectral information, and no prior image segmentation is required. The proposed method was evaluated on the publicly available remote sensing data using multitemporal Landsat optical images taken from the freely available Landsat program. The arid region around Biskra in Algeria is selected as a study area since it is well-known that desertification phenomena strongly influence this region. The VAE model was evaluated and compared with restricted Boltzmann machines, deep learning model, and binary clustering algorithms, including Agglomerative, BIRCH, expected maximization, k-mean clustering algorithms, and one-class support vector machine. The comparative results showed that the VAE consistently outperformed the other models for detecting changes to the land cover, mainly deserted regions. This study also showed that VAE outperformed the state-of-the-art algorithms.

Automatic change detection is an interesting research area in remote sensing (RS) technology aims to detect the changes in synthetic aperture radar (SAR) and multi-temporal hyperspectral images acquired at different time intervals. This method identifies the differences between the images and accomplishes the classification result into changed and unchanged areas. However, the existing algorithms are degraded due to noises present in the RS images. The main aim of the proposed method is the automatic semantic segmentation based change detection that produces a final change between the two input images. This paper proposes a feature learning method named deep lab dilated convolutional neural network (DL-DCNN) for the detection of changes from the images. The proposed approach consists of three stages: (i) pre-processing, (ii) semantic segmentation based change detection and (iii) accuracy assessment. Initially, preprocessing is performed to correct the errors and to obtain detailed information from the scene. Then, map the changes between the two images with the help of a trained network. The DCNN network performs fine-tuning and determines the relationship between two images as changed and unchanged pixel areas. The experimental analysis conducted on various datasets and compared with several existing algorithms. The experimental analysis is performed in terms of F-score, percentage correct classification, kappa coefficient, and overall error rate measures to show a better performance measure than the other state-of-art approaches.

In order to enhance the image understanding of different regions for national costume grayscale image automatic colorization, let coloring tasks take advantage of semantic conditions, also let it can apply the human parsing semantic segmentation method to the national costume grayscale image for semantic segmentation task. This paper proposes a semantic segmentation model for context embedding based on edge perceiving. Aiming at the features of national costume grayscale image, more optimizing the model and loss function. The national costume grayscale image semantic segmentation is different from semantic segmentation of the color image, this task is more difficult for the grayscale image has no color feature. In this paper, edge information and edge consistency constraints are used to improve the national costume grayscale image coloring effect. The experimental results show that the model designed in this paper can obtain more accurate fine-grained semantic segmentation results for the national costume grayscale image.

Unsupervised segmentation of hyperspectral images is an important problem in the field of image processing with many applications, including scene labelling of remote-sensing images and object detection in medical images. However, the high-dimensionality of hyperspectral images poses considerable challenges in this problem. In previous studies, to deal with high-dimensionality, band selection has been performed as a separate step before image segmentation. In this approach, however, information loss can occur because band selection methods do not consider the segmentation task when they select the bands. In addition, in most segmentation tasks, the fact that the number of segments of the image needs to be given before the segmentation task is also an issue. To address these challenges, we propose a method for simultaneous segmentation of hyperspectral images and selection of important bands that are highly involved in the segmentation task. The proposed method, called a mixture of finite maximum margin mixtures, combines a mixture of finite mixtures, which allows the number of segments to be automatically inferred from the data, and a maximum margin classifier, which allows a flexible segmentation and tractable inference on the segmentation labels. Furthermore, we simultaneously select useful bands that help with the segmentation during the segmentation procedure by introducing a latent binary vector that represents discriminating or non-discriminating bands. We demonstrate the advantages of the proposed method using four real datasets.

Abstract. Unsupervised and self-supervised deep learning networks for semantic segmentation of images have made impressive progress in the last years. They can be trained without any labelled data and yet are able to effectively segment RGB images into meaningful semantic groups. In remote sensing, supplementary information, such as elevation, improves class separation by differentiating classes based to their height above ground. We take SmooSeg, a recently developed, state-of-the-art unsupervised network for semantic segmentation, and guide its training process by infusing elevation information into its projector and smoothness prior. This ensures global label consistency across the entire dataset and improves the segmentation performance, since patches of the same semantic group often exhibit similar elevation characteristics. We also extend the Conditional Random Field (CRF) to refine the low-resolution segmentation results in a post-processing step with elevation information. We introduce a second pairwise potential that encourages neighboring pixels with similar elevation to have the same label, ensuring local label consistency. Our multi-modal training strategy remains unsupervised and improves the segmentation performance on the ISPRS Potsdam-3 dataset by +4.0% in mIoU over the RGB-only SmooSeg baseline and by 4.4% when also using the multi-modal CRF post-processing. Collectively, our approach surpasses all state-of-the-art unsupervised segmentation networks that rely solely on RGB data for the Potsdam-3 dataset, highlighting the important role of elevation data in label-free segmentation for remote sensing applications.

Chapter 12 - Semantic scene segmentation for robotics

We propose a novel approach to investigate and implement unsupervised image content understanding and segmentation of color industrial images like medical imaging, forensic imaging, security and surveillance imaging, biotechnical imaging, biometrics, mineral and mining imaging, material science imaging, and many more. In this particular work, our focus will be on medical images only. The aim is to develop a computer aided diagnosis (CAD) system based on a newly developed Multidimensional Spatially Variant Finite Mixture Model (MSVFMM) using Markov Random Fields (MRF) Model. Unsupervised means automatic discovery of classes or clusters in images rather than generating the class or cluster descriptions from training image sets. The aim of this work is to produce precise segmentation of color medical images on the basis of subtle color and texture variation. Finer segmentation of images has tremendous potential in medical imaging where subtle information related to color and texture is required to analyze the image accurately. In this particular work, we have used CIE-Luv and Daubechies wavelet transforms as color and texture descriptors respectively. Using the combined effect of a CIE-Luv color model and Daubechies transforms, we can segment color medical images precisely in a meaningful manner. The evaluation of the results is done through comparison of the segmentation quality with another similar alternative approach and it is found that the proposed approach is capable of producing more faithful segmentation.