Satellite image time series clustering under collaborative principal component analysis

Satellite Image Time Series (SITS) is a data set that includes satellite images across several years with a high acquisition rate. Radiometric normalization is a fundamental and important preprocessing method for remote sensing applications using SITS due to the radiometric distortion caused by noise between images. Normalizing the subject image based on the reference image is a general strategy when using traditional radiometric normalization methods to normalize multi-temporal imagery (usually two or three scenes in different time phases). However, these methods are unsuitable for calibrating SITS because they cannot minimize the radiometric distortion between any pair of images in SITS. The existing relative radiometric normalization methods for SITS are based on linear assumptions, which cannot effectively reduce nonlinear radiometric distortion caused by continuously changing noise in SITS. To overcome this problem and obtain a more accurate SITS, we propose a nonlinear radiometric normalization model (NMAG) for SITS based on Artificial Neural Networks (ANN) and Greedy Algorithm (GA). In this method, GA is used to determine the correction order of SITS and calculate the error between the image to be corrected and normalized images, which avoids the selection of a single reference image. ANN is used to obtain the optimal solution of error function, which minimizes the radiometric distortion between different images in SITS. The SITS composed of 21 Landsat-8 images in Tianjin, China, from October 2017 to January 2019 was selected to test the method. We compared NMAG with other two contrast methods (Contrast Method 1 (CM1) and Contrast Method 2 (CM2)), and found that the average root mean square error (μRMSE) of NMAG (497.22) is significantly smaller than those of CM1 (641.39) and CM2 (543.47), and the accuracy of normalized SITS obtained using NMAG increases by 22.4% and 8.5% compared with CM1 and CM2, respectively. These experimental results confirm the effectiveness of NMAG in reducing radiometric distortion caused by continuously changing noise between images in SITS.

Satellite Image Time Series (SITS) have become more accessible in recent years and SITS analysis has attracted increasing research interest. Given that labeled SITS training samples are time and effort consuming to acquire, clustering or unsupervised analysis methods need to be developed. Similarity measure is critical for clustering, however, currently established methods represented by Dynamic Time Warping (DTW) still exhibit several issues when coping with SITS, such as pathological alignment, sensitivity to spike noise, and limitation on capacity. In this paper, we introduce a new time series similarity measure method named time adaptive optimal transport (TAOT) to the application of SITS clustering. TAOT inherits several promising properties of optimal transport for the comparing of time series. Statistical and visual results on two real SITS datasets with two different settings demonstrate that TAOT can effectively alleviate the issues of DTW and further improve the clustering accuracy. Thus, TAOT can serve as a usable tool to explore the potential of precious SITS data.

Combining Sentinel-1 and Sentinel-2 Satellite Image Time Series for land cover mapping via a multi-source deep learning architecture

A spatiotemporal cube model for analyzing satellite image time series: Application to land-cover mapping and change detection

The dense Satellite Images Time Series (SITS) plays an important role in the agriculture semantic segmentation task. However, in real-world scenarios, cloud contamination and temporary sensor outages can lead to significant data missing in SITS, which declines the performance of models trained on ideal scenarios. A common approach is to reconstruct the complete SITS before the model’s prediction, where the reconstruction is independent of the prediction. This approach not only leads to the error accumulation from reconstruction to prediction, but also the detailed rebuilding of complete SITS may be redundant for the prediction. In this paper, we proposed a features reconstruction and prediction joint learning framework. The collaborative optimization of the two tasks aims to encourage the model to efficiently reconstruct complete features beneficial for prediction from incomplete SITS. Specifically, we simulate the data-missing scenarios with masks. The prediction task of masked data is supervised by labels. Meanwhile, by using the model that is well-trained on ideal scenarios as a teacher, we leverage its extracted temporal features from the data before masking as the target of the feature reconstruction task. The gradient flow of two tasks will be shared, which enables mutual supervision between them. Feature reconstruction prevents the model from acquiring incorrect reasoning ability caused by the shortest path problem during prediction, whereas prediction keeps reliability and reduces redundancy of reconstructed information. Furthermore, after training with the proposed framework, the model architecture remains unchanged and still maintains its robustness of complete SITS, which enhances the model's feasibility in practical applications. The experiments were conducted across multiple agricultural semantic segmentation datasets with incomplete SITS, sourced from Sentinel-2 and Planet satellites. We also validate its robustness for the common model architectures, and visualize the intermediate features to explore the mutual influence between the two tasks.

Estimating wheat yields in Australia using climate records, satellite image time series and machine learning methods

Mapping Earth land use and cover changes is crucial to understand agricultural dynamics. Recently, analysis of time series extracted from Earth observation satellite images has been widely used to produce land use and cover information. In time series analysis, clustering is a common technique performed to discover patterns on data sets. In this work, we evaluate the Growing Self-Organizing Maps algorithm for clustering satellite image time series and compare it with Self-Organizing Maps algorithm. This paper presents a case study using satellite image time series associated to samples of land use and cover classes, highlighting the advantage of providing a neutral factor (called spread factor) as a parameter for GSOM, instead of the SOM grid size.

Abstract. Semantic segmentation is essential in the field of remote sensing because it is used for various applications such as environmental monitoring and land cover classification. Recent advancements aim to collectively classify data from diverse sensors and epochs to improve predictive accuracy. With the availability of vast Satellite Image Time Series (SITS) data, supervised deep learning methods, such as Transformer models, become viable options. This paper introduces the Temporal Vision Transformer(ViT), designed to extract features from SITS. These features, capturing the temporal patterns of land cover classes, are integrated with features derived from aerial imagery to improve land cover classification. Drawing inspiration from the success of transformers in Natural language processing (NLP), Temporal ViT concurrently extracts spatial and temporal information from SITS data using tailored positional encoding strategies. The proposed approach fosters comprehensive feature learning across both domains, facilitating seamless integration of encoded data from SITS into aerial images. Furthermore, a training strategy is proposed that supports the Temporal ViT to focus on classes with a changing appearance over the year. Extensive experiments carried out in this work indicate the enhanced classification performance of Temporal ViT compared to existing state-of-the-art techniques for multi-modal land cover classification. Our model achieves a 3.8% increase in the mean IoU compared to the network solely relying on aerial images.

Satellite Image Time Series (SITS) provide us with precious information on land cover evolution. By studying these series of images we can both understand the changes of specific areas and discover global phenomena that spread over larger areas. Changes that can occur throughout the sensing time can spread over very long periods and may have different start time and end time depending on the location, which complicates the mining and the analysis of series of images. This work focuses on frequent sequential pattern mining (FSPM) methods, since this family of methods fits the above-mentioned issues. This family of methods consists of finding the most frequent evolution behaviors, and is actually able to extract long-term changes as well as short term ones, whenever the change may start and end. However, applying FSPM methods to SITS implies confronting two main challenges, related to the characteristics of SITS and the domain's constraints. First, satellite images associate multiple measures with a single pixel (the radiometric levels of different wavelengths corresponding to infra-red, red, etc.), which makes the search space multi-dimensional and thus requires specific mining algorithms. Furthermore, the non evolving regions, which are the vast majority and overwhelm the evolving ones, challenge the discovery of these patterns. We propose a SITS mining framework that enables discovery of these patterns despite these constraints and characteristics. Our proposal is inspired from FSPM and provides a relevant visualization principle. Experiments carried out on 35 images sensed over 20 years show the proposed approach makes it possible to extract relevant evolution behaviors.

In this paper, we present PaTSI, a tool for analyzing evolutions of objects in time series of satellite images. This tool is a plugin integrated in the KNIME Analytics Platform. PaTSI is a workflow composed of several nodes assembled together to form a whole KDD process (data selection, pre-processing, image segmentation, pattern mining and visualization). Input data consists of a time series of satellite images and GIS information on the studied area. This data is transformed in a single attributed directed acyclic graph (a-DAG), where nodes represent objects described by several attributes and edges represent temporal relationships. This graph is then mined to extract frequent evolutions (weighted path patterns) using an efficient graph mining algorithm. At the end of the process, extracted patterns can be filtered using regular expressions and displayed on the original images in order to facilitate the experts' interpretation of the results. In the present demo, the pertinence of PaTSI is illustrated through its application to soil erosion monitoring.

Temporal trajectories of land-cover change provide important information on landscape dynamics that are critical to our understanding of complex human–environment adaptive systems. The increasing availability of long time series of satellite images, especially the recent free release of multi-decadal Landsat satellite archive, presents a great opportunity to improve our ability to detect land-cover change over multiple dates and advance land change science. In this article, a spatial-temporal modeling approach is developed for reconstructing land-cover change trajectories from time series of satellite images. The change detection method represents an enhancement to the conventional post-classification comparison. The key innovation lies in the use of Markov random field theory to model spatial-temporal contextual information explicitly in the classification of time series images. When evaluated using a time series of seven Landsat images in a case study of southeast Ohio, the spatial-temporal modeling approach yielded significantly more accurate and consistent trajectories of land-cover change than conventional non-contextual approaches. The results from the case study demonstrate the effectiveness of the change detection method in reconstructing land-cover change trajectories and also highlight the utility of spatial-temporal contextual information in improving the accuracy and consistency of land-cover classifications across space and time.

The use of satellite image time series analysis and machine learning methods brings new opportunities and challenges for land use and cover changes (LUCC) mapping over large areas. One of these challenges is the need for samples that properly represent the high variability of land used and cover classes over large areas to train supervised machine learning methods and to produce accurate LUCC maps. This paper addresses this challenge and presents a method to identify spatiotemporal patterns in land use and cover samples to infer subclasses through the phenological and spectral information provided by satellite image time series. The proposed method uses self-organizing maps (SOMs) to reduce the data dimensionality creating primary clusters. From these primary clusters, it uses hierarchical clustering to create subclusters that recognize intra-class variability intrinsic to different regions and periods, mainly in large areas and multiple years. To show how the method works, we use MODIS image time series associated to samples of cropland and pasture classes over the Cerrado biome in Brazil. The results prove that the proposed method is suitable for identifying spatiotemporal patterns in land use and cover samples that can be used to infer subclasses, mainly for crop-types.

Image time series can be interpreted on a higher level. We use the information-bottleneck principle to demonstrate our approach.

Time series remote sensing images can be used to monitor the dynamic changes of forest lands. Due to consistent cloud cover and fog, a single sensor typically provides limited data for dynamic monitoring. This problem is solved by combining observations from multiple sensors to form a time series (a satellite image time series). In this paper, the pixel-based multi-source remote sensing image fusion (MulTiFuse) method is applied to combine the Landsat time series and Huanjing-1 A/B (HJ-1 A/B) data in the Fuling district of Chongqing, China. The fusion results are further corrected and improved with spatial features. Dynamic monitoring and analysis of the study area are subsequently performed on the improved time series data using the combination of Mann-Kendall trend detection method and Theil Sen Slope analysis. The monitoring results show that a majority of the forest land (60.08%) has experienced strong growth during the 1999–2013 period. Accuracy assessment indicates that the dynamic monitoring using the fused image time series produces results with relatively high accuracies.

Unsupervised classification of satellite image time series (SITS) is prominent in numerous multitemporal remote sensing applications. However, when optical images are concerned, a missing value reconstruction task becomes pivotal, due to the impact of cloud cover on the input SITS. This task is usually addressed as a pre-processing step before feeding the time series to a clustering model. In this paper, we propose a pixelwise SITS clustering algorithm which integrates missing value imputation jointly with the clustering task. Moreover, the clustering process is designed to jointly perform both representation learning and cluster assignment, enabling the proposed model to simultaneously tackle three duties (missing value imputation, representation learning, and cluster assignment), while being trained in an end-to-end manner. The experimental results show that the proposed model performs well compared to other models that address time series imputation and clustering independently. Furthermore, a visualization analysis suggests that the proposed model learns both the imputation and clustering effectively despite being trained simultaneously.