Classification of spatially enriched pixel time series with convolutional neural networks

Most of the existing hyperspectral image change detection methods based on deep learning utilize convolutional neural networks (CNN), recurrent neural networks (RNN), and long-short-term memory (LSTM). However, CNNs primarily focus on extracting spatial information from hyperspectral images, thereby failing to fully utilize channel and temporal information. In this paper, we propose a network called STSTNet that can jointly extract spatial, spectral, and temporal feature information. To achieve a richer feature representation space, we employ a spatial attention module to extract spatial feature information from feature maps, and a spectral transformer encoder structure to extract abundant spectral sequence information. Specifically, we design two temporal information extraction structures after each spatial information extraction structure or spectral information extraction structure to capture correlations and interactions within the hyperspectral images. Extensive experiments have been conducted on three hyperspectral datasets, and the experimental results demonstrate the network’s excellent performance.

In this paper, we present PaTSI, a tool for analyzing evolutions of objects in time series of satellite images. This tool is a plugin integrated in the KNIME Analytics Platform. PaTSI is a workflow composed of several nodes assembled together to form a whole KDD process (data selection, pre-processing, image segmentation, pattern mining and visualization). Input data consists of a time series of satellite images and GIS information on the studied area. This data is transformed in a single attributed directed acyclic graph (a-DAG), where nodes represent objects described by several attributes and edges represent temporal relationships. This graph is then mined to extract frequent evolutions (weighted path patterns) using an efficient graph mining algorithm. At the end of the process, extracted patterns can be filtered using regular expressions and displayed on the original images in order to facilitate the experts' interpretation of the results. In the present demo, the pertinence of PaTSI is illustrated through its application to soil erosion monitoring.

Satellite Image Time Series (SITS), such as the ones acquired by the new Sentinel-2 (S2), combine a large amount of information compared to previous satellite generations since a better trade-off in terms of spatial/spectral/temporal resolutions is guaranteed. The characteristics of S2 SITS become more relevant in the agricultural analysis, where availability of continuous and/or dense SITS is important to map and analyze the crops dynamics. So far, agriculture applications have been limited by the spatial and temporal resolutions to monthly or yearly analysis over large regions and homogeneous crops. Whereas, the high spatial resolution offered by S2 allows to separate and analyze single crop fields, even when their size is relatively small (few hectares), with a high temporal resolution. In the literature only few techniques exist that deal with the detection and analysis of crop fields at single field level. Thus, this paper proposes an approach to dynamic crop field mapping in S2 SITS. To this aim, spectral, spatial and temporal information are used. The high variability of the crop cycle is analyzed in time to extract spectral and spatial information for crop-field detection. Connected-component labeling is used to label and analyze each crop field independently. Experimental results obtained for S2 SITS acquired over Barrax, Spain, confirm the effectiveness of the proposed approach.

Compared with one single image, satellite image time series (SITS) can capture the dynamic changes in land cover types, thus achieving a more comprehensive and accurate land cover classification map. Due to decades of data acquisition and new high temporal resolution sensors, SITS is becoming more available. Corresponding SITS analysis techniques need to be further developed. Most satellite images are multispectral, namely, multivariate. However, multivariate time series analysis techniques are less mature compared with univariate time series. There seems to be a lack of a robust and accurate similarity measure between multivariate time series for SITS clustering. In this paper, we propose a novel method to transform multivariate SITS into univariate SITS while the useful information is kept as much as possible. And then advanced univariate time series similarity measures can be adopted to achieve SITS clustering. The proposed method is tested on Landsat-TM SITS dataset and shows a better clustering result than ordinary multivariate time series similarity measure. In addition, the overall computing time may be reduced due to dimension reduction.

Satellite Image Time Series (SITS) is a data set that includes satellite images across several years with a high acquisition rate. Radiometric normalization is a fundamental and important preprocessing method for remote sensing applications using SITS due to the radiometric distortion caused by noise between images. Normalizing the subject image based on the reference image is a general strategy when using traditional radiometric normalization methods to normalize multi-temporal imagery (usually two or three scenes in different time phases). However, these methods are unsuitable for calibrating SITS because they cannot minimize the radiometric distortion between any pair of images in SITS. The existing relative radiometric normalization methods for SITS are based on linear assumptions, which cannot effectively reduce nonlinear radiometric distortion caused by continuously changing noise in SITS. To overcome this problem and obtain a more accurate SITS, we propose a nonlinear radiometric normalization model (NMAG) for SITS based on Artificial Neural Networks (ANN) and Greedy Algorithm (GA). In this method, GA is used to determine the correction order of SITS and calculate the error between the image to be corrected and normalized images, which avoids the selection of a single reference image. ANN is used to obtain the optimal solution of error function, which minimizes the radiometric distortion between different images in SITS. The SITS composed of 21 Landsat-8 images in Tianjin, China, from October 2017 to January 2019 was selected to test the method. We compared NMAG with other two contrast methods (Contrast Method 1 (CM1) and Contrast Method 2 (CM2)), and found that the average root mean square error (μRMSE) of NMAG (497.22) is significantly smaller than those of CM1 (641.39) and CM2 (543.47), and the accuracy of normalized SITS obtained using NMAG increases by 22.4% and 8.5% compared with CM1 and CM2, respectively. These experimental results confirm the effectiveness of NMAG in reducing radiometric distortion caused by continuously changing noise between images in SITS.

Abstract. Semantic segmentation is essential in the field of remote sensing because it is used for various applications such as environmental monitoring and land cover classification. Recent advancements aim to collectively classify data from diverse sensors and epochs to improve predictive accuracy. With the availability of vast Satellite Image Time Series (SITS) data, supervised deep learning methods, such as Transformer models, become viable options. This paper introduces the Temporal Vision Transformer(ViT), designed to extract features from SITS. These features, capturing the temporal patterns of land cover classes, are integrated with features derived from aerial imagery to improve land cover classification. Drawing inspiration from the success of transformers in Natural language processing (NLP), Temporal ViT concurrently extracts spatial and temporal information from SITS data using tailored positional encoding strategies. The proposed approach fosters comprehensive feature learning across both domains, facilitating seamless integration of encoded data from SITS into aerial images. Furthermore, a training strategy is proposed that supports the Temporal ViT to focus on classes with a changing appearance over the year. Extensive experiments carried out in this work indicate the enhanced classification performance of Temporal ViT compared to existing state-of-the-art techniques for multi-modal land cover classification. Our model achieves a 3.8% increase in the mean IoU compared to the network solely relying on aerial images.

Modern industrial plants generally consist of multiple manufacturing units, and the local correlation within each unit can be used to effectively alleviate the effect of spurious correlation and meticulously reflect the operation status of the process system. Therefore, the local correlation, which is called spatial information here, should also be taken into consideration when developing the monitoring model. In this study, a cascaded monitoring network (MoniNet) method is proposed to develop the monitoring model with concurrent analytics of temporal and spatial information. By implementing convolutional operation to each variable, the temporal information that reveals dynamic correlation of process data and spatial information that reflects local characteristics within individual operation unit can be extracted simultaneously. For each convolutional feature, a submodel is developed and then all the submodels are integrated to generate a final monitoring model. Based on the developed model, the operation status of the newly collected sample can be identified by comparing the calculated statistics with their corresponding control limits. Similar to the convolutional neural network (CNN), the MoniNet can also expand its receptive field and capture deeper information by adding more convolutional layers. Besides, the filter selection and submodel development in MoniNet can be replaced to generalize the proposed network to many existing monitoring strategies. The performance of the proposed method is validated using two real industrial processes. The illustration results show that the proposed method can effectively detect process anomalies by concurrent analytics of temporal and spatial information.

Temporal trajectories of land-cover change provide important information on landscape dynamics that are critical to our understanding of complex human–environment adaptive systems. The increasing availability of long time series of satellite images, especially the recent free release of multi-decadal Landsat satellite archive, presents a great opportunity to improve our ability to detect land-cover change over multiple dates and advance land change science. In this article, a spatial-temporal modeling approach is developed for reconstructing land-cover change trajectories from time series of satellite images. The change detection method represents an enhancement to the conventional post-classification comparison. The key innovation lies in the use of Markov random field theory to model spatial-temporal contextual information explicitly in the classification of time series images. When evaluated using a time series of seven Landsat images in a case study of southeast Ohio, the spatial-temporal modeling approach yielded significantly more accurate and consistent trajectories of land-cover change than conventional non-contextual approaches. The results from the case study demonstrate the effectiveness of the change detection method in reconstructing land-cover change trajectories and also highlight the utility of spatial-temporal contextual information in improving the accuracy and consistency of land-cover classifications across space and time.

Marine debris is a severe environmental problem. It originates from many sources and causes a wide spectrum of environmental, economic, safety, health and cultural impacts. Millions of tonnes enter the oceans every year and tackling the issue is gaining momentum at all levels.Our project presents an approach to detect and track floating marine debris such as lost fishing nets, debris aggregations or patches based on high-resolution remote sensing time series, machine learning, and ocean current modeling. The goal is to obtain more reliable data regarding quantity, position, material properties and sources of litter as well as to address the lack of understanding of floating debris behavior in the open sea due to the limited monitoring capabilities (Garaba and Dierssen, 2018). In order to achieve this goal, a convolutional neural network was trained with a hand-labeled dataset of objects floating on the sea surface extracted from Planet satellite imagery, to learn the spatial characteristics of these objects. A software pipeline has been developed to automatically retrieve, process and analyze large amounts of satellite images and enable continuous monitoring. As identifying floating objects in the marine environment from space remains a challenging and difficult task and ground truthing is near to impossible, we further apply a mechanism to find matching objects on time series of satellite images using an ocean current model as well as different image processing techniques.

Assessment of radiometric correction techniques in analyzing vegetation variability and change using time series of Landsat images

Grouped Frequent Sequential patterns can be extracted in an unsupervised way from Image Time Series (ITS). Plotting the occurrence maps of these patterns allows to describe the dataset spatially and temporally while discarding random uncertainties. However these maps can be too numerous and a swap randomization ranking approach has been proposed recently to select the most promising patterns. This previous work experimented the technique on Satellite ITS, giving credit to the maps that are least likely to appear on a randomized ITS. In this paper, extraction and ranking of GFS patterns is performed on a motion field time series obtained by terrestrial photogrammetry over the Argentiere glacier. The focus is extended to the maps that are most likely to occur on the randomized time series and the experiment is repeated thousand times to assess the stability of the ranking.

There is context-related temporal and spatial information in video. How to make better use of these information is the key to improving the accuracy and speed of video object detection. The method of processing spatial and temporal information through 3D convolution is better than that of processing spatial information first and then processing time dimension information. There are two reasons for this assertion. One is that the 3D convolution can process spatial information and temporal information at the same time which will not divide them into two parts in feature extraction. The other is that the human visual derivation mechanism needs to memorize multi frames and process a video as a whole. After the training test on NFS dataset, it is verified that 3D convolution has stronger ability to extract temporal and spatial information. In addition, this paper also proposes a downward sampling method by connecting the low-resolution layer and the high-resolution layer, which can effectively improve the FPN model and enrich its high-level feature representation. After training and testing with different number of downward sample modules on MOT16 dataset, the improvement of model performance by down-sampling module is verified. Finally, this paper combines the downward sample module with the 3D network, and obtains the best test results on the NFS dataset.

Satellite Image Time Series (SITS) have become more accessible in recent years and SITS analysis has attracted increasing research interest. Given that labeled SITS training samples are time and effort consuming to acquire, clustering or unsupervised analysis methods need to be developed. Similarity measure is critical for clustering, however, currently established methods represented by Dynamic Time Warping (DTW) still exhibit several issues when coping with SITS, such as pathological alignment, sensitivity to spike noise, and limitation on capacity. In this paper, we introduce a new time series similarity measure method named time adaptive optimal transport (TAOT) to the application of SITS clustering. TAOT inherits several promising properties of optimal transport for the comparing of time series. Statistical and visual results on two real SITS datasets with two different settings demonstrate that TAOT can effectively alleviate the issues of DTW and further improve the clustering accuracy. Thus, TAOT can serve as a usable tool to explore the potential of precious SITS data.

<p>The development of tracking and surveillance devices makes extracting useful information efficiently. Head tracking is an efficient method to obtain then analyze trajectory data and make a decision based on the spatiotemporal information of videos. Many applications are based on head tracking such as diseases some diagnosis, the gestures languages, and drowsiness detection and so on. Abnormal head movement detection can be achieved using spatial information based on a single image (one frame) at a time without considering the temporal information over time. In this paper, a new method based on multi-images is proposed to track head in order to detect abnormal head movement depending on spatiotemporal Feature using Deep Neural Network DNN that employed the 3-Dimensional Convolution Neural Networks 3D CNN. The proposed method extracts the spatial information as well as the temporal information available in a video then analysis this information to make the decision based on time series (sequences of frames); these time series provides the tracking to the head overtime to make the decision. The new dataset created and gathered to implement with the proposed system and called Normal Abnormal Head Movement Dataset (NAHM) video dataset. The new dataset provides different subjects with different conditions that give more efficiency in the implementation of the proposed system. The accuracy of the training set that achieves by the proposed system reach to 88% and of validation set reaches to 86%. The values of loss function reach to 0.3 for the training set and 0.4 for the validation set.</p>

Time series remote sensing images can be used to monitor the dynamic changes of forest lands. Due to consistent cloud cover and fog, a single sensor typically provides limited data for dynamic monitoring. This problem is solved by combining observations from multiple sensors to form a time series (a satellite image time series). In this paper, the pixel-based multi-source remote sensing image fusion (MulTiFuse) method is applied to combine the Landsat time series and Huanjing-1 A/B (HJ-1 A/B) data in the Fuling district of Chongqing, China. The fusion results are further corrected and improved with spatial features. Dynamic monitoring and analysis of the study area are subsequently performed on the improved time series data using the combination of Mann-Kendall trend detection method and Theil Sen Slope analysis. The monitoring results show that a majority of the forest land (60.08%) has experienced strong growth during the 1999–2013 period. Accuracy assessment indicates that the dynamic monitoring using the fused image time series produces results with relatively high accuracies.