SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation [Software and Data Sets

Deep learning methods have become an integral part of computer vision and machine learning research by providing significant improvement performed in many tasks such as classification, regression, and detection. These gains have been also observed in the field of remote sensing for Earth observation where most of the state-of-the-art results are now achieved by deep neural networks. However, one downside of these methods is the need for large amounts of annotated data, requiring lots of labor-intensive and expensive human efforts, in particular for specific domains that require expert knowledge such as medical imaging or remote sensing. In order to limit the requirement on data annotations, several self-supervised representation learning methods have been proposed to learn unsupervised image representations that can consequently serve for downstream tasks such as image classification, object detection or semantic segmentation. As a result, self-supervised learning approaches have been considerably adopted in the remote sensing domain within the last few years. In this article, we review the underlying principles developed by various self-supervised methods with a focus on scene classification task. We highlight the main contributions and analyze the experiments, as well as summarize the key conclusions, from each study. We then conduct extensive experiments on two public scene classification datasets to benchmark and evaluate different self-supervised models. Based on comparative results, we investigate the impact of individual augmentations when applied to remote sensing data as well as the use of self-supervised pre-training to boost the classification performance with limited number of labeled samples. We finally underline the current trends and challenges, as well as perspectives of self-supervised scene classification.

Deep learning methodologies have significantly advanced the fields of computer vision and machine learning, enhancing performance across various tasks like classification, regression, and detection. In remote sensing for Earth observation, deep neural networks have propelled state-of-the-art results. However, a major drawback is their dependence on large annotated datasets, necessitating extensive human effort, especially in specialized domains like medical imaging or remote sensing. To mitigate this reliance on annotations, several self-supervised representation learning techniques have emerged, aiming to learn unsupervised image representations applicable to downstream tasks such as image classification, object detection, or semantic segmentation. Consequently, self-supervised learning approaches have gained traction in remote sensing. This article surveys the foundational principles of various self-supervised methods, focusing on scene classification tasks. We elucidate key contributions, analyze experimental setups, and synthesize findings from each study. Furthermore, we conduct comprehensive experiments on two public scene classification datasets to evaluate and benchmark different self-supervised models

The foundation model has recently garnered significant attention due to its potential to revolutionize the field of visual representation learning in a self-supervised manner. While most foundation models are tailored to effectively process RGB images for various visual tasks, there is a noticeable gap in research focused on spectral data, which offers valuable information for scene understanding, especially in remote sensing (RS) applications. To fill this gap, we created for the first time a universal RS foundation model, named SpectralGPT, which is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT). Compared to existing foundation models, SpectralGPT 1) accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS Big Data; 2) leverages 3D token generation for spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; and 4) trains on one million spectral RS images, yielding models with over 600 million parameters. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS Big Data applications within the field of geoscience across four downstream tasks: single/multi-label scene classification, semantic segmentation, and change detection.

Earth observation (EO) yields large-scale, multimodal datasets collected from various satellite missions, DEMs, land-use data, and textual metadata. Foundation models like 4M (Massively Multimodal Masked Modeling) can learn a joint embedding space that bridges modality gaps, mitigates missing data issues, and facilitates partial spatio-temporal alignment [1]. However, directly training such foundation models on the vast, high-dimensional original EO datasets is not only computationally intensive but also imposes substantial demands on storage resources.To address this, one can leverage VQ-VAE (Vector Quantized-Variational AutoEncoder) as neural compressors to transform high-dimension multimodal inputs into a few discrete indices, significantly reducing data volume while preserving critical information. By inverting the tokenization process, we can reconstruct the original high-dimensional data with minimal quality loss, aided by adversarial and perceptual losses that enhance reconstruction fidelity.Traditional VQ-based approaches, however, face challenges such as inefficient codebook utilization and limited latent space representation. To overcome these, we propose scaling strategies that complement 4M’s tokenizer-based architecture. By expanding the codebook size, latent dimensions, and network depth, our method captures the complexity of EO modalities more effectively. Specifically, we employ spherical quantization techniques like Grouped Spherical Quantization (GSQ) to address limitations in traditional approaches [2]. GSQ constrains codebook vectors to a spherical surface, stabilizing training, preventing code collapse, and promoting uniform codebook usage. Unlike standard VQ, GSQ uses spherical initialization and normalization to maintain consistent distances among codebook entries, ensuring robust latent space coverage even under extreme compression or large codebooks. From our empirical and ablation studies, alternative methods like LFQ (Lookup-Free Quantization), FSQ (Finite Scalar Quantization), and RVQ (Residual Vector Quantizer) often exhibit limitations, such as tightly coupling the latent dimension to codebook size or relying on specialized training losses. In contrast, spherical-based techniques effectively decouple latent dimensions from codebook vocabulary, providing greater flexibility and scalability as data demands increase.Our approach enables neural compressors to adapt to varying scales of compression and complexity without compromising performance. Comprehensive scalability experiments—examining large codebooks, deeper networks, and diverse compression ratios—assessed the generalizability of the proposed compression strategies and demonstrated their effectiveness on high-dimensional, large-scale EO data with minimal information loss. By integrating advanced compression techniques with scalable architectures, this framework establishes a robust foundation for addressing multimodal challenges in EO research that significantly reduces the difficulty of training foundation models on multimodal high-dimensional EO data.

Certainly, unsupervised learning plays a critical role in the vast majority of the remote sensing (RS) applications due to the labeling costs. This research study focuses on a self-supervised representation learning approach for RS change detection (CD) based on increasing mutual information between features gathered from various views of a common context (same patch) in a non-linear way, inspired from a recently introduced unsupervised Deep Learning (DL) approach named InfoMax (DIM). The investigated model can learn image representations in unsupervised fashion that significantly produce promising results on CD. Outstandingly, using self-supervised learning, DIM model can learn representations on multispectral, hyperspectral or radar data. Our research reveals promising evidence of the significant potential utility of deep unsupervised features learning in other RS tasks.

Earth observation (EO) data are remote sensing image data obtained by air-borne or space-borne sensors, which has the characters of distribution and heterogeneity. Because these data are owned by different space agencies. With Petabyte-scale data volumes of Earth Observation (EO) obtained, the EO data has been one of the most important part of big data. The characters of distribution and heterogeneity of big EO data storage and processing systems owned by different space agencies brings challenges to integrate and use these data. To meet the big EO data challenges, the new generation Spatial data infrastructure is needed to provide the solution and implementation technique to big EO data storage, access, process and analysis. The concept and architecture of Next-generation national EO Spatial Data Infrastructure (EO-SDI) is proposed in the paper. Key techniques are introduced to solve the complexity of distributed and heterogeneous EO data integration and on-demand processing. An interoperable national EO data system of systems is developed to integrate distributed EO archives of eight EO data centers.

Precision agriculture requires accurate discrimination between winter crops and weeds, but there is a lack of annotated image data for winter cropping systems. This paper investigates a two-stage deep learning approach that integrates self-supervised feature learning with supervised fine-tuning for winter crop and weed image classification. A new winter crop and weed image dataset, WinterCropWeedDB, is proposed and used in this paper, which contains 1,136 high-resolution images of six winter crop species and four weed species collected from agricultural fields in central India. In the first stage of self-supervised learning, an EfficientNet-B3 model is pre-trained using a SimCLR-style self-supervised learning approach with an InfoNCE loss function (temperature τ = 0.5) on the images. The average contrastive loss value reduces from 2.0712 in the first iteration to 1.6835 at the end of pretraining. In the second stage of supervised fine-tuning, the pre-trained EfficientNet-B3 model is fine-tuned with a supervised classifier head on the images and tested on a single internal validation split (30%) of the dataset. The fine-tuned model reaches a maximum validation accuracy of 98.27%, with a macro-averaged F1-score of 0.98. Gradient-weighted class activation mapping (Grad-CAM) and Grad-CAM++ are used on the fine-tuned model to provide a qualitative visualization of the image regions that contribute to class predictions. The experiment outcomes demonstrate the viability of using self-supervised pretraining and supervised fine-tuning for the classification of winter crop and weed images on a region-specific dataset, while also emphasizing the importance of additional testing on independent test sets.

AN ASSESSMENT OF SPATIAL VARIATION OF LAND SURFACE CHARACTERISTICS OF MINNA, NIGER STATE NIGERIA FOR SUSTAINABLE URBANIZATION USING GEOSPATIAL TECHNIQUES

<p>Precision Agriculture (PA) applied on a widespread basis can be a building block for reduced ecosystem degradation without compromising food security. One problem of farmers in the implementation of PA applications is the lack of high spatial resolution soil information.</p><p>The EU-funded research project ‘pH-BB: Precision liming in Brandenburg’ aims at developing innovative nutrient management strategies based on proximal soil sensing data.  For this study, the pH-BB project provided us the data of the 361 soil samples, which were taken at an 800-hectare farm in Brandenburg close to Frankfurt (Oder) and analyzed in the laboratory for the texture fractions clay, silt, and sand. We used the Google Earth Engine to process remote sensing earth observation (EO) data of 1474 Sentinel-1 (S1) SAR and 85 cloud free Sentinel-2 (S2) scenes available at the study site during the period 2016-03-01 - 2021-11-09. Vegetation and soil indices were computed with optical S2 data and backscatter in VV and VH polarization were extracted from the S1 datasets. To derive long-term persistent patterns in the EO data, simple statistical parameters such as coefficient of variation, standard deviation, maximum pixel value, etc. were calculated along the temporal domain of the EO data. Together with calculated terrain attributes, 24 covariate grids were finally available for model building. Reference samples (rs) were randomly divided into a training dataset (70% of rs) and a validation dataset (30% of rs). Pixel values of the covariate datasets at the sampling locations were added to the rs datasets.</p><p>A random forest machine learning algorithm was applied to the training dataset to train two individual models for the alr-transformed target variables silt and sand using the covariates. The developed models were then applied to the gridded datasets to predict maps for the alr-transformed target variables silt and sand. The final maps of all 3 texture fractions clay, silt, and sand were computed by back-transforming the predicted alr silt and sand grids.</p><p>In the derived models EO covariates showed the highest level of importance. Comparison of the prediction results with the validation data set showed that the spatial distribution, of the clay, silt, and sand fractions was predicted, with a root mean square error (rmse) of 6.2, 5.3, and 9.7 mass-%, respectively. A classification of the predicted maps according to the German KA5 scheme showed that especially in the sand dominated soil classes the prediction errors were lower, whereas they increased in the loamy soil classes (dependent on the clay content).</p><p>With a rmse of 5.3 - 9.7 mass-%, the performance of the approach shows good potential for surface soil texture assessments even at high resolution or for global applications or as an initial guess for soil mapping with high resolution proximal soil sensing devices.</p>

Self-supervised learning has demonstrated considerable success in hyperspectral image (HSI) classification when limited labeled data is available. However, inherent dissimilarities among HSIs require self-supervised pre-training from scratch for each HSI dataset. Pre-training on a large amount of unlabeled data can be time consuming. Additionally, the poor quality of some HSIs can limit the performance of self-supervised learning algorithms. To address these issues, we propose to enhance self-supervised pre-training on HSIs with transfer learning. We introduce a progressive self-supervised pre-training framework that acquires strong initialization for the final pre-training on the target HSI dataset by sequentially performing self-supervised pre-training on datasets that are increasingly similar to the target HSI, specifically, first on a large general vision dataset and then on a related HSI dataset. This sequential strategy enables the model to progressively learn from domain-general vision knowledge to target-specific hyperspectral knowledge. To mitigate the catastrophic forgetting in sequential training, we develop a regularization method, called self-supervised elastic weight consolidation, to impose adaptive constraints on the changes to model parameters. Thorough classification experiments on various HSI datasets demonstrate that our framework significantly and consistently improves the self-supervised pre-training on HSIs in terms of both convergence speed and representation quality. Furthermore, our framework exhibits high generalizability and can be applied to various self-supervised learning algorithms. Transfer learning continues to prove its usefulness in self-supervised settings.

Earth Observation (EO) data plays a crucial role in research and applications related to environmental monitoring, enabling informed decision-making. However, the continuously increasing volume and diversity of EO data, distributed across multiple platforms and varying formats, pose challenges for easy access and the development of scalable and reproducible workflows.openEO addresses these challenges by providing a community-driven, open standard for unified access to EO data and cloud-native processing capabilities. It supports researchers to develop interoperable, scalable and reproducible workflows that can be executed using various programming languages (Python, R or JavaScript).openEO has become a cornerstone technology across major initiatives in agriculture, natural capital accounting, and land-cover monitoring. In ESA’s WorldCereal project, it provides the scalable framework needed to process global Sentinel-1 and Sentinel-2 time series and integrate advanced machine-learning models, enabling dynamic 10-meter cropland and crop-type maps. It also supports the Copernicus Global Land Cover service and its tropical forestry component by delivering consistent and repeatable processing chains for annual 10-meter land-cover products, which are crucial for policy reporting and SDG monitoring. Beyond land cover, openEO supports efforts like ESA's World Ecosystem Extent Dynamics project by creating reproducible ecosystem-extent mapping and change detection maps — key elements for biodiversity and environmental management.Building on this foundation, the openEO Federation, now integrated within the Copernicus Data Space Ecosystem (CDSE), provides seamless access to distributed Earth observation data and processing resources through a single, unified interface. By connecting multiple backends, it removes the need to juggle separate accounts or APIs and enables cross-platform workflows over datasets hosted by platforms such as Terrascope and CDSE.openEO also strongly supports FAIR (Findable, Accessible, Interoperable, Reusable) principles. It exposes rich metadata, relies on standardised processes, and encourages the use of reusable workflow definitions. This promotes transparency, reproducibility, and the sharing of algorithms and data across research and operational communities. The approach has been validated in several large-scale implementations, including ESA’s WorldCereal and the JRC’s Copernicus Global Land Cover and Tropical Forestry Mapping and Monitoring Service (LCFM), demonstrating its maturity for both research and production environments.By enabling reusable, federated, and reproducible Earth observation workflows, openEO is helping to build a more interoperable and efficient computational ecosystem, one that supports scalable innovation, collaboration, and long-term operational monitoring. Therefore, in this session, we aim to spark discussion on how openEO enables federated, FAIR-compliant, and reproducible workflow approaches for large-scale Earth observation applications.

Scene change detection is the process of identifying the differences between the multi-temporal image scenes at the semantic level, which has significant potential in the application of urban development and land management. In this paper, we propose a novel deep convolution canonical correlation analysis neural network (DCCANet) architecture, which could consider the spectral-spatial-temporal correlation feature for scene change detection in remote sensing images. For this purpose, we put together the convolutional neural networks (CNNs) and the deep canonical correlation analysis (DCCA) into the end-to-end network. The CNN could get the spectral-spatial feature information for scene representation, while the later could enhance the temporal correlation by nonlinear high- dimensional transformations between the multi-temporal image scenes for scene change detection. Experiments with high-resolution remote sensing image scene datasets demonstrated that our proposed approach can get a better performance in scene classification and change detection.

Object detection plays a crucial role in scene understanding and has extensive practical applications. In the field of remote sensing object detection, both detection accuracy and robustness are of significant concern. Existing methods heavily rely on sophisticated adversarial training strategies that tend to improve robustness at the expense of accuracy. However, detection robustness is not always indicative of improved accuracy. Therefore, in this paper, we research how to enhance robustness, while still preserving high accuracy, or even improve both simultaneously, with simple vanilla adversarial training or even in the absence thereof. In pursuit of a solution, we first conduct an exploratory investigation by shifting our attention from adversarial training, referred to as adversarial fine-tuning, to adversarial pretraining. Specifically, we propose a novel pretraining paradigm, namely structured adversarial self-supervised (SASS) pretraining, to strengthen both clean accuracy and adversarial robustness for object detection in remote sensing images. At a high level, SASS pretraining aims to unify adversarial learning and self-supervised learning into pretraining and encode structured knowledge into pretrained representations for powerful transferability to downstream detection. Moreover, to fully explore the inherent robustness of vision Transformers and facilitate their pretraining efficiency, by leveraging the recent masked image modeling (MIM) as the pretext task, we further instantiate SASS pretraining into a concise end-to-end framework, named structured adversarial MIM (SA-MIM). SA-MIM consists of two pivotal components, structured adversarial attack and structured MIM (S-MIM). The former establishes structured adversaries for the context of adversarial pretraining, while the latter introduces a structured local-sampling global-masking strategy to adapt to hierarchical encoder architectures. Comprehensive experiments on three different datasets have demonstrated the significant superiority of the proposed pretraining paradigm over previous counterparts for remote sensing object detection. More importantly, regardless of with or without adversarial fine-tuning, it enables simultaneous improvements on detection accuracy and robustness as expected, promisingly alleviating the dependence on complicated adversarial fine-tuning.

The growing availability of Earth Observation (EO) data enables monitoring of terrestrial ecosystems at unprecedented spatio-temporal resolutions. In practice, however, effective use of EO data remains constrained by substantial technical barriers. Working with raw, multi-modal EO imagery requires specialised domain expertise, large data transfers, access to high-performance computing infrastructure, and advanced machine learning (ML) skills. These requirements limit the accessibility of EO-based analytics for many downstream applications.Geospatial foundation models (GFMs) provide a promising alternative by learning general-purpose representations from large volumes of unlabeled EO data. By decoupling representation learning from downstream task modelling, GFMs allow users to exploit expressive features from modern deep learning models with limited EO or deep learning expertise and modest computational resources.In this work, we investigate embedding-based GFM workflows for forest disturbance monitoring, where timely inference and regional customisation are often more critical than maximising absolute predictive accuracy. Forest disturbances such as logging, windthrows, fires, pests, and diseases can occur abruptly and require rapid detection to support conservation, policy-making, and risk-management efforts.Machine learning methods for forest disturbance detection from EO data are well established and have shown strong performance on regional benchmarks. However, much of this work remains confined to academic demonstrations and is rarely translated into operational monitoring systems. Existing forest monitoring tools, including those aggregated by Global Forest Watch, typically rely on region- and sensor-specific models with limited feature expressivity. These systems may benefit from the rich multi-modal and spatio-temporal representations learned by GFMs, provided such embeddings can be accessed through scalable and practical deployment pipelines.We build on a pipeline designed to deliver on-demand, location- and time-specific geospatial embeddings as a service. Embeddings are generated server-side from raw EO data, compressed, and distributed as lightweight representations. End users interact only with these embeddings, which can be analysed using simple models such as linear probes or small decoders. This approach removes the need for the user to manipulate raw EO data, download large multi-modal datasets, or train and deploy large deep learning models, enabling rapid adaptation to local contexts with limited annotations and modest computational resources.We present preliminary results demonstrating the feasibility of this approach for forest disturbance detection and discuss its strengths and limitations relative to bespoke, fully supervised image-based models. While GFMs may not be optimal for applications with abundant annotations and stringent accuracy requirements, embedding-based services are particularly well suited to time-sensitive and regionally adaptive monitoring scenarios. Overall, this work illustrates how releasing geospatial embeddings as a product or service can lower barriers to EO-based forest monitoring and support faster, more inclusive environmental decision-making.

The role of Earth observation (EO) data in addressing societal problems from environmental through to humanitarian should not be understated. Recent innovation in EO means provision of analysis ready data and data cubes, which allows for rapid use of EO data. This in combination with processing technologies, such as Google Earth Engine and open source algorithms/software for EO data integration and analyses, has afforded an explosion of information to answer research questions and/or inform policy making. However, there is still a need for both training and validation data within EO projects – often this can be challenging to obtain. It has been suggested that citizen science can help here to provide these data, yet there is some perceived hesitancy in using citizen science within EO projects. This paper reports on the Citizen Science 4 EO (Citizens4EO) project that aimed to obtain an in-depth understanding of researchers’ and practitioners’ experiences with citizen science data in EO within the UK. Through a mixed methods approach (online and in-depth surveys and a spotlight case study) it was found that although the benefits of using citizen science data in EO projects were many (and highlighted in the spotlighted “Slavery from Space” case study), there were a number of common concerns around using citizen science. These were around the mechanics of deploying citizen science and the unreliability of a potentially misinformed or undertrained citizen base. As such, comparing the results of this study with those of a similar survey undertaken in 2016, it is apparent that progress towards optimizing citizen science use in EO has been incremental but positive with evidence of the realization of the benefits of citizen science for EO (Citizens4EO). As such, we conclude by offering priority action areas to support further use of citizen science by the EO community within the UK, which ultimately should be adopted further afield.