Array DBMS in environmental science: Satellite sea surface height data in the cloud

A raster is the primary data type in Earth science, geology, remote sensing and other fields with tremendous growth of data volumes. An array DBMS is an option to tackle big raster data processing. However, raster data are traditionally stored in files, not in databases. Command line tools have long being developed to process raster files. Most tools are feature-rich and free but optimized for a single machine. This paper proposes new techniques for distributed processing of raster data directly in diverse file formats by delegating considerable portions of work to such tools. An N-dimensional array data model is proposed to maintain independence from the files and the tools. Also, a new scheme named GROUP–APPLY–FINALLY is presented to universally express the majority of raster data processing operations and streamline their distributed execution. New approaches make it possible to provide a rich collection of raster operations at scale and outperform SciDB over \(410\times \) on average on climate reanalysis data. SciDB is the only freely available distributed array DBMS to date. Experiments were carried out on 8- and 16-node clusters in Microsoft Azure Cloud.

Explosive growth of raster data volumes in numerical simulations, remote sensing and other fields stimulate the development of new efficient data processing techniques. For example, in-situ approach queries data in diverse file formats avoiding time-consuming import phase. However, after data are read from file, their further processing always takes place with code developed almost from scratch. Standalone command line tools are one of the most popular ways for in-situ processing of raster files. Decades of development and feedback resulted in numerous feature-rich, elaborate, free and quality-assured tools optimized mostly for a single machine. The paper reports current development state and first results on performance evaluation of ChronosServer – distributed system partially delegating in-situ raster data processing to external tools. The new delegation approach is anticipated to readily provide rich collection of raster operations at scale. ChronosServer already outperforms state-of-the-art array DBMS on single machine up to 193×.

Collection, Processing, and Documentation of Reliable Environmental Informations - Airborne close-range sensing data volumes require new data management concepts Computer-aided methods provide today the possibility to handle, process, analyse, and retrieve large information volumes. Both, remote and close-range sensing, collect extreme quantities of data, but with different focus. Therefore, different processing strategies need to be developed to combine and handle 2-dimensional spectral data covering wide areas or analog stereo models of small regions. TOPOGRAMM uses both, digital and analog acquisition and, merges local and global scopes combining conventional aerial photogrammetry with high precision measurements.The session is focussing on processing of vector and raster data, mass information storage, data banks, archiving, analysis, and documentation procedures. It describes existing projects and pilot applications.The term TOPOGRAMM stands for an innovative system to collect and process environmental and geo-scientific data. After the first close-range metric sensing proved workable, specialized Computer assisted procedures were developed to handle the new quality of data. Acquisition and stereo processing, combining micro-and meso-scales and different disciplines, requires new approaches and the exploitation of Visual inspection and differentiation powers. Digital and analog superimposition in high fidelity environments is another challenge.A navigated sensor-platform acquires point and area, raster and density informations. Their combination promises new insights and allows interdisciplinary approaches. The outstanding quality of the collected data results from repetition, selection, and measurements. This “reliable” information is object and milieu related and can be obtained at any time by a Visual “stereo” access to the archived models.Objective dimensional informations are processed in Computer systems to portray the spatial variability and analyse, model, map, and archive. Specific procedures are governed by media restrictions. Film, as one source media with high data capacity, competes with digital storage media, holding raster data and data-bases of geographic information systems (GIS).The system is suitable for engineering surveys and quantity surveying. The given accuracy can be favourably used for other mapping tasks or environmental applications like bio-monitoring or erosion protection.

The technology of multi-resolution pyramid is widely used in raster data (image and DEM) processing. However, the main weakness of such implementation is that redundant store exists among different resolution levels. To overcome this problem, we propose a novel organization for multi-resolution pyramid, called wavelet pyramid, which combines the strength of pyramid structure and wavelet transform. To integrate wavelet pyramid into Geoglobe (a Google Earth like system), we build the relationship between wavelet subbands and Geoglobe tiles. We also discuss the network transmission of wavelet pyramid in Geoglobe, and propose an incremental and progressive schedule method. Compared to tile pyramid, wavelet pyramid occupies less storage size, and its network schedule is more effective.

The rapid increase in the number of aerial and orbital Earth observation systems is generating a huge amount of remote sensing data that need to be readily transformed into useful information for policy and decision makers. This exposes an urgent demand for image interpretation tools that can deal efficiently with very large volumes of data. In this work, we introduce a set of methods that support distributed processing of georeferenced raster and vector data in a computer cluster, which may be a virtual cluster provided by cloud computing infrastructure services. The set of methods comprise a particular technique for indexing distributed georeferenced datasets, as well as strategies for distributing efficiently the processing of spatial context-aware operations. They provide the means for the development of scalable applications, capable of processing large volumes of geospatial data. We evaluated the proposed methods in a remote sensing image interpretation application, built on the MapReduce framework, and executed in a cloud computing infrastructure. The experimental results corroborate the capacity of the methods to support efficient handling of very large earth observation datasets.

The Antarctic Circumpolar Current (ACC) is a complex current system composed of multiple jets that is both unique to the world's oceans and relatively under observed compared with other current systems. Observations taken by current- and pressure-recording inverted echo sounders (CPIES) over four years, from November 2007 to November 2011, quantify the mean structure of one of the main jets of the ACC - the Polar Front - in a composite-mean sense. While the array of CPIES deployed in Drake Passage included a 3 x 7 local dynamics array, analysis of the Polar Front makes use of the line of CPIES that spanned the width of Drake Passage (C-Line). The Polar Front tends to prefer one of two locations, separated along the C-Line by 1_ of latitude, with the core of the jet centered on corresponding geopotential height contours (with a 17 cm difference between the northern and southern jets). Potential vorticity fields suggest that the Polar Front is susceptible to baroclinic instability, regardless of whether it is found upstream (farther south along the C-Line) or downstream (farther north along the C-Line) of the Shackleton Fracture Zone (SFZ), yet the core of the jet remains a barrier to smaller-scale mixing, as inferred from estimated mixing lengths. Within the local dynamics array of CPIES, the observed offset between eddy heat flux (EHF) and eddy kinetic energy (EKE) and the alignment of EHF with sea surface height (SSH) standard deviation motivates a proxy for depth-integrated EHF that can be estimated from available satellite SSH data. An eddy-resolving numerical model develops the statistics of a logarithmic fit between SSH standard deviation and cross-frontal EHF that is applied to the ACC in a circumglobal sense. We find 1.06 PW enters the ACC from the north and 0.02 PW exits towards Antarctica. The magnitude of the estimated EHF, along with contemporaneous estimates of the mean heat flux, suggests that the air-sea heat flux south of the PF is an overestimate. Long-term trends in EHF are calculated from January 1992 to December 2014 and reveal varying trends at the eight ACC EHF hot spots, with only three having statistically significant temporal trends of strengthening cross-frontal EHF. The dynamics of an oceanic storm track are investigated using CPIES observations in the local dynamics array to better understand the processes responsible for the spatial offset between EHF and EKE. Wave activity flux (W), calculated from the total geostrophic

Raster data represents a fundamental form of geospatial information, with satellite imagery serving as a prominent example. Domain scientists across multiple fields [3, 5, 15, 18, 23] rely extensively on raster data for their research. Currently, over 900 Earth observation satellites are active, with projections indicating this number will exceed 2,300 by 2032 [4]. Recent advances in deep learning [12] and foundation models [13, 17] have increasingly focused on satellite imagery applications, addressing critical tasks such as land cover classification and image segmentation using data from platforms like Sentinel-2 and Landsat 8. NASA projects that its Earth Science data will exceed 600 petabytes by 2030 [4], highlighting the exponential growth in data volume and the corresponding computational demands. As both the volume and complexity of raster data continue to increase, there is an urgent need for scalable processing frameworks. This paper summarizes the work on a large-scale raster data processing framework, including its optimization strategies, visualization, real-world applications, and future directions for benchmarking foundation models.

Immense volumes of geospatial arrays are generated daily. Examples of such include satellite imagery, numerical simulation, and derivative data avalanche. Array DBMS are one of the prominent tools for working with large geospatial arrays. Usually the arrays natively come as raster files. ChronosDB is a novel distributed, file based, geospatial array DBMS: http://chronosdb.gis.land/ ChronosDB operates directly on raster files, delegates array processing to existing elaborate command line tools, and outperforms SciDB by up to 75x on average. This demonstration will showcase three new components of ChronosDB enabling users to interact with the system and appreciate its benefits: (i) a Web GUI (edit, submit queries and get the output), (ii) an execution plan explainer (investigate the generated DAG), and (iii) a dataset visualizer (display ChronosDB arrays on an interactive web map).

Big geospatial raster data pose a grand challenge to data management technologies for effective big data query and processing. To address these challenges, various big data container solutions have been developed or enhanced to facilitate data storage, retrieval, and analysis. Data containers were also developed or enhanced to handle geospatial data. For example, Rasdaman was developed to handle raster data and GeoSpark/SpatialHadoop were enhanced from Spark/Hadoop to handle vector data. However, there are few studies to systematically compare and evaluate the features and performances of these popular data containers. This paper provides a comprehensive evaluation of six popular data containers (i.e., Rasdaman, SciDB, Spark, ClimateSpark, Hive, and MongoDB) for handling multi-dimensional, array-based geospatial raster datasets. Their architectures, technologies, capabilities, and performance are compared and evaluated from two perspectives: (a) system design and architecture (distributed architecture, logical data model, physical data model, and data operations); and (b) practical use experience and performance (data preprocessing, data uploading, query speed, and resource consumption). Four major conclusions are offered: (1) no data containers, except ClimateSpark, have good support for the HDF data format used in this paper, requiring time- and resource-consuming data preprocessing to load data; (2) SciDB, Rasdaman, and MongoDB handle small/mediate volumes of data query well, whereas Spark and ClimateSpark can handle large volumes of data with stable resource consumption; (3) SciDB and Rasdaman provide mature array-based data operation and analytical functions, while the others lack these functions for users; and (4) SciDB, Spark, and Hive have better support of user defined functions (UDFs) to extend the system capability.

Abstract. The continuous growth of remotely sensed data raises the need for efficient ways of accessing data archives. The classical model of accessing remote sensing (satellite) archives via distribution of large files is increasingly making way for a more dynamic and interactive data service. A challenge, though, is interoperability of such services, in particular when multi-dimensional data and advanced processing are involved. Individually crafted service interfaces typically do not allow substitution and combination of services. Open standards can provide a way forward if they are powerful enough to address both data and processing model. The OGC Web Coverage Service (WCS) is a modular service suite which provides high-level interface definitions for data access, subsetting, filtering, and processing of spatio-temporal raster data. WCS based service interfaces to data archives deliver data in their original semantics useful for further client-side processing, as opposed to the Web Map Service (WMS) (de la Beaujardière, 2006) which performs a pre-rendering into images only useful for display to humans. In this paper we present a case study where the OGC coverage data and service model defines the client/server interface for a climate data service. In particular, we show how flexible temporal analysis can be performed efficiently on massive spatio-temporal coverage objects. This service, which is operational on a several Terabyte data holding, has been established as part of the EarthServer initiative focusing on Big Data in the Earth and Planetary sciences.

Accelerating batch processing of spatial raster analysis using GPU

Cold-water coral (CWC) habitats are considered important centers of biodiversity in the deep sea, acting as spawning grounds and feeding area for many fish and invertebrates. Given their occurrence in remote parts of the planet, research on CWC habitats has largely been derived from remotely-sensed marine spatial data. However, with ever-developing marine data acquisition and processing methods and non-ubiquitous nature of infrastructure, many studies are completed in isolation resulting in large inconsistencies. Here, we present a concise review of marine remotely-sensed spatial raster data acquisition and processing methods in CWC habitats to highlight trends and knowledge gaps. Sixty-three studies that acquire and process marine spatial raster data since the year 2000 were reviewed, noting regional geographic location, data types (‘acquired data’) and how the data were analyzed (‘processing methods’). Results show that global efforts are not uniform with most studies concentrating in the NE Atlantic. Although side scan sonar was a popular mapping method between 2002 and 2012, since then, research has focused on the use of multibeam echosounder and photogrammetric methods. Despite advances in terrestrial mapping with machine learning, it is clear that manual processing methods are largely favored in marine mapping. On a broader scale, with large-scale mapping programs (INFOMAR, Mareano, Seabed2030), results from this review can help identify where more urgent research efforts can be concentrated for CWC habitats and other vulnerable marine ecosystems.

In recent years, many fields that experience a sudden proliferation of data, which increases the volume of data that must be processed and the variety of formats the data is stored in have been identified. This causes pressure on existing compute infrastructures and data analysis methods, as more and more data are considered as a useful source of information for making critical decisions in particular fields. Among these fields exist several areas related to human life, e.g., various branches of medicine, where the uncertainty of data complicates the data analysis, and where the inclusion of fuzzy expert knowledge in data processing brings many advantages. In this paper, we show how fuzzy techniques can be incorporated in big data analytics carried out with the declarative U-SQL language over a big data lake located on the cloud. We define the concept of big data lake together with the Extract, Process, and Store process performed while schematizing and processing data from the Data Lake, and while storing results of the processing. Our solution, developed as a Fuzzy Search Library for Data Lake, introduces the possibility of massively parallel, declarative querying of big data lake with simple and complex fuzzy search criteria, using fuzzy linguistic terms in various data transformations, and fuzzy grouping. Presented ideas are exemplified by a distributed analysis of large volumes of biomedical data on Microsoft Azure cloud. Results of performed tests confirm that the presented solution is highly scalable on the Cloud and is a successful step toward soft and declarative processing of data on a large scale. The solution presented in this paper directly addresses three characteristics of big data, i.e., volume , variety , and velocity , and indirectly addresses, veracity and value .

Remote sensing data has become increasingly important for domain scientists from diverse areas including agriculture. Traditional data analytics methods cannot scale to big-data as they normally run on a single machine. This led to new systems that can process satellite data on distributed environments. However, directly translating traditional query processing pipelines to distributed systems is still challenging due to the complexity of raster data processing, such as varying spatial and temporal resolutions, computational overhead from iterative operations, and dealing with excessive data volumes. This paper introduces FieldSAT, a real-life case study of scaling query pipelines from the agriculture field using a Spark-based raster processing system named RDPro. This work focuses on the BAITSSS model, which is a complex, yet very important, model for estimating evapotranspiration from raster data on large scale. The paper goes beyond straightforward implementation of the model and introduces key optimization techniques and lessons learned that can be broadly applied to similar applications. The optimizations introduced in FieldSAT include: managing prolonged Spark DAGs and dependency chains, minimizing shuffle volumes, implementing raster interpolation methods for extreme resolution differences, and selective data persistence to eliminate redundant processing. Our approach demonstrates how distributed frameworks can be effectively adapted to handle complex scientific analysis pipelines used by domain scientists while maintaining computational efficiency.

With the development of Earth observation technologies, geospatial raster data such as DEM and remote sensing data experienced explosive growth in last forty years, providing TB or even PB amount of data to research projects. However, storing and processing the large amount of data can be very challenge to many projects, especially these without access to high performance computers. Distributed computing can integrate existed computing and storage resources through the Internet, providing an alternative way to facilitate these projects on handling the raster data. This study presents a distributed DEM processing approach developed based on the Map/Reduce, which has been widely adopted in cloud computing applications. The approach allows users to store and process the raster type DEM data in a distributed storage regime to utilize computing and storage capabilities combined from many average computers, e.g., PC. The approach has been implemented in a prototyped system, which is developed by utilizing the Apache Hadoop. The prototype has been deployed in the experimental environment, and then 90-m resolution DEM for China and a DEM hillshade model have been ingested into the prototype to evaluate the prototype.KeywordsData BlockCloud StorageVirtual ImageRaster DataImage GeometryThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.