A new data science framework for analysing and mining geospatial big data

Simple SummaryForecasting dengue cases often face challenges from (1) time-effectiveness due to time-consuming satellite data downloading and processing, (2) weak spatial representation due to data dependence on administrative unit-based statistics or weather station-based observations, and (3) stagnant accuracy without historical dengue cases. With the advance of the geospatial big data cloud computing in Google Earth Engine and deep learning, this study proposed an efficient framework of dengue prediction at an epidemiological week basis using geospatial big data analysis in Google Earth Engine and Long Short Term Memory modeling. We focused on the dengue epidemics in the Federal District of Brazil during 2007–2019. Based on Google Earth Engine and epidemiological calendar, we computed the weekly composite for each dengue driving factor, and spatially aggregated the pixel values into dengue transmission areas to generate the time series of driving factors. A multi-step-ahead Long Short Term Memory modeling was used, and the time-differenced natural log-transformed dengue cases and the time series of driving factors were considered as outcomes and explantary factors, respectively, with two modeling scenarios (with and without historical cases). The performance is better when historical cases were used, and the 5-weeks-ahead forecast has the best performance.Timely and accurate forecasts of dengue cases are of great importance for guiding disease prevention strategies, but still face challenges from (1) time-effectiveness due to time-consuming satellite data downloading and processing, (2) weak spatial representation capability due to data dependence on administrative unit-based statistics or weather station-based observations, and (3) stagnant accuracy without the application of historical case information. Geospatial big data, cloud computing platforms (e.g., Google Earth Engine, GEE), and emerging deep learning algorithms (e.g., long short term memory, LSTM) provide new opportunities for advancing these efforts. Here, we focused on the dengue epidemics in the urban agglomeration of the Federal District of Brazil (FDB) during 2007–2019. A new framework was proposed using geospatial big data analysis in the Google Earth Engine (GEE) platform and long short term memory (LSTM) modeling for dengue case forecasts over an epidemiological week basis. We first defined a buffer zone around an impervious area as the main area of dengue transmission by considering the impervious area as a human-dominated area and used the maximum distance of the flight range of Aedes aegypti and Aedes albopictus as a buffer distance. Those zones were used as units for further attribution analyses of dengue epidemics by aggregating the pixel values into the zones. The near weekly composite of potential driving factors was generated in GEE using the epidemiological weeks during 2007–2019, from the relevant geospatial data with daily or sub-daily temporal resolution. A multi-step-ahead LSTM model was used, and the time-differenced natural log-transformed dengue cases were used as outcomes. Two modeling scenarios (with and without historical dengue cases) were set to examine the potential of historical information on dengue forecasts. The results indicate that the performance was better when historical dengue cases were used and the 5-weeks-ahead forecast had the best performance, and the peak of a large outbreak in 2019 was accurately forecasted. The proposed framework in this study suggests the potential of the GEE platform, the LSTM algorithm, as well as historical information for dengue risk forecasting, which can easily be extensively applied to other regions or globally for timely and practical dengue forecasts.

Abstract. With the wide application of Big Data, Artificial Intelligence and Internet of Things in geographic information technology and industry, geospatial big data arises at the historic moment. In addition to the traditional "5V" characteristics of big data, which are Volume, Velocity, Variety, Veracity and Valuable, geospatial big data also has the characteristics of "Location Attribute". At present, the study of geospatial big data are mainly concentrated in: knowledge mining and discovery of geospatial data, Spatiotemporal big data mining, the impact of geospatial big data on visualization, social perception and smart city, geospatial big data services for government decision-making support four aspects. Based on the connotation and extension of geospatial big data, this paper comprehensively defines geospatial big data comprehensively. The application of geospatial big data in location visualization, industrial thematic geographic information comprehensive service and geographic data science and knowledge service is introduced in detail. Furthermore, the key technologies and design indicators of the National Geospatial Big Data Platform are elaborated from the perspectives of infrastructure, functional requirements and non-functional requirements, and the design and application of the National Geospatial Public Service Big Data Platform are illustrated. The challenges and opportunities of geospatial big data are discussed from the perspectives of open resource sharing, management decision support and data security. Finally, the development trend and direction of geospatial big data are summarized and prospected, so as to build a high-quality geospatial big data platform and play a greater role in social public application services and administrative management decision-making.

Convolutional neural network (CNN)-based deep learning (DL) methods have transformed the analysis of geospatial, Earth observation, and geophysical data due to their ability to model spatial context information at multiple scales. Such methods are especially applicable to pixel-level classification or semantic segmentation tasks. A variety of R packages have been developed for processing and analyzing geospatial data. However, there are currently no packages available for implementing geospatial DL in the R language and data science environment. This paper introduces the geodl R package, which supports pixel-level classification applied to a wide range of geospatial or Earth science data that can be represented as multidimensional arrays where each channel or band holds a predictor variable. geodl is built on the torch package, which supports the implementation of DL using the R and C++ languages without the need for installing a Python/PyTorch environment. This greatly simplifies the software environment needed to implement DL in R. Using geodl, geospatial raster-based data with varying numbers of bands, spatial resolutions, and coordinate reference systems are read and processed using the terra package, which makes use of C++ and allows for processing raster grids that are too large to fit into memory. Training loops are implemented with the luz package. The geodl package provides utility functions for creating raster masks or labels from vector-based geospatial data and image chips and associated masks from larger files and extents. It also defines a torch dataset subclass for geospatial data for use with torch dataloaders. UNet-based models are provided with a variety of optional ancillary modules or modifications. Common assessment metrics (i.e., overall accuracy, class-level recalls or producer's accuracies, class-level precisions or user's accuracies, and class-level F1-scores) are implemented along with a modified version of the unified focal loss framework, which allows for defining a variety of loss metrics using one consistent implementation and set of hyperparameters. Users can assess models using standard geospatial and remote sensing metrics and methods and use trained models to predict to large spatial extents. This paper introduces the geodl workflow, design philosophy, and goals for future development.

Abstract. The use of intelligent technologies within 3D geospatial data analysis and management will decidedly open the door towards efficiency, cost transparency, and on-time schedules in planning processes. Furthermore, the mission of smart cities as a future option of urban development can lead to an environment that provides high-quality life along stable structures. However, neither geospatial information systems nor building information modelling systems seem to be well prepared for this new development. After a review of current approaches and a discussion of their limitations we present our approach on the way to an intelligent platform for the management and analysis of big 3D geospatial data focusing on infrastructure projects such as metro or railway tracks planning. three challenges are presented focusing on the management of big geospatial data with existing geo-database management systems, the integration of heterogeneous data, and the 3D visualization for database query formulation and query results. The approach for the development of a platform for big geospatial data analysis is discussed. Finally, we give an outlook on our future research supporting intelligent 3D city applications in the United Arab Emirates.

ABSTRACTEarth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.

With the speedy expansion of Internet of Spatial Things, the enormous volume of geospatial big data is produced by the IoT devices. It gives rise to the new challenges for real time geospatial data processing and storing of reliable data in cloud system. The traditional geospatial cloud computing system is not efficient enough to process large volumetric of concurrent geospatial data. Consequently, fog assisted cloud computing environment has come into picture for achieving secure geospatial big data deduplication scheme. In this paper, we introduce a novel scheme GeoBD2 which defines the geo-deduplication structure to build an efficient geospatial bigdata deduplication scheme on fog assisted cloud computing framework. It also regulates which fog node needs to be traversed to investigate duplicate geospatial data rather than to traverse all the fog nodes. This can substantially enhance the efficiency of geospatial big data deduplication in fog assisted cloud environment. It also executes the performance analysis of the proposed scheme. By the experimental results, it is found that the proposed scheme has minimum overhead cost than the existing big data deduplication scheme.

Developments in sensor technology have contributed immensely to the growth of big geospatial sensor data. Moreover, advances in telecommunications have made it possible to use in-situ sensors to capture and transfer data about the environment in near real-time. The combination of various sensor types within a predefined geographic space and the possibility of making measurements at high temporal resolution contributes to a better understanding of our environment while also generating big geospatial sensor data. Big data from multi-sensor networks, particularly those that capture dynamic characteristics of the environment, have not been spared the challenges that face other types of big data. Specifically, the quality of data and the propagation of uncertainty through the multi-sensor data processing workflows have remained a major concern in the big geospatial sensor data research community. Attempts to document, quantify and communicate the uncertainty associated with sensor data and related sensor network outputs have been made mainly in the context of individual projects. This paper aims to document the state-of-art in defining uncertainty with regard to multi-sensor geospatial data. In particular, we analyse the current literature to outline different types of uncertainty, and document methods for handling uncertainty in the different stages of multi-sensor geospatial data collection, processing and delivery.

Abstract. In recent years, geospatial big data has been generated at a very high speed, and the data volume is becoming increasingly massive. In order to realize the full potential of geospatial big data, there has been a strong requirement and push to embrace the value of open science. However, it is still challenging to preserve the privacy and integrity of geospatial data in data sharing and management. Blockchain as d distributed ledger technology has a series of good characteristics, such as decentralization, trust-free, transparency, tamper-free, consensus and security, etc. These characteristics of blockchain are beneficial for facilitating geospatial data sharing and management, and hence promoting the development of open GIS. In this paper, we provide a comprehensive review on the literature that involves how blockchain technology is applied to geospatial data, especially in geospatial data privacy and integrity preservation. First, the background knowledge on geospatial data privacy and blockchain technology are introduced. Then, we reviewed how blockchain technology is applied to geospatial data, followed by the conclusion of the topic. This review is beneficial for understanding how blockchain technology can be applied to geospatial domain by integrating geospatial technologies like GIS and remote sensing.

With the rapid development of Internet of Things (IoT) technologies, the increasing volume and diversity of sources of geospatial big data have created challenges in storing, managing, and processing data. In addition to the general characteristics of big data, the unique properties of spatial data make the handling of geospatial big data even more complicated. To facilitate users implementing geospatial big data applications in a MapReduce framework, several big data processing systems have extended the original Hadoop to support spatial properties. Most of those platforms, however, have included spatial functionalities by embedding them as a form of plug-in. Although offering a convenient way to add new features to an existing system, the plug-in has several limitations. In particular, while executing spatial and nonspatial operations by alternating between the existing system and the plug-in, additional read and write overheads have to be added to the workflow, significantly reducing performance efficiency. To address this issue, we have developed Marmot, a high-performance, geospatial big data processing system based on MapReduce. Marmot extends Hadoop at a low level to support seamless integration between spatial and nonspatial operations of a solid framework, allowing improved performance of geoprocessing workflow. This paper explains the overall architecture and data model of Marmot as well as the main algorithm for automatic construction of MapReduce jobs from a given spatial analysis task. To illustrate how Marmot transforms a sequence of operators for spatial analysis to map and reduce functions in a way to achieve better performance, this paper presents an example of spatial analysis retrieving the number of subway stations per city in Korea. This paper also experimentally demonstrates that Marmot generally outperforms SpatialHadoop, one of the top plug-in based spatial big data frameworks, particularly in dealing with complex and time-intensive queries involving spatial index.

Geoinformatics researchers and practitioners have been facing challenges of geospatial data processing ever since the debut of digital mapping, the predecessor term for today’s geospatial information systems. Over time, due to the advancement in data collection technologies (such as satellite imagery and GPS), challenges of geospatial data processing have become widespread. Today, every individual or organization that either routinely or sporadically must handle geospatial data has a better understanding and much appreciation for the complex nature of geospatial data processing. Of several characteristics of geospatial phenomena that make geospatial data to be complex for processing, data-intensive computing, one of the main characteristics of geospatial big data that can be addressed by employing special computing hardware and advanced computing techniques, is discussed in this chapter. To that end, since geospatial big data is defined and different approaches for addressing challenges of this emergent paradigm are discussed in all other chapters of this book, in this chapter high-performance computing (HPC), commonly considered as one main approach for handling data-intensive problems, is focused.

Geospatial technology is still facing a lack of “out of the box” distributed processing solutions which are suitable for the amount and heterogeneity of geodata, and particularly for use cases requiring a rapid response. Moreover, most of the current distributed computing frameworks have important limitations hindering the transparent and flexible control of processing (and/or storage) nodes and control of distribution of data chunks. We investigated the design of distributed processing systems and existing solutions related to Geospatial Big Data. This research area is highly dynamic in terms of new developments and the re-use of existing solutions (that is, the re-use of certain modules to implement further specific developments), with new implementations continuously emerging in areas such as disaster management, environmental monitoring and earth observation. The distributed processing of raster data sets is the focus of this paper, as we believe that the problem of raster data partitioning is far from trivial: a number of tiling and stitching requirements need to be addressed to be able to fulfil the needs of efficient image processing beyond pixel level. We attempt to compare the terms Big Data, Geospatial Big Data and the traditional Geospatial Data in order to clarify the typical differences, to compare them in terms of storage and processing backgrounds for different data representations and to categorize the common processing systems from the aspect of distributed raster processing. This clarification is necessary due to the fact that they behave differently on the processing side, and particular processing solutions need to be developed according to their characteristics. Furthermore, we compare parallel and distributed computing, taking into account the fact that these are used improperly in several cases. We also briefly assess the widely-known MapReduce paradigm in the context of geospatial applications. The second half of the article reports on a new processing framework initiative, currently at the concept and early development stages, which aims to be capable of processing raster, vector and point cloud data in a distributed IT ecosystem. The developed system is modular, has no limitations on programming language environment, and can execute scripts written in any development language (e.g. Python, R or C#).

Geospatial data analysis can be improved by using data-driven algorithms and techniques from the machine learning field. The aim of our research is to discover interrelationships among topographical data to support the decision-making process. In this paper, we extracted topographical geospatial data from digital elevation model (DEM) raster images, and we discovered hidden patterns among this data based on the K-means clustering algorithm, to uncover relationships and find clusters of elevation values for the area of Jordan. We introduce a method for querying and clustering geospatial data and we built an interactive map accordingly. The method discovers hidden patterns and uncovers relationships in given large datasets. We demonstrate the applicability of the method using the Jordan map and we report on geospatial data analysis and retrieval improvements. The results show that the optimal decision is in favor of four clusters (classes). The first class includes the high elevation values, the second class includes the very low elevation values, the third class includes the medium-high elevation values, and the fourth class includes the very high elevation values.

Intensive livestock farming has been linked to excessive pollution and contamination of land, water and air. A common body of knowledge is required in order to understand and assess this impact of agriculture, and take proper countermeasures with a focus on the sustainability and the protection of the physical environment, affecting agricultural productivity as little as possible. In this paper, we examine the problem of the environmental impact of livestock agriculture, targeting animal manure, which is a serious problem in various territories of Europe and the USA. Aiming to address this problem, we have developed AgriBigCAT, an online software platform that uses geophysical information from various diverse sources, employing geospatial and big data analysis, together with web technologies, in order to estimate the impact of the livestock agricultural sector on the environment, examining possible pollution of land, water and air, considering various scenarios and potential applications to mitigate the problem. The AgriBigCAT platform can assist both the farmers' decision-making processes and the administration planning and policy making, with the ultimate objective of meeting the challenge of increasing food production at a lower environmental impact.

Analysis of geospatial data has traditionally been model-based, with a mean model, customarily specified as a linear regression on the covariates, and a Gaussian process covariance model, encoding the spatial dependence. While nonlinear machine learning algorithms like neural networks are increasingly being used for spatial analysis, current approaches depart from the model-based setup and cannot explicitly incorporate spatial covariance. We propose NN-GLS, embedding neural networks directly within the traditional Gaussian process (GP) geostatistical model to accommodate nonlinear mean functions while retaining all other advantages of GP, like explicit modeling of the spatial covariance and predicting at new locations via kriging. In NN-GLS, estimation of the neural network parameters for the nonlinear mean of the Gaussian Process explicitly accounts for the spatial covariance through use of the generalized least squares (GLS) loss, thus, extending the linear case. We show that NN-GLS admits a representation as a special type of graph neural network (GNN). This connection facilitates the use of standard neural network computational techniques for irregular geospatial data, enabling novel and scalable mini-batching, backpropagation, and kriging schemes. We provide methodology to obtain uncertainty bounds for estimation and predictions from NN-GLS. Theoretically, we show that NN-GLS will be consistent for irregularly observed spatially correlated data processes. We also provide a finite sample concentration rate, which quantifies the need to accurately model the spatial covariance in neural networks for dependent data. To our knowledge, these are the first large-sample results for any neural network algorithm for irregular spatial data. We demonstrate the methodology through numerous simulations and an application to air pollution modeling. We develop a software implementation of NN-GLS in the Python package geospaNN. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

This research investigates the illegal Vegetation Suppression Authorization (VSA) for Legal Reserve Compensation (LRC) in Tocantins, Brazil, between 2012 and 2014. The VSA is a license issued by state environmental agencies that authorizes the removal or clearing of native vegetation. This infraction resulted from the state environmental agency’s mistaken interpretation of Resolution 07/2005 of the State Environmental Council and the Native Vegetation Protection Law, which operates nationally and takes precedence over state resolutions and laws. The study area focuses on eight municipalities contained in the Formoso, Pium, and Javaés river basins. The methodology examines irregular VSA-LRC cases through geospatial data analysis, multitemporal analysis using remote sensing data, and economic assessments in both the origin and destination areas of the compensatory reserve. The results showed that among the 217 VSA-LRC applications during the period, 93 had corresponding geospatial data, indicating that 57.1% of the geospatial data were missing. This discrepancy between the authorized area and the available geospatial information restricts the oversight capability of the regulatory agency. A decade-long deforestation analysis of VSA-LRCs using remote sensing data considered three phases: 2012–2014 (application and issuance of VSA-LRCs), 2015–2018 (operational issuance), and 2019–2022 (post-expiration of VSA-LRCs). The results revealed a total VSA-LRC area of 343.291 km2 (34,329.10 ha), with 229.1169 km2 (22,911.69 ha) of deforestation. Notably, 54.877 km2 (5487.70 ha) of this deforestation occurred in 2019–2022, after the VSA-LRC had expired. The analysis of the LRC origin and destination areas demonstrates a unidirectional strategy, from economically more valuable areas to less valuable ones, according to land market dynamics. Therefore, this investigation concludes the need for greater administrative transparency, the development of a reliable geospatial data system for monitoring, and policy changes and actions to enhance environmental protection.