A multilevel parallel and scalable single-host GPU cluster framework for large-scale geospatial data processing

Geospatial Big Data analytics are changing the way that businesses operate in many industries. Although a good number of research works have reported in the literature on geospatial data analytics and real-time data processing of large spatial data streams, only a few have addressed the full geospatial big data analytics project lifecycle and geospatial data science project lifecycle. Big data analysis differs from traditional data analysis primarily due to the volume, velocity and variety characteristics of the data being processed. One of a motivation of introducing new framework is to address these big data analysis challenges. Geospatial data science projects differ from most traditional data analysis projects because they could be complex and in need of advanced technologies in comparison to the traditional data analysis projects. For this reason, it is essential to have a process to govern the project and ensure that the project participants are competent enough to carry on the process. To this end, this paper presents, new geospatial big data mining and machine learning framework for geospatial data acquisition, data fusion, data storing, managing, processing, analysing, visualising and modelling and evaluation. Having a good process for data analysis and clear guidelines for comprehensive analysis is always a plus point for any data science project. It also helps to predict required time and resources early in the process to get a clear idea of the business problem to be solved.

With the speedy expansion of Internet of Spatial Things, the enormous volume of geospatial big data is produced by the IoT devices. It gives rise to the new challenges for real time geospatial data processing and storing of reliable data in cloud system. The traditional geospatial cloud computing system is not efficient enough to process large volumetric of concurrent geospatial data. Consequently, fog assisted cloud computing environment has come into picture for achieving secure geospatial big data deduplication scheme. In this paper, we introduce a novel scheme GeoBD2 which defines the geo-deduplication structure to build an efficient geospatial bigdata deduplication scheme on fog assisted cloud computing framework. It also regulates which fog node needs to be traversed to investigate duplicate geospatial data rather than to traverse all the fog nodes. This can substantially enhance the efficiency of geospatial big data deduplication in fog assisted cloud environment. It also executes the performance analysis of the proposed scheme. By the experimental results, it is found that the proposed scheme has minimum overhead cost than the existing big data deduplication scheme.

In this paper, the high-performance processing of massive geospatial data on many-core GPU (Graphic Processing Unit) is presented. We use CUDA (Compute Unified Device Architecture) pro ...

Geoinformatics researchers and practitioners have been facing challenges of geospatial data processing ever since the debut of digital mapping, the predecessor term for today’s geospatial information systems. Over time, due to the advancement in data collection technologies (such as satellite imagery and GPS), challenges of geospatial data processing have become widespread. Today, every individual or organization that either routinely or sporadically must handle geospatial data has a better understanding and much appreciation for the complex nature of geospatial data processing. Of several characteristics of geospatial phenomena that make geospatial data to be complex for processing, data-intensive computing, one of the main characteristics of geospatial big data that can be addressed by employing special computing hardware and advanced computing techniques, is discussed in this chapter. To that end, since geospatial big data is defined and different approaches for addressing challenges of this emergent paradigm are discussed in all other chapters of this book, in this chapter high-performance computing (HPC), commonly considered as one main approach for handling data-intensive problems, is focused.

Geospatial technology is still facing a lack of “out of the box” distributed processing solutions which are suitable for the amount and heterogeneity of geodata, and particularly for use cases requiring a rapid response. Moreover, most of the current distributed computing frameworks have important limitations hindering the transparent and flexible control of processing (and/or storage) nodes and control of distribution of data chunks. We investigated the design of distributed processing systems and existing solutions related to Geospatial Big Data. This research area is highly dynamic in terms of new developments and the re-use of existing solutions (that is, the re-use of certain modules to implement further specific developments), with new implementations continuously emerging in areas such as disaster management, environmental monitoring and earth observation. The distributed processing of raster data sets is the focus of this paper, as we believe that the problem of raster data partitioning is far from trivial: a number of tiling and stitching requirements need to be addressed to be able to fulfil the needs of efficient image processing beyond pixel level. We attempt to compare the terms Big Data, Geospatial Big Data and the traditional Geospatial Data in order to clarify the typical differences, to compare them in terms of storage and processing backgrounds for different data representations and to categorize the common processing systems from the aspect of distributed raster processing. This clarification is necessary due to the fact that they behave differently on the processing side, and particular processing solutions need to be developed according to their characteristics. Furthermore, we compare parallel and distributed computing, taking into account the fact that these are used improperly in several cases. We also briefly assess the widely-known MapReduce paradigm in the context of geospatial applications. The second half of the article reports on a new processing framework initiative, currently at the concept and early development stages, which aims to be capable of processing raster, vector and point cloud data in a distributed IT ecosystem. The developed system is modular, has no limitations on programming language environment, and can execute scripts written in any development language (e.g. Python, R or C#).

Developments in sensor technology have contributed immensely to the growth of big geospatial sensor data. Moreover, advances in telecommunications have made it possible to use in-situ sensors to capture and transfer data about the environment in near real-time. The combination of various sensor types within a predefined geographic space and the possibility of making measurements at high temporal resolution contributes to a better understanding of our environment while also generating big geospatial sensor data. Big data from multi-sensor networks, particularly those that capture dynamic characteristics of the environment, have not been spared the challenges that face other types of big data. Specifically, the quality of data and the propagation of uncertainty through the multi-sensor data processing workflows have remained a major concern in the big geospatial sensor data research community. Attempts to document, quantify and communicate the uncertainty associated with sensor data and related sensor network outputs have been made mainly in the context of individual projects. This paper aims to document the state-of-art in defining uncertainty with regard to multi-sensor geospatial data. In particular, we analyse the current literature to outline different types of uncertainty, and document methods for handling uncertainty in the different stages of multi-sensor geospatial data collection, processing and delivery.

Geospatial information has been indispensable for many application fields, including traffic planning, urban planning, and energy management. Geospatial data are mainly stored in relational databases that have been developed over several decades, and most geographic information applications are desktop applications. With the arrival of big data, geospatial information applications are also being modified into, e.g., mobile platforms and Geospatial Web Services, which require changeable data schemas, faster query response times, and more flexible scalability than traditional spatial relational databases currently have. To respond to these new requirements, NoSQL (Not only SQL) databases are now being adopted for geospatial data storage, management, and queries. This paper reviews state-of-the-art geospatial data processing in the 10 most popular NoSQL databases. We summarize the supported geometry objects, main geometry functions, spatial indexes, query languages, and data formats of these 10 NoSQL databases. Moreover, the pros and cons of these NoSQL databases are analyzed in terms of geospatial data processing. A literature review and analysis showed that current document databases may be more suitable for massive geospatial data processing than are other NoSQL databases due to their comprehensive support for geometry objects and data formats and their performance, geospatial functions, index methods, and academic development. However, depending on the application scenarios, graph databases, key-value, and wide column databases have their own advantages.

We evaluate a novel implementation of a Self-Organizing Map (SOM) on a Graphics Processing Unit (GPU) cluster. Using various combinations of OpenCL, CUDA, and two different graphics cards, we demonstrate the scalability of the SOM implementation on one to eight GPUs. Results indicate that while the algorithm scales well with the number of training samples and the map size, the benefits from using the data-parallel approaches offered by the GPU are severely limited when combined with the Message Passing Interface (MPI) in this setting, and comparable to speedups of GPU-based implementations as compared to optimized sequential code. Speedups achieved range from 3 to 32, for various map and training data sizes. We also observed a performance penalty for the OpenCL implementation as compared to CUDA.

With the rapid development of Internet of Things (IoT) technologies, the increasing volume and diversity of sources of geospatial big data have created challenges in storing, managing, and processing data. In addition to the general characteristics of big data, the unique properties of spatial data make the handling of geospatial big data even more complicated. To facilitate users implementing geospatial big data applications in a MapReduce framework, several big data processing systems have extended the original Hadoop to support spatial properties. Most of those platforms, however, have included spatial functionalities by embedding them as a form of plug-in. Although offering a convenient way to add new features to an existing system, the plug-in has several limitations. In particular, while executing spatial and nonspatial operations by alternating between the existing system and the plug-in, additional read and write overheads have to be added to the workflow, significantly reducing performance efficiency. To address this issue, we have developed Marmot, a high-performance, geospatial big data processing system based on MapReduce. Marmot extends Hadoop at a low level to support seamless integration between spatial and nonspatial operations of a solid framework, allowing improved performance of geoprocessing workflow. This paper explains the overall architecture and data model of Marmot as well as the main algorithm for automatic construction of MapReduce jobs from a given spatial analysis task. To illustrate how Marmot transforms a sequence of operators for spatial analysis to map and reduce functions in a way to achieve better performance, this paper presents an example of spatial analysis retrieving the number of subway stations per city in Korea. This paper also experimentally demonstrates that Marmot generally outperforms SpatialHadoop, one of the top plug-in based spatial big data frameworks, particularly in dealing with complex and time-intensive queries involving spatial index.

Geospatial data is increasingly being made available on the Web as knowledge graphs using Linked Data principles. This entails adopting the best practices for publishing, retrieving, and using data, providing relevant initiatives that play a prominent role in the Web of Data. Despite the appropriate progress related to the amount of geospatial data available, knowledge graphs still face significant limitations in the GIScience community since their use, consumption, and exploitation are scarce, especially considering that just a few developments retrieve and consume geospatial knowledge graphs from within GIS. To overcome these limitations and address some critical challenges of GIScience, standards and specific best practices for publishing, retrieving, and using geospatial data on the Web have appeared. Nevertheless, there are few developments and experiences that support the possibility of expressing queries across diverse knowledge graphs to retrieve and process geospatial data from different and distributed sources. In this scenario, we present an approach to request, retrieve, and consume (geospatial) knowledge graphs available at diverse and distributed platforms, prototypically implemented on Apache Marmotta, supporting SPARQL 1.1 and GeoSPARQL standards. Moreover, our approach enables the consumption of geospatial knowledge graphs through a lightweight web application or QGIS. The potential of this work is shown with two examples that use GeoSPARQL-based knowledge graphs.

<strong class="journal-contentHeaderColor">Abstract.</strong> With semiconductor technology gradually approaching its physical and thermal limits, Graphics processing unit (GPU) is becoming an attractive solution in many scientific applications due to their high performance. This paper presents an application of GPU accelerators in air quality model. We endeavor to demonstrate an approach that runs a PPM solver of horizontal advection (HADVPPM) for air quality model CAMx on GPU clusters. Specifically, we first convert the HADVPPM to a new Compute Unified Device Architecture C (CUDA C) code to make it computable on the GPU (GPU-HADVPPM). Then, a series of optimization measures are taken, including reducing the CPU-GPU communication frequency, increasing the size of data computation on GPU, optimizing the GPU memory access, and using thread and block indices in order to improve the overall computing performance of CAMx model coupled with GPU-HADVPPM (named as CAMx-CUDA model). Finally, a heterogeneous, hybrid programming paradigm is presented and utilized with the GPU-HADVPPM on GPU clusters with Massage Passing Interface (MPI) and CUDA. Offline experiment results show that running GPU-HADVPPM on one NVIDIA Tesla K40m and NVIDIA Tesla V100 GPU can achieve up to 845.4x and 1113.6x acceleration. By implementing a series of optimization schemes, the CAMx-CUDA model resulted in a 29.0x and 128.4x improvement in computational efficiency using a GPU accelerator card on a K40m and V100 cluster, respectively. In terms of the single-module computational efficiency of GPU-HADVPPM, it can achieve 1.3x and 19.4x speedup on NVIDIA Tesla K40m GPU and NVIDA Tesla V100 GPU respectively. The multi-GPU acceleration algorithm enables 4.5x speedup with 8 CPU cores and 8 GPU accelerators on V100 cluster.

<strong class="journal-contentHeaderColor">Abstract.</strong> With semiconductor technology gradually approaching its physical and thermal limits, Graphics processing unit (GPU) is becoming an attractive solution in many scientific applications due to their high performance. This paper presents an application of GPU accelerators in air quality model. We endeavor to demonstrate an approach that runs a PPM solver of horizontal advection (HADVPPM) for air quality model CAMx on GPU clusters. Specifically, we first convert the HADVPPM to a new Compute Unified Device Architecture C (CUDA C) code to make it computable on the GPU (GPU-HADVPPM). Then, a series of optimization measures are taken, including reducing the CPU-GPU communication frequency, increasing the size of data computation on GPU, optimizing the GPU memory access, and using thread and block indices in order to improve the overall computing performance of CAMx model coupled with GPU-HADVPPM (named as CAMx-CUDA model). Finally, a heterogeneous, hybrid programming paradigm is presented and utilized with the GPU-HADVPPM on GPU clusters with Massage Passing Interface (MPI) and CUDA. Offline experiment results show that running GPU-HADVPPM on one NVIDIA Tesla K40m and NVIDIA Tesla V100 GPU can achieve up to 845.4x and 1113.6x acceleration. By implementing a series of optimization schemes, the CAMx-CUDA model resulted in a 29.0x and 128.4x improvement in computational efficiency using a GPU accelerator card on a K40m and V100 cluster, respectively. In terms of the single-module computational efficiency of GPU-HADVPPM, it can achieve 1.3x and 19.4x speedup on NVIDIA Tesla K40m GPU and NVIDA Tesla V100 GPU respectively. The multi-GPU acceleration algorithm enables 4.5x speedup with 8 CPU cores and 8 GPU accelerators on V100 cluster.

In this paper,a scalable graphics processing unit (GPU) cluster solution for the acceleration of FDTD for large-scale simulations is proposed. The hardware and software implementations are described. To illustrate the speed performance of the cluster, the simulation results of a cubic resonator with PEC boundaries is presented. A realistic large-scale simulation performed using SEMCAD X from SPEAG, accelerated by Acceleware's software libraries and GPU cluster solution is also presented.

Overlapping computation and communication of three-dimensional FDTD on a GPU cluster

In recent years, geospatial databases have been commercialized and widely exposed to mass users. Current exponential growth in data generation and querying rates for these data highlights the importance of efficient techniques for streaming. Traditional database technology, which operates on persistent and less dynamic data objects does not meet the requirements for efficient geospatial data streaming. Geostreaming, the intersection of data stream processing and geospatial querying, is an ongoing research focus in this area. In this paper, we describe why cloud is the most appropriate infrastructure in which to support geospatial stream data processing. First, we argue that cloud best fits the requirements of a large-scale geostreaming application. Second, we propose ElaStream, a general cloud-based streaming infrastructure that enables huge parallelism by means of the divide, conquer, and combine paradigm. Third, we examine key related work in the data streaming and (geo)spatial database fields, and describe the challenges ahead to build scalable cloud-based geostreaming applications.