Introduction to Python for Geographic Data Analysis: A new, open resource for teachers and learners

Geospatial datasets are currently constructed, managed, and utilized individually according to the spatial scale of the real world, such as the ground/surface/underground or indoor/outdoor, as well the particular purpose of the geospatial data used for location-based services. In addition, LBS applications use an optimal data model and data format according to their particular purpose, and thus, various datasets exist to represent the same spatial features. Such duplicated geospatial datasets and geographical feature-based GIS data cause serious problems in the financial area, compatibility issues among LBS systems, and data integration problems among the various geospatial datasets generated independently for different systems. We propose a geospatial data fusion model called the topological relation-based data fusion model (TRDFM) using topological relations among spatial objects in order to integrate different geospatial datasets and different data formats. The proposed model is a geospatial data fusion model implemented in a spatial information application and is used to directly provide spatial information-based services without data conversion or exchange of geometric data generated by different data models. The proposed method was developed based on an extension of the AnchorNode concept of IndoorGML. The topological relationships among spatial objects are defined and described based upon the basic concept of IndoorGML. This paper describes the concept of the proposed TRDFM and shows an experimental implementation of the proposed data fusion model using commercial 3D GIS software. Finally, the limitations of this study and areas of future research are summarized.

Aggregation and visualization of geographical data are an important part of environmental data mining, environmental modelling, and agricultural management. However, it is difficult to aggregate geospatial data of the various formats such as maps, census and surveys. This paper presents a framework named PlaniSphere,which can aggregate the various geospatial datasets and synthesizes raw data. We developed an algorithm in PlaniSphere to aggregate remote sensing images with census data for classification and visualization of land use and land cover (LULC). The results show that the framework is able to classify geospatial data sets ofLULC from multiple formats. National census data sets can be used for calibration of remote sensing LULC classifications. This provides a new approach for the classification of remote sensing data. The approach proposed in this paper is useful for LULC classification in environmental spatial analysis.

Data is arguably the most valuable asset of the modern world. In this era, the success of any data-intensive solution relies on the quality of data that drives it. Among vast amount of data that are captured, managed, and analyzed everyday, geospatial data are one of the most interesting class of data that hold geographical information of real-world phenomena and can be visualized as digital maps. Geo-spatial data is the source of many enterprise solutions that provide local information and insights. In order to increase the quality of such solutions, companies continuously aggregate geospatial datasets from various sources. However, lack of a global standard model for geospatial datasets makes the task of merging and integrating datasets difficult and error-prone. Traditionally, domain experts manually validate the data integration process by merging new data sources and/or new versions of previous data against conflicts and other requirement violations. However, this approach is not scalable and is hinder toward rapid release, when dealing with frequently changing big datasets. Thus more automated approaches with limited interaction with domain experts is required. As a first step to tackle this problem, in this paper, we leverage Information Retrieval (IR) and geospatial search techniques to propose a systematic and automated conflict identification approach. To evaluate our approach, we conduct a case study in which we measure the accuracy of our approach in several real-world scenarios and we interview with software developers at Localintel Inc. (our industry partner) to get their feedbacks.

Flooding is a highly destructive natural disaster affecting millions of people annually, causing substantial global economic losses. Especially in the Global South, vulnerable populations with limited resources for flood protection and recovery suffer the most. However, effective flood impact analysis and standardized geospatial data processing and visualization is lacking, hindering effective disaster risk reduction. This study employs Python to develop a standardized, automated analysis and visualization model to address flood impacts after a high magnitude event anywhere in the world by using global geospatial data sets.In order to test its applicability two relevant cases, with different locations and scales, were selected, (Bangkok, Thailand, and Tula de Allende, Mexico). In both cases the automated flood model efficiently processed flood extent and depth data from ICEYE SAR images, added the population estimates from the German Aerospace Center, and key infrastructure elements from OpenStreetMap. The output of the automated process is presented as web-based interactive maps, offering insights into flooding severity and impacts with minimal user input. This quick approach (processing time between 41 s and 10,5 min) facilitates timely responses by first responders, with great potential to aid and improve the efficiency of international humanitarian efforts.By leveraging global high-resolution geospatial data for local-level analysis, this study demonstrates the versatility and time-saving benefits of this automated analysis and visualization. Automated models standardize results, minimizing human errors, and enabling consistent historical flood data comparison. Even though the databases are gathered from trusted sources, additional evaluation of uncertainties of the results with field data should be considered in further development of this tool. Nevertheless, this research highlights the worldwide potential, and especially the Global South, of automated global geospatial analysis to improve disaster impact reduction by enhancing efficient response efforts.

ABSTRACTEarth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.

Big geospatial raster data pose a grand challenge to data management technologies for effective big data query and processing. To address these challenges, various big data container solutions have been developed or enhanced to facilitate data storage, retrieval, and analysis. Data containers were also developed or enhanced to handle geospatial data. For example, Rasdaman was developed to handle raster data and GeoSpark/SpatialHadoop were enhanced from Spark/Hadoop to handle vector data. However, there are few studies to systematically compare and evaluate the features and performances of these popular data containers. This paper provides a comprehensive evaluation of six popular data containers (i.e., Rasdaman, SciDB, Spark, ClimateSpark, Hive, and MongoDB) for handling multi-dimensional, array-based geospatial raster datasets. Their architectures, technologies, capabilities, and performance are compared and evaluated from two perspectives: (a) system design and architecture (distributed architecture, logical data model, physical data model, and data operations); and (b) practical use experience and performance (data preprocessing, data uploading, query speed, and resource consumption). Four major conclusions are offered: (1) no data containers, except ClimateSpark, have good support for the HDF data format used in this paper, requiring time- and resource-consuming data preprocessing to load data; (2) SciDB, Rasdaman, and MongoDB handle small/mediate volumes of data query well, whereas Spark and ClimateSpark can handle large volumes of data with stable resource consumption; (3) SciDB and Rasdaman provide mature array-based data operation and analytical functions, while the others lack these functions for users; and (4) SciDB, Spark, and Hive have better support of user defined functions (UDFs) to extend the system capability.

Prototyping the visualization of geographic and sensor data for agriculture

Abstract. A set of mental strategies called "heuristics" – logical shortcuts that we use to make decisions under uncertainty – has become the subject of a growing number of studies. However, the process of heuristic reasoning about uncertain geospatial data remains relatively under-researched. With this study, we explored the relation between heuristics-driven decision-making and the visualization of geospatial data in states of uncertainty, with a specific focus on the visualization of borders, here termed "borderization". Therefore, we tested a set of cartographic techniques to visualize the boundaries of two types of natural hazards across a series of maps through a user survey. Respondents were asked to assess the safety and desirability of several housing locations potentially affected by air pollution or avalanches. Maps in the survey varied by "borderization" method, background color and type of information about uncertain data (e.g., extrinsic vs. intrinsic). Survey results, analyzed using a mixed quantitative-qualitative approach, confirmed previous suggestions that heuristics play a significant role in affecting users' map experience, and subsequent decision-making.

In geographic information science, semantic relatedness is important for Geographic Information Retrieval (GIR), Linked Geospatial Data, geoparsing, and geo-semantics. But computing the semantic similarity/relatedness of geographic terminology is still an urgent issue to tackle. The thesaurus is a ubiquitous and sophisticated knowledge representation tool existing in various domains. In this article, we combined the generic lexical database (WordNet or HowNet) with the Thesaurus for Geographic Science and proposed a thesaurus–lexical relatedness measure (TLRM) to compute the semantic relatedness of geographic terminology. This measure quantified the relationship between terminologies, interlinked the discrete term trees by using the generic lexical database, and realized the semantic relatedness computation of any two terminologies in the thesaurus. The TLRM was evaluated on a new relatedness baseline, namely, the Geo-Terminology Relatedness Dataset (GTRD) which was built by us, and the TLRM obtained a relatively high cognitive plausibility. Finally, we applied the TLRM on a geospatial data sharing portal to support data retrieval. The application results of the 30 most frequently used queries of the portal demonstrated that using TLRM could improve the recall of geospatial data retrieval in most situations and rank the retrieval results by the matching scores between the query of users and the geospatial dataset.

Input/output (I/O) of geospatial raster data often becomes the bottleneck of parallel geospatial processing due to the large data size and diverse formats of raster data. The open‐source Geospatial Data Abstraction Library (GDAL), which has been widely used to access diverse formats of geospatial raster data, has been applied recently to parallel geospatial raster processing. This article first explores the efficiency and feasibility of parallel raster I/O using GDAL under three common ways of domain decomposition: row‐wise, column‐wise, and block‐wise. Experimental results show that parallel raster I/O using GDAL under column‐wise or block‐wise domain decomposition is highly inefficient and cannot achieve correct output, although GDAL performs well under row‐wise domain decomposition. The reasons for this problem with GDAL are then analyzed and a two‐phase I/O strategy is proposed, designed to overcome this problem. A data redistribution module based on the proposed I/O strategy is implemented for GDAL using a message‐passing‐interface (MPI) programming model. Experimental results show that the data redistribution module is effective.

The development of the Global Navigation System and wireless networking technologies have changed the way we live, communicate, share information and even the collection of geospatial data in the field. Along with wireless networking technologies, the improvement in computational power of handheld devices such as smartphones, tablet PCs, ultra-mobile personal computers (UMPCs) and netbook computers allow field users to connect, store and stream large amounts of geospatial data from the web-server. Nowadays, geospatial data collection is more flexible and timely manner. In this paper we discuss field data collection using a smartphone and web-based GIS system, which collects, integrates, visualizes and analyzes the collected data in real-time. We built a web-GIS system for creating a user account, acquiring coordinates from GPS embedded devices or wireless access points, and providing a user-friendly survey form. The collected data can be visualized and analyzed by performing thematic mapping, labeling, symbolizing, querying and generating a summary report. We tested this system on a university campus management system, in which we collected information on illegal disposal sites and parking events within the university campus.

We present an Ontology-based data access technique for the integration of geospatial data from different sources and different formats using ontologies and mappings. Our technique is implemented in the system Ontop-spatial, which enables the execution of GeoSPARQL queries over geospatial relational databases storing vector or raster data by performing on-the-fiy GeoSPARQL-to-SQL translation using ontologies and mappings. Ontop-spatial is generally more efficient than traditional geospatial RDF stores. Linked geospatial data can then be visualised on the map using Sextant, an online tool for browsing and visualisation of linked geospatial data.

Open data published by various organizations is intended to make the data available to the public. All over the world, numerous organizations maintain a considerable number of open databases containing a lot of facts and numbers. However, most of them do not offer a concise and insightful data interpretation or visualization tool, which can help users to process all of the information in a consistently comparable way. Canadian Federal and Provincial Elections is an example of these databases. This information exists in numerous websites, as separate tables so that the user needs to traverse through a tree structure of scattered information on the site, and the user is left with the comparison, without providing proper tools, data-interpretation or visualizations. In this paper, we provide technical details of addressing this problem, by using the Canadian Elections data (since 1867) as a specific case study as it has numerous technical challenges. We hope that the methodology used here can help in developing similar tools to achieve some of the goals of publicly available datasets. The developed tool contains data visualization, trend analysis, and prediction components. The visualization enables the users to interact with the data through various techniques, including Geospatial visualization. To reproduce the results, we have open-sourced the tool.

As Geographic Information Systems (GIS) increasingly facilitate the analysis and sharing of geospatial data, the protection of sensitive information becomes paramount. This research explores the implementation of anonymization and differential privacy techniques to enhance security in GIS. Anonymization methods effectively remove or obscure personally identifiable information from geospatial datasets, while differential privacy introduces a mathematical framework that allows for the sharing of aggregate data without compromising individual privacy. This study evaluates the strengths and weaknesses of these techniques, demonstrating their effectiveness in maintaining the utility of geospatial data while safeguarding sensitive information. Through case studies and comparative analysis, we provide insights into best practices for integrating these privacy-preserving strategies into GIS applications, ensuring compliance with legal regulations and fostering public trust in geospatial technologies.

Federal, state and local government agencies develop, exchange and maintain geospatial information for the purpose of land-related legislation and administration. Geospatial information change continually and geospatial datasets become outdated and hence unsuitable for decision support because of inadequate quality of data, and are costly to maintain. There is a growing demand for accurate and consistent geospatial information in critical sectors such as emergency response. Examples of such geospatial information include the address locations, cadastre (land titles/lots), Environmental Planning Instruments (EPI), and roads and administrative boundaries. Often geospatial data are not synchronized within an agency or across agencies, leading to inaccurate data in repositories and infrequent updates (if at all). Vendor software, such as ArcGIS provides a basic replication solution. However such solutions are vendor environment dependant. Furthermore, real world spatial enterprises have many complexities that lead to replication interruption leaving replicated databases in spatially inconsistent states. In this paper we use a near real-time event driven asynchronous replication mechanism for selective heterogeneous geospatial data based on a mediation framework. Within this framework, we have introduced a failed-tuple blocking strategy that overcomes some of the limitations faced in the real-time operation of the framework. The proposed method was applied to synchronizing cadastral geospatial information across multiple repositories, including a repository that stores data spatio-temporally. The repositories are used in near real-time Internet and other mapping applications. The result of this application is briefly discussed.