GeoBeam: A distributed computing framework for spatial data

With the diversification of applications and the diversity of users, the era of big data is always producing various new types of data, such as graph data, time series data, and spatial data. When predicting the future trend of the system, we always need to learn from the historical working state, and time series data plays a huge value in this aspect. Hardware resources are becoming cheaper, computing power has made major breakthroughs, and multi-machine and multi-core scenarios have become commonplace. At present, the focus of large-scale time series data processing is either focused on data collection and storage, or on single-machine processing, which has been unable to meet the existing needs. In response to this problem, this paper designs and implements a distributed large-scale time series processing and analysis system based on the Spark platform. The system framework is mainly divided into storage layer, operator layer and algorithm layer. At the storage layer, the system completes the organization and indexing of large-scale time series data based on HDFS and Hive. At the operator layer, the system provides users with basic operations commonly used for time series data on the Spark platform, and allows users to directly use these operators to implement custom time series related processing algorithms. At the algorithm level, the system implements some commonly used time series analysis algorithms in the Spark platform, including time series similarity query, clustering and prediction, so that users can directly use these algorithms to analyze time series. Finally, the feasibility and practicability of this system are verified by testing the performance and function of the system.

Leveraging the power of multi-core platforms for large-scale geospatial data processing: Exemplified by generating DEM from massive LiDAR point clouds

Delays in delivering spatial information to support the decisions about maintenance of bridges introduce stoppages into the bridge management workflow. Previous studies show that spatial data collection and analysis are major components of these inspection programs. However, the time consuming spatial data processing cause days or weeks of delays in delivering the needed spatial information. Federal agencies, in many cases, either wait for the tedious spatial data processing, or use partial data processing results containing high uncertainties. Using the processing of 3D laser scanning point clouds as an example, this paper examines the technical feasibility and scientific challenges of quantifying the data processing time and the quality (e.g., accuracy) of derived information (e.g., minimum under-clearance of a bridge) useful for bridge management. Different decisions need different geometric attributes and relationships with various accuracy requirements. The optimal data processing strategy varies with the needed information quality and time limits. It is difficult to manually analyze the time-quality trade-offs of spatial data processing due to its ad-hoc nature and large number of possible parameter settings in data processing workflows. The authors proposed a computational framework that automatically records data processing histories of engineers along with the engineering needs. Analyzing those histories lead to insights into the trade-offs between required computational complexity/time and quality of delivered spatial information. This paper presents time-quality analysis results from two bridge inspection cases.

Real-time monitoring microblog data can find sensitive information in time and provide help for public sentiment management and control. However, it needs processing large-scale data stream. MapReduce is a framework of processing large-scale data in batch mode, its purpose is to increase throughput, but its real-time performance is limited. Aiming at the real-time performance limitation of MapReduce, RT-SSP Real-Time Staged Stream Processing, a hybrid staged real-time stream processing scheme both for batch and real-time processing was proposed. By this method large-scale high-speed data stream is locally processed in stages, the communication cost is reduced by storing intermediate results to local node, and key technologies such as cache optimization are used to realize high concurrent read and write. Experiments show that RT-SSP scheme can improve the real-time performance of processing large-scale microblog data stream and achieve speed-up ratio of about 2.3.

Over the last 20 years, the process of measuring and recording a place or position on the surface of the earth has gone through a revolutionary change. There has been a switch from alidades to GPS, from slide rules to pentium computers, from manual to automatic recording and processing of spatial data. The changes include introduction and implementation of two satellite surveying systems (Transit and GPS), transfer of computations to high-speed computers, and processing and graphic presentation of spatial data in real time. Within the GPS domain, techniques have been developed to overcome degradation of the data by the government and to permit positioning to an accuracy of 2–3 mm in some cases.

Spatial omics allows for the simultaneous measurement of multiple “omics” layers (genomics, transcriptomics, proteomics, metabolomics). The analysis reveals significant advances in tumor microenvironment characterization, biomarker discovery, and clinical applications. Spatial Multiomics is particularly crucial in cancer research because tumors are highly heterogeneous, and the spatial relationships between cancer cells, immune cells, and stromal cells within the tumor microenvironment (TME) are critical for understanding disease progression, metastasis, and treatment response. The convergence of large-scale spatial omics data with artificial intelligence is powering the next revolution in oncology research. The clinical applications of spatial multi-omics are expanding rapidly, from patient stratification for precision oncology to the design of more efficient clinical trials. Developing interpretable models that can guide treatment decisions based on spatial patterns of gene and protein expression will be a key focus of future research.

Spatial data refers to extracting and mining the hidden, implicit, valid, novel and interesting spatial or non-spatial patterns or rules from large-amount, incomplete, noisy, fuzzy, random, and practical spatial databases. In which an important issue but remains underdeveloped is to reveal and handle the uncertainties in spatial data mining. In This work, uncertainty of spatial data is briefly analyzed firstly, including the types and origins of uncertainty, their models of measurement and propagation. Then, some uncertainty factors in operation of spatial data are discussed and some uncertainty handling methods are adopted, including maximum variance data discretization and fuzzy belief function. Finally, we think the process of spatial data can be regarded as a complex system, a linear serial processing system in engineering control systems. An uncertainty propagation model of spatial data - fuzzy logic uncertainty propagation model with credibility factor is developed. Moreover, several key problems about uncertainty handling and propagation in spatial data are put forward.

Event Abstract Back to Event Reconstruction of the mouse brain regions from the spatial gene expression data Alexey Tsvetkov1* 1 Russian Scientific Centre, Kurchatov Institute, Nano-, Bio-, Info-, Cognitive (NBIC) Center Understanding the structural basis of nervous system still remains one of the primary challenges in neuroscience. New perspectives for the studies of brain anatomy are emerging with the availability of large-scale spatial gene expression data. Exploration of these data promises to deliver new insights into the understanding of relations between genes and brain structure. We performed an analysis of the gene expression for more than 4000 genes in the mouse brain with the spatial resolution of 200 micrometers. These data are freely available from the Allen Brain Atlas [1] project. After data filtering and scaling, distance matrices between all brain voxels (volumetric pixels) were calculated for a number of common distance measures. We also proposed and utilized a novel distance measure based on the locality of gene expression. Then the brain volume was hierarchically clustered with a variety of techniques [2]. In order to verify the resulting brain parcellations we compared them by means of a quantitative measure to the histologically-defined anatomical reference atlas and found significant overlap between automatically derived spatial clusters and anatomical structures. We also compared brain regions by the number of genes expressed. Here the number of simultaneously activated genes was considered to be proportional to the amount of accumulated evolutionary changes. This allows us to identify dentate gyrus and adjoining hippocampal regions as the structures with the richest expression and, hence, less conservative in the evolutionary sense. Other regions of the cerebral cortex show intermediate levels of expression variability while other brain structures have accumulated fewer changes in the process of evolution, with olfactory bulbs and cerebellum being most conservative. This work was supported by Russian FTP "Scientific and scientific-pedagogical personnel". [1] Ng L., Bernard A., Lau C. et. al., "An anatomic gene expression atlas of the adult mouse brain" (2009) Nature Neuroscience, 12 (3): 356-362. [2] Xu R. and Wunsch D., "Survey of clustering algorithms" (2005) IEEE Transactions on Neural Networks, 16 (3): 645-678. Keywords: General neuroinformatics Conference: 4th INCF Congress of Neuroinformatics, Boston, United States, 4 Sep - 6 Sep, 2011. Presentation Type: Poster Presentation Topic: General neuroinformatics Citation: Tsvetkov A (2011). Reconstruction of the mouse brain regions from the spatial gene expression data. Front. Neuroinform. Conference Abstract: 4th INCF Congress of Neuroinformatics. doi: 10.3389/conf.fninf.2011.08.00043 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 17 Oct 2011; Published Online: 19 Oct 2011. * Correspondence: Dr. Alexey Tsvetkov, Russian Scientific Centre, Kurchatov Institute, Nano-, Bio-, Info-, Cognitive (NBIC) Center, Moscow, alexis.tsvetkov@gmail.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Alexey Tsvetkov Google Alexey Tsvetkov Google Scholar Alexey Tsvetkov PubMed Alexey Tsvetkov Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Cloud computing technology is changing the mode of the spatial information industry which is applied and provides new ideas for it. Since Hadoop platform provides easy expansion, high performance, high fault tolerance and other advantages, we propose a novel vector spatial data storage schema based on it to solve the problems on how to use cloud computing technology to directly manage spatial data and present data topological relations. Firstly, vector spatial data storage schema is designed based on column-oriented storage structures and key/value mapping to express spatial topological relations. Secondly, we design middleware and merge with vector spatial data storage schema in order to directly store spatial data and present geospatial data access refinement schemes based on GeoTools toolkit. Thirdly, we verify the middleware and the data storage schema through Hadoop cluster experiments. Comprehensive experiments demonstrate that our proposal is efficient and applicable to directly storing large-scale vector spatial data and timely express spatial topological relations.

The article explores the current trends, challenges, and prospects of automated mapping in land management, focusing on the increasing demand for high-precision, up-to-date cadastral information. Traditional cartographic methods, based on manual data processing and analog technologies, are labor-intensive and time-consuming, often leading to inconsistencies and inaccuracies. Automated mapping is an important direction in the development of modern geographic information systems and technologies in land management. The introduction of automated methods allows to significantly increase the productivity, accuracy and reliability of cartographic work, as well as to ensure the prompt updating of cartographic information. The study emphasizes the advantages of automated mapping through the integration of modern geoinformation technologies, including remote sensing (RS), global navigation satellite systems (GNSS), geographic information systems (GIS), and unmanned aerial vehicles (UAVs). These technologies significantly enhance the efficiency, accuracy, and reliability of spatial data collection, processing, and visualization. The article reviews recent scientific studies on the application of automated methods for processing remote sensing data, including aerial and satellite imagery, in the creation and updating of cartographic materials. Special attention is given to the use of UAVs for high-precision orthophoto mapping and 3D terrain modeling. The research also highlights the importance of ensuring data compatibility and consistency between various sources, such as cadastral registers, GPS measurements, and geoinformation resources. Furthermore, the study addresses the regulatory and legal aspects of automated mapping in land management, analyzing key legislative acts and government regulations in Ukraine that define the standards for cadastral mapping and land use planning. It discusses the necessity of implementing automated data validation and quality control procedures to improve the credibility of cartographic information. The article concludes by identifying key benefits of automated mapping, including increased accuracy, reduced costs, faster data processing, and improved decision-making in land management. It provides recommendations for the integration of advanced geoinformation technologies in land administration, ensuring the timely and reliable update of cadastral data, optimizing land resource management, and supporting sustainable land use planning.

Biomedical Engineering, usually known as Bioengineering, is among the fastest developing and one of most important interdisciplinary fields today. It connects natural and technical sciences, for all of which biological and medical phenomena, computation, and data management play important roles in science and industry. Distributed computing and parallel algorithms have proved to be effective in solving the problem with high computational complexity in a wide range of domains, including areas of computational bioengineering. This special edition collects high-quality research papers in the field of application of distributed computing systems in bioengineering applications. This issue is the achievement of the developments of modern computer and biomedical technology. Most bioengineering or bioinformatics methods need to access and analyses large amounts of data. For that it is necessary to achieve an effective way that from Big data creates knowledge and useful applications. The nature of such scientific data/information/knowledge demands the use of a powerful computing and intelligent database management systems. Areas of current and future research include advanced topics in this special issue. The paper from Juhasz et al. present a novel, GPU-based streaming architecture that has the potential for drastically reducing execution times and at the same time providing simultaneous 2D and 3D visualization facilities. The system uses a highly-optimised and re-configurable pipeline of CPU and GPU cores that attempts to exploit the tremendous computing power whenever possible. The system can process live data arriving from an EEG device or data stored in EEG data files. The computer drives a large display wall system of four 46 monitors that provides a 4K-resolution drawing surface for visualizing raw EEG data, potential maps and various 3D views of the patient's head. Authors from the group of M. Gusev presents the scalable balancer in Cloud computing extended to Dew Computing level. Given a successful implementation of scalable low level load balancer, implemented on the network layer. The scalability is proved with series of experiments. The experiments showed that it adds small latency of several milliseconds and thus it slightly reduces the performance when the distributed system is underutilized. Rybicki et al. demonstrate how to realize a Software-as-a-Service solution for a variety of science software using container technologies. The presented solution utilized DARIAH-DE research infrastructure based on OpenStack and UNICORE grid to deliver an extensible solution for the digital humanities domain. The paper from authors Forer et al. describe the extension of Cloudflow to support ApacheSpark without any adaptions to already implemented pipelines. The described performance evaluation demonstrates that Spark can bring an additional boost for analyzing next generation sequencing(NGS) data to the eld of genetics. The Cloudflow framework is open source and freely available at https://github.com/genepi/cloudflow . The paper by Memon et al. describes an approach for executing computational jobs across HPC and HTC resources while operating on geographically dispersed data via a global federated file system. The solution is realized as a new framework that integrates UNICORE and GFFS to provide a standards-based environment to support large scale data intensive computations frequent in today's biomedical analyses. The paper Parallel algorithm for the state space exploration, authored by L. Allal et al., proposes a new automatic verification technique based on model checking that determines whether a given system satisfies its specification. Such a technique suffers from the state explosion problem when traversing all possible states of systems. A new synchronized parallel algorithm (SPA) of exploration is proposed based on a fixed number of threads. Exhaustive comparative studies between the standard parallel exploration algorithm in SPIN and the new SPA show that the SPA performs slightly better regarding the execution time and memory complexity. Smart systems in telemedicine frequently use intelligent sensor devices at large scale. Practitioners can monitor non-stop the vital parameters of hundreds of patients in real-time. The most important pillars of remote patient monitoring services are communication and data processing. Large scale data processing is done mainly using workflows. In the paper Eszter et al. give a brief overview of the different check pointing techniques and propose two new provenance based check pointing algorithms which uses the information stored in the workflow structure to dynamically change the frequency of check pointing and can be efficiently used for dynamic health care smart systems.

We have setup a new course on the large scale data processing using clusters. It introduces the concepts and design of distributed systems. Many newly developed ideas such as Google file system and MapReduce programming framework for processing large scale data sets are introduced. Students will gain practical experience with distributed programming technologies via several small labs and one large multi-week final project. Labs and projects will be completed using Hadoop, an open-source implementation of Google's distributed file system and MapReduce programming model. We have taught this class named Mass Data Processing Technology on Large Scale Clusters for two years. This paper will describe the design, perform of the course as well as the experiences and lessons learned.

The article explores the development and implementation of an automated system for constructing professionograms of artificial intelligence (AI) specialists using modern information technologies. Against the backdrop of an ever-evolving digital economy and rising demand for qualified AI professionals, the study highlights the need for accurate and scalable tools to identify and assess relevant competencies. The main objective of the study is to design an innovative digital platform that enables automated generation of professionograms by analyzing unstructured textual data—such as job descriptions, expert publications, and scientific articles—through machine learning algorithms. Special emphasis is placed on the use of TF-IDF (Term Frequency-Inverse Document Frequency) and LDA (Latent Dirichlet Alloca-tion), which allow for effective identification and classification of key terms, skills, and knowledge areas relevant to the AI domain. The methodology involves a comprehensive review of the labor market, an analysis of expert opin-ions, and the processing of large-scale textual data to develop a structured model of professional compe-tencies. The system is designed to function as a decision-support tool in HR analytics, enabling data-driven recruitment, optimization of corporate training programs, and personalized career planning. One of the core advantages of the proposed solution is the reduction of subjectivity in the recruit-ment and evaluation process, as well as its adaptability to various specializations within the AI field. The system architecture includes a web-based interface, an interactive testing module, and visualization tools such as profession graphs and thematic distribution diagrams. This allows stakeholders—from HR man-agers to educators and researchers—to intuitively explore competency models and apply them in real-world scenarios. Furthermore, the article discusses the broader implications of using data-driven approaches in strategic human capital management. It argues that automated professionogram systems can significantly improve workforce planning, reduce turnover rates, and align employee skill sets with evolving business needs. The research also opens pathways for extending the system to other professional domains, incor-porating soft skills and psychological profiling, and developing mobile applications for continuous self-assessment and skill tracking. In conclusion, the study presents a scalable and versatile framework that not only enhances the efficiency of personnel selection and development but also supports long-term organizational growth in the age of artificial intelligence.

In the realm of Internet of Things (IoT)-driven healthcare, diverse technologies, including wearable medical devices, mobile applications, and cloud-based health systems, generate substantial data streams, posing challenges in real-time operations, especially during emergencies. This study recommends a hybrid architecture utilizing Hadoop for real-time processing of extensive medical data within the IoT framework. By employing distributed machine learning models, the system analyzes health-related data streams ingested into Spark streams via Kafka threads, aiming to transform conventional machine learning methodologies within Spark's real-time processing, crafting scalable and efficient distributed approaches for predicting health statuses related to diabetes and heart disease while navigating the landscape of big data. Furthermore, the system provides real-time health status forecasts based on a multitude of input features, disseminates alert messages to caregivers, and stores this valuable information within a distributed database, which is instrumental in health data analysis and the production of flow reports. We compute a range of evaluation parameters to evaluate the proposed methods' efficacy. This assessment phase encompasses measuring the performance of the Spark-based machine learning algorithm in a distributed parallel computing environment.

The efficient acquisition and processing of large-scale terrain data has always been a focal point in the field of photogrammetry. Particularly in complex mountainous regions characterized by clouds, terrain, and airspace environments, the window for data collection is extremely limited. This paper investigates the use of airborne millimeter-wave InSAR systems for efficient terrain mapping under such challenging conditions. The system's potential for technical application is significant due to its minimal influence from cloud cover and its ability to acquire data in all-weather and all-day conditions. Focusing on the key factors in airborne InSAR data acquisition, this study explores advanced route planning and ground control measurement techniques. Leveraging radar observation geometry and global SRTM DEM data, we simulate layover and shadow effects to formulate an optimal flight path design. Additionally, the study examines methods to reduce synchronous ground control points in mountainous areas, thereby enhancing the rapid acquisition of terrain data. The results demonstrate that this approach not only significantly reduces field work and aviation costs but also ensures the accuracy of the mountain surface data generated by airborne millimeter-wave InSAR, offering substantial practical application value by reducing field work and aviation costs while maintaining data accuracy.