Massive AIS Data Management Based on HBase and Spark

There are nearly 50 million AIS data generated every day in the Yangtze River Basin. Efficient storage and query of AIS data become the key problem of smart maritime administration. To overcome this problem, a distributed storage and query scheme based on Hadoop platform is proposed. Firstly, the original AIS message is parsed, the abnormal data is removed and the parsed data is pushed to the Kafka message queue. Secondly, the messages in the queue are pushed to different servers to complete the storage of real-time cache, historical trajectory and regional information. Finally, the query interface is provided to realize the real-time query of ship status, trajectory and regional ship in a certain period of time. The real applications show that compared with the traditional relational database, the solution can meet the requirements of real-time query and has high scalability.

New generation of health care IT systems are collecting and storing more and more data of patients. Useful knowledge can be extracted from the data in EMR or PHR to provide medical advises to patients, while through data analysis the result statistics can be used to support the scientific research. However, RDBMSs-based framework is not able to support the requirements of massive health care data storage, management and analysis. To solve the problem, this paper proposes a massive data management and analysis solution based on Hadoop to archive better performance, scalability and fault tolerance. The data management framework is presented. Besides, 2 different data analysis methods based on MapReduce and Hive are proposed. Experiment results of data upload, data query and data analysis show that the performance of the proposed framework is greatly improved, and a brief summary of the performance and the differences between 2 methods of MapReduce and Hive is also discussed.

With the rapid development of water transportation industry, the number of ships is increasing, which leads to a sharp increase in AIS data. However, the efficient management of AIS data has become an important issue that needs to be solved. AIS data is a typical stream data. The management of AIS data is facing the problem of high efficiency of spatial-temporal query of massive data. A spatial-temporal hybrid index structure is built to optimize spatial-temporal query of AIS data. This spatial-temporal hybrid index structure is composed of two indexes, one is HBase-STR index for HBase and another is Redis-Geo index for Redis. Finally, the experiment results show that the hybrid index structure effectively improve the efficiency of querying AIS data.

Publishing biological data in XML formats is attractive for organizations who would like to provide their bioinformatics resources in an extensible and machine-readable format. In the era of big data, massive XML-based biological data management is emerged as a challengeable issue. With the continuous growth of the XML-based biological data sets, it is usually frustrating to use traditional declarative query languages to provide efficient query capabilities in terms of processing speed and scale. In this study, we report a novel platform to store and query massive XML-based biological data collections. A prototype tool for constructing HBase tables from XML-based biological data collections is first developed, and then a formal approach to transform the XML query model into the MapReduce query model is proposed. Finally, an evaluation of the query performance of the proposed approach on the existing XML-based biological databases is presented, showing that the performance advantages of the proposed solution. The source code of the massive XML-based biological data management platform is freely available at https://github.com/lyotvincent/X2H.

Owing to the rapid development of earth observation technology, the volume of spatial information is growing rapidly; therefore, improving query retrieval speed from large, rich data sources for remote-sensing data management systems is quite urgent. A global subdivision model, geographic coordinate subdivision grid with one-dimension integer coding on 2n-tree, which we propose as a solution, has been used in data management organizations. However, because a spatial object may cover several grids, ample data redundancy will occur when data are stored in relational databases. To solve this redundancy problem, we first combined the subdivision model with the spatial array database containing the inverted index. We proposed an improved approach for integrating and managing massive remote-sensing data. By adding a spatial code column in an array format in a database, spatial information in remote-sensing metadata can be stored and logically subdivided. We implemented our method in a Kingbase Enterprise Server database system and compared the results with the Oracle platform by simulating worldwide image data. Experimental results showed that our approach performed better than Oracle in terms of data integration and time and space efficiency. Our approach also offers an efficient storage management system for existing storage centers and management systems.

NoSQL and relational databases have been integrated in response to the increasing demand for scalable and efficient data management in hybrid cloud environments. The differences in data structures and query processing methods between these databases present both challenges and opportunities when designing an optimized hybrid system. This study explores the integration of NoSQL and relational databases to maximize data storage, retrieval, and processing efficiency across multiple cloud systems. NoSQL databases excel in handling unstructured and semi-structured data, offering flexibility and scalability, whereas relational databases provide robust consistency and structured query capabilities. Despite the wide availability of relational databases, certain applications require the dynamic adaptability of NoSQL systems, making integration a viable solution. The research evaluates key performance parameters, including query execution speed, scalability, data consistency, and resource utilization. Cloudsim is used for simulation, allowing for an in-depth comparison between standalone and hybrid database models. Experimental results indicate that the proposed hybrid model improves query performance by 30%, reduces latency by 20%, and enhances scalability by 40% compared to using relational or NoSQL databases alone. The novelty of this approach lies in its ability to overcome the limitations of each database type by their strengths in a dynamic cloud environment. The results highlight the effectiveness of hybrid database integration in optimizing cloud-based data management while ensuring seamless operation, adaptability, and resource efficiency.

Schema mappings establish a correspondence between data stored in two databases, called source and target respectively. Query processing under schema mappings has been investigated extensively in the two cases where each target atom is mapped to a query over the source (called GAV, global-as-view), and where each source atom is mapped to a query over the target (called LAV, local-as-view). The general case, called GLAV, in which queries over the source are mapped to queries over the target, has attracted a lot of attention recently, especially for data exchange. However, query processing for GLAV mappings has been considered only for the basic service of query answering, and mainly in the context of conjunctive queries (CQs) in relational databases. In this paper we study query processing for GLAV mappings in a wider sense, considering not only query answering, but also query rewriting, perfectness (the property of a rewriting to compute exactly the certain answers), and query containment relative to a mapping. We deal both with the relational case, and with graph databases, where the basic querying mechanism is that of regular path queries. Query answering in GLAV can be smoothly reduced to a combination of the LAV and GAV cases, and for CQs this reduction can be exploited also for the remaining query processing tasks. In contrast, as we show, GLAV query processing for graph databases is non-trivial and requires new insights and techniques. We obtain upper bounds for answering, rewriting, and perfectness, and show decidability of relative containment.

eXtensible Markup Language (XML) is one of the most used standards for information sharing between applications and devices, both on the internet and local network. However, relational database (RDB) has been used by many enterprises as their data management system and will require an amount of cost to change the system completely, if they are to change to XML technology solely. Thus, a mapping scheme is required to provide seamless integration on bridging XML technologies and RDBs. In this paper, an efficient model-based mapping scheme named XML-REG is proposed. The XML document will first be read and parsed into the parser, namely Streaming API for XML (StAX) parser. Then, each node will then be assigned with unique identification label to show the exact position of nodes in the document. Subsequently, by employing the proposed algorithm, data will then be transformed into tables in the RDB storage. As the result, two tables, namely (i) value table to store information carried by text node of the document, and (ii) path table to store the hierarchy structure of the document will be created. Experimental evaluations demonstrated that XML-REG outperformed some existing approaches, such as Mini-XML, XAncestor, XMap and XRecursive in terms of data storage size, mapping time and query retrieval time. In addition, the scalability test has also been conducted to show the capability of these approaches in supporting huge datasets, by scaling the DBLP dataset by times 5, times 10 and times 15. The results showed that XML-REG has the closest to linear graph compared to other existing approaches. On average, XML-REG showed the best performance in terms of query retrieval time and database storage size.

Relational database technology has enabled easy and efficient management of data in a variety of applications. This technology, however, has been used for computer simulations to a limited extent. In this context, users may need to analyze the results of multiple simulation runs, which typically capture interactions between various entities in the simulated world. To these users, storing such simulation data only in the form of relational tables may seem counter-intuitive or challenging, particularly if they are not familiar with database normalization theory. Furthermore, some complex queries, including those that examine the cumulative effect of an action (e.g., disease propagation after an initial outbreak), can neither be easily expressed nor efficiently executed on traditional database systems. We propose a new database approach that aims to achieve convenient and highly efficient storage and querying of simulation data by adopting a graph data model. We also discuss new challenges that arise in this research, with a focus on language design, coordination of simulations, data storage, and query processing.

데이터를 구조화하여 사용하는 관계형 데이터베이스가 현재까지 데이터 관리에 가장 많이 사용되고 있다. 그러나 관계형 데이터베이스는 데이터가 증가되면 데이터를 저장하거나 조회할 때 읽기, 쓰기 연산 수행에 제약 조건이 발생되어 서비스가 느려지는 현상이 나타난다. 또 새로운 업무가 추가되면 데이터베이스 내 데이터는 증가되고 결국 이를 해결하기 위해 하드웨어의 병렬 구성, CPU, 메모리, 네트워크 등 추가적인 인프라 구성을 필요로 하게 된다. 본 논문에서는 관계형 데이터베이스의 데이터 증가로 느려지는 웹 정보서비스 개선을 위해 기존 관계형 데이터베이스의 데이터를 하둡 HDFS로 전송하고 이를 일원화하여 데이터를 재구성한 후 사용자에게 하둡 데이터 처리로 대량의 데이터를 빠르고 안전하게 추출하는 모델을 구현한다. 본 시스템 적용을 위해 웹 기반 민원시스템과 비정형 데이터 처리인 이미지 파일 저장에 본 제안시스템을 적용하였다. 적용결과 관계형 데이터베이스 시스템보다 제안시스템 데이터 처리가 0.4초 더 빠른 결과를 얻을 수 있었고 기존 관계형 데이터베이스와 같은 대량의 데이터를 처리를 빅 데이터 기법인 하둡 데이터 처리로도 웹 정보서비스를 지원이 가능하였다. 또한 하둡은 오픈소스로 제공되어 소프트웨어 구매 비용을 줄여주는 장점이 있으며 기존 관계형 데이터베이스의 데이터 증가로 효율적인 대용량 데이터 처리를 요구하는 조직에게 도움을 줄 수 있을 것이다. Relational databases used by structuralizing data are the most widely used in data management at present. However, in relational databases, service becomes slower as the amount of data increases because of constraints in the reading and writing operations to save or query data. Furthermore, when a new task is added, the database grows and, consequently, requires additional infrastructure, such as parallel configuration of hardware, CPU, memory, and network, to support smooth operation. In this paper, in order to improve the web information services that are slowing down due to increase of data in the relational databases, we implemented a model to extract a large amount of data quickly and safely for users by processing Hadoop Distributed File System (HDFS) files after sending data to HDFSs and unifying and reconstructing the data. We implemented our model in a Web-based civil affairs system that stores image files, which is irregular data processing. Our proposed system's data processing was found to be 0.4 sec faster than that of a relational database system. Thus, we found that it is possible to support Web information services with a Hadoop-based big data processing technique in order to process a large amount of data, as in conventional relational databases. Furthermore, since Hadoop is open source, our model has the advantage of reducing software costs. The proposed system is expected to be used as a model for Web services that provide fast information processing for organizations that require efficient processing of big data because of the increase in the size of conventional relational databases.

Becoming development trend of GIS, three-dimensional visualization makes the amount of spatial data grown inevitably. Therefore, the study on storage and management of massive three-dimensional geospatial data is necessary and significant. The C/S framework, which consists of massive three-dimensional geospatial database for server and a large number of low-profile PC for clients, has become a basic idea to solve the problem. However, the development of server software witnesses plenty of challenges, such as error detection and recovery, intercurrent control and real time dispatch of spatial data etc. Based on ACE, how to apply the framework and pattern of communications software for building up a powerful application framework and constructing a high-performance server called adaptive communication server of massive three-dimensional geospatial data, is presented. The paper discusses massive three-dimensional data server related technology to achieve a thorough exploration and research, analyses the domain of 3D GIS thoroughly, identifies the problem space and the core design challenges, and proposes a feasible solution. Reducing the difficulty and workload of 3D GIS server development, this solution can have the developer focus on the development of data visualization. In addition, this paper described the object oriented architecture which supports multiple server policy configurations. Being a dynamically ordered structure, it can be ordered systematically these strategies and can be statically and dynamically change their behavior, so that it can provide the most effective strategies for a given software or hardware platform and work load.

Efficient storage and handling of data stored in a peer-to-peer (P2P) network, proves vital for various applications such as query processing and data mining. This paper presents a distributed, scalable and robust layered overlay (L-overlay) to index and manage multidimensional data in a dynamic P2P network. The proposed method distinguishes between the data and peer layers, with efficient mapping between the two. The data is organized such that semantically similar data objects are accessed hastily. Grid and tree structures are proposed for the peer layer. As application examples of L-overlay in query processing and data mining, k-nearest neighbors query processing and distributed Naive Bayes classification algorithms, are proposed. We show the effectiveness of our scheme in static and dynamic environments using simulation. L-overlay is shown to be more efficient than SSW, an available semantic overly, in terms of maintenance and query processing costs.

Modern large-scale information systems often use multiple database management systems, not all of which are necessarily relational. In recent years, NoSQL databases have gained acceptance in certain domains while relational databases remain de facto standard in many others. Many "legacy" information systems also use relational databases. Unlike relational database systems, NoSQL databases do not have a common data model or query language, making it difficult for users to access data in a uniform manner when using a combination of relational and NoSQL databases or simply several different NoSQL database systems. Therefore, the need for uniform data access from such a variety of data sources becomes one of the central problems for data integration. In this paper we provide an overview of the main problems, methods, and solutions for data integration between relational and NoSQL databases, as well as between different NoSQL databases. We focus mainly on the problems of structural, syntactic, and semantic heterogeneity and on proposed solutions for uniform data access, emphasizing some of the more recent proposals.

As various geospatial data are providing people with more and more convenience of daily lives, it's significant to research on managing the increasingly growing geospatial data. This paper proposes a grid-based approach to geospatial data management. There are three layers in our approach. At the storage layer, a shared grid environment is built for massive geospatial data storage. At the logical layer, a geo-middleware that consists of a server, a geospatial operating library and a client toolkit, is developed to enable the geospatial data access and integration. At the application layer, using some mature open-source projects, a prototype system is designed to demonstrate our approach. The feasibility and practicability of our approach is verified by a practical cased. In a word, this grid-based approach has great potential to be improved as a novel solution for massive geospatial data management in the future.

Power grid operation and maintenance decision-making reform is an important part of power system reform. With the construction of massive historical quasi real-time data management platform, the reform of power system is also advancing. However, in the face of massive data explosion, the business level and business logic become disorganized and redundant. Based on the actual situation of Shenzhen Power Supply Bureau, the sg-erp data center is composed of structured data center, massive data center, unstructured data center and power grid GIS data center. With the unprecedented growth of business application data, the data center can improve business logic and promote power system reform. The experimental results show that big data technology has a broad application prospect in the reform of power industry.