Massive AIS data storage and query based on Hadoop platform

With the rapid development of water transportation industry, the number of ships is increasing, which leads to a sharp increase in AIS data. However, the efficient management of AIS data has become an important issue that needs to be solved. AIS data is a typical stream data. The management of AIS data is facing the problem of high efficiency of spatial-temporal query of massive data. A spatial-temporal hybrid index structure is built to optimize spatial-temporal query of AIS data. This spatial-temporal hybrid index structure is composed of two indexes, one is HBase-STR index for HBase and another is Redis-Geo index for Redis. Finally, the experiment results show that the hybrid index structure effectively improve the efficiency of querying AIS data.

There is high risk of collision between passenger ferries and other nearby ships in Inland River when the ferries are crossing the channel. Some serious accidents might happen and result in catastrophic fatalities. Therefore, researches on the behavior feature of ferries are performed in this paper. Most of the information related to ship's encountering can be extracted from AIS data, which provides with effective assistance to the investigation on the behavior features of ferries. As a result, the AIS data of ships in Jiangsu section of the Yangtze River is introduced, and the linear interpolation is applied after data preprocessing to obtain the navigation data of other ships near ferries at the same time. Based on this, some statistical analysis, including the encounter situation, ship type, size, navigation speed and course when the passenger ferries are encountered with target ships, are analyzed so that the detailed navigation features about the crossing behaviors can be identified in more detail. This study presents a theoretical basis for ensuring the navigation safety of passenger ferries and reducing the probability of accidents.

With the popularization of AIS technology on ships, the scale of maritime traffic data are rapidly increasing. The traditional management system of AIS data based on relational databases is faced with difficulties in expansion and low access efficiency. In view of that, we present AISHS, a distributed system based HBase and Spark to offer efficient storage and near-real-time query to massive AIS Data. In order to avoid the communication overhead caused by regroup the trajectory in query process, AISHS stores all data of a ship in one Region of HBase, constructs a global secondary index by the spatial-temporal attributes of AIS data. Based on this, we implement the co-location between HBase Regions and Spark RDD partitions to ensure efficient spatial-temporal query. Experiments on real datasets show that AISHS is suitable for the storage and query of AIS data.

A novel MP-LSTM method for ship trajectory prediction based on AIS data

Efficient storage and handling of data stored in a peer-to-peer (P2P) network, proves vital for various applications such as query processing and data mining. This paper presents a distributed, scalable and robust layered overlay (L-overlay) to index and manage multidimensional data in a dynamic P2P network. The proposed method distinguishes between the data and peer layers, with efficient mapping between the two. The data is organized such that semantically similar data objects are accessed hastily. Grid and tree structures are proposed for the peer layer. As application examples of L-overlay in query processing and data mining, k-nearest neighbors query processing and distributed Naive Bayes classification algorithms, are proposed. We show the effectiveness of our scheme in static and dynamic environments using simulation. L-overlay is shown to be more efficient than SSW, an available semantic overly, in terms of maintenance and query processing costs.

Protein-protein interaction (PPI) data are often stored in publicly available databases that are often queried by the use of simple query interfaces allowing only key-based queries. Such databases enable the retrieval of one or more proteins that interact with a target protein, using a target protein identifier. Nevertheless, a lot of biological information is available and is spread on different sources and encoded in different ontologies (e.g. Gene Ontology). Annotating existing PPI databases with biological information may result in richer querying interfaces and successively could enable the development of novel algorithms that may use biological information. The OntoPIN project showed the effectiveness of the introduction of a framework for the ontology-based management and querying of Protein-Protein Interaction Data. The OntoPIN framework first merges PPI data with annotations extracted from existing ontologies (e.g. Gene Ontology) and stores annotated data into a database. Then, a semantic-based query interface enables users to query these data by using biological concepts. OntoPIN allows: (a) to extend existing PPI databases by using ontologies, (b) to enable a key-based querying of annotated data, and (c) to offer a novel query interface based on semantic similarity among annotations. This paper presents the current state of the OntoPIN project discussing a case study on human protein interaction data.

Protein-protein interaction (PPI) data stored in publicly available databases are queried by the use of simple query interfaces allowing only key-based queries. A typical query on such databases is based on the use of protein identifiers and enables the retrieval of one or more proteins. Nevertheless, a lot of biological information is available and is spread on different sources and encoded in different ontologies such as Gene Ontology. The integration of existing PPI databases and biological information may result in richer querying interfaces and successively could enable the development of novel algorithms that may use biological information. The OntoPIN project showed the effectiveness of the introduction of a framework for the ontology-based management and querying of Protein-Protein Interaction Data. The OntoPIN framework first merges PPI data with annotations extracted from existing ontologies (e.g. Gene Ontology) and stores annotated data into a database. Then, a semantic-based query interface enables users to query these data by using biological concepts. OntoPIN allows: (a) to extend existing PPI databases by using ontologies, (b) to enable a key-based querying of annotated data, and (c) to offer a novel query interface based on semantic similarity among annotations.

A distributed remote sensing data storage system based on Hadoop platform is designed to solve the problem of low storage efficiency and low security of mass remote sensing data. Uesd virtualization technology to PC as hardware based virtual computing node the Hadoop Cluster required, without increasing the dedicated server clusters and does not change the original PC use conditions, built the Hadoop platform by the virtual machine cluster. Using LandSat-8 satellite image data as the experimental object, through the design of file tree structure, the remote sensing image data according to the band and split into several blocks of image coordinates, combined with the Hadoop distributed file system (HDFS) characteristics of block storage data block, the remote sensing data stored in the distribution of the whole virtual computer cluster in. Build Hadoop platform remote sensing data distributed storage system with low cost.

With the development of agricultural space-time localisation, sensor network and cloud computing, the amount of agricultural data is increasing rapidly and the data structure becomes more complicated and changeable. Currently, the widely used agricultural database is the relational database. This database handles a large amount of data with very limited throughput and is not suitable for organisation and management of distributed data. HBase is a non-relational database of distributed file storage built on Hadoop platform. HBase is suitable for unstructured data storage database and it can handle large volume of data with high scalability. To better store agricultural big data in HBase, we propose a special agricultural data buffer structure which stores data based on the data category and a two-level indexing memory organisation strategy on HBase. The method proposed saves more than a quarter of the time compared to traditional buffering methods. Experimental results show the higher efficiency of the agricultural data buffer structure and memory-organisation strategy.

Proposes a flexible and friendly user interface for data queries using data visualization. There are a variety of ways how computers help users search data, and query interfaces are expected to allow users to choose these ways according to their purposes. These ways are classified into two aspects, filtering and browsing. While filtering is a function which searches data on behalf of users, browsing is function which helps users to search data. In the proposed query interface, both browsing function and filtering function are integrated as follows. The browsing function consists of data visualizing and image zooming. With data visualizing, the original data can be transformed into representations from views, which helps users understand the data more easily. Then the image zooming function displays the visualized representation in various scale factors, which enables users to investigate the outline of the entire data or the detail of partial data. The filtering function is also enhanced by providing several ways of post-processing the filtered results. These include marking the results on the visualized representation as well as extracting them. We implemented an experimental system using the proposed query interface. The system shows that it works well as a post-processing unit to conventional data query interfaces, and it is powerful for flexible handling of the data which are roughly searched with the conventional query interfaces.

Research on the summertime runoff variations and their connections with Asian summer monsoons can give insights for explanation of the hydrological processes and climate change in the Yangtze River basin. Currently, regional studies are focused on the relationships between Asian summer monsoons and meteorological elements. However, research on the runoff variations and their connections to Asian summer monsoons is still scarce. With the help of continuous wavelet transform, cross-wavelet, and wavelet coherence analysis methods, this research explored multiscale summertime runoff variations and their connections with Asian summer monsoons during 1957–2012 in the Yangtze River basin. The results indicate that periodical characteristics of summertime runoff along the mainstream of the Yangtze River basin have distinct differences. Upstream flow is characterized by interannual (1- to 3-year), and downstream by decadal (7- to 10-year) oscillations over certain time periods. In the source region, summertime runoff is primarily influenced by the South Asian summer monsoons (SASM), and mainly in-phase relationships are detected between the summertime runoff and SASM indices. In the midstream and downstream regions, summertime runoff is primarily influenced by the East Asian summer monsoons (EASM), and mainly anti-phase relationships are detected between the summertime runoff and EASM indices.

This study constructs a secure and efficient data storage and sharing architecture for the Internet of Vehicles. A practical Byzantine fault-tolerant consensus algorithm using reputation was proposed to address the shortcomings of roadside unit equipment in the architecture. The experimental results showed that in the performance verification of reinforced data storage, the storage proportion of the research scheme was always lower than that of the comparison algorithm. In the simulated 5000 transactions, it was always lower than 40 MB, while the comparison algorithm reached a maximum of 72 MB. The efficiency of querying registration information and vehicle data increased by 38.2% and 49.1%, respectively. In the shared performance verification, the fully loaded roadside unit devices decreased by a maximum of 2.8%. Under heavy load conditions, the fully loaded nodes decreased by a maximum of 20.7%. In the validation of the proposed consensus algorithm, its performance was more stable in terms of throughput, and the number of transactions processed per second by d (10) remained basically between 380 and 640. Overall, the constructed secure and efficient data storage and sharing architecture for the Internet of Vehicles is efficient and secure, which can be effectively applied in practical IoV systems.

Using panel data derived from 8 cities along Yangtze river in China' s Jiangsu Province, the paper analyzes the impact of industrial agglomeration on environmental pollution on the basis of elaborating their influencing mechanism, applying threshold model and drawing lessons from developed countries to make targeted recommendations. Estimated results are as follows. First, industrial agglomeration exacerbates industrial pollution emissions overall, but the negative environmental effects were weakened following Jiangsu's entering into the new period of time. Second, FDI can significantly reduce industrial pollutant emission. Third, investment in pollution control does not fully play its role in improving environmental quality.

In view of the existing recommendation system in the Big Data have two insufficiencies: poor scalability of the data storage and poor expansibility of the recommendation algorithm, research and analysis the IBCF algorithm and the working principle of Hadoop and HBase platform, a scheme for optimizing the design of personalized recommendation system based on Hadoop and HBase platform is proposed. The experimental results show that, using the HBase database can effectively solve the problem of mass data storage, using the MapReduce programming model of Hadoop platform parallel processing recommendation problem, can significantly improve the efficiency of the algorithm, so as to further improve the performance of personalized recommendation system.

In the vast valley of the Yellow River in the northern part of China, where wheat production has been a primary cultivation for its agriculture, and thereby wheaten food (or wheat powder made food) has been a traditional food for the livelihood of the local peeple since a long time ago, which is in the sharply contrast to the production form and lifestyle in the Yangtze River basin, southern part of China. The wheaten foods in the northern of China are vafious, but mainly are composed of steamed bread, noodles, steamed stuffed bun, and dumpling (jiaozi). This paper focuses on the history regarding mantou and the civil folk culture concerned. In north China, mantou is not only the most popular ＂food＂, but also a usually perceived ＂sacrifice＂ used at the ceremony for Gods and ancestors. Meanwhile, it is also an important ＂gift＂ for the important festivals and rituals, such as the ceremonies of born day, matured year, marriage day, and funerals. The more important is its civil root feature derived from the prolonged logic line of folk lifestyle, presenting as food-sacrifice-gift. Then it becomes an ＂artwork＂, taking the forms as ＂flower mantou ＂or called ＂wheaten flower＂. The ＂flower mantou＂ or ＂Wheaten flower＂ is an real art but exists in a flash time in the lifestyle of the local people. Its symbolic meanings are, however, very rich in representing perfectly the values of the local people, including their outlooks on life, views on happiness and aesthetic awareness. Dangjia village, one of the villages of Hancheng City, Shaanxi Province, is located in the northeast part about 9 kilometers from the city center, the Yellow River is 3.5 kilometers to the east of the village and the No.108 national road is 1.5 kilometers to the west of the village. The surrounding geography of Dangjia village is the joint area of the Loess Plateau and the Qin plain, so it is a typical loess landscape. The village's location is on a river tableland and faces a small river named Bishui river, which finds its way to the Yellow River. There are many fruit trees, Chinese prickly ash trees, vegetable plots and arable land on the both sides of the river and the plateau of the southern part of the village. There are two family names mainly, Dang and Jia, but a few other family names in this village, marriage linkage has been kept between these two families for generations, so, it is a typical village for its clan marriage. In 2006, the households there were 392 and the total population was 1373, with total farmland of 2220 Chinese mu, averagely 7 mu for one family. Apart from the income from agricultural activities ( as productions of wheat, corn and Chinese prickly ash for the major income, most of the villagers heavily rely on their family members who work outside of the village or young laborers who do the odd-jobs in small towns or large cities. The middle reaches of the Yellow River where Dangjia village locates is one of the important wheat production areas in China, wherefore wheaten food is the main food for the local people there. Over a long period of time, the local females have created a very special culture presented by various ritual steamed bread or flower steamed bread. In this paper, steamed bread (Mantou) refers to the steamed food made of leavened dough, which is named ＂Mo＂ or ＂Memo＂ in Shaanxi dialect. ＂Huamo＂(flower steamed bread) means the steamed bread with various models or various colors, whereas, ritual steamed bread refers to the steamed bread or flower steamed bread used as gifts among the local people, and the so called steamed food is the general name for the all kinds of the ＂Mo＂ mentioned above.