Scalable Strategies for Computing with Massive Data

Overlay analysis is a common task in geographic computing that is widely used in geographic information systems, computer graphics, and computer science. With the breakthroughs in Earth observation technologies, particularly the emergence of high-resolution satellite remote-sensing technology, geographic data have demonstrated explosive growth. The overlay analysis of massive and complex geographic data has become a computationally intensive task. Distributed parallel processing in a cloud environment provides an efficient solution to this problem. The cloud computing paradigm represented by Spark has become the standard for massive data processing in the industry and academia due to its large-scale and low-latency characteristics. The cloud computing paradigm has attracted further attention for the purpose of solving the overlay analysis of massive data. These studies mainly focus on how to implement parallel overlay analysis in a cloud computing paradigm but pay less attention to the impact of spatial data graphics complexity on parallel computing efficiency, especially the data skew caused by the difference in the graphic complexity. Geographic polygons often have complex graphical structures, such as many vertices, composite structures including holes and islands. When the Spark paradigm is used to solve the overlay analysis of massive geographic polygons, its calculation efficiency is closely related to factors such as data organization and algorithm design. Considering the influence of the shape complexity of polygons on the performance of overlay analysis, we design and implement a parallel processing algorithm based on the Spark paradigm in this paper. Based on the analysis of the shape complexity of polygons, the overlay analysis speed is improved via reasonable data partition, distributed spatial index, a minimum boundary rectangular filter and other optimization processes, and the high speed and parallel efficiency are maintained.

In order to solve the low prediction accuracy and prediction rate of traditional time series prediction algorithm in the analysis of massive data, according to the characteristics of communication network performance index, time series prediction algorithm based on the big data method is proposed. In addition, based on the traditional three components of time series feature extraction, the concept of emergency component is introduced, and the outlier detection and processing as well as prediction analysis are carried out on the basis of the extraction results. The results show that the algorithm based on big data, through the analysis, fitting, modelling, and prediction of massive data, smaller granularity decomposition of the time series value is conducted, which significantly improved the credibility and accuracy of prediction. Based on the above findings, it is summarized that the time series prediction algorithm has good performance.

Schedulers for symmetric multiprocessing (SMP) machines use sophisticated algorithms to schedule processes onto the available processor cores. Hardware-dependent code and the use of locks to protect shared data structures from simultaneous access lead to poor portability, the difficulty to prove correctness, and a myriad of problems associated with locking such as limiting the available parallelism, deadlocks, starvation, interrupt handling, and so on. In this work we explore what can be achieved in terms of portability and simplicity in an SMP scheduler that achieves similar performance to state-of-the-art schedulers. By strictly limiting ourselves to only lock-free data structures in the scheduler, the problems associated with locking vanish altogether. We show that by employing implicit cooperative scheduling, additional guarantees can be made that allow novel and very efficient implementations of memory-efficient unbounded lock-free queues. Cooperative multitasking has the additional benefit that it provides an extensive hardware independence. It even allows the scheduler to be used as a runtime library for applications running on top of standard operating systems. In a comparison against Windows Server and Linux running on up to 64 cores we analyze the performance of the lock-free scheduler and show that it matches or even outperforms the performance of these two state-of-the-art schedulers in a variety of benchmarks.

New generation of health care IT systems are collecting and storing more and more data of patients. Useful knowledge can be extracted from the data in EMR or PHR to provide medical advises to patients, while through data analysis the result statistics can be used to support the scientific research. However, RDBMSs-based framework is not able to support the requirements of massive health care data storage, management and analysis. To solve the problem, this paper proposes a massive data management and analysis solution based on Hadoop to archive better performance, scalability and fault tolerance. The data management framework is presented. Besides, 2 different data analysis methods based on MapReduce and Hive are proposed. Experiment results of data upload, data query and data analysis show that the performance of the proposed framework is greatly improved, and a brief summary of the performance and the differences between 2 methods of MapReduce and Hive is also discussed.

Next-generation sequencing (NGS) techniques produce high quantities of DNA and RNA sequencing data, enabling variation analysis at a whole genome level. Due to the massive availability of NGS data, the processing and analysis now constitute a serious challenge for research in life sciences. This necessitates the development of novel, user-friendly and scalable software tools, which are able to jointly handle large NGS data sets in order to break the bottleneck that has already moved from data generation to data analysis in high-throughput biology. Here we introduce a highly efficient NGS analysis and visualization software tool (Rikurator), which is designed for researchers who are struggling with NGS data analysis and may not have access to a dedicated bioinformatics team. Rikurator can be applied to a multitude of genomic analysis tasks, including discovery of pathogenic mutations in familial or sporadic sample sets, analysis of structural variants and somatic characterization of cancer genomes. To these ends the software implements interactive variant filtering and annotation methods based on sample comparison, control data (e.g. ExAC), amino acid changes and quality scores, which provide an intuitive view at the studied samples. Shared candidate genes in a set of dozens of individuals can be discovered in a matter of minutes with an ordinary laptop computer. Importantly, the software is able to analyze hundreds of whole-genome sequenced samples using only a modest amount of RAM. The tool also visualizes and integrates genes, variants (VCF file format), sequence reads (BAM) and user defined features (BED/GFF). In addition, Rikurator is not limited to human studies, supporting reference genomes and genome annotations in standard formats. The software supports targeted (e.g. exome), whole-genome, RNA, Oxford Nanopore and PacBio sequencing data. Rikurator can be run on Windows, Linux and Mac, and it will be freely available after publication. Citation Format: Riku Katainen, Veli Mäkinen, Lauri A. Aaltonen, Esa Pitkänen. Fast and scalable software for comparative variant analysis and visualization of massive next-generation sequencing data. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 5281.

With Moore's law supplying billions of transistors on-chip, embedded systems are undergoing a transition from single-core to multi-core to exploit this high transistor density for high performance. However, there exists a plethora of multi-core architectures and the suitability of these multi-core architectures for different embedded domains (e.g., distributed, real-time, reliability-constrained) requires investigation. Despite the diversity of embedded domains, one of the critical applications in many embedded domains (especially distributed embedded domains) is information fusion. Furthermore, many other applications consist of various kernels, such as Gaussian elimination (used in network coding), that dominate the execution time. In this paper, we evaluate two embedded systems multi-core architectural paradigms: symmetric multiprocessors (SMPs) and tiled multi-core architectures (TMAs). We base our evaluation on a parallelized information fusion application and benchmarks that are used as building blocks in applications for SMPs and TMAs. We compare and analyze the performance of an Intel-based SMP and Tilera's TILEPro64 TMA based on our parallelized benchmarks for the following performance metrics: runtime, speedup, efficiency, cost, scalability, and performance per watt. Results reveal that TMAs are more suitable for applications requiring integer manipulation of data with little communication between the parallelized tasks (e.g., information fusion) whereas SMPs are more suitable for applications with floating point computations and a large amount of communication between processor cores.

Multicore machines are an extension of the single core personal computer that include several processors (cores) and a shared memory. As such, they are suitable for running parallel programs, including ParC, that use shared memory. Multicore machines replace single processor personal computers and are thus widely used. In this work we will use the term “core” instead of processor or CPU to indicate that several cores are packed in a single chip or actually in one die, functioning as a multi-processor machine. Basically, the parallelism available in multicore machines is used by the operating system to execute several unrelated processes in parallel (e.g., running two compilations and a web search on different cores). We will demonstrate that multicore machines can be used to execute parallel programs efficiently, programs that spawn many threads all communicating through shared memory. However, as an extension of a single core machine designed for personal computers, the mechanism using the shared memory is not as effective as it would have been had it been designed from scratch as a parallel machine. In fact, the shared memory of multicore machines is basically a simulation of shared memory over the single port memory module of the single core personal computer. Thus, it is important to understand how the shared memory of multicore machines works in order to determine how ParC can be implemented and used by multicore machines.KeywordsShared MemoryMain MemoryMemory ReferenceMemory ModuleCache LineThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The data in the power Internet of Things has the characteristics of multiple types, large amounts of data, and obvious heterogeneity. In order to solve the storage and scheduling problem of massive heterogeneous data in the power grid, this paper designs a hybrid heterogeneous data storage scheduling architecture based on container technology. In order to solve the problem of sensor data analysis with unknown types and dynamic changes, this paper uses the K-Means clustering method to propose a data analysis process based on the idea of unsupervised learning to realize the analysis and adaptation of massive sensor data. In addition, in order to improve scheduling efficiency in big data scenarios, a two-layer scheduling architecture based on YARN and Kubernetes is proposed to achieve unified scheduling at the task level and node level, thereby improving cluster resource utilization. The heterogeneous data storage scheduling architecture proposed in this paper combines the container technology to meet the needs of massive data storage and processing for low latency and improved interrupt response rate and can be applied to power Internet of Things scenarios.KeywordsBig dataTask schedulingContainerKubernetesYARN

Blasting vibration is one of the main negative effects of blasting construction. It is influenced by many external factors and has a complex nonlinear relationship with them. How to predict and reduce the blasting vibration effect while completing the blasting construction is a long-term concern in the blasting industry. With the development of the Internet of Things and mobile devices, various types of data collection in blasting projects have become quick and varied. Many massive, multi-source, heterogeneous blasting vibrations, and geological data have been gathered. Such a large-scale data set implies potentially useful information and presents significant challenge relevant to traditional computation theories and methods as well. Based on the massive data related to blasting vibration, this paper uses Hadoop to establish a distributed computing framework, establishes a big data analysis platform to carry out massive data mining and analysis, and builds a parallel SVM blasting vibration intelligent prediction model based on the Spark parallel computing framework. The blasting vibration velocity and frequency of blasting construction of a project in southwest are selected and compared with the calculated and measured values of the traditional Sadovski's formula to verify the intelligent blasting prediction model in the big data environment. The practice shows that blasting vibration intelligent prediction has high efficiency and higher accuracy than traditional models based on parallel SVM used in large data environment. In this paper, the parallel computing and data mining technologies of big data clusters are applied to intelligent prediction of blasting vibrations, changed the traditional random sampling and single mining analysis models, and extracted the useful information for management, decision-making and production from massive data, improved the efficiency and precision of the intelligent blasting prediction models.

Sharing memory spaces between parallel processes is a common practice in a real-time simulation environment. In an environment where parallel processes can exist on the same machine or on different machines, it is a challenge to develop and maintain reusable software that interfaces with shared memory spaces. The two main problems involved in dealing with shared memory spaces are: 1) providing platform independent access to the memory space, and 2) providing to all processes a consistent view of the structure and content of the memory space. Object-oriented techniques were used to create a software architecture designed to manage shared memory spaces. Object-oriented design patterns were used to present client code with a conceptual model of shared memory spaces while concealing the underlying implementation details. The resulting architecture is largely computing platform independent, with platform specific code being isolated to a few selected classes. The design was implemented in C++ for the NASA Langley Standard Real-Time Simulation (LaSRS++) Application Framework. This paper provides an overview of the design and implementation of the LaSRS++ shared memory management software.

The use of clusters of symmetric multiprocessor (SMP) configurations in database processing has become a key factor in allowing greater scalability. It has also posed many challenges in the implementation of one of the most costly operations within relational algebra: the join operation. When massive data is involved, usually the join cannot be performed in-memory and is processed out of core. In this case, performance depends on an effective use of the memory hierarchy, such that I/O and memory contention are minimized. In this paper, we propose a parallel algorithm for out of core join processing that dynamically adapts its behavior to the resources available in the system. We evaluate and compare our proposal against other parallel approaches in a real SMP cluster in a major commercial database, the IBM/spl reg/ DB2P Universal Database7 product (DB2 UDB). Results show that our proposal outperforms previous work significantly.

The accuracy of power load forecasting is sensitive to various characteristic factors. For the massive historical load data, how to mine the correlation between different attributes in the characteristic data set is the key to improve the accuracy of load forecasting. In this paper, a load forecasting method based on Bi-LSTM network considering data feature analysis is proposed. Firstly, the attribute correlation feature analysis of massive historical load data is realized by using random forest algorithm, and the features with strong correlation with load are selected as the input of the model; Secondly, Bi-LSTM network with good data fitting accuracy is used for load forecasting modeling. Finally, the actual load data of a city is used for simulation analysis. The simulation results show that this method has more advantages in accuracy than traditional forecasting methods, which verifies the effectiveness and practicability of this method.KeywordsMultidimensional characteristicsLoad forecastingRandom forestBi-LSTMFeature selection

With massive high-dimensional data now common-place in research and industry, there is a strong and growing demand for more scalable computational techniques for data analysis and knowledge discovery. In this paper, we review scalable algorithms for learning statistical models on high-dimensional data. Especially, we introduce two techniques of lossless and lossy compressions. The first one is a method using grammar compression. Grammar compression is a lossless compression for texts and has been successfully applied to binary data matrices for scalable learning of statistical models. The second one is a method of lossy compressions named feature maps (FMs). Recently, quite a few number of FMs for kernel approximations have been proposed and have been used in practical applications. Those methods, of which we present a brief survey in this paper, open the door for large-scale analyses of massive and high-dimensional data.

Nowadays there is a boom in social network data streaming from various fields of interest related to finance, engineering, medicine, and general sciences. All these data are modeled as graphs for better analysis. Community detection is one such mechanism for the analysis of such massive data. Many community detection algorithms exist in literature. The existing algorithms are compared by using either real-world or artificial networks (modeled as graphs) but not both. This paper aims to make a comparative study of two popular existing community detection algorithms both on real-world and synthetic data and verify their performance. The approach in this paper makes good use of recent advances in graphical modeling of different social networks. We generated a random graph that represents most of the observed properties of a real-world dataset. The experimental results are tabulated and the computed metrics help in inferring the suitability or scalability of an algorithm for small or massive datasets.

This work is devoted to establishing a comparatively accurate classification model between symptoms, constitutions, and regimens for traditional Chinese medicine (TCM) constitution analysis to provide preliminary screening and decision support for clinical diagnosis. However, for the analysis of massive distributed medical data in a cloud platform, the traditional data mining methods have the problems of low mining efficiency and large memory consumption, and long tuning time, an association rules method for TCM constitution analysis (ARA-TCM) is proposed that based on FP-growth algorithm and the open-source distributed file system in Hadoop framework (HDFS) to make full use of its powerful parallel processing capability. Firstly, the proposed method was used to explore the association rules between the 9 kinds of TCM constitutions and symptoms, as well as the regimen treatment plans, so as to discover the rules of typical clinical symptoms and treatment rules of different constitutions and to conduct an evidence-based medical evaluation of TCM effects in constitution-related chronic disease health management. Secondly, experiments were applied on a self-built TCM clinical records database with a total of 30,071 entries and it is found that the top three constitutions are mid constitution (42.3%), hot and humid constitution (31.3%), and inherited special constitution (26.2%), respectively. What is more, there are obvious promotions in the precision and recall rate compared with the Apriori algorithm, which indicates that the proposed method is suitable for the classification of TCM constitutions. This work is mainly focused on uncovering the rules of “disease symptoms constitution regimen” in TCM medical records, but tongue image and pulse signal are also very important to TCM constitution analysis. Therefore, this additional information should be considered into further studies to be more in line with the actual clinical needs.