Multi-Q: Multiple Queries Optimization Based on MapReduce in Cloud

Map Reduce Programming for Electronic Medical Records Data Analysis on Cloud Using Apache Hadoop, Hive and Sqoop

High-performance processing scientific data has enabled the development of digital resources and web-based services that facilitate uses of data beyond those that may have been envisioned by the original data producers. Scientific data processing systems must handle scientific data coming from real time, high-throughput applications. Timely processing of scientific data is important and requires sufficient available resources to achieve high throughput and deliver accurate output results. Cloud Computing provides a low-priced way for small and medium sized enterprises to process scientific data. Based on Hadoop MapReduce of cloud computing, we propose the detailed procedure of scientific data processing algorithm which can improve the overall performance under the shared environment while retaining compatibility with the native Hadoop MapReduce in this paper.

Achieving Accountable MapReduce in cloud computing

In this digital world, as organizations are evolving rapidly with data centric asset the explosion of data and size of the databases have been growing exponentially. Data is generated from different sources like business processes, transactions, social networking sites, web servers, etc. and remains in structured as well as unstructured form. The term ― Big data is used for large data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data varies in size ranging from a few dozen terabytes to many petabytes of data in a single data set. Difficulties include capture, storage, search, sharing, analytics and visualizing. Big data is available in structured, unstructured and semi-structured data format. Relational database fails to store this multi-structured data. Apache Hadoop is efficient, robust, reliable and scalable framework to store, process, transforms and extracts big data. Hadoop framework is open source and fee software which is available at Apache Software Foundation. In this paper we will present Hadoop, HDFS, Map Reduce and c-means big data algorithm to minimize efforts of big data analysis using Map Reduce code. The objective of this paper is to summarize the state-of-the-art efforts in clinical big data analytics and highlight what might be needed to enhance the outcomes of clinical big data analytics tools and related fields.

Metric databases are databases where a metric distance function is defined for pairs of database objects. In such databases, similarity queries in the form of range queries or k-nearest neighbor queries are the most important queries. In traditional query processing, single queries are issued independently by different users. In many data mining applications, however, the database is typically explored by iteratively asking similarity queries for answers of previous similarity queries. In this paper, we introduce a generic scheme for such data mining algorithms and we investigate two orthogonal approaches, reducing I/O cost as well as CPU cost, to speed-up the processing of multiple similarity queries. The proposed techniques apply to any type of similarity query and to an implementation based on an index or using a sequential scan. Parallelization yields an additional impressive speed-up. An extensive performance evaluation confirms the efficiency of our approach.

A Reverse Top-k Geo-Social Keyword Query (RkGSKQ) aims to find all the users who have a given geo-social object in their top-k geo-social keyword query results. This query is practical in detecting prospective customers for online business in social networks. Existing work on RkGSKQ only explored efficient approaches in answering a single query per time, which could not be efficient in processing multiple queries in a query batch. In many real-life applications, multiple RkGSKQs for multiple query objects can be issued at the same time. To this end, in this paper, we focus on the efficient batch processing algorithm for multiple RkGSKQs. To reduce the overall cost and find concurrently results of multiple queries, we present a group processing framework based on the current state-of-the-art indexing and group pruning strategies to answer multiple RkGSKQs by sharing common CPU and I/O costs. Extensive experiments on three data sets demonstrate the effectiveness and efficiency of our proposed methods.

We have witnessed the fast-growing deployment of Hadoop, an open-source implementation of the MapReduce programming model, for purpose of data-intensive computing in the cloud. However, Hadoop was not originally designed to run transient jobs in which users need to move data back and forth between storage and computing facilities. As a result, Hadoop is inefficient and wastes resources when operating in the cloud. This paper discusses the inefficiency of MapReduce in the cloud. We study the causes of this inefficiency and propose a solution. Inefficiency mainly occurs during data movement. Transferring large data to computing nodes is very time-consuming and also violates the rationale of Hadoop, which is to move computation to the data. To address this issue, we developed a distributed cache system and virtual machine scheduler. We show that our prototype can improve performance significantly when running different applications.

Graph Analytics (GA) is a rapidly growing field that holds significant importance in numerous applications that focuses on analysing the complex relations in a network. One of the crucial concepts in GA is Query Reachability which refers to the ability to find a path connecting two nodes (u, v) in graph G. The multiple attributes in reachability query processing play a significant role to discover some interesting patterns. Existing studies did not deal with processing complex queries in attributed graphs. To fulfill some of the challenges, We contributed to this paper with three ideas for solving complex queries, finding paths, and processing multiple queries in parallel and distributed paradigms. Therefore, 1) Proposed Distributed Set Reachability Query in Attributed Networks (DSRQ_AN) algorithm using distributed computing through the Neo4j graph data platform and Cypher Query Language, 2) Finding the path to the given complex queries, and 3) Proposed a Multiple Query Processing in Parallel and Distributed (MQP_PD) algorithm to execute in a parallel environment. This paper assessed the performance of the proposed approaches through a parallel paradigm on Neo4j slaves. Furthermore, the demonstrated results of parallel execution outperform sequential execution for multiple queries on attributed graphs.

Distributed data processing is becoming a reality. Businesses want to do it for many reasons, and they often must do it in order to stay competitive. While much of the infrastructure for distributed data processing is already there (e.g., modern network technology), a number of issues make distributed data processing still a complex undertaking: (1) distributed systems can become very large, involving thousands of heterogeneous sites including PCs and mainframe server machines; (2) the state of a distributed system changes rapidly because the load of sites varies over time and new sites are added to the system; (3) legacy systems need to be integrated—such legacy systems usually have not been designed for distributed data processing and now need to interact with other (modern) systems in a distributed environment. This paper presents the state of the art of query processing for distributed database and information systems. The paper presents the “textbook” architecture for distributed query processing and a series of techniques that are particularly useful for distributed database systems. These techniques include special join techniques, techniques to exploit intraquery paralleli sm, techniques to reduce communication costs, and techniques to exploit caching and replication of data. Furthermore, the paper discusses different kinds of distributed systems such as client-server, middleware (multitier), and heterogeneous database systems, and shows how query processing works in these systems.

Big Data consists of huge volume of complex growing data sets from several independent sources. With the rapid development of data collection and storage capacity, big data are expanding in all science and engineering domains. The most fundamental challenge for big data applications is to scrutinize the large amount of data and extract required information or knowledge for future usage which is beyond the limit of relational databases with respect to storage and processing of massive quantity of data. Intent of this paper is by considering the big data repository as Twitter, dynamically mine the recent tweets related to Kapoor and Sons movie and perform the data mining operation and analytics on it by overcoming the challenges categorized with respect to the HACE theorem. To handle the massive amount of tweets we have used Hadoop Map Reduce framework to perform data mining analytic operations such as data cleansing, data classification and data clustering. Prediction model for the movie review is built by using Naive Bayes Classifier and accuracy of the prediction is calculated with the help of binomial test as it conforms to the Bernoulli distribution. Clustering of Tweets are obtained on the basis of Location and Hash Tags. Additionally privacy for the user tweets are preserved by using Data mining anomaly Technique, results are displayed with the help of intelligent graphs. As a Performance Evaluation of Map Reduce, the predictive analysis is done by using Map Reduce as well as without using Map Reduce, based on the execution time comparison performance graph is obtained to prove Map Reduce is an Efficient framework for huge volume of data.

Given a very large multimedia database, how to process k-nearest-neighbor queries efficiently? While the sequential scan is one of the most obvious solutions for small-to-moderate multimedia databases, it becomes practically infeasible when the database size grows. Concomitant with the volume and velocity of data, multimedia databases are frequently endowed with a complex distance-based similarity model that supports content-based data access in an adjustable and adaptive manner. Typical for many state-of-the-art distance-based similarity models is an at least quadratic computation time complexity for a single distance evaluation between two multimedia objects. Thus the search for the most query-like multimedia objects is still one of the major challenges.In this paper, we address the problem of optimal k-nearest-neighbor query processing via multiple lower bound approximations in very large multimedia databases. To this end, we propose the concepts of filter-optimality and refinement-optimality and present the Cascading Multi-Step Algorithm and the Interleaved Multi-Step Algorithm for fast query processing. Besides the algorithms’ properties, we study their query processing performance with respect to the number of CPU and I/O operations on large-scale benchmark multimedia databases. Our performance analysis shows how to process k-nearest-neighbor queries in multimedia databases efficiently and provides a guide for further research.

Rapid development of e-commerce and mobile communication opens a new era of big data. In this article, the authors put big data and e-commerce security together. They construct electronic commerce security system from these aspects: the creation of database, the security of information storage, the mining of information based on big data environment thoroughly. The second-generation product distributed platform- Apache Hadoop which is more popular and instant has been brought in. what's more, this article expounds the structure and working process. On the base of this platform, this article analyses the certainty and security of e-commerce transactions data developed on the condition of big data. It puts forward a construction view that people should guide and monitor the behavior of e-commerce, and improve the security system of electronic commerce on the base of data.

For speeding up query processing on Big Data, frequent sub-queries or views may be materialized such that the query processing cost is minimized with optimum cost of maintaining the materialized views and/or queries. Materializing frequent sub-queries and views means that resultant data set of the views reside in the memory of one or more nodes in the cluster, so that it reduces the MapReduce cost, submission and scheduling cost of Distributed File System jobs for query processing. We have defined materialized views as resultant data of frequent sub-queries and aggregation functions of a set of Big Data warehousing queries that are saved for enhancing query performance. The problem is defined as a multi-objective optimization problem for minimizing the total query processing MapReduce cost, MapReduce cost for maintaining the materialized views and the number of views selected for materializing with maximized total size of the views selected. We applied Differential Evolution algorithm and NSGA-II to study their performances for developing a recommendation system for selecting views for materializing in Big Data warehousing.

In the contemporary world, with the emergence of information technologies, Big data is one of the rising IT trends. An overwhelming amount of data and information is generated every day throughout the world. Storing and processing this huge volume of data is named by a ubiquitous term: Big Data Management. The traditional database architectures are not designed to face the challenges associated with huge data. The Apache Hadoop is the well-known software library framework that allows distributed processing of large data sets across clusters of low-end computers using simple programming models. Hadoop is designed to scale up from single servers to thousands of machines, each offering local computation and storage. When HDFS takes in data, it breaks the raw data into separate blocks and distributes them to different nodes(DataNode) in a cluster for parallel processing. But there is no scope to determine, which block is stored in which DataNode. Thus if there is a need to run a query on a specific item/keyword/key, the Hadoop runs the query or analytics program to all its cluster, which somehow increase the overall cost. To overcome these issues, We propose a scheme where we store the data in such a way that the system responses faster to each query and overall processing time decreases. Our motivation is to deliver an efficient and reliable storage and retrieval system in the health-care industry.

Large amounts of data are generated daily, according to the wide usage of social media websites and scientific data. These data need to be stored and analyzed to help decision makers but the traditional database concepts are insufficient. Data warehouse and OLAP are useful technologies in the storage and analysis of big data. Using MapReduce will help to save processing time, using cloud computing will help in saving resources and storage. In this paper, we propose a system that integrates the OLAP and MapReduce over cloud (considering workload balance) in order to enhance the performance of query processing over big data. The proposed system is applied to large amounts of data stored in cubes located in a Peer-to-peer cloud; this process is done using an allocation approach to save resources and query processing times. The proposed system achieves enhancements as time saving in query processing and in resources usage.