- Research Article
7
- 10.1002/cpe.4517
Cloud computing and big data: Technologies and applications
- May 20, 2018
- Concurrency and Computation: Practice and Experience
- Mostapha Zbakh + 3 more +3
Cloud computing and big data: Technologies and applications
Large quantities of data have been generated from multiple sources at exponential rates in the last few years. These data are generated at high velocity as real time and streaming data in variety of formats. These characteristics give rise to challenges in its modeling, computation, and processing. Hadoop MapReduce (MR) is a well known data-intensive distributed processing framework using the distributed file system (DFS) for Big Data. Current implementations of MR only support execution of a single algorithm in the entire Hadoop cluster. In this paper, we propose MapReducePack (MRPack), a variation of MR that supports execution of a set of related algorithms in a single MR job. We exploit the computational capability of a cluster by increasing the compute-intensiveness of MapReduce while maintaining its data-intensive approach. It uses the available computing resources by dynamically managing the task assignment and intermediate data. Intermediate data from multiple algorithms are managed using multi-key and skew mitigation strategies. The performance study of the proposed system shows that it is time, I/O, and memory efficient compared to the default MapReduce. The proposed approach reduces the execution time by 200% with an approximate 50% decrease in I/O cost. Complexity and qualitative results analysis shows significant performance improvement.
Loading PDF
Cloud computing and big data: Technologies and applications
Cloud computing and big data: Technologies and applications
A Maturity Analysis of Big Data Technologies
Big Data OverviewDriven by the need to generate business value, the enterprise has started to adopt Big Data as a solution, migrating from the classical databases and data stores which lack the flexibility and are not optimized enough [1].The changes in the environment make big data analytics attractive to all types of organizations, while the market conditions make it practical. The combination of simplified models for development, commoditization, a wider palette of data management tools, and low-cost utility computing has effectively lowered the barrier to entry. [2]. The concept addresses large volumes of complex data, rapid growing data sets that may come from different autonomous sources.In recent approaches, Big Data is characterized by principles known as the 4VVolume, Variety, Velocity and Veracity [3]. There are opinions about accepting other principles as Big Data characteristics, such as Value.Each day more businesses realize that Big Data is relevant as the applications generate large volumes of data generated automatically, from different data sources, centralized or autonomous. As traditional databases hit limitations when the need of analyzing this data, dedicated solutions must be considered.Important BigData Solutions:* Apache HBase/Hadoop is based on Google's BigTable distributed storage system, which runs on top of Hadoop as a distributed and scalable big data store. This means that HBase can leverage the distributed processing paradigm of the Hadoop Distributed File System (HDFS) and benefit from Hadoop's MapReduce programming model. It combines the scalability of Hadoop with real-time data access as a key/value store and deep analytic capabilities of Map Reduce [4].HBase allows to query for individual records as well as derive aggregate analytic reports across a massive amount of data. It can host large tables with billions of rows, millions of columns and run across a cluster of commodity hardware. HBase is composed of three types of servers in a master slave type of architecture. Region servers are responsible to serve data for reads and writes. When accessing data, clients communicate with HBase Region Servers directly. Region assignment, DDL (create, delete tables) operations are handled by the HBase Master process.* Apache Cassandra is a distributed database used for the administration and management of large amounts of structured data across multiple servers, while providing highly available service and no single point of failure. It provides features such as continuous availability, linear scale performance, data distribution across multiple data centers and cloud availability zones. Cassandra inherits its data architecture from Google's BigTable and it borrows its distribution mechanisms from Amazon's Dynamo.The nodes in a Cassandra cluster are completely symmetrical, all having identical responsibilities. Cassandra also employs consistent hashing to partition and replicate data. it has the capability of handling large amounts of data and thousands of concurrent users or operations per second across multiple data centers.Cassandra has a hierarchy of caching mechanisms and carefully orchestrated disk I/O which ensures speed and data safety. Write operations are sent first to a persistent commit log (ensuring a durable write), then to a write-back cache called a memTable; when the memTable fills, it is flushed to a sorted string table - SSTable - on disk. A Cassandra cluster is organized as a ring, and it uses a partitioning strategy to distribute data evenly.* Redis represents an in-memory data structure store used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperlogs and geospatial indexes with radius queries. Redis stores all data in RAM, allowing lightning fast reads and writes. It runs extremely efficiently in memory and handles high-velocity data, needing simple standard servers to deliver millions of operations per second with submillisecond latency. …
Read moreSurvey on Scientific Data Processing Using Hadoop MapReduce in Cloud Environments
High-performance processing scientific data has enabled the development of digital resources and web-based services that facilitate uses of data beyond those that may have been envisioned by the original data producers. Scientific data processing systems must handle scientific data coming from real time, high-throughput applications. Timely processing of scientific data is important and requires sufficient available resources to achieve high throughput and deliver accurate output results. Cloud Computing provides a low-priced way for small and medium sized enterprises to process scientific data. Based on Hadoop MapReduce of cloud computing, we propose the detailed procedure of scientific data processing algorithm which can improve the overall performance under the shared environment while retaining compatibility with the native Hadoop MapReduce in this paper.
Read moreIEEE IRI 2014 tutorial: Big data management and mining
These tutorials discuss the following: Big Data Management and Mining; Nature-Inspired Cyber Security.
Big Data Processing on Cloud Computing Using Hadoop Mapreduce and Apache Spark
Size of the data used by enterprises has been growing at exponential rates since last few years; handling such huge data from various sources is a challenge for Businesses. In addition, Big Data becomes one of the major areas of research for Cloud Service providers due to a large amount of data produced every day, and the inefficiency of traditional algorithms and technologies to handle these large amounts of data. In order to resolve the aforementioned problems and to meet the increasing demand for high-speed and data-intensive computing, several solutions have been developed by researches and developers. Among these solutions, there are Cloud Computing tools such as Hadoop MapReduce and Apache Spark, which work on the principles of parallel computing. This chapter focuses on how big data processing challenges can be handled by using Cloud Computing frameworks and the importance of using Cloud Computing by businesses
Read moreA hybrid mapreduce model for prolog
Interest in the Map Reduce programming model has been rekindled by Google in the past 10 years; its popularity is mostly due to the convenient abstraction for parallelization details this framework provides. State-of-the-art systems such as Google's, Hadoop or SAGA often provide added features like a distributed file system, fault tolerance mechanisms, data redundancy and portability to the basic Map Reduce framework. However, these features pose an additional overhead in terms of system performance. In this work, we present a Map Reduce design for Prolog which can potentially take advantage of hybrid parallel environments; this combination allies the easy declarative syntax of logic programming with its suitability to represent and handle multi-relational data due to its first order logic basis. Map Reduce for Prolog addresses efficiency issues by performing load balancing on data with different granularity and allowing for parallelization in shared memory, as well as across machines. In an era where multicore processors have become common, taking advantage of a cluster's full capabilities requires the hybrid use of parallelism.
Read moreA framework for clustering and adaptive topic tracking on evolving text and social media data streams.
Recent advances and widespread usage of online web services and social media platforms, coupled with ubiquitous low cost devices, mobile technologies, and increasing capacity of lower cost storage, has led to a proliferation of Big data, ranging from, news, e-commerce clickstreams, and online business transactions to continuous event logs and social media expressions. These large amounts of online data, often referred to as data streams, because they get generated at extremely high throughputs or velocity, can make conventional and classical data analytics methodologies obsolete. For these reasons, the issues of management and analysis of data streams have been researched extensively in recent years. The special case of social media Big Data brings additional challenges, particularly because of the unstructured nature of the data, specifically free text. One classical approach to mine text data has been Topic Modeling. Topic Models are statistical models that can be used for discovering the abstract ``topics'' that may occur in a corpus of documents. Topic models have emerged as a powerful technique in machine learning and data science, providing a great balance between simplicity and complexity. They also provide sophisticated insight without the need for real natural language understanding. However they have not been designed to cope with the type of text data that is abundant on social media platforms, but rather for traditional medium size corpora consisting of longer documents, adhering to a specific language and typically spanning a stable set of topics. Unlike traditional document corpora, social media messages tend to be very short, sparse, noisy, and do not adhere to a standard vocabulary, linguistic patterns, or stable topic distributions. They are also generated at high velocity that impose high demands on topic modeling; and their evolving or dynamic nature, makes any set of results from topic modeling quickly become stale in the face of changes in the textual content and topics discussed within social media streams. In this dissertation, we propose an integrated topic modeling framework built on top of an existing stream-clustering framework called Stream-Dashboard, which can extract, isolate, and track topics over any given time period. In this new framework, Stream Dashboard first clusters the data stream points into homogeneous groups. Then data from each group is ushered to the topic modeling framework which extracts finer topics from the group. The proposed framework tracks the evolution of the clusters over time to detect milestones corresponding to changes in topic evolution,
Read moreVideo Search Using Map Reduce Framework in an Education Cloud
Video search systems have become popular in recent years. The system prompts users to give a string query and retrieves the matched video quickly from the user for playing. There is still no established search method available for scalable fast search in large distributed video databases. In video search systems, when the number of online users reaches a certain scale, it will greatly reduce the response from the server. There are many video resources stored in distributed databases which should be accessible to all the users. If many users access the videos at the same time, it may lead to increase in load on the server. To solve this problem, cloud computing technology is used. A distributed database is used for storing and indexing videos. This system uses Map Reduce paradigm for retrieving the videos in distributed fashion. Map Reduce approach allows splitting the tasks into sub-tasks and then assigning it to various virtual nodes present in the cloud, which are then processed and consolidated to give the final output. Thus, the processing speed is increased while the processing time is greatly reduced.KeywordsCloud computingDistributed databaseMap reduce
Read moreText Classification Using Genetic Programming with Implementation of Map Reduce and Scraping
Classification of text documents on online media is a big data problem and requires automation. Text classification accuracy can decrease if there are many ambiguous terms between classes. Hadoop Map Reduce is a parallel processing framework for big data that has been widely used for text processing on big data. The study presented text classification using genetic programming by pre-processing text using Hadoop map-reduce and collecting data using web scraping. Genetic programming is used to perform association rule mining (ARM) before text classification to analyze big data patterns. The data used are articles from science-direct with the three keywords. This study aims to perform text classification with ARM-based data pattern analysis and data collection system through web-scraping, pre-processing using map-reduce, and text classification using genetic programming. Through web scraping, data has been collected by reducing duplicates as much as 17718. Map-reduce has tokenized and stopped-word removal with 36639 terms with 5189 unique terms and 31450 common terms. Evaluation of ARM with different amounts of multi-tree data can produce more and longer rules and better support. The multi-tree also produces more specific rules and better ARM performance than a single tree. Text classification evaluation shows that a single tree produces better accuracy (0.7042) than a decision tree (0.6892), and the lowest is a multi-tree(0.6754). The evaluation also shows that the ARM results are not in line with the classification results, where a multi-tree shows the best result (0.3904) from the decision tree (0.3588), and the lowest is a single tree (0.356).
Read moreA SURVEY ON PRIVACY PRESERVATIONTECHNIQUES FOR DATA CLUSTERING K-MEANS OVER LARGE-SCALE DATASET
Cloud computing supports different handling of Big-Data applications in such divisions like human services and Sports and so on. Data sets like electronic well being records is regularly contain protection touchy data, which achieves security concerns possibly if the data is discharged/shared to outsiders in cloud. A functional and broadly received procedure for protection safeguarding is to anonymize information by means of speculation to fulfill a given security demonstrates. In this paper, we propose a viable security safeguarding K-implies grouping plan that can be effectively outsourced to cloud servers. The present work permits cloud servers to perform bunching specifically finished encoded data sets, while achieving comparable computational complexity and accuracy compared with clusters over unencrypted ones. In addition to existing techniques, Map Reduce approach also combined in this paper, which makes this work greatly appropriate for Map Reduce condition. Deferentially security approach ensures the results of questions to a database, which will expand the versatility and time proficiency over existing methodologies.
Read moreFedLoop: Looping on Federated MapReduce
The challenges of the Big Data era has motivated many organizations to turn towards distributed, large-scale processing platforms to deal with their data. Map Reduce, and its open-source implementation, Hadoop, has grown to be highly popular with its successful programming model for simplified cluster processing. As a result, many organizations deploy their own Map Reduce/Hadoop clusters to store and process large amounts of useful data. This multicluster setting is gradually growing attention. Numerous previous works have researched on how to execute Map Reduce across geographically distributed data in this setting. However, an important class of applications have not been explored for multicluster Map Reduce: iterative computation. In this paper, we propose Fed Loop, a composite system aimed at providing iterative Map Reduce computation for geographically distributed data in multicluster settings. Fed Loop is capable of transparently executing both iterative and non-iterative Map Reduce jobs on either a single cluster or multiple clusters. For our performance evaluation, two well-known iterative algorithms was executed over 4 independent clusters (16 physical nodes in total) using Fed Loop: K-Means and Page Rank. Results helped us discover how different iterative applications may differ in execution efficiency for mutlicluster environments and how iterative multicluster computation systems like Fed Loop can be optimized.
Read moreA New Architecture for Real Time Data Stream Processing
Processing a data stream in real time is a crucial issue for several applications, however processing a large amount of data from different sources, such as sensor networks, web traffic, social media, video streams and other sources, represents a huge challenge. The main problem is that the big data system is based on Hadoop technology, especially MapReduce for processing. This latter is a high scalability and fault tolerant framework. It also processes a large amount of data in batches and provides perception blast insight of older data, but it can only process a limited set of data. MapReduce is not appropriate for real time stream processing, and is very important to process data the moment they arrive at a fast response and a good decision making. Ergo the need for a new architecture that allows real-time data processing with high speed along with low latency. The major aim of the paper at hand is to give a clear survey of the different open sources technologies that exist for real-time data stream processing including their system architectures. We shall also provide a brand new architecture which is mainly based on previous comparisons of real-time processing powered with machine learning and storm technology.
Read moreAn efficient framework of data mining and its analytics on massive streams of big data repositories
Big Data consists of huge volume of complex growing data sets from several independent sources. With the rapid development of data collection and storage capacity, big data are expanding in all science and engineering domains. The most fundamental challenge for big data applications is to scrutinize the large amount of data and extract required information or knowledge for future usage which is beyond the limit of relational databases with respect to storage and processing of massive quantity of data. Intent of this paper is by considering the big data repository as Twitter, dynamically mine the recent tweets related to Kapoor and Sons movie and perform the data mining operation and analytics on it by overcoming the challenges categorized with respect to the HACE theorem. To handle the massive amount of tweets we have used Hadoop Map Reduce framework to perform data mining analytic operations such as data cleansing, data classification and data clustering. Prediction model for the movie review is built by using Naive Bayes Classifier and accuracy of the prediction is calculated with the help of binomial test as it conforms to the Bernoulli distribution. Clustering of Tweets are obtained on the basis of Location and Hash Tags. Additionally privacy for the user tweets are preserved by using Data mining anomaly Technique, results are displayed with the help of intelligent graphs. As a Performance Evaluation of Map Reduce, the predictive analysis is done by using Map Reduce as well as without using Map Reduce, based on the execution time comparison performance graph is obtained to prove Map Reduce is an Efficient framework for huge volume of data.
Read moreTaking a byte out of big data
Taking a byte out of big data
Implementation of Multi Node Hadoop Virtual Cluster on Open Stack Cloud Environments
Nowadays computing plays a vital role in information technology and all other fields. Yes, the cloud computing is one of the biggest milestone in most leading next generation technology and booming up in IT filed and business sectors. In our day to day life the data is being generated is enormous amount such as tera (TB), peta (PB), zeta (ZB) bytes. Hadoop Map Reduce is the popular distributed computing paradigm to process data intensive jobs in cloud. Completion time goals for deadline of map reduce jobs set by users are becoming crucial in existing cloud based data processing environments like Hadoop. In this paper proposes a real-time implementation of multi node Hadoop virtual cluster on open stack cloud environments and also it processes the huge data sets in parallel different virtual machines (VMs) and it compares average execution time for different node virtual clusters and various size inputs.
Read more