Adapting scientific computing problems to clouds using MapReduce

Scientific Computing deals with solving complex scientific problems by applying resource-hungry computer simulation and modeling tasks on-top of supercomputers, grids and clusters. Typical scientific computing applications can take months to create and debug when applying de facto parallelization solutions like Message Passing Interface (MPI), in which the bulk of the parallelization details have to be handled by the users. Frameworks based on the MapReduce model, like Hadoop, can greatly simplify creating distributed applications by handling most of the parallelization and fault recovery details automatically for the user. However, Hadoop is strictly designed for simple, embarrassingly parallel algorithms and is not suitable for complex and especially iterative algorithms often used in scientific computing. The goal of this work is to analyze alternative MapReduce frameworks to evaluate how well they suit for solving resource hungry scientific computing problems in comparison to the assumed worst (Hadoop MapReduce) and best case (MPI) implementations for iterative algorithms.

This paper describes Jaql, a declarative scripting language for analyzing large semistructured datasets in parallel using Hadoop's MapReduce framework. Jaql is currently used in IBM's InfoSphere BigInsights [5] and Cognos Consumer Insight [9] products. Jaql's design features are: (1) a flexible data model, (2) reusability, (3) varying levels of abstraction, and (4) scalability. Jaql's data model is inspired by JSON and can be used to represent datasets that vary from flat, relational tables to collections of semistructured documents. A Jaql script can start without any schema and evolve over time from a partial to a rigid schema. Reusability is provided through the use of higher-order functions and by packaging related functions into modules. Most Jaql scripts work at a high level of abstraction for concise specification of logical operations (e.g., join), but Jaql's notion of physical transparency also provides a lower level of abstraction if necessary. This allows users to pin down the evaluation plan of a script for greater control or even add new operators. The Jaql compiler automatically rewrites Jaql scripts so they can run in parallel on Hadoop. In addition to describing Jaql's design, we present the results of scale-up experiments on Hadoop running Jaql scripts for intranet data analysis and log processing.

Solving systems of linear algebraic equations (SLAE) is a problem often encountered in fields like engineering, physics, computer science and economics. As the number of unknowns in the linear system grows, the runtime and the memory requirement of solving SLAE increases dramatically. To manage this, the execution of the solver should be parallelizable and be performed in distributed environments like cloud. However, to fully take the advantage of cloud infrastructure, one should adapt the SLAE to frameworks that can successfully exploit the cloud resources like the MapReduce framework, which provides automatic parallelism, scalability and fault tolerance. With this goal, in our previous work we have adapted a SLAE algorithm Conjugate Gradient (CG) to Hadoop MapReduce framework. However, the relative complexity and the iterative structure of the CG algorithm makes it unsuited for Hadoop, which is designed for embarrassingly parallel data intensive tasks. One of the most widely used types of embarrassingly parallel algorithms are algorithms based on the Monte Carlo method. This paper presents a Monte Carlo based linear system solver that is adapted to the MapReduce model, and compares the resulting parallel efficiency and scalability to the CG implementation. The detailed analysis shows that the algorithm performs better than the Hadoop CG implementation, however loses to Twister, an alternative MapReduce implementation.

'Data mining' has transformed into a ubiquitous term in the world of IT and Computer Science in recent times. Developments in this field have been countless. Using one of Apriori algorithm's numerous variants with a couple of insightful additions can significantly improve upon the existing standard of Data Mining. In this paper a new approach to considerably reduce the time complexity of the database scan has been proposed. This has been achieved by using the MapReduce framework for Hadoop Distributed File System (HDFS). Coupled with Cloud computing, which handles large data sets and processing remotely, the resultant system - that uses MapReduce for the full table scan, the Pincer-Search Algorithm, and Cloud Computing - is a force to reckon with.

Purpose: The accuracy of medical physics calculations is largely limited by the availability of fast computer hardware, which is in turn strongly limited by budget considerations. This research explores the use of the Cloud Computing paradigm to replace or supplement existing computing infrastructure for medical physics calculations. Cloud computing services allow for on‐demand virtual computing clusters with a pay‐as‐you‐go pricing model. This model has the potential to make large scale Monte Carlo simulations routine in clinical and research activities. Methods and materials: To test the feasibility of medical physics calculations in a cloud computing environment, a distributed Monte Carlo dose calculation framework was implemented with Amazon's EC2 cloud computing service, the Fluka 2008.3b Monte Carlo package, a distributed processing framework written in Python, and the Hadoop MapReduce framework. A variety of relevant calculations were performed: depth dose profiles of proton, electron, and photon beams, along with a simple proton treatment plan using a voxel patient geometry (Zubal phantom). The performance of EC2 was tested by running calculations on 1 to 200 virtual nodes with the different distributed processing frameworks. EC2 prices were compared with costs of comparable local hardware. Results: Relevant medical physics calculations were successfully carried out using the EC2 service. Heavy charged particle, electron, and photon depth dose profiles and a simple treatment plan were successfully calculated remotely. Performance data demonstrated the expected 1/n speed‐up as a function of node number. Two different distributed processing schemes were implemented in the cloud. Conclusion: Our implementations demonstrate the power of the existing cloud computing model. The relatively cheap pay‐as‐you‐go pricing for infrastructure on demand offers an extremely promising new paradigm for clinical and research computing, available to anyone with a network connection.

Cloud computing and big data: Technologies and applications

Purpose: To conceptually prove the cloud‐computing application on large‐scale medical physics problems and to develop a fast, reliable and distributed Cone‐Beam (CB) computed tomography (CT) reconstruction technique using MapReduce, a cloud‐computing framework. Methods: We propose a CBCT reconstruction algorithm based on Hadoop, an open source MapReduce model. Two functions are used to complete the process: a Map function that reconstructs partial results from each projection and a Reduce function that aggregates those partial results to form the whole volume. The map function and the reduce function are automatically parallelized on a large cluster of machines. In addition, the proposed algorithm accelerates the reconstruction in two ways: (1) Typedbytes format speeds up data transferring; (2) Efficient partial results grouping scheme optimizes communication effort. The proposed CBCT algorithm was executed on a commercial computer cloud.Results: 2.3× speedup of reconstruction time is achieved using 18 computing machines (nodes) compared with that on a single‐threaded node. Mean square error between the results using cloud and a single machine is within 0.04%. The linearity between the time and the number of nodes implies higher efficiency is expected with more computing nodes deployed. Hadoop cloud computing is also proved to be fault tolerant; the reconstruction completes successfully with identical results even half of the nodes were terminated manually during the process. Conclusions: Cloud computing provides researchers in medical physics with an efficient and reliable means of solving large‐scale problems. A fast, reliable and scalable CBCT reconstruction algorithm is developed with the MapReduce framework.

Search Based Software Testing (SBST) is one of the most explored fields in Software Testing. It suffers from the optimization problem to execute the Software Under Test (SUT). This problem is addressed mostly using Genetic Algorithm (GA) and it involves three operations namely selection, crossover and mutation to accomplish a global search to yield fitness solution to run the SUT successfully. In existing work, GA is combined with Hadoop MapReduce to give Parallel Genetic Algorithm (PGA). Here, mapper function performs parallel fitness computation and reducer function performs the GA. This PGA generates test suite that makes the entire SUT to get executed. This paper makes an attempt to existing by parallelizing fitness calculation and GA operations to generate search test data for the SUT based on branch coverage criteria.

In this paper, a HadoopMapReduce framework is presented in order to perform parallel processing used for an online CBIR application. HadoopMapReduce framework is used with the intention of integrating an image analysis algorithm into the text-based image search engines without degrading their response time. Therefore, the main objective of the study is a distribution of the image data over a large number of nodes. Some of the techniques used in the paper includes: image indexing and retrieval, parallel processing of downloading images, indexing, and comparing the similarity of retrieved images from various sources.

Scalable and efficient algorithms for the propagation of uncertainty from data through inference to prediction for large-scale problems, with application to flow of the Antarctic ice sheet

Public cloud infrastructure is widely used by enterprises to store and process big data. Cloud and its distributed computing phenomena not only provides scalable, available and affordable solution for storage and compute services but also raises security concerns. Many security solutions that came into existence encrypt data and allow accessing plaintext for data analytics in the confines of secure hardware. However, the fact remains that the large volumes of data is processed in distributed environment involving hundreds of commodity machines. There exist numerous communications between machines in MapReduce computing model. In the process, compromised MapReduce machines or functions are vulnerable to query based inference attacks on big data that lead to leakage of sensitive information. The main focus of this paper is to overcome the problem aforementioned. Towards this end, a methodology is proposed with an underlying algorithm for defeating query based inference attacks on big data in Hadoop. The proposed algorithm is known as Multi-Model Defence Against Query Based Inference Attacks (MMD-QBIA). A realistic attack model is considered for validating the effectiveness of the proposed methodology. Then an integrated framework for security and privacy to big data is evaluated. Cloudera Distribution Hadoop (CDH) is the environment used for empirical study. The experimental results revealed that the proposed solution prevents different kinds of query based inference attacks on big data besides security to big data in Hadoop MapReduce framework.

In recent years, "cloud computing" is one of the most popular terminologies in computer society. Many IT providers and enterprises are eager to move closer to “cloud” in order to solve the current bottlenecks encountered in various fields such as scientific computing. In common, scientific computing needs a huge number of computers available to perform large-scale experiments. These should not only require enormous money to construct, but also need a lot of follow-up time and manpower to maintain and operate. Cloud computing provides a new computing pattern for scientific computing, which could make scientists dynamically access to computing infrastructure on demand, such as computing, storage resources and applications. The emergence of this new computing pattern has brought new opportunities and challenges for computational science. This thesis starts with the introduction of cloud computing, and then probes into the application prospect and results of cloud computing in scientific computing.

Cloud computing has gradually evolved into an infrastructural tool for a variety of scientific research and computing. It has become a trend that lots of products have been migrated from local servers to cloud by many institutions and organizations. One of the challenges in cloud computing now is how to run software efficiently on cloud platforms since lots of original codes are not capable of being executed in parallel on cloud contexts, resulting in that the power of clouds cannot be exerted well. It is costly to redesign and convert current sequential codes into cloud platform. Thus, automatic translation from sequential code to cloud code is one of the directions that could be taken to resolve the problem of code migration in cloud infrastructure. In this paper, a new Java to MapReduce (J2M) translator is developed to achieve the automatic translation from sequential Java to cloud for specific data-parallel code with large loops. This paper will provide details about the design of our translator and evaluate our performance through experiments. The experimental results not only indicate that the translator can precisely translate the sequential Java into cloud codes, but also show that it can achieve very good speedup in performance, and we expect that an almost linear speedup is possible if larger enough data is processed. It is believed that the J2M translator is an ideal stereotype for code migration and will play an important role in the transition era of cloud computing.

In order to solve the problem that how to improve the efficiency of query and calculation in massive data, a method of building quotient cubes in Hadoop plateform which combined the advantage of the quotient cube and MapReduce model is proposed in this paper. At first, all cubes will be established and their aggregate value will be calculated in the Mapping stage. All the key/value pair formed in Mapping stage will be passed to Reducing stage. Equivalence partitioning will be carried out In this stage, and the minimum aggregation cube of each equivalence partitioning will be the key with its aggregate value. According to the minimum aggregation cubes, we can get the quotient cubes. In order to improve the speed of parallel computing and reduce network traffic, equivalence class division will be executed locally after the Map stage, it is named as combiner stage. In this paper, MapReduce model is used to improve the efficiency of building quotient cube because of its ability of parallel computing in a large amount of data. In addition, the experiment proved that, under certain circumstances, increasing the number of Mapper/Reducer task can reduce the building time effectively, and improve the construction efficiency.

An efficient approach to the map-reduce framework and genetic algorithm based whale optimization algorithm for task scheduling in cloud computing environment