Modeling and Verification of MRSCAN Based on MapReduce Framework

Map-Reduce is a powerful solution for processing and analyzing large-scale data. Just as Hadoop and Spark are able to deal with terabyte data and even more. Users only need to complete map and reduce function, the Map-Reduce framework can finish variety jobs. But many machine learning and data mining algorithms cannot leverage the Map-Reduce framework or it would take large efforts to modify the algorithm itself. This issue can be explained by the following ways: 1. Map-Reduce is a batch operation so that most of Map-Reduce frameworks do not built-in to support iteration. 2. Map-Reduce is absolutely parallel, each vertex cannot obtain all records, so none of them could get the global optimal model. In this paper, we proposed a job management tool to enable the Map-Reduce framework to support iteration, called de-parallel. This make the Map-Reduce framework like Hadoop so that Map-Reduce could run more algorithms and support more various tasks. In addition, our tool does not modify the Map-Reduce framework itself. In face MR-Runner interacts with Map-Reduce framework like a client, therefore MR-Runner could be deployed in any single PC instead of Map-Reduce cluster. We also abstract the mainly interface related to Map-Reduce frameworks, this makes our tool portable to the representative Map-Reduce frameworks.

MapReduce is the preferred cloud computing framework used in large data analysis and application processing. MapReduce frameworks currently in place suffer performance degradation due to the adoption of sequential processing approaches with little modification and thus exhibit underutilization of cloud resources. To overcome this drawback and reduce costs, we introduce a Parallel MapReduce (PMR) framework in this paper. We design a novel parallel execution strategy of Map and Reduce worker nodes. Our strategy enables further performance improvement and efficient utilization of cloud resources execution of Map and Reduce functions to utilize multicore environments available with computing nodes. We explain in detail makespan modeling and working principle of the PMR framework in the paper. Performance of PMR is compared with Hadoop through experiments considering three biomedical applications. Experiments conducted for BLAST, CAP3, and DeepBind biomedical applications report makespan time reduction of 38.92%, 18.00%, and 34.62% considering the PMR framework against Hadoop framework. Experiments' results prove that the PMR cloud computing platform proposed is robust, cost-effective, and scalable, which sufficiently supports diverse applications on public and private cloud platforms. Consequently, overall presentation and results indicate that there is good matching between theoretical makespan modeling presented and experimental values investigated.

Due to the rapid growth of network technology, huge volume and distinct data sent via networks is expanding constantly. The situation shows how complex and dense cyber attacks and hazards are developing. Due to the rapid advancement in network density, cyber security specialists find it difficult to monitor all network activity. Due to frequent and sophisticated cyber attacks, it is becoming more challenging to detect and identify abnormalities in network events. The use of deep learning provides a variety of tools and strategies for automated cyber-attack detection as well as quick attack-type prediction as well as evaluation. This work introduces a novel cyber-attack detection and mitigation process under the following phases including preprocessing, feature extraction via the Map Reduce framework that handles the big data, feature selection, attack detection and mitigation. The Improved Normalisation process is achieved on the preprocessing phase. The work is examined from a big data perspective; hence Map Reduce framework is utilised for this. As a result, the framework will manage the feature extraction process, where features including statistical features, raw features, improved correlation-based features, and info gain-based features will be extracted. Following feature extraction, the Improved Recursive Feature Elimination procedure is processed that selects the relevant features. The hybrid detection model, which combines Recurrent Neural Networks (RNN) Deep and Belief Networks (DBN) is used to detect the attacks. Once an attack has been detected, the attacker must be mitigated. To accomplish this, an improved BAIT-based mitigation procedure is used. The two datasets used in this work are, namely, Intrusion Detection Systems (IDS) 2018 Intrusion CSVs (CSE-CIC-IDS2018) and UNSW_NB15. Finally, the suggested model and the alternative methods are contrasted using a variety of measures such as accuracy, sensitivity, specificity, precision, FDR, FNR and FPR.

Adapting scientific computing problems to clouds using MapReduce

 Abstract—This paper presents a monitoring and notification system based on risk analysis using map-reduce framework that can do risk prediction using big data. By using social media and access records of their users, the proposed system determines their risks. Also, it monitors the restricted area through setting them and notifies the administrator as soon as the intrusions are detected. So, it will provide a control situation for administrator in real time and can detect the risky moments that may occur in advance. Therefore it can reduce the probability of risk of security penetration. Also, since it receives the control situation and provides real-time control screen via smartphone, it is possible to improve the convenience of intrusion management in a very fast and effective way.

The traditional MapReduce paradigm lacks in expressivity. However it is popular for its fault-tolerance, automatic parallelism, and the ease of distributed computing. Our work aims to tackle this by introducing an expressive MapReduce framework. Expressivity has its theoretical foundations in description logics. It allows for reasoning tasks such as subsumption. We have picked description logic eλ+ for the expressivity due to its feasible computational complexity. In this work we will show how a tree structure be used as the basis for expressive eλ+ MapReduce. Trees will be pruned to sub-trees and re-assembled in parallel and distributed MapReduce framework. As a result, large scale linked data reasoning on MapReduce framework would be possible in the expressive MapReduce framework.

A big-data oriented recommendation method based on multi-objective optimization

The tremendous increase in the amount of available research documents impels researchers to propose topic models to extract the latent semantic themes of a documents collection. However, how to extract the hidden topics of the documents collection has become a crucial task for many topic model applications. Moreover, conventional topic modeling approaches suffer from the scalability problem when the size of documents collection increases. In this paper, the Correlated Topic Model with variational Expectation-Maximization algorithm is implemented in MapReduce framework to solve the scalability problem. The proposed approach utilizes the dataset crawled from the public digital library. In addition, the full-texts of the crawled documents are analysed to enhance the accuracy of MapReduce CTM. The experiments are conducted to demonstrate the performance of the proposed algorithm. From the evaluation, the proposed approach has a comparable performance in terms of topic coherences with LDA implemented in MapReduce framework.

An efficient approach to the map-reduce framework and genetic algorithm based whale optimization algorithm for task scheduling in cloud computing environment

One of the prominent issues in Genetic Algorithm (GA) is premature convergence on local optima. This restricts the enhanced optimal solution searching in the entire search space. Population size is one of the influencing factors in Genetic Algorithm. Increasing the population size will improvise the randomized searching and maintains the diversity in the population. It also increases its computational complexity. Especially in GA Biclustering (GABiC), the search should be randomized to find more optimal patterns. In this paper, a novel approach for population setup in MapReduce framework is proposed. The maximal population is split into population sets, and these groups will proceed searching in parallel using MapReduce framework. This approach is attempted for biclustering the gene expression dataset in this paper. The performance of this proposed work seems promising on comparing its results with those obtained from previous hybridized optimization approaches. This approach will also handle data scalability issues and applicable to the big data biclustering problems.

Pillar K-means cluster is one of clustering algorithm developed from K-means algorithm which is more effective than another similar algorithm. The advanced development of electronic data has brought two main impacts in data clustering algorithm, including how to store big data and how to process this type of data. This paper discussed the analysis and implementation of Pillar K-means algorithm in a distributed system using MapReduce Framework. We cited the existing Pillar K-Means Algorithm and implemented it by using MapReduce Framework. Various functions had been implemented simultaneously such as Mapper and Reducer, which are part of MapReduce Framework. The result showed that there was a positive performance regarding efficiency and scalability of Pillar K-means by using synthetic datasets.

Clustering plays a fundamental task in the process of data mining, which remains more demanding due to the ever-increasing dimension of accessible datasets. Big data is considered more populous as it has the ability to handle various sources and formats of data under numerous highly developed technologies. This paper devises a robust and effective optimization-based Internet of Things (IoT) routing technique, named Student Psychology Based Optimization (SPBO) -based routing for the big data clustering. When the routing phase is done, big data clustering is carried out using the Deep Fractional Calculus-Improved Invasive Weed Optimization fuzzy clustering (Deep FC-IIWO fuzzy clustering) approach. Here, the Mapreduce framework is used to minimizing the over fitting issues during big data clustering. The process of feature selection is performed in the mapper phase in order to select the major features using Minkowski distance, whereas the clustering procedure is carried out in the reducer phase by Deep FC-IIWO fuzzy clustering, where the FC-IIWO technique is designed by the hybridization of Improved Invasive Weed Optimizer (IIWO) and Fractional Calculus (FC). The developed SPBO-based routing approach achieved effective performance in terms of energy, clustering accuracy, jaccard coefficient, rand coefficient, computational time and space complexity of 0.605 J, 0.935, 0.947, 0.954, 2100.6 s and 72KB respectively.

We design and implement Mars, a MapReduce framework, on graphics processors (GPUs). MapReduce is a distributed programming framework originally proposed by Google for the ease of development of web search applications on a large number of commodity CPUs. Compared with CPUs, GPUs have an order of magnitude higher computation power and memory bandwidth, but are harder to program since their architectures are designed as a special-purpose co-processor and their programming interfaces are typically for graphics applications. As the first attempt to harness GPU's power for MapReduce, we developed Mars on an NVIDIA G80 GPU, which contains over one hundred processors, and evaluated it in comparison with Phoenix, the state-of-the-art MapReduce framework on multi-core CPUs. Mars hides the programming complexity of the GPU behind the simple and familiar MapReduce interface. It is up to 16 times faster than its CPU-based counterpart for six common web applications on a quad-core machine.

The MapReduce framework has been widely adopted for processing Big Data in the cloud. While efficient, MapReduce offers very complicated (if any) means for users to request nodes that satisfy certain security and privacy requirements to process their data.In this paper, we propose a novel approach to seamlessly integrate node selection control to the MapReduce framework for increasing data security. We define a succinct yet expressive policy language for MapReduce environments, according to which users can specify their security and privacy concerns over their data. Then, we propose corresponding data preprocessing techniques and node verification protocols to achieve strong policy enforcement. Our experimental study demonstrates that, compared to the traditional MapReduce framework, our policy control mechanism allows to achieve data privacy without introducing significant overhead.KeywordsMapReduceNode selectionAccess control

Filtering uninteresting data is important to utilize “big data”. Skyline query is popular technique to filter uninteresting data, in which it selects a set of objects that are not dominated by another from a given large database. However, a skyline query often retrieves too many objects to analyze intensively especially for high-dimensional dataset. To solve the problem, k-dominant skyline queries have been introduced. The size of databases sometimes become too large to compute in a centralized environment. Conventional algorithms for computing k-dominant skyline queries are not well suited for parallel and distributed environments, such as the MapReduce framework. In this paper, we consider an efficient parallel algorithm to process k-dominant skyline query in MapReduce framework. Extensive experiments demonstrate the scalability of proposed algorithm for synthetic big datasets under different settings of data distribution, dimensionality, and cardinality.