The research and design of parallel recommendation algorithm based on mapreduce

In the past three decades, with the fast development of China's economy, the business data of electric power industry also showed rapid growth and the data size is continual expanding. Thus, the management systems of electric enterprise groups encountered the performance bottlenecks of massive data processing. Hadoop is an open-source software framework developed for reliable, scalable, and efficient distributed computing and storage, and is widely used in massive data processing. Based on the study of the proposed massive business data processing of electric industry, this paper introduces a novel architecture, which is based on Hadoop ecosystem, for management systems of electric enterprise groups to crack the present performance bottlenecks. First, the business logic and processing produce of management system is analyzed. Then new architecture with Hadoop based distributed processing model is introduced in detailed. At last, experimental results with actual business scenarios and data prove the architecture can effectively resolve present performance bottlenecks of massive business data processing, and greatly improve the performance and efficiency of the management system of electric enterprise groups.

Cloud security is one of the issues that cloud platform needs to focus on. Cloud platform has powerful computing ability and storage resources, which makes it become the target of hackers. Therefore, in addition to the traditional network security configuration, it is necessary to adopt a more complete security defense measure to protect the data processing platform. In this paper, it proposes a distributed anomaly traffic detection technology based on classifier combination according to the characteristics of massive network data processing in Hadoop platform, which can improve the security of massive network data processing platform.

Decision tree is the traditional method to process health-related data. However, there are challenges in developing the distributed decision tree algorithms for massive health data using Hadoop. First, as the datasets are massive, the process of building a decision tree is time-consuming. The second problem is splitting. To solve these issues, we propose MapReduce-based Heart Disease Prediction System (MRHDP) for constructing a decision tree to predict heart disease with the Map Reduce programming model using the Hadoop platform. We develop distributed algorithms using the MapReduce framework, and also we conducted a large-scale experiment on massive data sets. Finally, the results indicate that the MRHDP algorithm exhibits better prediction results for heart disease, indicates better processing with less time productivity and versatility.

Electronic circuits: by E. J. Angelo, Jr. 450 pages, diagrams, 6 × 9 in. New York, McGraw-Hill Book Co. Inc., 1958.Price, $9.00.

The procedure of matching vehicle location data onto road map is very essential for many ITS (Intelligent Transportation System) applications. However, with the boosting deployment of GPS devices in vehicles, the accumulation of huge amount of GPS data caused great challenge on the efficiency and scalability of traditional serial map matching algorithm. In this paper we address the challenge by presenting a novel parallel map matching algorithm to realize high-performance processing of GPS data. The main idea is to adapt the serial map matching algorithm for cloud computing environment by reforming its' data-intensive or I/O-intensive computing stages using MapReduce paradigm. We implemented the algorithm in Hadoop platform and tested its performance by a large GPS dataset exceeds 120 billion GPS records. Experimental results show that our approach is highly efficient and scalable for massive historical GPS data processing.

The sine cosine algorithm’s main idea is the sine and cosine-based vacillation outwards or towards the best solution. The first main contribution of this paper proposes an enhanced version of the SCA algorithm called as ESCA algorithm. The supremacy of the proposed algorithm over a set of state-of-the-art algorithms in terms of solution accuracy and convergence speed will be demonstrated by experimental tests. When these algorithms are transferred to the business sector, they must meet time requirements dependent on the industrial process. If these temporal requirements are not met, an efficient solution is to speed them up by designing parallel algorithms. The second major contribution of this work is the design of several parallel algorithms for efficiently exploiting current multicore processor architectures. First, one-level synchronous and asynchronous parallel ESCA algorithms are designed. They have two favors; retain the proposed algorithm’s behavior and provide excellent parallel performance by combining coarse-grained parallelism with fine-grained parallelism. Moreover, the parallel scalability of the proposed algorithms is further improved by employing a two-level parallel strategy. Indeed, the experimental results suggest that the one-level parallel ESCA algorithms reduce the computing time, on average, by 87.4% and 90.8%, respectively, using 12 physical processing cores. The two-level parallel algorithms provide extra reductions of the computing time by 91.4%, 93.1%, and 94.5% with 16, 20, and 24 processing cores, including physical and logical cores. Comparison analysis is carried out on 30 unconstrained benchmark functions and three challenging engineering design problems. The experimental outcomes show that the proposed ESCA algorithm behaves outstandingly well in terms of exploration and exploitation behaviors, local optima avoidance, and convergence speed toward the optimum. The overall performance of the proposed algorithm is statistically validated using three non-parametric statistical tests, namely Friedman, Friedman aligned, and Quade tests.

As multiple processor systems become more widely accepted the importance of parallel programming increases. In this paper, approaches to the design and analysis of parallel algorithms are investigated. Through several examples, the importance of interprocessor communication in parallel processing is demonstrated. Various techniques that are applicable in the design and analysis of parallel algorithms are examined with emphasis on those techniques that incorporate communication aspects. The paper discusses several models of synchronous and asynchronous parallel computation and their use in analyzing algorithms. Relatively primitive methodologies for designing parallel algorithms are discussed and the need for more general and practical methodologies is indicated.

Overlay analysis is a common task in geographic computing that is widely used in geographic information systems, computer graphics, and computer science. With the breakthroughs in Earth observation technologies, particularly the emergence of high-resolution satellite remote-sensing technology, geographic data have demonstrated explosive growth. The overlay analysis of massive and complex geographic data has become a computationally intensive task. Distributed parallel processing in a cloud environment provides an efficient solution to this problem. The cloud computing paradigm represented by Spark has become the standard for massive data processing in the industry and academia due to its large-scale and low-latency characteristics. The cloud computing paradigm has attracted further attention for the purpose of solving the overlay analysis of massive data. These studies mainly focus on how to implement parallel overlay analysis in a cloud computing paradigm but pay less attention to the impact of spatial data graphics complexity on parallel computing efficiency, especially the data skew caused by the difference in the graphic complexity. Geographic polygons often have complex graphical structures, such as many vertices, composite structures including holes and islands. When the Spark paradigm is used to solve the overlay analysis of massive geographic polygons, its calculation efficiency is closely related to factors such as data organization and algorithm design. Considering the influence of the shape complexity of polygons on the performance of overlay analysis, we design and implement a parallel processing algorithm based on the Spark paradigm in this paper. Based on the analysis of the shape complexity of polygons, the overlay analysis speed is improved via reasonable data partition, distributed spatial index, a minimum boundary rectangular filter and other optimization processes, and the high speed and parallel efficiency are maintained.

SummaryIn this chapter we will survey the techniques of the design of algorithms that effectively solve the computational problems of the previous chapters on parallel computers, reaching a substantial (and frequently dramatic) acceleration of the known sequential algorithms. In some cases this requires us to revise the sequential case approach or even to change it completely. We systematically review the state of the art in this area, in particular, the computational complexity estimates, the algorithms supporting them, and the major techniques for the design of effective parallel algorithms. We include several recent research results and techniques, particularly in sections 4 and 6 and in the appendices. The results of Appendices B and C on computing a maximal independent subset of a vector set and the techniques of the design of parallel algorithms demonstrated in Appendix A may be of interest for the designers of combinatorial and graph algorithms.

Personalised recommendation has gradually become an effective way to solve the problem of information overload in the era of big data. Therefore, in order to improve the efficiency of online learning, this paper discusses the design of online learning resource recommendation algorithm based on improved BP neural network, and the results show that it has high value for popularisation and application. Based on the transmission network, the improved BP neural network of momentum factor can achieve more efficient data mining. After training learning resources and user data, it can match the real score and the predicted score, so as to ensure the accuracy of personalised recommendation. The main contribution of this paper is to propose a recommendation algorithm to online learning resources through improved BP neural network algorithm, and the feasibility of the algorithm is verified. The research method of this paper provides a reference for the research of personalised recommendation algorithm of online resources.

Metalearning and Recommender Systems: A literature review and empirical study on the algorithm selection problem for Collaborative Filtering


 
 
 
 Apriori algorithm is a classical algorithm of association rule mining and widely used for generating frequent item sets. However, the original Apriori algorithm has some limitation such as it needs to scan the dataset many times to discover all frequent itemset and generate huge number of candidate itemset. To overcome these limitations, researchers have made a lot of improvements to the Apriori such as candidate generation, without candidate generation, transaction reduction, partitioning, and sampling. When it comes to mine massive data, these algorithms failed to prove efficiency because limitation of the processing capacity, storage capacity, and main memory constraints. Therefore, parallel and distributed algorithms are developed to perform large-scale computing in ARM on multiple processors. However, the problems with most of the parallel and distributed framework are overheads of managing distributed system, lack of high level parallel programming language, and node failures. Hadoop-MapReduce is an efficient, scalable, and simplified programming model for massive data processing and it also available on cloud environment. Cloud computing offers huge computing resources, and capacities to solve big data challenges. Recently many parallel algorithms have been proposed on Hadoop-MapReduce to enhance the performance of Apriori algorithm but there are some drawbacks: since multiple scan over the dataset is needed to generate candidate itemset, it consume more execution time. The aim of this study is to propose a parallel Transaction Reduction MapReduce Apriori algorithm (TRMR-Apriori) which is reduce unnecessary transaction values and transactions from the dataset in parallel manner to overcome above problems. The experiments show that TRMR-Apriori is able to achieve better execution time to discover frequent itemset those of previous sequential ARM algorithms such as Apriori, AprioriTid, Eclat, and FP-Growth and the previous parallel algorithms such as PApriori, MRApriori, and Modified Apriori with different condition on homogeneous computing environment using Hadoop-MapReduce platform in cloud. Overall, the TRMR-Apriori shows the strength to extract the frequent itemset from massive dataset in cloud.
 
 
 
 

Text categorisation (TC) is the task of assigning predefined categories to text. Most of the machine learning algorithms are sensitive to the features fed into it. Feature selection (FS) is an important preprocessing step to remove redundant and irrelevant terms from the training corpus. This paper proposes an ant colony optimisation (ACO) algorithm to select features for categorising longer documents to closely related categories. Heuristic value for each word is computed by the statistical dependency of the term to a category and its compactness value. Experiments were conducted with documents from 20newsgroup and Reuters-21578 benchmarks. The selected features were fed into the Naïve Bayes classifier. It was observed that the performance of the classifier improves with the features selected by the proposed method. The processes involved in algorithm are time intensive and demand parallelism. Hence, the ACO algorithm was parallelised using the MapReduce programming model of Hadoop.

The authors provide optimal parallel solutions to several fundamental link distance problems set in trapezoided rectilinear polygons. All parallel algorithms are deterministic, run in logarithmic time, have an optimal time-processor product and are designed to run on EREW PRAM. The authors develop techniques (e.g. rectilinear window partition) for solving link distance problems in parallel which are expected to find applications in the design of other parallel computational geometry algorithms. They employ these parallel techniques for example to compute the link diameter, link center, and central diagonal of a rectilinear polygon. Their results also imply an optimal linear-time sequential algorithm for constructing a data structure to support rectilinear link distance queries between points.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This paper examines the use of binary trees in the design of efficient parallel algorithms. Using binary trees, we develop efficient algorithms for several scheduling problems. The shared memory model for parallel computation is used. Our success in using binary trees for parallel computations, indicates that the binary tree is an important and useful design tool for parallel algorithms.