Apriori Parallel Improved Algorithm Based on MapReduce Distributed Architecture


 
 
 
 Apriori algorithm is a classical algorithm of association rule mining and widely used for generating frequent item sets. However, the original Apriori algorithm has some limitation such as it needs to scan the dataset many times to discover all frequent itemset and generate huge number of candidate itemset. To overcome these limitations, researchers have made a lot of improvements to the Apriori such as candidate generation, without candidate generation, transaction reduction, partitioning, and sampling. When it comes to mine massive data, these algorithms failed to prove efficiency because limitation of the processing capacity, storage capacity, and main memory constraints. Therefore, parallel and distributed algorithms are developed to perform large-scale computing in ARM on multiple processors. However, the problems with most of the parallel and distributed framework are overheads of managing distributed system, lack of high level parallel programming language, and node failures. Hadoop-MapReduce is an efficient, scalable, and simplified programming model for massive data processing and it also available on cloud environment. Cloud computing offers huge computing resources, and capacities to solve big data challenges. Recently many parallel algorithms have been proposed on Hadoop-MapReduce to enhance the performance of Apriori algorithm but there are some drawbacks: since multiple scan over the dataset is needed to generate candidate itemset, it consume more execution time. The aim of this study is to propose a parallel Transaction Reduction MapReduce Apriori algorithm (TRMR-Apriori) which is reduce unnecessary transaction values and transactions from the dataset in parallel manner to overcome above problems. The experiments show that TRMR-Apriori is able to achieve better execution time to discover frequent itemset those of previous sequential ARM algorithms such as Apriori, AprioriTid, Eclat, and FP-Growth and the previous parallel algorithms such as PApriori, MRApriori, and Modified Apriori with different condition on homogeneous computing environment using Hadoop-MapReduce platform in cloud. Overall, the TRMR-Apriori shows the strength to extract the frequent itemset from massive dataset in cloud.
 
 
 
 

With the advent of the era of big data, intelligent manufacturing is becoming the developing direction of textile and clothing industry. Many textile enterprises are exploring the intelligent techniques to improve production efficiency and product quality. This paper focuses on yarn quality prediction based on the big data of spinning production. First, the association rules algorithms, Aproiori and I_Apriori algorithms, which are commonly used for intelligent prediction in spinning production, are analyzed. Then, aiming to overcome their disadvantages such as low efficiency, time consuming, big error of prediction results under big data, a global optimization strategy based on genetic algorithm is proposed. This strategy optimizes the global search process of Aprioir algorithm for pruning the frequent itemsets by introducing the genetic algorithm, which can avoid the local optimal solution in the search process. Finally, based on the big real production data collected from the spinning factory, the effectiveness of the proposed Apriori algorithm are investigated and compared with the normal Apriori algorithm. The result indicates that the improved algorithm has better efficiency and more accurate prediction results and is good at dealing with the big data environment.

With the advent of the era of big data, social development has fully entered a highly informatized state. The application of information technology in various fields of life and production provides a lot of convenience for people's life and work. However, in the big data environment, how to dig out valuable information from massive data while ensuring the safety and reliability of the information storage, transmission and use process is a key issue that needs to be considered in the information age. This article will combine the characteristics of the big data technology environment to analyze the development status of computer information processing technology. The author conducts research and discussion on the specific application of computer information processing technology under the big data environment.

Blasting vibration is one of the main negative effects of blasting construction. It is influenced by many external factors and has a complex nonlinear relationship with them. How to predict and reduce the blasting vibration effect while completing the blasting construction is a long-term concern in the blasting industry. With the development of the Internet of Things and mobile devices, various types of data collection in blasting projects have become quick and varied. Many massive, multi-source, heterogeneous blasting vibrations, and geological data have been gathered. Such a large-scale data set implies potentially useful information and presents significant challenge relevant to traditional computation theories and methods as well. Based on the massive data related to blasting vibration, this paper uses Hadoop to establish a distributed computing framework, establishes a big data analysis platform to carry out massive data mining and analysis, and builds a parallel SVM blasting vibration intelligent prediction model based on the Spark parallel computing framework. The blasting vibration velocity and frequency of blasting construction of a project in southwest are selected and compared with the calculated and measured values of the traditional Sadovski's formula to verify the intelligent blasting prediction model in the big data environment. The practice shows that blasting vibration intelligent prediction has high efficiency and higher accuracy than traditional models based on parallel SVM used in large data environment. In this paper, the parallel computing and data mining technologies of big data clusters are applied to intelligent prediction of blasting vibrations, changed the traditional random sampling and single mining analysis models, and extracted the useful information for management, decision-making and production from massive data, improved the efficiency and precision of the intelligent blasting prediction models.

Data mining is aimed to extract hidden acknowledge from large dataset, in order to exploit it for predicting future trends and make decisions. Extracting meaningful and useful candidate optimally is handled by several algorithms, mainly those based on exploring incoming data, which can lose information. To address this issue, this paper proposes an algorithm named Incremental Apriori (IncA) for discovering frequent itemsets in transaction databases, which is in fact a variant of the well-known Apriori algorithm. In IncA, we introduce a notion of promising items generated from the original database, an incremental technique applied on incremental database and a health check process to ensure candidate generation completeness. On the theoretical side, our algorithm exhibits the best computational complexity compared to the recent state-of-the-art algorithms. On the other hand, we tested the proposed approach on large synthetic databases. The obtained results prove that IncA reduces the running time as well as the search space and also show that our algorithm performs better than the Apriori algorithm.

Analysis of Transaction Database Using Apriori Algorithm. This study examines the use of the Apriori algorithm for analysing transaction databases. The Apriori algorithm is a fundamental technique in data mining that allows for the efficient discovery of frequent patterns and association rules in large datasets. The Apriori algorithm employs a two-step approach. Initially, it identifies frequent items in the database based on a user-defined minimum support threshold. Subsequently, it generates association rules that describe relationships between these frequent items based on metrics such as confidence and lift. This paper provides an in-depth explanation of the Apriori algorithm, emphasizing its strengths and limitations. Additionally, it presents various applications of the Apriori algorithm in real-world scenarios, such as shopping cart analysis, cross-selling and upselling, and customer segmentation. The significance of this study lies in its comprehensive analysis of the Apriori algorithm and its practical relevance in diverse data mining tasks. It serves as a valuable resource for researchers, practitioners, and anyone seeking to understand the fundamentals and applications of association rule mining. Keywords: Datamining, OLTP, Apriori algorithm, Item sets and Market Basket Analysis

In view of the existing defects, based on the analysis of the ideas and methods improved by predecessors, and combining with the characteristics of mobile platform teaching evaluation, the original classic Apriori algorithm is improved. The improved database only needs to scan the original database once, store the transactions belonging to each frequent itemset, and store these collections in the newly added database. Therefore, when counting the candidate item set generated by self-join, only the intersection of the transaction set stored in the subset of the candidate item set is required to obtain the number of supported items in the candidate item set. The improved algorithm can reduce the scanning times of the original database and compress the scanning range, thus improving the efficiency of the algorithm. The teaching evaluation system based on mobile platform meets the needs of all the teachers and students and provides great convenience. Therefore, it can greatly improve the participation rate and make the data more comprehensive and complete.

At present, storage technology cannot save data completely. Therefore, in such a big data environment, data mining technology needs to be optimized for intelligent data. Firstly, in the face of massive intelligent data, the potential relationship between data items in the database is firstly described by association rules. The data items are measured by support degree and confidence level, and the data set with minimum support is found. At the same time, strong association rules are obtained according to the given confidence level of users. Secondly, in order to effectively improve the scanning speed of data items, an optimized association data mining technology based on hash technology and optimized transaction compression technology is proposed. A hash function is used to count the item set in the set of waiting options, and the count is less than its support, then the pruning is done, and then the object compression technique is used to delete the item and the transaction which is unrelated to the item set, so as to improve the processing efficiency of the association rules. Experiments show that the optimized data mining technology can significantly improve the efficiency of obtaining valuable intelligent data.

Frequent Itemset Mining FIM is one of the most investigated fields of data mining. The goal of Frequent Itemset Mining FIM is to find the most frequently-occurring subsets from the transactions within a database. Many methods have been proposed to solve this problem, and the Apriori algorithm is one of the best known methods for frequent Itemset mining FIM in a transactional database. In this paper, a parallel Frequent Itemset Mining Algorithm, called Accelerating Parallel Frequent Itemset Mining on Graphic Processors with Sorting APFMS, is presented. This algorithm utilizes new-generation graphic processing units GPUs to accelerate the mining process. In it, massive processing units of GPU were used to speed up the frequent item verification procedure on the OpenCL platform. The experimental results demonstrated that the proposed algorithm had dramatically reduced computation time compared with previous methods.

The goal of Frequent Item set Mining (FIM) is to find the biggest number of frequently used subsets from a big transaction database. In previous studies, using the advantage of multicore computing, the execution time of an Apriori algorithm was sharply decreased: when the size of a data set was more than TBs and a single host had been unable to afford a large number of operations by using a number of computers connected into a super computer to speed up execution as being the obvious solution. Some parallel Apriori algorithms, based on the MapReduce framework, have been proposed. However, with these algorithms, memory would be quickly exhausted and communication cost would rise sharply. This would greatly reduce execution efficiency. In this paper, we present an improved reformative Apriori algorithm that uses the length of each transaction to determine the size of the maximum merge candidate item sets. By reducing the production of low frequency item sets in Map function, memory exhaustion is ameliorated, greatly improving execution efficiency.

Aiming at the problem of product family assembly line (PFAL) evolution balancing, an evolution balancing model for PFAL is established and an improved algorithm based on NSGA_II is also proposed. Firstly, the product family evolution and assembly line characteristics are researched and analyzed in big data environment. Tasks on PFAL are divided into platform and personality tasks, and the stability of assembly tasks is mainly considered especially. In the optimization process, a chromosome encoding based on TOP sorting algorithm is adopted, and a new density selection and decoding algorithm is proposed to make up for the deficiencies in traditional algorithms. Finally, an example of PFAL planning is given to verify the effectiveness and feasibility of the improved NSGA_II.

Sonar is a general term for various underwater acoustic devices, which are used to perform underwater target detection, positioning, identification, tracking and underwater communication, navigation, measurement and other tasks. All sonar devices including active sonar and passive sonar both require a signal processor to execute underwater acoustic signal processing. Beamforming is the main and core component of the sonar signal processor, and its processing speed affects the real-time performance of sonar. Thus, this paper first implements a parallel sonar beamforming algorithm to improve the processing speed of the beamforming algorithm. The proposed algorithm was carried out several experiments, and the experimental results demonstrate that the parallelism of multi-core CPU can bring better speedup ratio and efficiency than the traditional serial CPU algorithm.

Financial institutions have long sought to reduce the risk of consumer loans by improving their credit assessment methods. As new information and network technologies enable massive data collections from many different sources, credit assessment has become a challenge in the big data environment. Complicated processing is required to deal with vast, messy data sources and ever-changing loan regulations. This study proposes a decision tree credit assessment approach (DTCAA) to solve the credit assessment problem in a big data environment. Decision tree models offer good interpretability and easily understood rules, with competitive performance capabilities. In addition, DTCAA features various data consolidation methods to eliminate some of the noise in raw data and facilitate the construction of decision tree. By using a large volume data set from one of the biggest car collateral loan companies in Taiwan, this study verifies the efficiency and validity of DTCAA. The results indicate that DTCAA is competitive in various situations and across multiple factors, in support of the applicability of DTCAA to credit assessment practices.

HEPM: High-efficiency pattern mining

A provided data from a transactional database, has supported technical development which can automatically find product association or items saved in the database. This finding association rules between saved product in database known as mining association rules. There are so many theories and algorithm developed for conducting mining association rules. One of algorithm developed is Apriori algorithm. This method, has a main goal to find the maximum frequent itemset. Next, this frequent itemset will be generated into associative rules which are not shown before in the database, become valuable information for considering materials in the decision process. Apriori algorithm is a interpretation technique of mining association rules, will be implemented into a web based software. On the software test which use in some different data, it’s concluded that time for mining association rules depends on the presence of every item in every transaction, total of transaction, minimum support and minimum confidence. For smaller value of minimum support and minimum confidence that entered, program will generate more association rules, vice versa.