Design and Development of Apriori Algorithm for Sequential to concurrent mining using MPI


 
 
 
 Apriori algorithm is a classical algorithm of association rule mining and widely used for generating frequent item sets. However, the original Apriori algorithm has some limitation such as it needs to scan the dataset many times to discover all frequent itemset and generate huge number of candidate itemset. To overcome these limitations, researchers have made a lot of improvements to the Apriori such as candidate generation, without candidate generation, transaction reduction, partitioning, and sampling. When it comes to mine massive data, these algorithms failed to prove efficiency because limitation of the processing capacity, storage capacity, and main memory constraints. Therefore, parallel and distributed algorithms are developed to perform large-scale computing in ARM on multiple processors. However, the problems with most of the parallel and distributed framework are overheads of managing distributed system, lack of high level parallel programming language, and node failures. Hadoop-MapReduce is an efficient, scalable, and simplified programming model for massive data processing and it also available on cloud environment. Cloud computing offers huge computing resources, and capacities to solve big data challenges. Recently many parallel algorithms have been proposed on Hadoop-MapReduce to enhance the performance of Apriori algorithm but there are some drawbacks: since multiple scan over the dataset is needed to generate candidate itemset, it consume more execution time. The aim of this study is to propose a parallel Transaction Reduction MapReduce Apriori algorithm (TRMR-Apriori) which is reduce unnecessary transaction values and transactions from the dataset in parallel manner to overcome above problems. The experiments show that TRMR-Apriori is able to achieve better execution time to discover frequent itemset those of previous sequential ARM algorithms such as Apriori, AprioriTid, Eclat, and FP-Growth and the previous parallel algorithms such as PApriori, MRApriori, and Modified Apriori with different condition on homogeneous computing environment using Hadoop-MapReduce platform in cloud. Overall, the TRMR-Apriori shows the strength to extract the frequent itemset from massive dataset in cloud.
 
 
 
 

This paper discusses the massive data processing method-Apriori algorithm for mining association rules. The Apriori algorithm repeatedly scans the database for shortcomings. Based on a Hadoop platform to block huge amounts of data, we provide the massive data processing method-Apriori algorithm for mining association rules.

Some problems of mining association rules with linguistic terms are discussed. First, an incremental updating algorithm of association rules with linguistic terms is presented. The collection of frequent linguistic attribute sets and its negative border along with their support count are maintained, which makes scan the entire database once at most in the process of updating association rules. The experiment shows that the updating algorithm can not only update association rules effectively but also avoid the repeated cost. Secondly, the parallel algorithm for mining association rules with linguistic terms is presented. The Boolean parallel mining algorithm is improved to discover frequent linguistic attribute sets, and the association rules with at least confidence are generated on all processors. This parallel mining algorithm has fine scale-up, size-up and speed-up.

The classic Apriori algorithm for discovering frequent itemsets scans the database many times and the pattern matching between candidate itemsets and transactions is used repeatedly, so a large number of candidate itemsets were produced, which results in low efficiency of the algorithm. The improved Apriori algorithm improved it from three aspects: firstly, the strategy of the join step and the prune step was improved when candidate frequent (k+1)-itemsets were generated from frequent k-itemsets; secondly, the method of dealing with transaction was improved to reduce the time of pattern matching to be used in the Apriori algorithm; in the end, the method of dealing with database was improved, which lead to only once scanning of the database during the whole course of the algorithm. According to these improvements, an improved algorithm was introduced. The efficiency of Apriori algorithm got improvement both in time and in space. The experimental results of the improved algorithm show that the improved algorithm is more efficient than the original.

To come over the limitations of Apriori algorithm and association rule mining algorithm based on Genetic Algorithm (GA), this paper proposed a new association rule mining algorithm based on the population-based incremental algorithm (PBIL), which is a kind of distribution estimation algorithms. The proposed association rule-mining algorithm keeps the advantages of GA mining association rules in coding and the fitness function. Through using probability vector possessing learning properties to update the population, the algorithm increases the convergence speed and enhances the searching ability, compared to GA. In the experiment of mining association, rules in blood indices data, PBIL algorithm performs better not only in running time, convergence speed, but also achieve better searching results. Meanwhile, this paper proposed a parallel algorithm for association rule mining based on PBIL and designed a system architecture based on cloud computing for blood indices analysis, providing a good example to apply the new algorithm to cloud computing.

In this paper, the proposed method reduces CPU computation time by reducing transaction scan. The Concept infrequent count is based on minimum threshold support and 2-way searching to reduce execution time during scanning of transaction is introduced in proposed method. There exist several data mining algorithms for finding association rules but one of the candidate generation algorithms named Apriori algorithm is considered for the proposed work.

Discovering the relationship between behavior and personality of learner in the web-based learning environment is a key to guide learners in the learning process. This paper proposes a new concept called personality mining to find the “deep” personality through the observed data about the behavior. First, a learner model which includes personality model and behavior model is proposed. Second, we have designed and implemented an improved algorithm, which is based on Apriori algorithm widely used in market basket analysis, to identify the relationship. Third, we have discussed various issues like constructing the learner model, unifying the value domain of heterogeneous model attributes, and improving Apriori algorithm with decision domain. Experiment result indicated that this algorithm for mining association rules between behavior and personality is feasible and efficient. The algorithm has been used in a web-based learning environment developed at Xi’an Jiaotong University.

Association rule is an important research topic in data mining and knowledge discovery. Traditional algorithms for mining association rules are built on the binary attributes databases, which has three limitations. Firstly, it can not concern quantitative attributes; secondly, it treats each item with the same significance although different item may have different significance; thirdly, only the direct association rules are discovered. Mining fuzzy association rules has been proposed to address the first limitation. In this paper, we put forward an idea for mining indirect weighted association rules to resolve the other two limitations, and a discovery algorithm for mining both direct and indirect weighted fuzzy association rules by integrating these three extensions.

A central part of many algorithms for mining association rules in large data sets is a procedure that finds so called frequent itemsets. This paper proposes a new approach to finding frequent itemsets. The approach reduces a number of passes through an input data set and generalises a number of strategies proposed so far. The idea is to analyse a variable number n of itemset lattice levels in p scans through an input data set. It is shown that for certain values of parameters (n,p) this method provides more flexible utilisation of fast access transient memory and faster elimination of itemsets with low support factor. The paper presents the results of experiments conducted to find how the performance of the association rule mining algorithm depends on the values of parameters (n,p).

Under the background of Internet big data era, the amount of data information in all walks of life is increasing day by day. How to filter, mine and manage different kinds of data information has become an important topic of network communication, public service data analysis, privacy protection and other activities. This paper mainly discusses the algorithm of association rules mining based on Apriori algorithm. The association algorithm is used to process the raw data and privacy data of different network data sets, and dig out the multi-level frequent itemsets in the database.

Association rule mining (ARM) is well-known issue in data mining. It is a combinatorial optimization problem purpose to extract the correlations between items in sizable data-sets. According to the literature study, bio-inspired prove their efficiency in term of time, memory and quality of generated rules. This paper investigates multi-population cooperative version of bat algorithm for association rule mining (BAT-ARM) named MPB-ARM which is based on bat inspired algorithm. The advantage of bat algorithm is the power combination between population-based algorithm and the local search, however, it more powerful in local search. The main factor to judge optimization algorithms is ensuring the interaction between global diverse exploration and local intensive exploitation. To maintain the diversity of bats, in our proposed approach, we introduce a cooperative master-slave strategy between the subpopulations. The experimental results shows that our proposal outperforms other bio-inspired algorithms already exist and cited in the literature including our previous work BAT-ARM.

SufRec, an algorithm for mining association rules: Recursivity and task parallelism

It presents an efficient mining algorithm FreqtTree for discovering all frequent patterns from XML data, and then considers mining global frequent patterns from XML data in distributed environment in this paper. First of all, the XML files are transferred to DOM tree, and then it mines all the frequent patterns from the DOM tree. It's a high efficient algorithm because it adopts the right extension technology and scans the DOM tree only one time. After that, it describes the distributed association rule data mining algorithm DFreqtTree based on DOM tree. At last, this algorithm is implemented and analyzed by Java language.

In the past three decades, with the fast development of China's economy, the business data of electric power industry also showed rapid growth and the data size is continual expanding. Thus, the management systems of electric enterprise groups encountered the performance bottlenecks of massive data processing. Hadoop is an open-source software framework developed for reliable, scalable, and efficient distributed computing and storage, and is widely used in massive data processing. Based on the study of the proposed massive business data processing of electric industry, this paper introduces a novel architecture, which is based on Hadoop ecosystem, for management systems of electric enterprise groups to crack the present performance bottlenecks. First, the business logic and processing produce of management system is analyzed. Then new architecture with Hadoop based distributed processing model is introduced in detailed. At last, experimental results with actual business scenarios and data prove the architecture can effectively resolve present performance bottlenecks of massive business data processing, and greatly improve the performance and efficiency of the management system of electric enterprise groups.

This article delves into the communication quality monitoring system driven by big data under the C/S (client/server) architecture, aiming to provide a comprehensive, real-time, and efficient security monitoring solution for factory production and manufacturing environments. The system integrates LoRa wireless communication technology, big data processing and analysis technology, and innovative dynamic expansion algorithms to achieve real-time monitoring and rapid response to various safety hazards in the production environment. The core of the system lies in the construction of LoRa wireless communication network, which adopts an innovative umbrella shaped network structure to ensure stable and fast transmission of sensor monitoring data to the server. By using the STM32 development board with LoRa module, dynamic expansion of hardware integration modules has been achieved, enhancing the flexibility and scalability of the system. At the level of data transmission and processing, introducing Kafka distributed message queue as a data cache effectively alleviates the pressure of processing massive real-time data. Combining the Spark Streaming streaming data processing framework, a distributed data processing model was constructed to achieve real-time consumption and efficient parallel processing of messages in Kafka queues. At the same time, the designed dynamic extension algorithm model can automatically persist the data of new monitoring points, ensuring the system's rapid adaptation to data changes. In terms of data storage and visualization, a data persistence layer architecture for a big data platform has been constructed. Real time data, historical data, and log data are stored separately in Redis, MySQL, and HDFS systems through a data streaming model, improving data storage efficiency and system performance. Based on Tomcat network server and SSM architecture, a B/S structured web server visualization platform has been developed, which supports users to remotely query the security status of the production environment through computer or mobile browser, and receive alarm notifications in case of abnormalities. The system integration test results show that the communication quality monitoring system performs excellently in wireless communication quality, data processing speed, data storage efficiency, and data visualization, and can meet the complex monitoring needs of factory production and manufacturing environments, providing solid technical support for enterprise safety production. However, there is still room for improvement in real-time data collection loss rate, alarm function completeness, and mobile access experience. Future research will focus on optimizing data transmission software models, adding multi-channel alarm functions, and developing mobile apps to enhance user experience.