GM-Tree: An efficient frequent pattern mining technique for dynamic database

Frequent pattern mining is a classic data mining technique, generally applicable to a wide range of application domains, and a mature area of research. The fundamental challenge arises from the combinatorial nature of frequent itemsets, scaling exponentially with respect to the number of unique items. Apriori-based and FPTree-based algorithms have dominated the space thus far. Initial phases of this research relied on the Apriori algorithm and utilized a distributed computing environment; we proposed the Cartesian Scheduler to manage Apriori's candidate generation process. To address the limitation of bottom-up frequent pattern mining algorithms such as Apriori and FPGrowth, we propose the Frequent Hierarchical Pattern Tree (FHPTree): a tree structure and new frequent pattern mining paradigm. The classic problem is redefined as frequent hierarchical pattern mining where the goal is to detect frequent maximal pattern covers. Under the proposed paradigm, compressed representations of maximal patterns are mined using a top-down FHPTree traversal, FHPGrowth, which detects large patterns before their subsets, thus yielding significant reductions in computation time. The FHPTree memory footprint is small; the number of nodes in the structure scales linearly with respect to the number of unique items. Additionally, the FHPTree serves as a persistent, dynamic data structure to index frequent patterns and enable efficient searches. When the search space is exponential, efficient targeted mining capabilities are paramount; this is one of the key contributions of the FHPTree. This dissertation will demonstrate the performance of FHPGrowth, achieving a 300x speed up over state-of-the-art maximal pattern mining algorithms and approximately a 2400x speedup when utilizing FHPGrowth in a distributed computing environment. In addition, we allude to future research opportunities, and suggest various modifications to further optimize the FHPTree and FHPGrowth. Moreover, the methods we offer will have an impact on other data mining research areas including contrast set mining as well as spatial and temporal mining.

Closed frequent similar pattern mining: Reducing the number of frequent similar patterns without information loss

With the phenomenal growth of electronic data and information, there are many demands for developments of efficient and effective systems (tools) to address the issue of performing data mining tasks on data warehouses or multidimensional databases. Association rules describe associations between itemsets (i.e., sets of data items) (or granules). Association mining (or called association rule mining) finds interesting or useful association rules in databases, which is the crucial technique for the development of data mining. Association mining can be used in many application areas, for example, the discovery of associations between customers’ locations and shopping behaviours in market basket analysis. Association mining includes two phases. The first phase is called pattern mining that is the discovery of frequent patterns. The second phase is called rule generation that is the discovery of the interesting and useful association rules in the discovered patterns. The first phase, however, often takes a long time to find all frequent patterns that also include much noise as well (Pei and Han, 2002). The second phase is also a time consuming activity (Han and Kamber, 2000) and can generate many redundant rules (Zaki, 2004) (Xu and Li, 2007). To reduce search spaces, user constraintbased techniques attempt to find knowledge that meet some sorts of constraints. There are two interesting concepts that have been used in user constraint-based techniques: meta-rules (Han and Kamber, 2000) and granule mining (Li et al., 2006). The aim of this chapter is to present the latest research results about data warehousing techniques that can be used for improving the performance of association mining. The chapter will introduce two important approaches based on user constraint-based techniques. The first approach requests users to inputs their meta-rules that describe their desires for certain data dimensions. It then creates data cubes based these meta-rules and then provides interesting association rules. The second approach firstly requests users to provide condition and decision attributes that used to describe the antecedent and consequence of rules, respectively. It then finds all possible data granules based condition attributes and decision attributes. It also creates a multi-tier structure to store the associations between granules, and association mappings to provide interesting rules.

Traditional frequent pattern mining considers equal profit/weight value of every item. Weighted Frequent Pattern (WFP) mining becomes an important research issue in data mining and knowledge discovery by considering different weights for different items. Existing algorithms in this area are based on fixed weight. But in our real world scenarios the price/weight/importance of a pattern may vary frequently due to some unavoidable situations. Tracking these dynamic changes is very necessary in different application area such as retail market basket data analysis and web click stream management. In this paper, we propose a novel concept of dynamic weight and an algorithm DWFPM (dynamic weighted frequent pattern mining). Our algorithm can handle the situation where price/weight of a pattern may vary dynamically. It scans the database exactly once and also eligible for real time data processing. To our knowledge, this is the first research work to mine weighted frequent patterns using dynamic weights. Extensive performance analyses show that our algorithm is very efficient and scalable for WFP mining using dynamic weights.

Frequent pattern mining is a basic problem in data mining and knowledge discovery. The discovered patterns can be used as the input for analyzing association rules, mining sequential patterns, recognizing clusters, and so on. However, discovering frequent patterns in large scale datasets is an extremely time consuming task. Most research in the area of association rule discovery has focused on the method of efficient frequent pattern discovery, e.g. Park, Chen & Yu (1995); Savasere, Omiecinski & Navathe (1995); Han, Pei & Yin (2000); Pei, Han, Lu, Nishio, Tang & Yang (2001). When seeking all associations that satisfy constraints on support and confidence, once frequent patterns have been identified, generating the association rules is trivial. In the last decade, various algorithms have been proposed on this problem, e.g. maximal pattern mining - Grahne & Zhu (2003); closed pattern mining - Pei, Han & Mao (2000); Grahne & Zhu (2003); mining the most interesting frequent patterns - Fu, Kwong & Tang (2000); Han, Wang, Lu & Tzvetkov (2002, 2005); Hirate, Iwahashi & Yamana (2004). However, some challenges are still existed and need to be overcome. In this paper, a mathematical space will be introduced with some new related concepts and propositions to design a new algorithm for solving frequent patterns mining problem. It is hoped that such an improved algorithm will be simple to implement and more efficient.

Data Mining: Mining Frequent Patterns, Associations Rules, and Correlations

Association rule mining is one of the most popular pattern discovery methods used in data mining. Frequent pattern extraction is an essential step in association rule mining. Most of the proposed algorithms for extracting frequent patterns are based on the downward closure lemma concept utilizing the support and confidence framework. In this chapter we investigate an alternative method for mining frequent patterns in a transactional database. Self-Organizing Map (SOM) is an unsupervised neural network that effectively creates spatially organized internal representations of the features and abstractions detected in the input space. It is one of the most popular clustering techniques, and it reveals existing similarities in the input space by performing a topology-preserving mapping. These promising properties indicate that such a clustering technique can be used to detect frequent patterns in a top-down manner as opposed to the traditional approach that employs a bottom-up lattice search. Issues that are frequently raised when using clustering technique for the purpose of finding association rules are: (i) the completeness of association rule set, (ii) the support level for the rules generated, and (iii) the confidence level for the rules generated. We present some case studies analyzing the relationships between the SOM approach and the traditional association rule framework, and propose a way to constrain the clustering technique so that the traditional support constraint can be approximated. Throughout our experiments, we have demonstrated how a clustering approach can be used for discovering frequent patterns.

Mining of frequent patterns is one of the popular knowledge discovery and data mining (KDD) tasks. It also plays an essential role in the mining of many other patterns such as correlation, sequences, and association rules. Hence, it has been the subject of numerous studies since its introduction. Most of these studies find all the frequent patterns from collection of precise data, in which the items within each datum or transaction are definitely known and precise. However, there are many real-life situations in which the user is interested in only some tiny portions of these frequent patterns. Finding all frequent patterns would then be redundant and waste lots of computation. This calls for constrained mining , which aims to find only those frequent patterns that are interesting to the user. Moreover, there are also many real-life situations in which the data are uncertain. This calls for uncertain data mining . In this article, we propose algorithms to efficiently find constrained frequent patterns from collections of uncertain data .

Privacy is very important in Data Mining. Association rule mining, is a important technique, has already been applied in a wide range of areas. Large databases cannot be secured by encryption or other techniques because of complexities. These security issues can be divided into two categories: data hiding and knowledge hiding. This paper deals with the problem of association rule mining which preserves the confidentiality of each database i.e. Knowledge hiding. Knowledge hiding is concerned with the sanitization of confidential knowledge from the data. There are many methods to solve this problem. The one, presented in the paper is called support-based and confidence-based blocking schemes. Apriori algorithm is used for mining rule from database. Knowledge hiding is achieved using ISL and DSR. The improvement over sensitive rule hiding is proposed to offer more accuracy and security with smaller negative impact. In ever changing business environment like enterprises e-businesses, the old fashioned disclosure and database inference protection techniques are not capable to ensure complete data privacy. Privacy is important for the on line disclosure of private information Organizations should be able to evaluate the risk of disclosing information and should adopt new more efficient approaches for information disclosure control, in order to maintain their competitive edge in the market. Not only the data but also the hidden knowledge in the data should be made secure. The sensitive information can be extracted in the form of association rules with association rule mining tools. But this can jeopardize the privacy of the customers. Some other rule could be very critical for the company itself such as buying patterns of very rich customers. For example, an Internet based company may give up selling hardware and may concentrate on selling books and videos. Therefore a rule relating the customer buying patterns of hardware may no longer be sensitive for that company. It is desirable to apply a trivial algorithm that hides data by deleting it randomly. Data Mining is the process of discovering new patterns from large data sets involving methods from statistics and artificial intelligence but also database management. Privacy has become an important issue in Data Mining. Many methods have been brought out to solve this problem. This paper focuses on the problem of association rule mining, which hides the knowledge from each database. This paper reviews the major method of knowledge hiding. Association rule mining, as a very important technique, has already been applied in a wide range of areas. Knowledge hiding is concerned with the sanitization of confidential knowledge from the data. As a result of association rule mining, many useful association rules will be discovered, but at the same time, many privacy rules will also be exposed which do not want others to know. To solve this, limit the mining process, in order to keep these sensitive rules being hidden. There are so many methods to solve this problem. The one that is covered in the paper is just one kind of them, called support-based and confidence-based blocking schemes. Algorithm used For Knowledge Hiding is ISL algorithm or DSR algorithm (2). The main disadvantage of a block in algorithm is the fact that the dataset, apart from the blocked values, is not distorted. Thus, an adversary can disclose the hidden rules by identifying those generating item sets that contain question marks and lead to rules with a maximum confidence that lies above the minimum confidence threshold. If the number of these rules is small then the probability of identifying the sensitive ones among them becomes high. To avoid this problem, a method for selectively removing individual values from a database is introduced to prevent the discovery of a set of rules, while preserving the data for other applications. The problem of building privacy preserving algorithms is considered for one category of data mining techniques, the association rule mining. New metrics is introduced in order to demonstrate how security issues can be taken into consideration in the general framework of association rule mining (1).

Frequent pattern mining techniques treat all items in the database equally by taking into consideration only the presence of an item within a transaction. However, the customer may purchase more than one of the same item, and the unit price may vary among items. High utility pattern mining approaches have been proposed to overcome this problem. As a result, it becomes a very important research issue in data mining and knowledge discovery. On the other hand, incremental and interactive data mining provides the ability to use previous data structures and mining results in order to reduce unnecessary calculations when the database is updated, or when the minimum threshold is changed. In this paper, we propose a tree structure, called IIUT (incremental and interactive utility tree), to solve these problems together. It uses a pattern growth mining approach to avoid the level-wise candidate set generation-and-test problem, and it can efficiently capture the incremental data without any restructuring operation. Moreover, IIUT has the build once mine many property and therefore it is highly suitable for interactive mining. Experimental results show that our tree structure is very efficient and scalable for incremental and interactive high utility pattern mining.

Mining frequent patterns is a general and important issue in data mining. Complex and unstructured (or semi-structured) datasets have appeared in major data mining applications, including text mining, web mining and bioinformatics. Mining patterns from these datasets is the focus of many of the current data mining approaches. We focus on labeled ordered trees, typical datasets of semi-structured data in data mining, and propose a new probabilistic model and its efficient learning scheme for mining labeled ordered trees. The proposed approach significantly improves the time and space complexity of an existing probabilistic modeling for labeled ordered trees, while maintaining its expressive power. We evaluated the performance of the proposed model, comparing it with that of the existing model, using synthetic as well as real datasets from the field of glycobiology. Experimental results showed that the proposed model drastically reduced the computation time of the competing model, keeping the predictive power and avoiding overfitting to the training data. Finally, we assessed our results using the proposed model on real data from a variety of biological viewpoints, verifying known facts in glycobiology.

Mining frequent patterns in transaction databases has been a popular subject in data mining research. Common activities include finding patterns in database transactions, times-series, and exceptions. The Apriori algorithm is a widely accepted method of generating frequent patterns. The algorithm can require many scans of the database and can seriously tax resources. New methods of finding association rules, such as the Frequent Pattern Tree (FP-Tree) have improved performance, but still have problems when new data becomes available and require two scans of the database.This paper proposes a new method, which requires only one scan of the database and supports update of patterns when new data becomes available. We design a new structure called Pattern Repository (PR), which stores all of the relevant information in a highly compact form and allows direct derivation of the FP-Tree and association rules quickly with a minimum of resources. In addition, it supports run-time generation of association rules by considering only those patterns that meet on-line data requirements.

Frequent pattern mining and consequently association rule mining is a useful technique for discovering relationships between items in databases. However, as the size of the data to be analyzed increases or the values of the pruning thresholds decrease, larger number of frequent pattern and more association rules will be generated with little information about the association rules in relation to each other. This research paper discusses a method to segment rules into different sets with no internal conflicts. The goal is to establish an effective method to reduce the difficulty for businesses to review the association rules of different customer segments, and track the behaviors of market segments based on their buying behaviors. The method established in this paper has the advantage of not needing customer information, thus removing the need for businesses to obtain customer information. This removes the threat of intrusions into customer privacy. The method also generates the rule sets based on conflicting rules, and dividing rules based on customer behaviors is more accurate than customer characteristics. The proposed method has been validated by running some tests.

Internet of Things (IoT)-backed smart shopping carts are generating an extensive amount of data in shopping markets around the world. This data can be cleaned and utilized for setting business goals and strategies. Artificial intelligence (AI) methods are used to efficiently extract meaningful patterns or insights from such huge amounts of data or big data. One such technique is Association Rule Mining (ARM) which is used to extract strategic information from the data. The crucial step in ARM is Frequent Itemsets Mining (FIM) followed by association rule generation. The FIM process starts by tuning the support threshold parameter from the user to produce the number of required frequent patterns. To perform the FIM process, the user applies hit and trial methods to rerun the aforesaid routine in order to receive the required number of patterns. The research community has shifted its focus towards the development of top-K most frequent patterns not using the support threshold parameter tuned by the user. Top-K most frequent patterns mining is considered a harder task than user-tuned support-threshold-based FIM. One of the reasons why top-K most frequent patterns mining techniques are computationally intensive is the fact that they produce a large number of candidate itemsets. These methods also do not use any explicit pruning mechanism apart from the internally auto-maintained support threshold parameter. Therefore, we propose an efficient TKIFIs Miner algorithm that uses depth-first search strategy for top-K identical frequent patterns mining. The TKIFIs Miner uses specialized one- and two-itemsets-based pruning techniques for topmost patterns mining. Comparative analysis is performed on special benchmark datasets, for example, Retail with 16,469 items, T40I10D100K and T10I4D100K with 1000 items each, etc. The evaluation results have proven that the TKIFIs Miner is at the top of the line, compared to recently available topmost patterns mining methods not using the support threshold parameter.

Data Mining is a technique for finding, searching, or extracting new information or knowledge from a very large set of data, by integration or merging with other disciplines such as statistics, artificial intelligence, and machine learning, making Data Mining as one of the tools to analyze data and then produce useful information. Association Rule is a process in Data Mining to determine all associative rules that meet the minimum requirements for support (minsup) and confidence (minconf) in a database. In Association Rule, there are 2 methods that can be used, namely a priori method and FP-Growth method, where FP-Growth method is the development of a priori method where a priori method there are still some shortcomings such as there are many patterns of data combinations that often appear (many frequent patterns), many types of items but low minimum support fulfillment, it takes quite a long time because database scanning is done repeatedly to get the ideal frequent pattern. In this study the method used is a priori algorithm method, a priori algorithm method is one of the alternative ways to find the most frequently appearing data sets (frequent itemset) without using candidate generation that is suitable for analyzing a transaction data. Coffee analysis is a Cafe Shop engaged in the sale of food and beverages that many food and beverage sales transactions. Open on November 7, 2021 coffee analysis penetrates 245 sales transactions and this transaction data continues to grow every day.