Constrained Frequent Pattern Mining from Big Data Via Crowdsourcing

Closed frequent similar pattern mining: Reducing the number of frequent similar patterns without information loss

Frequent pattern mining is a classic data mining technique, generally applicable to a wide range of application domains, and a mature area of research. The fundamental challenge arises from the combinatorial nature of frequent itemsets, scaling exponentially with respect to the number of unique items. Apriori-based and FPTree-based algorithms have dominated the space thus far. Initial phases of this research relied on the Apriori algorithm and utilized a distributed computing environment; we proposed the Cartesian Scheduler to manage Apriori's candidate generation process. To address the limitation of bottom-up frequent pattern mining algorithms such as Apriori and FPGrowth, we propose the Frequent Hierarchical Pattern Tree (FHPTree): a tree structure and new frequent pattern mining paradigm. The classic problem is redefined as frequent hierarchical pattern mining where the goal is to detect frequent maximal pattern covers. Under the proposed paradigm, compressed representations of maximal patterns are mined using a top-down FHPTree traversal, FHPGrowth, which detects large patterns before their subsets, thus yielding significant reductions in computation time. The FHPTree memory footprint is small; the number of nodes in the structure scales linearly with respect to the number of unique items. Additionally, the FHPTree serves as a persistent, dynamic data structure to index frequent patterns and enable efficient searches. When the search space is exponential, efficient targeted mining capabilities are paramount; this is one of the key contributions of the FHPTree. This dissertation will demonstrate the performance of FHPGrowth, achieving a 300x speed up over state-of-the-art maximal pattern mining algorithms and approximately a 2400x speedup when utilizing FHPGrowth in a distributed computing environment. In addition, we allude to future research opportunities, and suggest various modifications to further optimize the FHPTree and FHPGrowth. Moreover, the methods we offer will have an impact on other data mining research areas including contrast set mining as well as spatial and temporal mining.

As we are living in the era of big data, huge volumes of a wide variety of complex data-which can be of different levels of veracity-are generated or collected at a high velocity from rich sources of data in various real-life applications. A rich source of these big data sources is the Internet of Things (IoT), which include a collection of sensors, smartphones and other mobile devices, wearable devices, as well as other "things" that are capable to operate within the existing Internet infrastructure. Embedded in these big data are valuable knowledge and useful information. Hence, the research problem of data mining from big IoT data have drawn attention of many researchers as it aims to discover implicit, previously unknown and potentially useful information and knowledge from the data. For instance, frequent pattern mining finds sets of frequently co-occurring items in the IoT domains. Associative classification discovers rules revealing relationships among items within the frequent patterns and their associations with the corresponding class labels. Induction based classification uses decision tree or random forest to learn from old big IoT for classifying or making predictions on new data. Over the past quarter of a century, many serial, distributed, parallel, and MapReduce-based (Hadook-based and Spark-based) big data mining algorithms have been proposed. These algorithms are run in local computers, distributed and parallel environments, clusters, grids, clouds and/or data centers. In this paper, we review some of these algorithms, discuss issues and research prospective in mining classification patterns from these big IoT data in fog. Our case study on a real-life application shows the feasibility on classifying real-life big IoT data over fog for urban analytics.

Big data analysis and mining aims to discover implicit, previously unknown, and potentially useful information and knowledge from big databases that contain high volumes of valuable veracious data collected or generated at a high velocity from a wide variety of data sources. Among different big data mining tasks, this chapter focuses on big data analysis and mining for frequent patterns. By relying on the MapReduce programming model, researchers only need to specify the “map” and “reduce” functions to discover frequent patterns from (1) big databases of precise data in a breadth-first manner or in a depth-first manner and/or from (2) big databases of uncertain data. Such a big data analysis and mining process can be sped up. The resulting (constrained or unconstrained) frequent patterns mined from big databases provide users with new insights and a sound understanding of users' patterns. Such knowledge is useful is many real-life information science and technology applications.

Cloud computing and big data: Technologies and applications

Big data analytics and mining aims to discover implicit, previously unknown, and potentially useful information and knowledge from big data sets that contain huge volumes of valuable veracious data collected or generated at a high velocity from a wide variety of rich data sources. Among different big data analytic and mining tasks, this chapter focuses on frequent pattern mining. By relying on the MapReduce programming model, researchers only need to specify the “map” and “reduce” functions to discover (organizational) knowledge from (i) big data sets of precise data in a breadth-first manner or depth-first manner and/or from (ii) big data sets of uncertain data. Such a big data analytics process can be sped up by focusing the mining according to the user-specified constraints that express the user interests. The resulting (constrained or unconstrained) frequent patterns mined from big data sets provide users with new insights and a sound understanding of users' patterns. Such (organizational) knowledge is useful is many real-life information science and technology applications.

Big data analytics and mining aims to discover implicit, previously unknown, and potentially useful information and knowledge from big data sets that contain huge volumes of valuable veracious data collected or generated at a high velocity from a wide variety of rich data sources. Among different big data analytic and mining tasks, this chapter focuses on frequent pattern mining. By relying on the MapReduce programming model, researchers only need to specify the “map” and “reduce” functions to discover (organizational) knowledge from (i) big data sets of precise data in a breadth-first manner or depth-first manner and/or from (ii) big data sets of uncertain data. Such a big data analytics process can be sped up by focusing the mining according to the user-specified constraints that express the user interests. The resulting (constrained or unconstrained) frequent patterns mined from big data sets provide users with new insights and a sound understanding of users' patterns. Such (organizational) knowledge is useful is many real-life information science and technology applications.

Internet of Things (IoT)-backed smart shopping carts are generating an extensive amount of data in shopping markets around the world. This data can be cleaned and utilized for setting business goals and strategies. Artificial intelligence (AI) methods are used to efficiently extract meaningful patterns or insights from such huge amounts of data or big data. One such technique is Association Rule Mining (ARM) which is used to extract strategic information from the data. The crucial step in ARM is Frequent Itemsets Mining (FIM) followed by association rule generation. The FIM process starts by tuning the support threshold parameter from the user to produce the number of required frequent patterns. To perform the FIM process, the user applies hit and trial methods to rerun the aforesaid routine in order to receive the required number of patterns. The research community has shifted its focus towards the development of top-K most frequent patterns not using the support threshold parameter tuned by the user. Top-K most frequent patterns mining is considered a harder task than user-tuned support-threshold-based FIM. One of the reasons why top-K most frequent patterns mining techniques are computationally intensive is the fact that they produce a large number of candidate itemsets. These methods also do not use any explicit pruning mechanism apart from the internally auto-maintained support threshold parameter. Therefore, we propose an efficient TKIFIs Miner algorithm that uses depth-first search strategy for top-K identical frequent patterns mining. The TKIFIs Miner uses specialized one- and two-itemsets-based pruning techniques for topmost patterns mining. Comparative analysis is performed on special benchmark datasets, for example, Retail with 16,469 items, T40I10D100K and T10I4D100K with 1000 items each, etc. The evaluation results have proven that the TKIFIs Miner is at the top of the line, compared to recently available topmost patterns mining methods not using the support threshold parameter.

Recent research in frequent pattern mining (FPM) has shifted from obtaining the complete set of frequent patterns to generating only a representative (summary) subset of frequent patterns. Most of the existing approaches to this problem adopt a two-step solution; in the first step, they obtain all the frequent patterns, and in the second step, some form of clustering is used to obtain the summary pattern set. However, the twostep method is inefficient and sometimes infeasible since the first step itself may fail to finish in a reasonable amount of time. In this paper, we propose an alternative approach to mining frequent pattern representatives based on a uniform sampling of the output space. Our new algorithm, Musk, obtains representative patterns by sampling uniformly from the pool of all frequent maximal patterns; uniformity is achieved by a variant of Markov Chain Monte Carlo (MCMC) algorithm. Musk simulates a random walk on the frequent pattern partial order graph with a prescribed transition probability matrix, whose values are computed locally during the simulation. In the stationary distribution of the random walk, all maximal frequent pattern nodes in the partial order graph are sampled uniformly. Experiments on various kind of graph and itemset databases validate the effectiveness of our approach.

The objective of this book is to introduce the basic concepts of big data computing and then to describe the total solution of big data problems using HPCC, an open-source computing platform. The book comprises 15 chapters broken into three parts. The first part, Big Data Technologies, includes introductions to big data concepts and techniques; big data analytics; and visualization and learning techniques. The second part, LexisNexis Risk Solution to Big Data, focuses on specific technologies and techniques developed at LexisNexis to solve critical problems that use big data analytics. It covers the open source High Performance Computing Cluster (HPCC Systems) platform and its architecture, as well as parallel data languages ECL and KEL, developed to effectively solve big data problems. The third part, Big Data Applications, describes various data intensive applications solved on HPCC Systems. It includes applications such as cyber security, social network analytics including fraud, Ebola spread modeling using big data analytics, unsupervised learning, and image classification. The book is intended for a wide variety of people including researchers, scientists, programmers, engineers, designers, developers, educators, and students. This book can also be beneficial for business managers, entrepreneurs, and investors.

Discovery of frequent patterns has been studied in a variety of data mining settings. In its simplest form, known from association rule mining, the task is to discover all frequent itemsets, i.e., ...

Modern High Performance Computing (HPC) applications, such as Earth science simulations, produce large amounts of data due to the surging of computing power, while big data applications have become more compute-intensive due to increasingly sophisticated analysis algorithms. The needs of both HPC and big data technologies for advanced HPC and big data applications create a demand for integrated system support. In this study, we introduce Scientific Data Processing (SciDP) to support both HPC and big data applications via integrated scientific data processing. SciDP can directly process scientific data stored on a Parallel File System (PFS), which is typically deployed in an HPC environment, in a big data programming environment running atop Hadoop Distributed File System (HDFS). SciDP seamlessly integrates PFS, HDFS, and the widely-used R data analysis system to support highly efficient processing of scientific data. It utilizes the merits of both PFS and HDFS for fast data transfer, overlaps computing with data accessing, and integrates R into the data transfer process. Experimental results show that SciDP accelerates analysis and visualization of a production NASA Center for Climate Simulation (NCCS) climate and weather application by 6x to 8x when compared to existing solutions.

Frequent pattern mining is a basic problem in data mining and knowledge discovery. The discovered patterns can be used as the input for analyzing association rules, mining sequential patterns, recognizing clusters, and so on. However, discovering frequent patterns in large scale datasets is an extremely time consuming task. Most research in the area of association rule discovery has focused on the method of efficient frequent pattern discovery, e.g. Park, Chen & Yu (1995); Savasere, Omiecinski & Navathe (1995); Han, Pei & Yin (2000); Pei, Han, Lu, Nishio, Tang & Yang (2001). When seeking all associations that satisfy constraints on support and confidence, once frequent patterns have been identified, generating the association rules is trivial. In the last decade, various algorithms have been proposed on this problem, e.g. maximal pattern mining - Grahne & Zhu (2003); closed pattern mining - Pei, Han & Mao (2000); Grahne & Zhu (2003); mining the most interesting frequent patterns - Fu, Kwong & Tang (2000); Han, Wang, Lu & Tzvetkov (2002, 2005); Hirate, Iwahashi & Yamana (2004). However, some challenges are still existed and need to be overcome. In this paper, a mathematical space will be introduced with some new related concepts and propositions to design a new algorithm for solving frequent patterns mining problem. It is hoped that such an improved algorithm will be simple to implement and more efficient.

In this paper, we focus on frequent pattern mining using non Boolean similarity functions Several properties and propositions that allow pruning the search space of frequent similar patterns, are proposed Based on these properties, an algorithm for mining frequent similar patterns using non Boolean similarity functions is also introduced We evaluate the quality of the frequent similar patterns computed by our algorithm by means of a supervised classifier based on frequent patterns.

Big data has evolved as a most challenging area in scientific study and research. It has drawn much attention during the last few years. It influences our modern society, business, government, healthcare, research and almost every discipline. In this data-driven era, where data is continuously acquired from a verity of sources for different purposes, the ability to make timely decisions based on available data is a very critical task. The massive data size, variety, velocity, accuracy and high dimensionality presents new challenges to big data. This paper attempts to present some challenges of big data. In addition, a study on the conceptual design of big data architecture presented on specific big data applications. In this paper, we presented research work on big data analytics techniques in the areas of text, sentiment, video, social media and predictive analytics. A comparative analysis on selected big data applications has been presented in detail.