A Distributed Framework for Parallel Data Mining Using HPJava

Data Mining extracts the knowledge or interesting information from large set of structured data that are from different sources. Data mining applications are used in a range of areas; they are financial data analysis, retail and telecommunication industries, banking, health care and medicine. In health care, the data mining is mainly used for disease prediction. In data mining, there are several techniques have been developed and used for predicting the diseases that includes data preprocessing, classification, clustering, association rules and sequential patterns. This paper analyses the performance of two classification techniques such as Bayesian and Lazy classifiers for hepatitis and thyroiddataset.This classification task helps to classify the hepatitis dataset into two classes namely live and die and also to classify the thyroid dataset into two classes hyperthyroid or hypothyroid.

Peer-to-Peer (P2P) networks are gaining increasing popularity in many distributed applications such as file-sharing, network storage, web caching, sear- ching and indexing of relevant documents and P2P network-threat analysis. Many of these applications require scalable analysis of data over a P2P network. This paper starts by offering a brief overview of distributed data mining applications and algorithms for P2P environments. Next it discusses some of the privacy concerns with P2P data mining and points out the problems of existing privacy-preserving multi-party data mining techniques. It further points out that most of the nice assumptions of these existing privacy preserving techniques fall apart in real-life applications of privacy-preserving distributed data mining (PPDM). The paper offers a more realistic formulation of the PPDM problem as a multi-party game and points out some recent results.KeywordsData MiningPrivacy PreserveCheap TalkGood NodeDistribute Data MiningThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Nowadays, the process of data mining is one of the most important topics in scientific and business problems. Grid Computing can be used as infrastructures to provide an effective computational support for distributed data mining applications. The paper proposed a data mining architecture from large-scale distributed and semantically heterogeneous data resources in order to efficiently support knowledge acquisition. Grid technology and semantic web technology were combined to solve the difficult problem of distributed data mining. Data mining in the Grid environment can provide highly efficient and powerful data analysis and knowledge discovery solution and semantic web can effectively solve the heterogeneity of data sources and make them available for data mining.

Thanks to the rapid advances in Internet-based applications, data acquisition and storage technologies, petabyte-sized network data collections are becoming more and more common, thus prompting the need for scalable data analysis solutions. By leveraging today's ubiquitous many-core computer architectures and the increasingly popular cloud computing paradigm, the applicability of data mining algorithms to these large volumes of network data can be scaled up to gain interesting insights. This paper proposes NEMICO, a comprehensive Big Data mining system targeted to network traffic flow analyses (e.g., Traffic flow characterization, anomaly detection, multiple-level pattern mining). NEMICO comprises new approaches that contribute to a paradigm-shift in distributed data mining by addressing most challenging issues related to Big Data, such as data sparsity, horizontal scaling, and parallel computation.

Many widely used, latency sensitive, data-parallel distributed systems, such as HDFS, Hive, and Spark choose to use the Java Virtual Machine (JVM), despite debate on the overhead of doing so. This paper analyzes the extent and causes of the JVM performance overhead in the above mentioned systems. Surprisingly, we find that the warm-up overhead, i.e., class loading and interpretation of bytecode, is frequently the bottleneck. For example, even an I/O intensive, 1GB read on HDFS spends 33% of its execution time in JVM warm-up, and Spark queries spend an average of 21 seconds in warm-up.The findings on JVM warm-up overhead reveal a contradiction between the principle of parallelization, i.e., speeding up long running jobs by parallelizing them into short tasks, and amortizing JVM warm-up overhead through long tasks. We solve this problem by designing HotTub, a new JVM that amortizes the warm-up overhead over the lifetime of a cluster node instead of over a single job by reusing a pool of already warm JVMs across multiple applications. The speed-up is significant. For example, using HotTub results in up to 1.8X speedups for Spark queries, despite not adhering to the JVM specification in edge cases.

A significant amount of attention appears to be focusing on how to better collect, analyze, and disseminate information. In doing so, technology is commonly and increasingly looked upon as both a tool, and, in some cases, a substitute, for human resources. One such technology that is playing a prominent role in homeland security initiatives is data mining. Similar to the concept of homeland security, while data mining is widely mentioned in a growing number of bills, laws, reports, and other policy documents, an agreed upon definition or conceptualization of data mining appears to be generally lacking within the policy community (Relyea, 2002). While data mining initiatives are usually purported to provide insightful, carefully constructed analysis, at various times data mining itself is alternatively described as a technology, a process, and/or a productivity tool. In other words, data mining, or factual data analysis, or predictive analytics, as it also is sometimes referred to, means different things to different people. Regardless of which definition one prefers, a common theme is the ability to collect and combine, virtually if not physically, multiple data sources, for the purposes of analyzing the actions of individuals. In other words, there is an implicit belief in the power of information, suggesting a continuing trend in the growth of “dataveillance,” or the monitoring and collection of the data trails left by a person’s activities (Clarke, 1988). More importantly, it is clear that there are high expectations for data mining, or factual data analysis, being an effective tool. Data mining is not a new technology but its use is growing significantly in both the private and public sectors. Industries such as banking, insurance, medicine, and retailing commonly use data mining to reduce costs, enhance research, and increase sales. In the public sector, data mining applications initially were used as a means to detect fraud and waste, but have grown to also be used for purposes such as measuring and improving program performance. While not completely without controversy, these types of data mining applications have gained greater acceptance. However, some national defense/homeland security data mining applications represent a significant expansion in the quantity and scope of data to be analyzed. Moreover, due to their security-related nature, the details of these initiatives (e.g., data sources, analytical techniques, access and retention practices, etc.) are usually less transparent.

A comparative study of MPI implementations on a cluster of SMP workstations

Data mining (DM) is a powerful information technology (IT) tool in today’s competitive business world, especially as our human society entered the Big Data era. From academic point of view, it is an area of the intersection of human intervention, machine learning, mathematical modeling and databases. In recent years, data mining applications have become an important business strategy for most companies that want to attract new customers and retain existing ones. Using mathematical techniques, such as, neural networks, decision trees, mathematical programming, fuzzy logic and statistics, data mining software can help the company discover previously unknown, valid, and actionable information from various and large sources (either databases or open data sources like internet) for crucial business decisions. The algorithms of the mathematical models are implemented through some sort of computer languages, such as C++, JAVA, structured query language (SQL), on-line analysis processing (OLAP) and R. The process of data mining can be categorized as selecting, transforming, mining, and interpreting data. The ultimate goal of doing data mining is to find knowledge from data to support user’s decision. Therefore, data mining is strongly related with knowledge and knowledge management.

Grid is a new solution to computationally and data intensive computing problems. Since the distributed knowledge discovery process is both data and computational intensive, the Grid is a natural platform for deploying a high performance data mining service. In order to improve the performance of data mining applications, an effective method is task parallelization. Existing mechanisms of data mining parallelization are based on NOW or SMP, it is necessary to develop new parallel mechanism for grid feature. In this paper, we present a framework for high performance DDM applications in Computational Grid environments called Data Mining Grid, with the function for decomposing data mining application into subtasks and then combine those subtasks to form directed acyclic graph. This kind of parallel mechanism decomposes application according to the actual computation power of each node in dynamic Grid environment.

Data mining provides the methodology and technology to transform huge amount of data into useful information for decision making. It is a powerful process to extract knowledge and discover new patterns embedded in large data sets. Data mining has been increasingly used in medicine, particularly in cardiology. In fact, data mining applications can greatly benefits all parts involved in cardiology such as patients, cardiologists and nurses. This article aims to perform a systematic mapping study so as to analyze and synthesize empirical studies on the application of data mining techniques in cardiology. A total of 142 articles published between 2000 and 2015 were therefore selected, studied and analyzed according to the four following criteria: year and channel of publication, research type, medical task and empirical type. The results of this mapping study are discussed and a list of recommendations for researchers and cardiologists is provided.

With the increasing computational power and the decreasing cost of high bandwidth networks resulted in Distibuted Systems. Distributed Data Mining is being used to analyze and monitor data in distributed systems. In the past, distributed technologies like Java RMI, CORBA were used for data mining but the result was a more tightly coupled system. Using web services a loosely coupled, interoperable distributed computing framework can be built. The topic of this thesis is to investigate the use of web service in distributed data mining. This thesis involves the design, development and implementation of distributed data mining using web services as well as an in-depth look at technical aspects and future implication of such framework. A working framework will be created allowing a user to dynamically locate and run mining algorithms on data services or vice versa. The algorithm and data will be deployed as web services. The created web services will be registered at public registry servers. Two distributed data mining architectures will be presented, Data to Algorithm and Algorithm to Data. Finally, performance of the both the architectures will be compared with varying data using different public registry servers.

Data Mining refers to a set of methods applicable to large and complex databases to eliminate the randomness and discover the hidden pattern. Datamining (DM), also known as knowledge discovery from databases (KDD), is the extraction of new knowledge from huge databases. Data mining involves the use of sophisticated data analysis tools to discover previously unknown, valid patterns and relationships in large data sets. Data mining tools can forecast the future trends and activities to support the decision of people. The scope of datamining is associated with Uncovering trends and patterns are a great power for the businesses of all sectors and industries. Modern intrusion detection applications are confronted with a variety of issues. These applications must be reliable, extensible, manageable, and minimal in maintenance costs. Data mining-based intrusion detection systems (IDSs) have shown high accuracy, good generalisation to novel types of intrusion, and stable behaviour in a changing environment in recent years. The number of hidden layers in various neural network topologies is evaluated in order to discover the best neural network. The technique of attempting to discover instances of network attacks by comparing current behaviour to the expected actions of an intruder is known as misuse detection. Artificial neural networks have the ability to detect and classify network activity using data that is limited, incomplete, and nonlinear. The main purpose of this work is to identify privacy and security concerns among cloud computing participants and consumers in a distributed environment. Techniques like Machine Learning, Natural Language Processing (NLP), and Data Mining are combined to automatically identify and uncover patterns from many sorts of materials. Predictive analytics is capable of dealing with both continuous and discontinuous changes. Classification, prediction, and to some extent, affinity analysis constitute the analytical methods employed in predictive analytics. The contemporary study in text or document mining is focusing on syntactic components and the semantic environment. In order to accomplish this, and with the motivation gained from our previous research contributions, we investigated a mining model to classify documents based on the Order of Context, Concept, and Semantic Relations (OCCSR). The use of data mining techniques based on Cloud computing will enable users to retrieve meaningful information from virtually integrated data warehouses, lowering infrastructure and storage costs. Data mining can extract useful and potentially useful information from the cloud. Big Data is typically defined by three characteristics known as the 3Vs (Volume, Velocity and Variety). The surveys approaches, environments, and technologies in key areas for Big Data analytics capabilities and discusses how they aid in the development of analytics solutions for Clouds. The clustering technique belongs to an unsupervised learning and it is used to discover a new set of categories. Grid-based clustering has the shortest processing time, which is typically determined by the size of the grid rather than the data. We compare the performance of three clustering algorithms: hierarchical clustering, density-based clustering, and K Means clustering. The majority of current approaches to detecting misuse involve the use of rule-based expert systems to identify indicators of known attacks. We provide a brief overview of the use of various Artificial Intelligence techniques and their advancements in the design, development, and application of Intrusion Detection Systems (IDS) for protecting computer and communication networks from intruders. The goal of Knowledge Discovery in Data (KDD) is to extract information that is not obvious by using careful and detailed analysis and interpretation. To drive decisions and actions, analytics employs KDD, data mining, text mining, statistical and quantitative analysis, explanatory and predictive models, and advanced and interactive visualisation techniques.

The online version contains supplementary material available at 10.1007/s40200-021-00884-2.

Data mining has become a major academic research area over the last ten years. However, in the transition from academic prototypes to commercial products there have been few successes with commercial data mining applications failing to make any significant impact in the marketplace. In this paper it is argued that most data mining applications concentrate on the model-building phase of the data mining process and rarely engage the user in other stages. The paper reviews the challenges of producing successful data mining applications and in particular the role of information visualisation. Visualisation in data mining tends to be used to present final results, rather than playing an important part throughout the entire process. We argue that an immersive environment may provide the user with a more suitable interface than is commonly offered. We present a virtual data mining environment which attempts to integrate a data mining application interface and information visualisation in a seamless manner, using the concept of 'liquid data'.

Mining large data sets is no longer the prerogative of computer scientists - specialists in a wide variety of domains are performing analytics as a day-to-day activity. Often such analyses are specific to the domain and analysts are required to devise new algorithms or techniques. For such scenarios, providing a high-level programming environment that delivers high performance on clusters is a challenge. We propose a framework that supports high-level programming using domain abstractions in data mining while delivering scalable performance on commodity clusters i.e. clusters of multi-core workstations. This framework includes a domain specific programming language, DWARF, to enable data mining specialists to rapidly prototype algorithms. DWARF is supported by a compiler that automatically parallelizes code by identifying domain specific patterns and translating them to parallel code that exploits data parallelism and task parallelism. The compiler generates code for a hybrid virtual machine supporting distributed memory model at the top level and shared memory model nested within. The code generated by the compiler can be scheduled on commodity clusters. We compare the proposed framework with other frameworks commonly used for data mining on distributed platforms.