Research Data Mining Architecture and Task Allocation Model Algorithm Based on Distributional Database

Data mining tasks are often composed by multiple stages that may be linked each other to form various execution flows. Moreover, data mining tasks are often distributed since they involve data and tools located over geographically distributed environments, like the Grid. Therefore, it is fundamental to exploit effective formalisms, such as workflows, to model data mining tasks that are both multi-staged and distributed. The goal of this work is defining a workflow formalism and providing a visual software environment, named DIS3GNO, to design and execute distributed data mining tasks over the Knowledge Grid, a service-oriented framework for distributed data mining on the Grid. DIS3GNO supports all the phases of a distributed data mining task, including composition, execution, and results visualization. The paper provides a description of DIS3GNO, some relevant use cases implemented by it, and a performance evaluation of the system.KeywordsData MiningComputing NodeGrid ResourceExecution PlanData Mining TaskThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

With the increasing computational power and the decreasing cost of high bandwidth networks resulted in Distibuted Systems. Distributed Data Mining is being used to analyze and monitor data in distributed systems. In the past, distributed technologies like Java RMI, CORBA were used for data mining but the result was a more tightly coupled system. Using web services a loosely coupled, interoperable distributed computing framework can be built. The topic of this thesis is to investigate the use of web service in distributed data mining. This thesis involves the design, development and implementation of distributed data mining using web services as well as an in-depth look at technical aspects and future implication of such framework. A working framework will be created allowing a user to dynamically locate and run mining algorithms on data services or vice versa. The algorithm and data will be deployed as web services. The created web services will be registered at public registry servers. Two distributed data mining architectures will be presented, Data to Algorithm and Algorithm to Data. Finally, performance of the both the architectures will be compared with varying data using different public registry servers.

Parallel system is mainly composed of parallel algorithms which are cost optimal. In this paper a parallel algorithm the hash partitioned apriori (HPA) is taken into consideration. HPA partitions the candidate itemsets among processors using a hash function, like the hash join in relational databases. HPA effectively utilizes the whole memory space of all the processors, hence it works well for large scale data mining in a parallel and distributed environment. The optimization technique of dynamic data allocation is discussed for the execution of this application. This technique is applied in a parallel and distributed environment. Writing parallel data mining algorithms in a distributed environment is a non-trivial task. The main purpose of the proposed method is to meet certain challenges associated with parallel and distributed data mining such as (i) minimizing I/O (ii) Increasing processing speed (iii) Communication cost.

Nowadays, the process of data mining is one of the most important topics in scientific and business problems. Grid Computing can be used as infrastructures to provide an effective computational support for distributed data mining applications. The paper proposed a data mining architecture from large-scale distributed and semantically heterogeneous data resources in order to efficiently support knowledge acquisition. Grid technology and semantic web technology were combined to solve the difficult problem of distributed data mining. Data mining in the Grid environment can provide highly efficient and powerful data analysis and knowledge discovery solution and semantic web can effectively solve the heterogeneity of data sources and make them available for data mining.

Distributed Data Mining (DDM) approaches are considered most suitable for managing large datasets. DDM can reduce the communication cost, computational cost, and storage cost. However a key issue for DDM is accuracy of knowledge discovery. To mitigate the accuracy issue, it is necessary to provide reliable network. In practice, it is assumed that the network is available all the time but in reality, it may not available most of the time. In this study, we considered a sporadic wireless sensor network (SWSN), SWSN is a wireless network where most of the time a complete path is not available from a source to a destination, or such a path is highly unreliable. It is also possible in this type of network a communication path is available for a very short time and it may breaks when try to send data over it and limited storage capacity. The objective of this study is to overcome the above mentioned issues by providing a reliable and intelligent protocols for DDM architecture in a SWSN.

Distributed Data Mining (DDM) has been proposed as a means to deal with the analysis of distributed data, where DDM discovers patterns and implements prediction based on multiple distributed data sources. However, DDM faces several problems in terms of autonomy, privacy, performance and implementation. DDM requires homogeneity regarding environment, control, administration and the classification algorithm(s), and such that requirements are too strict and inflexible in many applications. In this paper, we propose the employment of a Multi-Agent System (MAS) to be combined with DDM (MAS-DDM). MAS is a mechanism for creating goal-oriented autonomous agents within shared environments with communication and coordination facilities. We shall show that MAS-DDM is both desirable and beneficial. In MAS-DDM, agents could communicate their beliefs (calculated classification) by covering private and non-sharable data, and other agents decide whether the use of such beliefs in classifying instances and adjusting their prior assumptions about each class of data. In MAS-DDM, we will develop and use a modified Naive Bayesian algorithm because (1) Naive Bayesian has been shown to be the most used algorithm to deal with uncertain data, and (2) to show that even if all agents in MAS-DDM use the same algorithm, MAS-DDM preforms better than DDM approaches with non-communicating processes. Point (2) provide an evidence that the exchange of information between agents helps in increasing the accuracy of the classification task significantly.

A hybrid-security model for privacy-enhanced distributed data mining

Due to today’s advancement in technology and businesses, fraud detection has become a critical component of financial transactions. Considering vast amounts of data in large datasets, it becomes more difficult to detect fraud transactions manually. In this research, we propose a combined method using both data mining and statistical tasks, utilizing feature selection, resampling and cost-sensitive learning for credit card fraud detection. In the first step, useful features are identified using genetic algorithm. Next, the optimal resampling strategy is determined based on the design of experiments (DOE) and response surface methodologies. Finally, the cost sensitive C4.5 algorithm is used as the base learner in the Adaboost algorithm. Using a real-time data set, results show that applying the proposed method significantly reduces the misclassification cost by at least 14% compared with Decision tree, Naive bayes, Bayesian Network, Neural network and Artificial immune system.

Distributed Data Mining (DDM) is the process of mining distributed and heterogeneous datasets. DDM is widely seen as a means of addressing the scalability issue of mining large data sets. Consequently, there is an emerging focus on optimisation of the DDM process. In this paper we present cost formulae for estimating the communication and computation time for different distributed data mining scenarios.

Recent advances in computing, communications, digital storage technologies, and high-throughput data-acquisition technologies, make it possible to gather and store incredible volumes of data. It creates unprecedented opportunities for large-scale knowledge discovery from huge database. Data mining (DM) technology has emerged as a means of performing this discovery. There are countless researchers working on designing efficient data mining techniques, methods, and algorithms. Many data mining methods and algorithms have been developed and applied in a lot of application fields [1]. Unfortunately, most data mining researchers pay much attention to technique problems for developing data mining models and methods, while little to basic issues of data mining [2]. In this talk, some basic issues of data mining are addressed. What is data mining? What is the product of a data mining process? What are we doing in a data mining process? What is the rule we should obey in a data mining process? Through analyzing existing data mining methods, and domain-driven (or user-driven) data mining models [3-5], we find that we should take a data mining process as a process of knowledge transformation. Based on this understanding of data mining, a conceptual data mining model of domain-oriented data-driven data mining (3DM) is proposed [2]. The relationship between traditional domain-driven (or user-driven) data mining models and our proposed 3DM model is also analyzed. Some domain-oriented data-driven data mining algorithms for mining such knowledge as default rule [6], decision tree [7], and concept lattice [8] from database are proposed. The experiment results for these algorithms are also shown to illustrate the efficiency and performance of the knowledge acquired by our 3DM data mining algorithms.

In knowledge management process, data mining technique can be used to extract and discover the valuable and meaningful knowledge from a large amount of data. Nowadays, data mining has given a great deal of concern and attention in the information industry and in society as a whole. This technique is an approach that is currently receiving great attention in data analysis and it has been recognized as a newly emerging analysis tool (Osei-Bryson, 2010; Park, 2001; Sinha, 2008; Tso & Yau, 2007; Wan, 2009; Zanakis, 2005; Zhuang et al., 2009). Additionally, among the major tasks in data mining are classification and prediction; concept description; rule association; cluster analysis; outlier analysis; trend and evaluation analysis; statistical analysis and others. Classification and prediction tasks are among the popular tasks in data mining; and widely used in many areas especially for trend analysis and future planning. In fact, classification technique is supervised learning, which is the class level or prediction target is already known. As a result, the classification model which is represented through rules structures will be constructed in the classification process. In this case, the constructed model will be representing the precious knowledge and it can be used for future planning. There are many areas which adapted this approach to solve their problems such as in finance, medical, marketing, stock, telecommunication, manufacturing, health care, customer relationship and etc. However, the data mining application has not attracted much attention from people in Human Resource (HR) field (Chien & Chen, 2008; Ranjan, 2008). Besides that, in our previous study, most of the prediction applications are used to predict stock, demand, rate, risk, event and others; but there are quite limited studies on human prediction. In addition prediction applications are mainly developed in business and industrious fields; and quite restricted studies involved human talent in an organization (Jantan et al., 2009). HR data can provide a rich resource for knowledge discovery and for decision support system development. Recently, an organization has to struggle effectively in term of cost, quality, service or innovation. All these depend on having enough right people with the right skills, employed

In the information age of the 21st century, a large amount of information is collected and applied. However, due to the heterogeneity of system environment for data storage and computing, how to mine these distributed data sources has become a valuable research topic that attracted more and more attention. In this paper, we firstly presented the problem scenario and main challenges confronting with the problem of distributed data mining on multiple sourced heterogeneous data sets. Then, we surveyed research works related to the problem and elicited their main features on different technology domains to show current distributed solutions for different data mining algorithm categories. Finally, we reviewed in detail the research works and discussed the challenges remained in the distributed data mining problem for multiple sourced heterogeneous data sets.

Modern e-commerce is vigorously developing. It makes the resources and services on Internet more richly colorful. Every day these rich resources and services can produce many new mass data which contain a large number of important information. These data are often heterogeneous, uncertain and unstructured. Their complexity has far exceeded the analysis and understanding ability which the humanity has at present. In order to resolve the problem, related technologies of distributed data mining in e-commerce environment are studied. Web services technology and agent technology are introduced in the e-commerce data mining. The distributed data mining structure in e-commerce environment based on Web services and mobile agent is put forward.

We describe the application of data mining algorithms to research problems in astronomy. We posit that data mining has always been fundamental to astronomical research, since data mining is the basis of evidence-based discovery, including classification, clustering, and novelty discovery. These algorithms represent a major set of computational tools for discovery in large databases, which will be increasingly essential in the era of data-intensive astronomy. Historical examples of data mining in astronomy are reviewed, followed by a discussion of one of the largest data-producing projects anticipated for the coming decade: the Large Synoptic Survey Telescope (LSST). To facilitate data-driven discoveries in astronomy, we envision a new data-oriented research paradigm for astronomy and astrophysics -- astroinformatics. Astroinformatics is described as both a research approach and an educational imperative for modern data-intensive astronomy. An important application area for large time-domain sky surveys (such as LSST) is the rapid identification, characterization, and classification of real-time sky events (including moving objects, photometrically variable objects, and the appearance of transients). We describe one possible implementation of a classification broker for such events, which incorporates several astroinformatics techniques: user annotation, semantic tagging, metadata markup, heterogeneous data integration, and distributed data mining. Examples of these types of collaborative classification and discovery approaches within other science disciplines are presented.

Java has become a language of choice for applications executing in heterogeneous environments utilising distributed objects and multithreading. To handle large data sets, scalable and efficient implementations of data mining approaches are required, generally employing computationally intensive algorithms. Conventional Java implementations do not directly provide support for the data structures often encountered in such algorithms, and they also lack repeatability in numerical precision across platforms. This paper describes a distributed framework employing task and data parallelism, and implemented in high performance Java (HPJava). Issues of interest for data mining algorithms are identified, and possible solutions discussed for overcoming limitations in the Java Virtual Machine. The framework supports parallelism across workstation clusters, using the message-passing interface as middleware, and can support different analysis algorithms, wrapped as Java objects, and linked to various databases using the Java database connectivity interface. Guidelines are provided for implementing parallel and distributed data mining on large data sets, and a proof-of-concept data mining application is analysed using a neural network.