Distributed data mining using web services.

Modern e-commerce is vigorously developing. It makes the resources and services on Internet more richly colorful. Every day these rich resources and services can produce many new mass data which contain a large number of important information. These data are often heterogeneous, uncertain and unstructured. Their complexity has far exceeded the analysis and understanding ability which the humanity has at present. In order to resolve the problem, related technologies of distributed data mining in e-commerce environment are studied. Web services technology and agent technology are introduced in the e-commerce data mining. The distributed data mining structure in e-commerce environment based on Web services and mobile agent is put forward.

Distributed Data Mining (DDM) has been proposed as a means to deal with the analysis of distributed data, where DDM discovers patterns and implements prediction based on multiple distributed data sources. However, DDM faces several problems in terms of autonomy, privacy, performance and implementation. DDM requires homogeneity regarding environment, control, administration and the classification algorithm(s), and such that requirements are too strict and inflexible in many applications. In this paper, we propose the employment of a Multi-Agent System (MAS) to be combined with DDM (MAS-DDM). MAS is a mechanism for creating goal-oriented autonomous agents within shared environments with communication and coordination facilities. We shall show that MAS-DDM is both desirable and beneficial. In MAS-DDM, agents could communicate their beliefs (calculated classification) by covering private and non-sharable data, and other agents decide whether the use of such beliefs in classifying instances and adjusting their prior assumptions about each class of data. In MAS-DDM, we will develop and use a modified Naive Bayesian algorithm because (1) Naive Bayesian has been shown to be the most used algorithm to deal with uncertain data, and (2) to show that even if all agents in MAS-DDM use the same algorithm, MAS-DDM preforms better than DDM approaches with non-communicating processes. Point (2) provide an evidence that the exchange of information between agents helps in increasing the accuracy of the classification task significantly.

Nowadays, the process of data mining is one of the most important topics in scientific and business problems. Grid Computing can be used as infrastructures to provide an effective computational support for distributed data mining applications. The paper proposed a data mining architecture from large-scale distributed and semantically heterogeneous data resources in order to efficiently support knowledge acquisition. Grid technology and semantic web technology were combined to solve the difficult problem of distributed data mining. Data mining in the Grid environment can provide highly efficient and powerful data analysis and knowledge discovery solution and semantic web can effectively solve the heterogeneity of data sources and make them available for data mining.

Distributed Data Mining (DDM) approaches are considered most suitable for managing large datasets. DDM can reduce the communication cost, computational cost, and storage cost. However a key issue for DDM is accuracy of knowledge discovery. To mitigate the accuracy issue, it is necessary to provide reliable network. In practice, it is assumed that the network is available all the time but in reality, it may not available most of the time. In this study, we considered a sporadic wireless sensor network (SWSN), SWSN is a wireless network where most of the time a complete path is not available from a source to a destination, or such a path is highly unreliable. It is also possible in this type of network a communication path is available for a very short time and it may breaks when try to send data over it and limited storage capacity. The objective of this study is to overcome the above mentioned issues by providing a reliable and intelligent protocols for DDM architecture in a SWSN.

A hybrid-security model for privacy-enhanced distributed data mining

With the accelerative development of modern society informationization, data and databases has rapidly increased. That demands the abroad application of distributed data mining (DDM) methods that can parallel to get the results in time. However the problem of minimizing the communications costs across limited bandwidth interconnection network in DDM is crucial now. Considering that, this paper proposes the architecture of distributed data mining, model of scheduling in distributed data mining and its algorithm to deal with it. To more optimize the result of the scheduling algorithm, genetic algorithm is also used in this paper.

In the information age of the 21st century, a large amount of information is collected and applied. However, due to the heterogeneity of system environment for data storage and computing, how to mine these distributed data sources has become a valuable research topic that attracted more and more attention. In this paper, we firstly presented the problem scenario and main challenges confronting with the problem of distributed data mining on multiple sourced heterogeneous data sets. Then, we surveyed research works related to the problem and elicited their main features on different technology domains to show current distributed solutions for different data mining algorithm categories. Finally, we reviewed in detail the research works and discussed the challenges remained in the distributed data mining problem for multiple sourced heterogeneous data sets.

Distributed Data Mining (DDM) is the process of mining distributed and heterogeneous datasets. DDM is widely seen as a means of addressing the scalability issue of mining large data sets. Consequently, there is an emerging focus on optimisation of the DDM process. In this paper we present cost formulae for estimating the communication and computation time for different distributed data mining scenarios.

This final report summarizes our accomplishments and findings and includes recent publications occurring in the final period of this award. One of our research goals was to develop algorithms and services for remote data analysis and distributed data mining which scaled from the commodity internet to high performance networks. When we began the project there was no effective mechanisms to achieve high end to end performance for data intensive applications over wide area, high bandwidth networks. For this reason, we developed algorithms and services for Layers 2,3, and 4 in the simple data web application stack below. We describe our research accomplishments for each of these layers in turn: Layer 4--Data Web Applications; Layer 3--Data Web Services; Layer 2--Network Protocol Services; Layer 1--IP.

Peer-to-Peer (P2P) networks are gaining increasing popularity in many distributed applications such as file-sharing, network storage, web caching, sear- ching and indexing of relevant documents and P2P network-threat analysis. Many of these applications require scalable analysis of data over a P2P network. This paper starts by offering a brief overview of distributed data mining applications and algorithms for P2P environments. Next it discusses some of the privacy concerns with P2P data mining and points out the problems of existing privacy-preserving multi-party data mining techniques. It further points out that most of the nice assumptions of these existing privacy preserving techniques fall apart in real-life applications of privacy-preserving distributed data mining (PPDM). The paper offers a more realistic formulation of the PPDM problem as a multi-party game and points out some recent results.KeywordsData MiningPrivacy PreserveCheap TalkGood NodeDistribute Data MiningThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We describe the application of data mining algorithms to research problems in astronomy. We posit that data mining has always been fundamental to astronomical research, since data mining is the basis of evidence-based discovery, including classification, clustering, and novelty discovery. These algorithms represent a major set of computational tools for discovery in large databases, which will be increasingly essential in the era of data-intensive astronomy. Historical examples of data mining in astronomy are reviewed, followed by a discussion of one of the largest data-producing projects anticipated for the coming decade: the Large Synoptic Survey Telescope (LSST). To facilitate data-driven discoveries in astronomy, we envision a new data-oriented research paradigm for astronomy and astrophysics -- astroinformatics. Astroinformatics is described as both a research approach and an educational imperative for modern data-intensive astronomy. An important application area for large time-domain sky surveys (such as LSST) is the rapid identification, characterization, and classification of real-time sky events (including moving objects, photometrically variable objects, and the appearance of transients). We describe one possible implementation of a classification broker for such events, which incorporates several astroinformatics techniques: user annotation, semantic tagging, metadata markup, heterogeneous data integration, and distributed data mining. Examples of these types of collaborative classification and discovery approaches within other science disciplines are presented.

Java has become a language of choice for applications executing in heterogeneous environments utilising distributed objects and multithreading. To handle large data sets, scalable and efficient implementations of data mining approaches are required, generally employing computationally intensive algorithms. Conventional Java implementations do not directly provide support for the data structures often encountered in such algorithms, and they also lack repeatability in numerical precision across platforms. This paper describes a distributed framework employing task and data parallelism, and implemented in high performance Java (HPJava). Issues of interest for data mining algorithms are identified, and possible solutions discussed for overcoming limitations in the Java Virtual Machine. The framework supports parallelism across workstation clusters, using the message-passing interface as middleware, and can support different analysis algorithms, wrapped as Java objects, and linked to various databases using the Java database connectivity interface. Guidelines are provided for implementing parallel and distributed data mining on large data sets, and a proof-of-concept data mining application is analysed using a neural network.

In this paper, we describe the design of web service with model-view-controller for implement the system of classify type of rice. This idea using LDA and SVM to estimate a rice surface area, and then we present this idea by create web service architecture and present prototype used in mobile. The process of making a model are choosing the literature of rice attribute compare with a hundred pieces of rice in different sizes and type, and then analyzed all of example by create web service that implement under spring MVC idea. In testing method of technique, we create prototype of web service and test function by user. The result shows that the model gives accurate result with very small errors and is also flexible to any change in all process.

Data mining and scoring tool providers require users to use provider-specific ways to invoke their services. The provider-specific approach could be a major factor affecting why data mining tools and applications are not currently as widespread as one might hope. However, SOA can implements the loose coupling between the services provider and services consumer. Using Web service protocol (such as SOAP, WSDL and UDDI), the data mining task can be encapsulated into a standard Web services. Services consumer can dynamically bound different services provider based on Web services, which implements the integration of data mining applications with data mining and scoring tool. The Java Data Mining API (JDM) is the first attempt to create a standard Java API to access data-mining tools from Java applications. A new service discovery model where quality of service is taken as constraints when searching for Web services would give some confidence to the Web service consumers about the quality of the service they are about to invoke.

Thanks to the rapid advances in Internet-based applications, data acquisition and storage technologies, petabyte-sized network data collections are becoming more and more common, thus prompting the need for scalable data analysis solutions. By leveraging today's ubiquitous many-core computer architectures and the increasingly popular cloud computing paradigm, the applicability of data mining algorithms to these large volumes of network data can be scaled up to gain interesting insights. This paper proposes NEMICO, a comprehensive Big Data mining system targeted to network traffic flow analyses (e.g., Traffic flow characterization, anomaly detection, multiple-level pattern mining). NEMICO comprises new approaches that contribute to a paradigm-shift in distributed data mining by addressing most challenging issues related to Big Data, such as data sparsity, horizontal scaling, and parallel computation.