Clustering Algorithm on Spatiotemporal Trajectories

The explosion of very large databases has created extraordinary opportunities for monitoring, analyzing and predicting global economical, geographical, demographic, medical, political, and other processes in the world. Statistical analysis and data mining techniques have emerged for these purposes. Data mining is the process of discovering previously unknown but potentially useful patterns, rules, or associations from huge quantity of data. Data mining can be performed on different data repositories such as relational databases, data warehouses, transactional databases, sequence databases, spatial databases, spatio-temporal databases, and text databases, etc. Typically, data mining functionalities can be classified into two categories: descriptive and predictive. Descriptive mining tasks aim at characterizing the general properties of the data in the databases, while predictive mining tasks perform inherence on the current data in order to make prediction in future.KeywordsRoot Mean Square ErrorData MiningAssociation RuleIntrusion DetectionAnomaly DetectionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preparing a data set for analysis is generally the most time consuming task in a data mining project, requiring many complex SQL queries, joining tables and aggregating columns. Existing SQL aggregations have limitations to prepare data sets because they return one column per aggregated group. In general, a significant manual effort is required to build data sets, where a horizontal layout is required. We propose simple, yet powerful, methods to generate SQL code to return aggregated columns in a horizontal tabular layout, returning a set of numbers instead of one number per row. This new class of functions is called horizontal aggregations. Horizontal aggregations build data sets with a horizontal denormalized layout (e.g. point-dimension, observation-variable, instance-feature), which is the standard layout required by most data mining algorithms. We propose three fundamental methods to evaluate horizontal aggregations: CASE: Exploiting the programming CASE construct; SPJ: Based on standard relational algebra operators (SPJ queries); PIVOT: Using the PIVOT operator, which is offered by some DBMSs. Experiments with large tables compare the proposed query evaluation methods. Our CASE method has similar speed to the PIVOT operator and it is much faster than the SPJ method. In general, the CASE and PIVOT methods exhibit linear scalability, where as the SPJ method does not. In a relational database, especially with normalized tables, a significant effort is required to prepare a summary data set that can be used as input for a data mining or statistical algorithm. Most algorithms require as input a data set with a horizontal layout, with several Records and one variable or dimension per column. That is the case with models like clustering, classification, regression and PCA; consult. Each research discipline uses different terminology to describe the data set. In data mining the common terms are point-dimension. Statistics literature generally uses observation-variable. Machine learning research uses instance-feature. This article introduces a new class of aggregate functions that can be used to build data sets in a horizontal layout (denormalized with aggregations), automating SQL query writing and extending SQL capabilities. We show evaluating horizontal aggregations is a challenging and interesting problem and we introduced alternative methods and optimizations for their efficient evaluation. II. MOTIVATION As mentioned above, building a suitable data set for data mining purposes is a time- consuming task. This task generally requires writing long SQL statements or customizing SQL Code if it is automatically generated by some tool. There are two main ingredients in such SQL code: joins and aggregations; we focus on the second one. The most widely- known aggregation is the sum of a column over groups of rows. Some other aggregations return the average, maximum, minimum or row count over groups of rows. There exist many aggregations functions and operators in SQL. Unfortunately, all these aggregations have limitations to build data sets for data mining purposes. The main reason is that, in general, data sets that are stored in a relational database (or a data warehouse) come

A fuzzy logic based resource manager (RM) has been developed that automatically allocates electronic attack resources in real-time over many dissimilar platforms. Two different data mining algorithms have been developed to determine rules, strategies, and fuzzy decision tree structure. The first data mining algorithm uses a genetic algorithm as a data mining function and is called from an electronic game. The game allows a human expert to play against the resource manager in a simulated battlespace with each of the defending platforms being exclusively directed by the fuzzy resource manager and the attacking platforms being controlled by the human expert or operating autonomously under their own logic. This approach automates the data mining problem. The game automatically creates a database reflecting the domain expert's knowledge. It calls a data mining function, a genetic algorithm, for data mining of the database as required and allows easy evaluation of the information mined in the second step. The criterion for re- optimization is discussed as well as experimental results. Then a second data mining algorithm that uses a genetic program as a data mining function is introduced to automatically discover fuzzy decision tree structures. Finally, a fuzzy decision tree generated through this process is discussed.

Recent advances in computing, communications, digital storage technologies, and high-throughput data-acquisition technologies, make it possible to gather and store incredible volumes of data. It creates unprecedented opportunities for large-scale knowledge discovery from huge database. Data mining (DM) technology has emerged as a means of performing this discovery. There are countless researchers working on designing efficient data mining techniques, methods, and algorithms. Many data mining methods and algorithms have been developed and applied in a lot of application fields [1]. Unfortunately, most data mining researchers pay much attention to technique problems for developing data mining models and methods, while little to basic issues of data mining [2]. In this talk, some basic issues of data mining are addressed. What is data mining? What is the product of a data mining process? What are we doing in a data mining process? What is the rule we should obey in a data mining process? Through analyzing existing data mining methods, and domain-driven (or user-driven) data mining models [3-5], we find that we should take a data mining process as a process of knowledge transformation. Based on this understanding of data mining, a conceptual data mining model of domain-oriented data-driven data mining (3DM) is proposed [2]. The relationship between traditional domain-driven (or user-driven) data mining models and our proposed 3DM model is also analyzed. Some domain-oriented data-driven data mining algorithms for mining such knowledge as default rule [6], decision tree [7], and concept lattice [8] from database are proposed. The experiment results for these algorithms are also shown to illustrate the efficiency and performance of the knowledge acquired by our 3DM data mining algorithms.

A fuzzy logic based expert system has been developed that automatically allocates resources in realtime over many dissimilar platforms. The platforms can be very general, e.g., ships, planes, etc. Potential foes can also be general. The resource manager has been embedded in an electronic game environment. This co-evolutionary game fully automates the data mining problem allowing determination of parameters essential to the resource manager. The game allows the resource manager to learn from human experts or computerized enemies. The game does not determine the structure of fuzzy decision trees. A new data mining algorithm that uses a genetic program, an algorithm that evolves other computer programs, as a data mining function has been developed to solve this problem. It not only determines the fuzzy decision tree structure it also creates fuzzy rules while mining scenario data bases. Finally, experimental results are discussed related to both data mining algorithms.

Data mining (DM) research has successfully developed advanced DM techniques and algorithms over the last few decades, and many organisations have great expectations to take more benefit of their data warehouses in decision making. Currently, the strong focus of most DM-researchers is still only on technology-oriented topics. Commonly the DM research has several stakeholders, the major of which can be divided into internal and external ones each having their own point of view, and which are at least partly conflicting. The most important internal groups of stakeholders are the DM research community and academics in other disciplines. The most important external stakeholder groups are managers and domain experts who have their own utility-based interests to DM and DM research results. In this paper we discuss these practice-oriented points of view towards DM research and suggest broader discussions inside the DM research community about who should do that kind of research. We bring in the discussion several topics developed in the information systems (IS) discipline and show some similarities between IS and DM systems. DM systems have also their own peculiarities and we conclude that researchers who take into account human and organisational aspects related to DM systems need to have also some understanding about DM. This makes us suggest that the research area inside the DM community should be made broader than the current heavily technology-oriented one.

Dingdong Buying Vegetable is a cohort of emerging entities that have swiftly gained prominence within the business domain in recent times. The data mining function of its data platform APP is undeniably linked to the underlying factor contributing to its commercial success. This article examines the fundamental principles, practical manifestations, and pertinent research instances of data mining. It specifically centers on the primary interface of the Dingdong Buying Vegetable APP, scrutinizing its design and distinctive attributes tailored to specific customers. Furthermore, it conducts an in-depth analysis of the correlation between the platforms commercial success and its data mining functionality. The primary aspect in which the data mining function of the Dingdong Buying Vegetable APP is expected to be manifested is through extensive data mining. The process of mining client data and conducting a full comparison and analysis of sales data for all products sold is a highly intricate and exhaustive endeavor. The use of these data mining functions serves multiple purposes. Firstly, it efficiently identifies customers genuine requirements on the client side, enabling the recommendation of suitable products and fostering customer reliance on the application. Additionally, it facilitates accurate decision-making in marketing products on the product side by comparing and analyzing diverse data pertaining to the sold products. This approach helps prevent the sale of unsought items and effectively minimizes company expenditure.

This paper describes an approach to identify the state of a power system using Data Mining Algorithms. With the advent of Phasor Measurement Units (PMU), a huge amount of power system data is being stored day-to-day in the Phasor Data Collectors (PDCs). Knowledge discovery and machine learning techniques can make use of this data to extract valuable information and interesting patterns in these databases. In this paper, Symbolic Aggregate ApproXimation (SAX) is used to convert the time stamped phasor data from the PMUs into symbolic strings and Data Mining (DM) algorithms are used to predict the current state of a power system. Along with the state, the location and cause of disturbance is also identified in minimal time. The effectiveness of different DM algorithms for determining state of a power system is also shown using the measures of accuracy, precision and recall.

: A data mining based procedure for automated reverse engineering has been developed. The data mining algorithm for reverse engineering uses a genetic program (GP) as a data mining function. A genetic program is an algorithm based on the theory of evolution that automatically evolves populations of computer programs or mathematical expressions, eventually selecting one that is optimal in the sense it maximizes a measure of effectiveness, referred to as a fitness function. The system to be reverse engineered is typically a sensor. Design documents for the sensor are not available and conditions prevent the sensor from being taken apart. The sensor is used to create a database of input signals and output measurements. Rules about the likely design properties of the sensor are collected from experts. The rules are used to create a fitness function for the genetic program. Genetic program based data mining is then conducted. This procedure incorporates not only the experts? rules into the fitness function, but also the information in the database. The information extracted through this process is the internal design specifications of the sensor. Significant mathematical formalism and experimental results related to GP based data mining for reverse engineering will be provided.

데이터를 구조화하여 사용하는 관계형 데이터베이스가 현재까지 데이터 관리에 가장 많이 사용되고 있다. 그러나 관계형 데이터베이스는 데이터가 증가되면 데이터를 저장하거나 조회할 때 읽기, 쓰기 연산 수행에 제약 조건이 발생되어 서비스가 느려지는 현상이 나타난다. 또 새로운 업무가 추가되면 데이터베이스 내 데이터는 증가되고 결국 이를 해결하기 위해 하드웨어의 병렬 구성, CPU, 메모리, 네트워크 등 추가적인 인프라 구성을 필요로 하게 된다. 본 논문에서는 관계형 데이터베이스의 데이터 증가로 느려지는 웹 정보서비스 개선을 위해 기존 관계형 데이터베이스의 데이터를 하둡 HDFS로 전송하고 이를 일원화하여 데이터를 재구성한 후 사용자에게 하둡 데이터 처리로 대량의 데이터를 빠르고 안전하게 추출하는 모델을 구현한다. 본 시스템 적용을 위해 웹 기반 민원시스템과 비정형 데이터 처리인 이미지 파일 저장에 본 제안시스템을 적용하였다. 적용결과 관계형 데이터베이스 시스템보다 제안시스템 데이터 처리가 0.4초 더 빠른 결과를 얻을 수 있었고 기존 관계형 데이터베이스와 같은 대량의 데이터를 처리를 빅 데이터 기법인 하둡 데이터 처리로도 웹 정보서비스를 지원이 가능하였다. 또한 하둡은 오픈소스로 제공되어 소프트웨어 구매 비용을 줄여주는 장점이 있으며 기존 관계형 데이터베이스의 데이터 증가로 효율적인 대용량 데이터 처리를 요구하는 조직에게 도움을 줄 수 있을 것이다. Relational databases used by structuralizing data are the most widely used in data management at present. However, in relational databases, service becomes slower as the amount of data increases because of constraints in the reading and writing operations to save or query data. Furthermore, when a new task is added, the database grows and, consequently, requires additional infrastructure, such as parallel configuration of hardware, CPU, memory, and network, to support smooth operation. In this paper, in order to improve the web information services that are slowing down due to increase of data in the relational databases, we implemented a model to extract a large amount of data quickly and safely for users by processing Hadoop Distributed File System (HDFS) files after sending data to HDFSs and unifying and reconstructing the data. We implemented our model in a Web-based civil affairs system that stores image files, which is irregular data processing. Our proposed system's data processing was found to be 0.4 sec faster than that of a relational database system. Thus, we found that it is possible to support Web information services with a Hadoop-based big data processing technique in order to process a large amount of data, as in conventional relational databases. Furthermore, since Hadoop is open source, our model has the advantage of reducing software costs. The proposed system is expected to be used as a model for Web services that provide fast information processing for organizations that require efficient processing of big data because of the increase in the size of conventional relational databases.

The subfield of technologies within the discipline of computer science is known as "data mining" Moreover, statistics are put to use in order to identify trends within the database. Essentially, the primary goal of the data mining methods is to extract from the database any information that will be valuable in the future or that will be needed at some point in the future and transform it into a usable format. Understandable framework for future usage. There are several approaches to choose from, each of which may be helpful in carrying out data mining operations in an effective manner. There is some information accessible in this document concerning comparative studies or analyses of various data mining approaches and certain data mining algorithms. This information may be found in this publication. This means that reading this paper will assist us in gaining knowledge regarding data mining technologies, data mining algorithms, as well as some data of corporations that have upgraded or enhanced their profits by modifying their data mining technologies and have obtained impressive output as a result of doing so.

A data mining based procedure for automated reverse engineering and defect discovery has been developed. The data mining algorithm for reverse engineering uses a genetic program (GP) as a data mining function. A GP is an evolutionary algorithm that automatically evolves populations of computer programs or mathematical expressions, eventually selecting one that is optimal in the sense it maximizes a fitness function. The system to be reverse engineered is typically a sensor that may not be disassembled and for which there are no design documents. The sensor is used to create a database of input signals and output measurements. Rules about the likely design properties of the sensor are collected from experts. The rules are used to create a fitness function for the GP allowing GP based data mining. This procedure incorporates not only the experts’ rules into the fitness function, but also the information in the database. The information extracted through this process is the internal design specifications of the sensor. These design properties can be used to create a fitness function for a genetic algorithm, which is in turn used to search for defects in the digital logic design. Significant theoretical and experimental results are provided.

Mining and analyzing social networks is now becoming a very popular research area not only for data mining and web mining but also social network analysis. Data mining is a technique that has the ability to process and analyze large amounts of data and by this to discover valuable information from the data. In recent years, due to the booming of social communications and social network-based web services, data mining has become a very important and powerful technique to process and analyze such large amounts of data. Recently, many researchers are focusing on developing new data mining techniques and algorithms, or devoting to improve traditional mining techniques for social network analysis. One key test of this discovered data is how it can be used in real-world decision-making situations. Social data are the aggregations of the communications, interactions and experiences of people, and it is useful to leverage this type of data for decision-making. Thus, is an important time to balance the research focus and real world applications, such as decision support and recommender systems. In this special issue, we invited authors to submit extended versions of their papers, which were deemed top-quality papers from MISNC 2015 (The 2nd Multidisciplinary International Social Networks Conference, Matsuyama, Japan), MSNDS 2015 (The 6th International Workshop on Mining and Analyzing Social Networks for Decision Support, Paris, France) and ASE SocialInformatics 2015 (Kaohsiung, Taiwan). In addition to the invited papers, we also called for public submissions. Each submission was reviewed by at least two experts and revised according to reviewers’ comments to ensure the quality of the papers. The first paper is entitled “Impersonate human decision making process: an interactive context-aware recommender system” and was authored by Chen-Shu Wang, Shiang-Lin Lin and Heng-Li Yang. The authors alter traditional context-aware-recommender systems (CAR) to interactive CAR (iCAR) in order to improve the recommendation accuracy. A car rental J Intell Inf Syst (2016) 47:193–194 DOI 10.1007/s10844-016-0425-4

Data mining is one of the most important steps of the knowledge discovery in databases process and is considered as significant subfield in knowledge management. Research in data mining continues growing in business and in learning organization over coming decades. This review paper explores the applications of data mining techniques which have been developed to support knowledge management process. The journal articles indexed in ScienceDirect Database from 2007 to 2012 are analyzed and classified. The discussion on the findings is divided into 4 topics: (i) knowledge resource; (ii) knowledge types and/or knowledge datasets; (iii) data mining tasks; and (iv) data mining techniques and applications used in knowledge management. The article first briefly describes the definition of data mining and data mining functionality. Then the knowledge management rationale and major knowledge management tools integrated in knowledge management cycle are described. Finally, the applications of data mining techniques in the process of knowledge management are summarized and discussed.

Chapter 19 - Summary