Hortizontal Aggregation in SQL for Data Mining Analysis to Prepare Data Sets

Preparing a data set for analysis is generally the most time consuming task in a data mining project, requiring many complex SQL queries, joining tables, and aggregating columns. Existing SQL aggregations have limitations to prepare data sets because they return one column per aggregated group. In general, a significant manual effort is required to build data sets, where a horizontal layout is required. We propose simple, yet powerful, methods to generate SQL code to return aggregated columns in a horizontal tabular layout, returning a set of numbers instead of one number per row. This new class of functions is called horizontal aggregations. Horizontal aggregations build data sets with a horizontal denormalized layout (e.g., point-dimension, observation-variable, instance-feature), which is the standard layout required by most data mining algorithms. We propose three fundamental methods to evaluate horizontal aggregations: CASE: Exploiting the programming CASE construct; SPJ: Based on standard relational algebra operators (SPJ queries); PIVOT: Using the PIVOT operator, which is offered by some DBMSs. Experiments with large tables compare the proposed query evaluation methods. Our CASE method has similar speed to the PIVOT operator and it is much faster than the SPJ method. In general, the CASE and PIVOT methods exhibit linear scalability, whereas the SPJ method does not.

Recent advances in computing, communications, digital storage technologies, and high-throughput data-acquisition technologies, make it possible to gather and store incredible volumes of data. It creates unprecedented opportunities for large-scale knowledge discovery from huge database. Data mining (DM) technology has emerged as a means of performing this discovery. There are countless researchers working on designing efficient data mining techniques, methods, and algorithms. Many data mining methods and algorithms have been developed and applied in a lot of application fields [1]. Unfortunately, most data mining researchers pay much attention to technique problems for developing data mining models and methods, while little to basic issues of data mining [2]. In this talk, some basic issues of data mining are addressed. What is data mining? What is the product of a data mining process? What are we doing in a data mining process? What is the rule we should obey in a data mining process? Through analyzing existing data mining methods, and domain-driven (or user-driven) data mining models [3-5], we find that we should take a data mining process as a process of knowledge transformation. Based on this understanding of data mining, a conceptual data mining model of domain-oriented data-driven data mining (3DM) is proposed [2]. The relationship between traditional domain-driven (or user-driven) data mining models and our proposed 3DM model is also analyzed. Some domain-oriented data-driven data mining algorithms for mining such knowledge as default rule [6], decision tree [7], and concept lattice [8] from database are proposed. The experiment results for these algorithms are also shown to illustrate the efficiency and performance of the knowledge acquired by our 3DM data mining algorithms.

Objectives: Clustering algorithm is broadly used as spectral algorithm in social media, where a reference of contact is used and mined further for various combinations of suggested friends and lookups. This paper Identifies key lead generation techniques to be used in customer relationship management for sales marketing to decide data gathering. Also to define key merits and demerits of these techniques and to prepare a Matrix of comparison of these techniques to justify the data source and data set. M-Cluster algorithm is used for lead qualification i.e. training set preparation and data evaluation. Methods: Todefine a training set based on various attributes/fields in the data given for classification. This training set is used to run the data process and to produce expected result. This is assessed and accepted for definite data set processing or additional run for interim training data set preparation. Findings: This study is taken to customize clustering algorithm for data mining process in a customer relationship management field as the space of data is more and variant. Also proving the usability of customized clustering algorithm in data mining and the efficiency in processing mechanism compared to other methods used in current situation of data mining in customer relationship management is the part of this study. The customized algorithm developed as part of this study considers the two major areas of data mining using clusters. Applications: The results produced tremendous trends that the clustered algorithm suits to any data mining process when scaling and data classification are diversified and less in control.Keywords: Classification, Clustering Algorithm, Data Mining, K-Means, Lead Generation, M-Clustering Algorithm

The choice of data mining algorithms has non-trivial nature. How to help users efficiently select an appropriate data mining algorithms draws a lot of researchers' attention nowadays in which the amount of data grows exponentially. This paper proposed a data mining web service recommendation engine which integrated collaborative filtering and association rule algorithms. It allows users share the experience on choosing data mining algorithms each other and it also can recommend appropriate data mining web services to users timely and effectively.

--Heart disease is one of the notable causes of death in worldwide everyday divination of heart disease is a critical challenge in area of medical data analysis. Machine learning (ML) has been shown to be successful in helping in making decision and prediction from huge quantity of data produced by medical health care industry. Medical data mining is an important area of data mining and considered as one of the important research field due to the application in medical health care domain. In medical data mining classification and prediction of medical dataset process challenges. It is difficult to medical practice to predict the heart attack as it complex task. The health sector at present contain the information that are hidden and which are important in making decision. Data mining algorithms such as Naive Bayes algorithm, decision tree algorithm and random forest are applied in the research for heart disease prediction the result show the comparison between the three algorithm and selecting the best one among three, the Random forest algorithm will provide the more accuracy can compared to all the three algorithms.

This paper describes an approach to identify the state of a power system using Data Mining Algorithms. With the advent of Phasor Measurement Units (PMU), a huge amount of power system data is being stored day-to-day in the Phasor Data Collectors (PDCs). Knowledge discovery and machine learning techniques can make use of this data to extract valuable information and interesting patterns in these databases. In this paper, Symbolic Aggregate ApproXimation (SAX) is used to convert the time stamped phasor data from the PMUs into symbolic strings and Data Mining (DM) algorithms are used to predict the current state of a power system. Along with the state, the location and cause of disturbance is also identified in minimal time. The effectiveness of different DM algorithms for determining state of a power system is also shown using the measures of accuracy, precision and recall.

In recent years, research on Educational Data Mining (EDM) has developed rapidly. However, most researches focus on data source issues, and ignore the importance of data preprocessing and data mining algorithms. This paper has studied EDM, with a special focus on educational big data mining algorithms. Firstly, it analyzed the relevant elements of EDM and introduces big data technology based on the requirements of educational data application. Then it introduced the common educational big data mining algorithms and their applications, and finally discussed the development trend of educational big data mining algorithms.

Liver cirrhosis or hepatic cirrhosis is termed as condition in which the liver of patient not working properly owing to damage by various factors. In a present day, data mining in the field of medical is an important tool to explore the sign patterns in term of medical data sets. The aim of current study is to investigate a method for the classification system of liver cirrhosis in data findings in databases utilizing data mining method which are used in present time. In the data mining system and research work, the Tanagra data mining device, an accurate classifier algorithm and an open source project is used. By using different classification algorithms, we find a pattern which is formed in it. Fischer filtering was good feature in feature selection algorithm. Limited number of attributes in filtering algorithm along with better classification system were chosen and executed. The architecture model flow includes: training dataset, feature relevance analysis, classification process i.e. classification and regression tree, random tree binary logistic regression, support vector machine accuracy analysis. The research work was accomplished on the basis classification algorithms including training and test datsets along with well obtained characterized data. The classification process categorized the liver cirrhosis in dataset and provides better results. The feature relevance analysis displays the different important characteristics which are intended for. The experiment is projected to prove the efficiency of the presentation of numerous classifiers to the liver cirrhosis and dataset used in it. Primary diagnosis of any.

Most data mining algorithms and tools stop at discovered customer models, producing distribution information on customer profiles. Such techniques, when applied to industrial problems such as customer relationship management (CRM), are useful in pointing out customers who are likely attritors and customers who are loyal, but they require human experts to postprocess the discovered knowledge manually. Most of the postprocessing techniques have been limited to producing visualization results and interestingness ranking, but they do not directly suggest actions that would lead to an increase in the objective function such as profit. In this paper, we present novel algorithms that suggest actions to change customers from an undesired status (such as attritors) to a desired one (such as loyal) while maximizing an objective function: the expected net profit. These algorithms can discover cost-effective actions to transform customers from undesirable classes to desirable ones. The approach we take integrates data mining and decision making tightly by formulating the decision making problems directly on top of the data mining results in a postprocessing step. To improve the effectiveness of the approach, we also present an ensemble of decision trees which is shown to be more robust when the training data changes. Empirical tests are conducted on both a realistic insurance application domain and UCI benchmark data

Use of machine learning (ML) and data mining (DM) algorithms has surfaced more often in the recent years for disease risk prediction problems in the healthcare communities. Several traditional feature selection models are combined with the DM and ML algorithms to improve accuracy of the disease risk prediction. In this study, a new Bio-inspired Ensemble Feature Selection (BEFS) model is introduced which is applied with the DM and ML algorithms. In the BEFS model, the most relevant and highly contributing features in the prediction are determined with a bio-inspired algorithm i.e., genetic algorithm, and an ensemble algorithm i.e., random forest algorithm. These important features obtained from the proposed model are then combined in various combinations and applied with the DM and ML algorithms, here logistic regression (LR) and random forest (RF), and the results obtained are promising. The experiment is executed using the famous ML language R. To accomplish this objective, the Breast Cancer Wisconsin (Diagnostic) dataset of UCI (University of California, Irvine) ML repository is utilized. In the experimental outcomes, the highest accuracy attained with the BEFS model is 96.49%, the AUC (Area Under Curve) achieved is 96%, and the sensitivity is 98.11%. These results, which greatly improve the disease risk prediction, are higher than several other existing works, while utilizing only six most relevant features out of the thirty two features of the dataset.

SummaryOne of the most important challenges of African countries is the effective use of information and communication technologies and its generalization in the different sectors (education, economic, political, and so on). This will have a great impact on different aspects of society and economic activities by making everyday procedures easier and more efficient. In this same context, we are interested in this article by the proposition of a decision support framework based on the use of data mining (DM) techniques. Using DM in e‐government is the process of translating data to appropriate knowledge which can be useful for decision‐making. We propose a framework whose goal is the generation of association rules (ARs) for better decision‐making. This framework includes two approaches: (1) the first approach applies different DM algorithms and (2) the second approach optimizes the first one by considering two different metaheuristics: the Genetic algorithm and the Cuckoo search algorithm. A new relevance measure called “Weighted Dominance” has been considered to evaluate the quality of the generated ARs. Extensive experiments have been conducted using different datasets. The results obtained demonstrated the effectiveness of combining DM and optimization algorithms compared to the first approach. Finally, a case study related to an Algerian mobile phone company has been presented illustrating the use of our framework in the decision‐making process.

A fuzzy logic based resource manager (RM) has been developed that automatically allocates electronic attack resources in real-time over many dissimilar platforms. Two different data mining algorithms have been developed to determine rules, strategies, and fuzzy decision tree structure. The first data mining algorithm uses a genetic algorithm as a data mining function and is called from an electronic game. The game allows a human expert to play against the resource manager in a simulated battlespace with each of the defending platforms being exclusively directed by the fuzzy resource manager and the attacking platforms being controlled by the human expert or operating autonomously under their own logic. This approach automates the data mining problem. The game automatically creates a database reflecting the domain expert's knowledge. It calls a data mining function, a genetic algorithm, for data mining of the database as required and allows easy evaluation of the information mined in the second step. The criterion for re- optimization is discussed as well as experimental results. Then a second data mining algorithm that uses a genetic program as a data mining function is introduced to automatically discover fuzzy decision tree structures. Finally, a fuzzy decision tree generated through this process is discussed.

As a result of the huge amounts of data available in any subject of research, data mining technologies confront considerable obstacles. Due to the large volume of data, most existing data mining methods are inapplicable to many real-world problems. Data mining methods become ineffective when the problem size becomes too large. Scalability is an issue that must be addressed in data mining algorithms in order to construct high-performance, efficient, and scalable data mining algorithms. A new scalability technique is proposed and applied to several data mining problems in this research. The proposed scaling methodology is developed using a cascade approach. The approach begins with the collection of a large data set from several sources, which is then preprocessed. Once the dataset has been preprocessed, decompose it into smaller data sets of equal size subsets. Then apply a data mining approach to each subset, with the identical data mining method stated for each subset. The findings of the data mining approach on all subsets are pooled and aggregated for the final output. The performance of the suggested algorithm is assessed using a number of criteria, including accuracy, precision, recall, F-score, and execution time. Social advertising and bank marketing are the two datasets on which the proposed method is tested. The suggested approach's performance is compared with non-scale data mining methods, and it is found that the scaling method outperformed non-scale data mining methods on all measures.

Aufgrund fallender Preise zur Speicherung von Daten kann man derzeit eine explosionsartige Zunahme in der Menge der verfügbaren Daten beobachten. Diese Entwicklung gibt Unternehmen und wissenschaftliche Institutionen die Möglichkeit empirische Daten in ungekannter Größenordnung zu analysieren. Für viele Firmen ist die Analyse der gesammelten Daten aus ihrem operationalen Geschäft längst zu einem zentralen strategischen Aspekt geworden. Im Gegensatz zu der seit längerem schon betriebenen Business Intelligence, bestehen diese Analysen nicht mehr nur aus traditionellen relationalen Anfragen. In zunehmendem Anteil kommen komplexe Algorithmen aus den Bereichen Data Mining und Maschinelles Lernen hinzu, um versteckte Muster in den Daten zu erkennen, oder Vorhersagemodelle zu trainieren. Mit zunehmender Datenmenge und Komplexität der Analysen wird jedoch eine neue Generation von Systemen benötigt, die diese Kombination aus Anfragekomplexität und Datenvolumen gewachsen sind. Relationale Datenbanken waren lange Zeit das Zugpferd der Datenanalyse im großen Stil. Grund dafür war zum großen Teil ihre deklarativen Anfragesprache, welche es ermöglichte die logischen und physischen Aspekte der Datenspeicherung und Verarbeitung zu trennen, und Anfragen automatisch zu optimieren. Das starres Datenmodell und ihre beschränkte Menge von möglichen Operationen schränken jedoch die Anwendbarkeit von relationalen Datenbanken für viele der neueren analytischen Probleme stark ein. Diese Erkenntnis hat die Entwicklung einer neuen Generation von Systemen und Architekturen eingeläutet, die sich durch sehr generische Abstraktionen für parallelisierbare analytische Programme auszeichnen; MapReduce kann hier beispielhaft genannt werden, als der zweifelsohne prominenteste Vertreter dieser Systeme. Zwar vereinfachte und erschloss diese neue Generation von Systemen die Datenanalyse in diversen neuen Anwendungsfeldern, sie ist jedoch nicht in der Lage komplexe Anwendungen aus den Bereichen Data Mining und Maschinelles Lernen effizient abzubilden, ohne sich dabei extrem auf spezifische Anwendungen zu spezialisieren. Verglichen mit den relationalen Datenbanken haben MapReduce und vergleichbare Systeme außerdem die deklarative Abstraktion aufgegeben und zwingen den Anwender dazu systemnahe Programme zu schreiben und diese manuell zu optimieren. In dieser Dissertation werden verschiedene Techniken vorgestellt, die es ermöglichen etliche der zentralen Eigenschaften von relationalen Datenbanken im Kontext dieser neuen Generation von daten-parallelen Analysesystemen zu realisieren. Mithilfe dieser Techniken ist es möglich ein Analysesystem zu beschreiben, dessen Programme gleichzeitig sowohl generische und ausdrucksstark, als auch prägnant und deklarativ sind. Im einzelnen stellen wir folgende Techniken vor: Erstens, eine Programmierabstraktion die generisch ist und mit komplexen Datenmodellen umgehen kann, aber gleichzeitig viele der deklarativen Eigenschaften der relationalen Algebra erhält. Programme, die gegen dies Abstraktion entwickelt werden können ähnlich optimiert werden wie relationale Anfragen. Zweitens stellen wir eine Abstraktion für iterative daten-parallele Algorithmen vor. Die Abstraktion unterstützt inkrementelle (delta-basierte) Berechnungen und geht mit zustandsbehafteteten Berechnungen transparent um. Wir beschreiben wie man einen relationalen Anfrageoptimierer erweitern kann so dass dieser iterative Anfragen effektiv optimiert. Wir zeigen dabei dass der Optimierer dadurch in die Lage versetzt wird automatisch Ausführungspläne zu erzeugen, die wohlbekannten, manuell erstellten Programmen entsprechen. Die Abstraktion subsumiert dadurch spezialisierte Systeme (wie Pregel) und bietet vergleichbare Performanz. Drittens stellen wir Methoden vor, um die Programmierabstraktion in eine funktionale Sprachen einzubetten. Diese Integration ermögliche es prägnante Programme zu schreiben und einfach wiederzuverwendenden Komponenten und Bibliotheken, sowie Domänenspezifische Sprachen, zu erstellen. Wir legen dar wie man die Übersetzung und Optimierung des daten-parallelen Programms mit dem Sprachübersetzer der funktionalen Sprache so integriert, dass maximales Optimierungspotenzial besteht.

This paper presents a hybrid approach that integrates a genetic algorithm (GA) and data mining to produce control signatures. The control signatures define the best parameter intervals leading to a desired outcome. This hybrid method integrates multiple rule sets generated by a data mining algorithm with the fitness function of a GA. The solutions of the GA represent intersections among rules providing tight parameter bounds. The integration of intuitive rules provides an explanation for each generated control setting and it provides insights into the decision making process. The ability to analyze parameter trends and the feasible solutions generated by the GA with respect to the outcomes is another benefit of the proposed hybrid method. The presented approach for deriving control signatures is applicable to various domains, such as energy, medical protocols, manufacturing, airline operations, customer service, and so on. Control signatures were developed and tested for control of a power plant boiler. These signatures discovered insightful relationships among parameters. The results and benefits of the proposed method for the power plant boiler are discussed in the paper.