Regression Based Data Pre-processing Technique for Predicting Missing Values

Recent advances in computing, communications, digital storage technologies, and high-throughput data-acquisition technologies, make it possible to gather and store incredible volumes of data. It creates unprecedented opportunities for large-scale knowledge discovery from huge database. Data mining (DM) technology has emerged as a means of performing this discovery. There are countless researchers working on designing efficient data mining techniques, methods, and algorithms. Many data mining methods and algorithms have been developed and applied in a lot of application fields [1]. Unfortunately, most data mining researchers pay much attention to technique problems for developing data mining models and methods, while little to basic issues of data mining [2]. In this talk, some basic issues of data mining are addressed. What is data mining? What is the product of a data mining process? What are we doing in a data mining process? What is the rule we should obey in a data mining process? Through analyzing existing data mining methods, and domain-driven (or user-driven) data mining models [3-5], we find that we should take a data mining process as a process of knowledge transformation. Based on this understanding of data mining, a conceptual data mining model of domain-oriented data-driven data mining (3DM) is proposed [2]. The relationship between traditional domain-driven (or user-driven) data mining models and our proposed 3DM model is also analyzed. Some domain-oriented data-driven data mining algorithms for mining such knowledge as default rule [6], decision tree [7], and concept lattice [8] from database are proposed. The experiment results for these algorithms are also shown to illustrate the efficiency and performance of the knowledge acquired by our 3DM data mining algorithms.

In this research, deep neural networks (CNNs) were used to estimate missing values ​​in a univariate time series dataset and compared with DES/Holt and (SVR) estimation methods to determine their accuracy in handling missing values ​​in the dataset. These missing values ​​directly affect the process of building mathematical and statistical models, affecting the accuracy of the final results that enable us to make the right decisions in the future. The presence of missing values ​​in the dataset is due to problems that occur during sampling, such as a malfunction in the measuring device, security attacks, or communication errors. The results indicated that the CNN model outperformed other methods by using a simulation approach that generated data randomly. Three different sample sizes (60, 100, 300) were selected with missing data percentages of (10%, 15%, 20%) for each, meeting the "missing at random" (MAR) condition. The Box-Jenkins MA (1) model was applied once with β = 0.5 and again with β = 0.9. The accuracy of the estimation methods was evaluated using the accuracy criteria mean sum of squared error (MSE) and root mean squared error (RMSE).

This paper describes an approach to identify the state of a power system using Data Mining Algorithms. With the advent of Phasor Measurement Units (PMU), a huge amount of power system data is being stored day-to-day in the Phasor Data Collectors (PDCs). Knowledge discovery and machine learning techniques can make use of this data to extract valuable information and interesting patterns in these databases. In this paper, Symbolic Aggregate ApproXimation (SAX) is used to convert the time stamped phasor data from the PMUs into symbolic strings and Data Mining (DM) algorithms are used to predict the current state of a power system. Along with the state, the location and cause of disturbance is also identified in minimal time. The effectiveness of different DM algorithms for determining state of a power system is also shown using the measures of accuracy, precision and recall.

Preparing a data set for analysis is generally the most time consuming task in a data mining project, requiring many complex SQL queries, joining tables and aggregating columns. Existing SQL aggregations have limitations to prepare data sets because they return one column per aggregated group. In general, a significant manual effort is required to build data sets, where a horizontal layout is required. We propose simple, yet powerful, methods to generate SQL code to return aggregated columns in a horizontal tabular layout, returning a set of numbers instead of one number per row. This new class of functions is called horizontal aggregations. Horizontal aggregations build data sets with a horizontal denormalized layout (e.g. point-dimension, observation-variable, instance-feature), which is the standard layout required by most data mining algorithms. We propose three fundamental methods to evaluate horizontal aggregations: CASE: Exploiting the programming CASE construct; SPJ: Based on standard relational algebra operators (SPJ queries); PIVOT: Using the PIVOT operator, which is offered by some DBMSs. Experiments with large tables compare the proposed query evaluation methods. Our CASE method has similar speed to the PIVOT operator and it is much faster than the SPJ method. In general, the CASE and PIVOT methods exhibit linear scalability, where as the SPJ method does not. In a relational database, especially with normalized tables, a significant effort is required to prepare a summary data set that can be used as input for a data mining or statistical algorithm. Most algorithms require as input a data set with a horizontal layout, with several Records and one variable or dimension per column. That is the case with models like clustering, classification, regression and PCA; consult. Each research discipline uses different terminology to describe the data set. In data mining the common terms are point-dimension. Statistics literature generally uses observation-variable. Machine learning research uses instance-feature. This article introduces a new class of aggregate functions that can be used to build data sets in a horizontal layout (denormalized with aggregations), automating SQL query writing and extending SQL capabilities. We show evaluating horizontal aggregations is a challenging and interesting problem and we introduced alternative methods and optimizations for their efficient evaluation. II. MOTIVATION As mentioned above, building a suitable data set for data mining purposes is a time- consuming task. This task generally requires writing long SQL statements or customizing SQL Code if it is automatically generated by some tool. There are two main ingredients in such SQL code: joins and aggregations; we focus on the second one. The most widely- known aggregation is the sum of a column over groups of rows. Some other aggregations return the average, maximum, minimum or row count over groups of rows. There exist many aggregations functions and operators in SQL. Unfortunately, all these aggregations have limitations to build data sets for data mining purposes. The main reason is that, in general, data sets that are stored in a relational database (or a data warehouse) come

Multiply imputing missing values in data sets with mixed measurement scales using a sequence of generalised linear models

Missing values in datasets form a very relevant and often overlooked problem in many fields. Most algorithms are not able to handle missing values for training a predictive model or analyzing a dataset. For this reason, records with missing values are either rejected or repaired. However, both repairing and rejecting affects the dataset and the final results, creating bias and uncertainty. Therefore, knowledge about the nature of missing values and the underlying mechanisms behind them are of vital importance. To gain more in-depth insight into the underlying structures and patterns of missing values, the concept of Monotone Mixture Patterns is introduced and used to analyze the patterns of missing values in datasets. Several visualization methods are proposed to present the “patterns of missingness” in an informative way. Finally, an algorithm to generate missing values in datasets is provided to form the basis of a benchmarking tool. This algorithm can generate a large variety of missing value patterns for testing and comparing different algorithms that handle missing values.

This paper presents a hybrid approach that integrates a genetic algorithm (GA) and data mining to produce control signatures. The control signatures define the best parameter intervals leading to a desired outcome. This hybrid method integrates multiple rule sets generated by a data mining algorithm with the fitness function of a GA. The solutions of the GA represent intersections among rules providing tight parameter bounds. The integration of intuitive rules provides an explanation for each generated control setting and it provides insights into the decision making process. The ability to analyze parameter trends and the feasible solutions generated by the GA with respect to the outcomes is another benefit of the proposed hybrid method. The presented approach for deriving control signatures is applicable to various domains, such as energy, medical protocols, manufacturing, airline operations, customer service, and so on. Control signatures were developed and tested for control of a power plant boiler. These signatures discovered insightful relationships among parameters. The results and benefits of the proposed method for the power plant boiler are discussed in the paper.

Use of machine learning (ML) and data mining (DM) algorithms has surfaced more often in the recent years for disease risk prediction problems in the healthcare communities. Several traditional feature selection models are combined with the DM and ML algorithms to improve accuracy of the disease risk prediction. In this study, a new Bio-inspired Ensemble Feature Selection (BEFS) model is introduced which is applied with the DM and ML algorithms. In the BEFS model, the most relevant and highly contributing features in the prediction are determined with a bio-inspired algorithm i.e., genetic algorithm, and an ensemble algorithm i.e., random forest algorithm. These important features obtained from the proposed model are then combined in various combinations and applied with the DM and ML algorithms, here logistic regression (LR) and random forest (RF), and the results obtained are promising. The experiment is executed using the famous ML language R. To accomplish this objective, the Breast Cancer Wisconsin (Diagnostic) dataset of UCI (University of California, Irvine) ML repository is utilized. In the experimental outcomes, the highest accuracy attained with the BEFS model is 96.49%, the AUC (Area Under Curve) achieved is 96%, and the sensitivity is 98.11%. These results, which greatly improve the disease risk prediction, are higher than several other existing works, while utilizing only six most relevant features out of the thirty two features of the dataset.

Skin cancer is dangerous since the beginning of its existence. Till now it is a big question mark to scientists that are trying hard to figure out the ultimate solution to discover skin cancer. Researches are made for fifty years but still, the experts are unable to detect skin cancer with a hundred percent accuracy in this modern era of technology. Thus, people's data from all around the world is gathered to cancer diagnosis centers. Back then, the data was recorded manually but due to the advancement of technology and the behavior of the computer systems, it is recorded digitally and it is manipulated in a way so that a conclusion could be achieved. This paper presents a study to compare skin cancer using data mining techniques and algorithms in the most accurate form. With the help of data mining, people's data from around the world is preprocessed in order to eliminate redundancy and missing values. Then clusterization is done on pre-processed data by clustering different algorithms to separate like and unlike data. These compared tables will help out for implementation of data mining algorithms for this disease.

With the huge amount of information available online, the World Wide Web is a fertile area for data mining research. The data mining research is at the cross road of research from several research communities, such as database, information retrieval, and within AI, especially the sub-areas of machine learning and data integrity. Every E-commerce and social website in World Wide Web uses the Classification is one of the data mining problems receiving great attention recently in the database community. Neural network is not suitable for data mining directly, because how the classifications were made is not explicitly stated as symbolic rules that are suitable for verification or interpretation by humans. Different concise symbolic rules with high accuracy can be extracted from a neural network with the proposed approach. The neural network is first trained to achieve the required accuracy in data mining. In this paper we are going to combine neural network with the three different algorithms which are commonly used in data mining to improve the data mining result. These three algorithms are CHARM Algorithm, Top K Rules mining and CM SPAM Algorithm. The different datasets of online e-commerce website filpkart and Amazon are used to train the neural network and to use in data mining. The results of all three data mining algorithm with neural network techniques then tested on the available datasets and result are compared by computational complexity of the algorithm.

The data-dependent acquisition in mass spectrometry-based proteomics combined with quantitative analysis using isobaric labeling (iTRAQ and TMT) inevitably introduces missing values in proteomic experiments where a number of LC-runs are combined, especially in the growing field of shotgun clinical proteomics, where the protein profiles from the proteomics analysis of several hundred patient samples are compared and correlated to clinical traits such as a specific disease or disease treatment in order to link specific outcomes to one or more proteins. In the context of clinical research it is evident that missing values in such datasets reduce the power of the downstream statistical analysis therefore may hampers the linking of the expression of disease traits to the expression of specific proteins that may be useful for prognostic, diagnostic, or predictive purposes. In our study, we tested three data imputation approaches initially developed for microarray data for the imputation of missing values in datasets that are generated by several runs of shotgun proteomic experiments and where the data were relative protein abundances based on isobaric tags (iTRAQ and TMT). Our conclusion is that imputation methods based on k Nearest Neighbors successfully impute missing values in datasets with up to 50% missing values.

Electronic payments is something that is currently in high demand by investors today, but transactions are often constrained due to various problems, especially from third parties. For this reason, cryptocurrency emerged, which is one of the solutions for conducting electronic payment transactions. Some types of cryptocurrency that are most in demand by investors are bitcoin, ethereum, and ripple. The fluctuation value of cryptocurrency is very difficult to predict so that investors often experience losses when making transactions. This study aims to predict cryptocurrency prices such as bitcoin, ethereum and ripple using data mining algorithms. The data mining algorithm used in this prediction process is K-NN, Neural Network, SVM, Linear Regression, Random Forest and Decision Tree. Data mining modeling is done by dividing the dataset into each type of commodity and then analyzed using each algorithm. The results of this study indicate that the accuracy value obtained from some data mining algorithms is good enough to predict cryptocurrency prices

The presence of missing data is a common problem that affects almost all clinical datasets. Since most available data mining and machine learning algorithms require complete datasets, accurately imputing (i.e. "filling in") the missing data is an essential step. This paper presents a methodology for the missing data imputation of longitudinal clinical data based on the integration of linear interpolation and a weighted K-Nearest Neighbours (KNN) algorithm. The Maximal Information Coefficient (MIC) values among features are employed as weights for the distance computation in the KNN algorithm in order to integrate intra- and inter-patient information. An interpolation-based imputation approach was also employed and tested both independently and in combination with the KNN algorithm. The final imputation is carried out by applying the best performing method for each feature. The methodology was validated on a dataset of clinical laboratory test results of 13 commonly measured analytes of patients in an intensive care unit (ICU) setting. The performance results are compared with those of 3D-MICE, a state-of-the-art imputation method for cross-sectional and longitudinal patient data. This work was presented in the context of the 2019 ICHI Data Analytics Challenge on Missing data Imputation (DACMI).

Computer-assisted data mining algorithms (DMAs) are being studied to screen spontaneous reporting databases for signals of novel adverse events. The performance characteristics and optimum deployment of these techniques remain to be established. To explore issues in the practical evaluation and deployment of DMAs by comparing findings from an empirical Bayesian DMA with those from a traditional drug safety surveillance program. Published findings from early postmarketing safety surveillance of thalidomide were compared with findings from an empirical Bayesian DMA. Differential results were used to explore practical issues in the evaluation and deployment of DMAs. Most adverse events highlighted by each method were compatible with the product labeling or natural history/complications of reported treatment indications. Traditional surveillance highlighted 4 potentially serious and unexpected adverse events (Stevens-Johnson syndrome, toxic epidermal necrolysis, seizures, skin ulcers) warranting labeling amendments or close monitoring. None of these adverse event terms generated a signal using the DMA. The DMA would not have enhanced early postmarketing surveillance in this particular setting. While the results cannot be used to draw inferences about the global performance of DMAs, they illustrate the following: (1) DMA performance may be highly situation dependent; (2) over-reliance on these methods may have deleterious consequences, especially with so-called "designated medical events"; and (3) the most appropriate selection of pharmacovigilance tools needs to be tailored to each situation, being mindful of the numerous factors that may influence comparative performance and incremental utility of DMAs.

A fuzzy logic based resource manager (RM) has been developed that automatically allocates electronic attack resources in real-time over many dissimilar platforms. Two different data mining algorithms have been developed to determine rules, strategies, and fuzzy decision tree structure. The first data mining algorithm uses a genetic algorithm as a data mining function and is called from an electronic game. The game allows a human expert to play against the resource manager in a simulated battlespace with each of the defending platforms being exclusively directed by the fuzzy resource manager and the attacking platforms being controlled by the human expert or operating autonomously under their own logic. This approach automates the data mining problem. The game automatically creates a database reflecting the domain expert's knowledge. It calls a data mining function, a genetic algorithm, for data mining of the database as required and allows easy evaluation of the information mined in the second step. The criterion for re- optimization is discussed as well as experimental results. Then a second data mining algorithm that uses a genetic program as a data mining function is introduced to automatically discover fuzzy decision tree structures. Finally, a fuzzy decision tree generated through this process is discussed.