Soft Clustering Based Missing Value Imputation

Data preprocessing and cleansing play a vital role in data mining by ensuring good quality of data. Data-cleansing tasks include imputation of missing values, identification of outliers, and identification and correction of noisy data. In this paper, we present a novel technique called A Fuzzy Expectation Maximization and Fuzzy Clustering-based Missing Value Imputation Framework for Data Pre-processing (FEMI). It imputes numerical and categorical missing values by making an educated guess based on records that are similar to the record having a missing value. While identifying a group of similar records and making a guess based on the group, it applies a fuzzy clustering approach and our novel fuzzy expectation maximization algorithm. We evaluate FEMI on eight publicly available natural data sets by comparing its performance with the performance of five high-quality existing techniques, namely EMI, GkNN, FKMI, SVR and IBLLS. We use thirty-two types (patterns) of missing values for each data set. Two evaluation criteria namely root mean squared error and mean absolute error are used. Our experimental results indicate (according to a confidence interval and $$t$$t test analysis) that FEMI performs significantly better than EMI, GkNN, FKMI, SVR, and IBLLS.

Microarray data often contains multiple missing genetic expression values that degrade the performance of statistical and machine learning algorithms. This paper presents a K ranked diagonal covariance-based missing value estimation algorithm (KRCOV) that has demonstrated significantly superior performance compared to the more commonly used K-nearest neighbour (KNN) imputation algorithm when it is applied to estimate missing values of BRCA1, BRCA2 and Sporadic genetic mutation samples present in ovarian cancer. Experimental results confirm KRCOV outperformed both KNN and zero imputation techniques in terms of their classification accuracies when used toimpute randomly missing values from 1% to 5%.The classifier used for this purpose was the Generalized Regression Neural Network.The paper also provides a hypothesis for why KRCOV performs better than KNN not only for bioinformatics data but also for other data types having strong correlated values.

Trial Quality in Nephrology: How Are We Measuring Up?

The raw data obtained from the logs may be noisy, incomplete, and inconsistent that’s why data pre-processing is an essential step in data mining. The quality of data plays a vital role during the evaluation process. The results of the evaluation process primarily depend upon the quality of the data input. So, data pre-processing is the primary and most crucial step before knowledge discovery. This paper is based on two main steps- data pre-processing techniques and results after applied data pre-processing on mutual funds’ data. Data pre-processing transforms the raw data into a structured, understandable format. Moreover, data pre-processing performs not only the transformation of data but also makes it understandable according to need. It is mainly divided into four steps, i.e., data integration, data cleaning, data transformation, and data reduction. This paper takes the fifteen-year NAV data of twenty mutual funds for analysis propose. This paper explains several techniques of data pre-processing to transform the raw data into an understandable format.

BackgroundMissing data are a common problem in prospective studies with a long follow-up, and the volume, pattern and reasons for missing data may be relevant when estimating the cost of illness. We aimed to evaluate the effects of different methods for dealing with missing longitudinal cost data and for costing caregiver time on total societal costs in Alzheimer’s disease (AD).MethodsGERAS is an 18-month observational study of costs associated with AD. Total societal costs included patient health and social care costs, and caregiver health and informal care costs. Missing data were classified as missing completely at random (MCAR), missing at random (MAR) or missing not at random (MNAR). Simulation datasets were generated from baseline data with 10–40 % missing total cost data for each missing data mechanism. Datasets were also simulated to reflect the missing cost data pattern at 18 months using MAR and MNAR assumptions. Naïve and multiple imputation (MI) methods were applied to each dataset and results compared with complete GERAS 18-month cost data. Opportunity and replacement cost approaches were used for caregiver time, which was costed with and without supervision included and with time for working caregivers only being costed.ResultsTotal costs were available for 99.4 % of 1497 patients at baseline. For MCAR datasets, naïve methods performed as well as MI methods. For MAR, MI methods performed better than naïve methods. All imputation approaches were poor for MNAR data. For all approaches, percentage bias increased with missing data volume. For datasets reflecting 18-month patterns, a combination of imputation methods provided more accurate cost estimates (e.g. bias: −1 % vs −6 % for single MI method), although different approaches to costing caregiver time had a greater impact on estimated costs (29–43 % increase over base case estimate).ConclusionsMethods used to impute missing cost data in AD will impact on accuracy of cost estimates although varying approaches to costing informal caregiver time has the greatest impact on total costs. Tailoring imputation methods to the reason for missing data will further our understanding of the best analytical approach for studies involving cost outcomes.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-016-0188-1) contains supplementary material, which is available to authorized users.

Functional diversity (FD) is an important component of biodiversity that quantifies the difference in functional traits between organisms. However, FD studies are often limited by the availability of trait data and FD indices are sensitive to data gaps. The distribution of species abundance and trait data, and its transformation, may further affect the accuracy of indices when data is incomplete. Using an existing approach, we simulated the effects of missing trait data by gradually removing data from a plant, an ant and a bird community dataset (12, 59, and 8 plots containing 62, 297 and 238 species respectively). We ranked plots by FD values calculated from full datasets and then from our increasingly incomplete datasets and compared the ranking between the original and virtually reduced datasets to assess the accuracy of FD indices when used on datasets with increasingly missing data. Finally, we tested the accuracy of FD indices with and without data transformation, and the effect of missing trait data per plot or per the whole pool of species. FD indices became less accurate as the amount of missing data increased, with the loss of accuracy depending on the index. But, where transformation improved the normality of the trait data, FD values from incomplete datasets were more accurate than before transformation. The distribution of data and its transformation are therefore as important as data completeness and can even mitigate the effect of missing data. Since the effect of missing trait values pool-wise or plot-wise depends on the data distribution, the method should be decided case by case. Data distribution and data transformation should be given more careful consideration when designing, analysing and interpreting FD studies, especially where trait data are missing. To this end, we provide the R package “traitor” to facilitate assessments of missing trait data.

Racial and Ethnic Disparities in Predictive Accuracy of Machine Learning Algorithms Developed Using a National Database for 30-Day Complications Following Total Joint Arthroplasty

The success of blockchain projects depends, to a large extent, on the developer's involvement. Source code-commit privilege to the project is typically restricted to a few developers known as committers. Project leaders continuously search for new committers to evolve into a high-quality blockchain. However, promoting developers into committers is risky, mainly when the promoted committers exhibit low involvement behaviour. Hence, the committer assessment process is critical to the successful evolution of blockchain projects. The phenomenon of developer induction as code committers has previously been explored in the literature. However, previous studies employed objective measures such as the number of bugs report and code patches to appraise the developers activities for promotion purposes. Although these approaches are significant, behavioural tendencies influencing developers' activities are ignored. There have, however, been comparatively few investigations on a subjective measure that evaluate developers' perception of their activities for promotion purposes. This study aims to appraise blockchain developers' perceptions for predicting future committers in blockchain projects. In this study, 173 blockchain developers’ perceptions were gathered. The study employed hybrid analyses such as Structural Equation Modelling (SEM) to first analyse the survey data and Machine Learning (ML) algorithms to predict future committers. The performance of various ML algorithms was evaluated compared to classification performance indicators such as the F1 score, accuracy, precision, and AUC. In addition, the study also investigates the most important factors that predict future committers. The results indicate that XGBoost algorithm achieves the best performance with an accuracy of 0.94. Moreover, the most important factors in predicting future committers in a blockchain project are project desertion, developer involvement, decision-right delegation and system integration.

Machine learning for Big Data analytics in plants.

Restriction site‐associated DNA sequencing (RADseq) has emerged as a useful tool in systematics and population genomics. A common feature of RADseq data sets is that they contain missing data that arise from multiple sources including genealogical sampling bias, assembly methodology and sequencing error. Many RADseq studies have demonstrated that allowing sites (single nucleotide polymorphisms, SNPs) with missing data can increase support for phylogenetic hypotheses. Two non‐mutually exclusive explanations for this observation are that (a) larger data sets contain more phylogenetic information; and (b) excluding missing data disproportionally removes sites with the highest mutation rates, causing the exclusion of characters that are likely variable and informative. Using a RADseq data set derived from the East African banana frog, Afrixalus fornasini (up to 1.1 million SNPs), we found that missing data thresholds were positively correlated with the proportion of parsimony‐informative sites and mean branch support. Using three proxies for estimating site‐specific rate, we found that the most conservative missing data strategies excluded rapidly evolving sites, with four‐state sites present only when allowing ≥60% missing data per SNP. Topological similarity among estimated phylogenies was highest for the data sets with ≥60% missing data per SNP. Our results suggest that several desirable phylogenetic qualities were observed when allowing ≥60% missing data per SNP. However, at the highest missing data thresholds (80% and 90% missing data per SNP), we observed differences in performance between high‐ and mixed‐weight DNA extraction samples, which may indicate there are trade‐offs to consider when using degraded genomic template with RADseq protocols.

Real-world databases are highly susceptible to noisy, missing, and inconsistent data due to their typically huge size, which is a prevailing problem in data analysis. The easiest way to handle such data sets in classification is to discard data with missing and extreme values. Since this complete case approach may result in a loss of valuable information and reduced data set size, preprocessing techniques are used in this paper. These techniques include data cleaning, data transformations and data reduction. A new feature selection for data reduction is introduced, which uses the fast fuzzy clustering algorithm in classification problems. The experiments show the advantages of the proposed methods for data preprocessing in a real world problem: the prediction of bankruptcy. The data set used in this study has missing values and extreme values. The data set also presents a much smaller bankruptcy class than the not bankruptcy class

Machine learning-based imputation soft computing approach for large missing scale and non-reference data imputation

The liver, a life-sustaining organ, plays a substantial role in many body functions. Liver diseases have become an important world health problem in terms of prevalence, incidences, and mortalities. Liver fibrosis/cirrhosis is great of importance, because if not treated in time liver cancer could be occurred and spread to other parts of the body. For this reason, early diagnosis of liver fibrosis/cirrhosis gives significance. Accordingly, this study investigated the performances of different machine learning algorithms for prediction of liver fibrosis/cirrhosis based on demographic and blood values. In this context, random forest, k nearest neighbour, C4.5 decision tree, K-star, random tree and reduced error pruning tree algorithms were used. Two distinct approaches were employed to evaluate the performances of machine learning algorithms. In the first approach, the entire features of dataset were utilized, while in the second approach, only the features selected through principal component analysis were used. Each approach was rigorously assessed using both 10-fold cross-validation and data splitting (70% train and 30% test) techniques. By conducting separate evaluations for each approach, a comprehensive understanding of the effectiveness of utilizing all features versus extracted features based principal component analysis was attained, providing valuable insights into the impact of feature dimensionality reduction on model performance. In this study, all analyses were implemented on WEKA data mining tool. In the first approach, the classification accuracies of random forest algorithm were 89.72% and 90.75% with the application of data splitting (70%-30%) and cross-validation techniques, respectively. In the second approach, where feature reduction is performed using principal component analysis technique, the accuracy values obtained from data splitting and cross-validation techniques of random forest algorithm were 88.61% and 88.83%, respectively. The obtained results revealed out that random forest algorithm outperformed for both approaches. Besides, the application of principal component analysis technique negatively affected the classification performance of used machine learning algorithms. It is thought that the proposed model will guide specialist physicians in making appropriate treatment decisions for patients with liver fibrosis/cirrhosis, potentially leading to death in its advanced stages.

Pressure gradient (PG) in liquid-liquid flow is one of the key components to design an energy-efficient transportation system for wellbores. This study aims to develop five robust machine learning (ML) algorithms and their fusions for a wide range of flow patterns (FP) regimes. The MLs include Support Vector Machine (SVM), Gaussian Process (GP), Random Forest (RF), Artificial Neural Network (ANN), k-Nearest Neighbor (kNN), and fusions of these five MLs. A total of eleven hundred experimental data points for nine FPs (two stratified and seven dispersed patterns) in horizontal wellbores are used to develop the MLs. The MLs' performance is evaluated using the metrics including mean absolute percentage error (MAPE), median absolute percentage error (MdAPE), coefficient of variation of root mean squared error (CV-RMSE), and adjusted coefficient of determination. The evaluation metrics are cross-validated using a repeated train-test split strategy. Seven important predictor variables are identified using a supervised feature selection approach: oil and water velocities, FP, input diameter, oil and water density, and oil viscosity. The results show that the high PG prediction accuracy can be achieved using GP compared to other MLs except for the ML-fusions (p < 0.05). A Friedman's test and Wilcoxon Sign-Rank post hoc analysis with Bonferroni correction show that PG prediction errors using GP are significantly lower than using the ANN model (p < 0.05). The values are 18.44 % and 23.9 % for CV-RMSE, 11.6 % and 10.06 % for MAPE, and 7.5 % and 6.75 % for MdAPE, using ANN and GP, respectively. While the previous studies mostly used ANN to demonstrate the capability of MLs to predict PG over the mechanistic or correlation-based models, the present research has shown that GP is even better than ANN using a wide range of FPs and a large data set. • Machine-learning algorithms are developed to predict the pressure gradient. • The Gaussian Process model produces significantly lower prediction error. • The median absolute percentage error using Gaussian Process is 6.75 %. • More than 95 % variation of pressure gradient can be explained using seven variables.

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Contribution:</i> This article presents evidence that electrical engineering, computer science, and data science students, participating in introduction to machine learning (ML) courses, fail to interpret the performance of ML algorithms correctly, since they fail to consider the application domain. This phenomenon is referred to as the domain neglect bias and is discussed in this article from the cognitive psychology perspective. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>Background:</i> ML algorithms are common components in electrical engineering, computer science, and data science curricula. Learning ML is challenging not only due to its complex mathematical and algorithmic aspects, but also due to: 1) the complexity of using these algorithms in the context of real-life situations correctly and 2) the comprehension of related social and ethical issues. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>Research Question:</i> To what extent do students fail to correctly interpret the performance of ML algorithms due to the domain neglect bias and how can this bias be explained? <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>Methodology:</i> A questionnaire containing two questions was designed, in which the participants were asked to compare the performance of ML algorithms in the contexts of different application domains. The questions were designed to trigger the domain neglect bias which in turn, allowed quantification of its prevalence. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>Findings:</i> One third of the students failed to recognize the outperforming ML algorithm due to the domain neglect bias. Educators should note this bias and develop new pedagogical methods for teaching ML performance in order to mitigate the bias.