Imputing missing value through ensemble concept based on statistical measures

Background . Properly constructed decision support systems (DSS) for modelling and forecasting behaviour of dynamic systems provide a possibility for taking into consideration uncertainties of probabilistic, statistical and structural types what results in higher quality of developed models and estimated forecasts. Objective . To consider general reasons for loosing (missing) data in statistical data analysis; to provide categorization of missing data into several groups, and identify the reasons for missing measurements; to provide stepwise system methodology for uncertainty analysis and selection of data imputation techniques; to give an insight into some popular missing values imputation techniques regarding their possible applications. Methods . To solve the problems mentioned the following methods have been used: data categorization approach from business or practical point of view that is necessary for discovering the reasons for availability of systemic and/or random missing values; the modern systemic methodology was hired for analysis of uncertainty causes and missing values imputation; the decision tree based imputation procedures; EM algorithm and regression model approach to forecasting missing data using forecasting functions. Results . The main results of the study are in categorization of the missing data into groups; development of systemic methodology for analysis of uncertainty causes and missing values imputation; providing an analysis for possibilities of missing values imputation with decision trees, EM algorithm and regression models; development of multistep forecasting functions on the basis of autoregression models; illustration of application of some selected perspective methods for missing data imputation. Conclusions . We proposed the six steps system methodology for data imputation which stresses that selection of correct method for imputation is tightly connected with the step-by-step analysis of the gaps causes and finding an appropriate technique for their imputation. The results of imputation sometimes are rather far from the existing data and should be smoothed or even broken from the sample due to their incorrectness. For such cases it should be proposed a new probabilistic-regression method which allows define parameters of the probability interval for the regression aiming missing data imputation. A series of computing experiments performed with EM algorithm, forecast regression based imputation technique and some other approaches shows that it is possible to reach high quality results regarding correct processing of data with missing values.

 Electrocardiogram signal analysis is very difficult to classify cardiac arrhythmia using machine learning methods. The ECG datasets normally come with multiple missing values. The reason for the missing values is the faults or distortion. When performing data mining, missing value imputation is the biggest task for data preprocessing. This problem could arise due to incomplete medical datasets if the incomplete missing values and cases were removed from the original database. To produce a good quality dataset for better analyzing the clinical trials, the suitable missing value imputation method is used. In this paper, we explore the different machine-learning techniques for the computed missing value in the electrocardiogram dataset. To estimate the missing imputation values, the collected data contains feature dimensions with their attributes. The experiments to compute the missing values in the dataset are carried out by using the four feature selection methods and imputation methods. The implemented results are shown by combined features using IG (information gain), GA (genetic algorithm) and the different machine learning classifiers such as NB (naïve bayes), KNN (K-nearest neighbor), MLP (Multilayer perception), and RF (Random forest). The GA (genetic algorithm) and IG (information gain) are the best suitable methods for obtaining the results on lower dimensional datasets with RMSE (Root mean square error. It efficiently calculates the best results for missing values. These four classifiers are used to analyze the impact of imputation methods. The best results for missing rate 10% to 40% are obtained by NB that is 0.657, 0.6541, 0.66, 0.657, and 0.657, as computed by RMSE (Root mean Square error). It means that error will efficiently reduced by naïve bayes classifier.

Neglecting missing values in diagnostic models can result in unreliable and suboptimal performance on new data. In this study, the authors imputed missing values for the CA-125 tumor marker in a large data set of ovarian tumors that was used to develop models for predicting malignancy. Four imputation techniques were applied: regression imputation, expectation-maximization, data augmentation, and hotdeck. Models using the imputed data sets were compared with models without CA-125 to investigate the important clinical issue concerning the necessity of CA-125 information for diagnostic models and with models using only complete cases to investigate differences between imputation and complete case strategies for missing values. The models are based on Bayesian generalized linear models (GLMs) and Bayesian least squares support vector machines. indicate that the use of CA-125 resulted in small, clinically nonsignificant increases in the AUC of diagnostic models. Minor differences between imputation methods were observed, and imputing CA-125 resulted in minor differences in the AUC compared with complete case analysis (CCA). However, GLM parameter estimates of predictor variables often differed between CCA and models based on imputation. The authors conclude that CA-125 is not indispensable in diagnostic models for ovarian tumors and that missing value imputation is preferred over CCA.

Enhancing endangered species monitoring by lowering data entry requirements with imputation techniques as a preprocessing step for the footprint identification technology (FIT)

A new framework for missing data estimation and reconstruction based on the geographical input information, data mining, and multi-criteria decision-making; theory and application in missing groundwater data of Damghan Plain, Iran

(1) Background: Gene-expression data usually contain missing values (MVs). Numerous methods focused on how to estimate MVs have been proposed in the past few years. Recent studies show that those imputation algorithms made little difference in classification. Thus, some scholars believe that how to select the informative genes for downstream classification is more important than how to impute MVs. However, most feature-selection (FS) algorithms need beforehand imputation, and the impact of beforehand MV imputation on downstream FS performance is seldom considered. (2) Method: A modified chi-square test-based FS is introduced for gene-expression data. To deal with the challenge of a small sample size of gene-expression data, a heuristic method called recursive element aggregation is proposed in this study. Our approach can directly handle incomplete data without any imputation methods or missing-data assumptions. The most informative genes can be selected through a threshold. After that, the best-first search strategy is utilized to find optimal feature subsets for classification. (3) Results: We compare our method with several FS algorithms. Evaluation is performed on twelve original incomplete cancer gene-expression datasets. We demonstrate that MV imputation on an incomplete dataset impacts subsequent FS in terms of classification tasks. Through directly conducting FS on incomplete data, our method can avoid potential disturbances on subsequent FS procedures caused by MV imputation. An experiment on small, round blue cell tumor (SRBCT) dataset showed that our method found additional genes besides many common genes with the two compared existing methods.

A massive amount of data is emerging continuously in the era of big data; the missing value is a common yet challenging problem. Missing data or missing value can be described as the data whose values in any given dataset are unknown. It is a recurrent phenomenon and can give biased results from the data to be observed and affect the value of the learning process. Therefore, it is mandatory to use missing value imputation techniques. Missing value imputation provides (MVI) optimal solutions for missing values in a dataset. There are diverse missing value imputation techniques such as statistical method and machine learning methods proposed till date, each having their own significances and flaws. Reasonably, good imputation results may be produced by machine learning MVI approaches but they take greater imputation times than statistical approaches usually. In this paper, we have reviewed some of the missing value imputation approaches proposed. In order to show the efficiency of their proposed approaches, they had compared their proposed methods with some baseline imputation algorithms like mode/median, k-nearest neighbor, Naive Bayes, support vector machine. The outcome of each proposed method is analyzed and their extension scopes from the perspectives of research focus. The main idea of this literature is to give a general review of missing value imputation approaches.

Currently, many databases contain missing values, especially in medical data. Statistical and data mining approaches often require complete data conditions, where these two approaches will not provide adequate performance if the data contains missing values. Several techniques have been made to overcome missing values, one of which is by deleting data containing missing values. However, this approach will omit a lot of information if the data found includes many missing values. This study used an imputation approach (filling in the missing attributes) with a clustering approach. One of the most common clustering approaches is K-Means Clustering. In K-means clustering, the value of the centroid gets from the closest observed value. In this study, we propose updating the centroid value based on the harmonic average of the distance across all observations per centroid. This method is known as K-Harmonic Means Clustering (KHM). We proposed a new program approach for a mixed dataset on three scenarios for missing values of 10%, 20%, and 30%. From the experiments conducted on experimental data sets containing missing values, we get a small proportion of missing values (10%) with a small number of clusters or K, which gives a smaller RMSE value compared to other scenarios.

Real-time flood warning systems as part of digital and innovative non-structural solutions have been widely used to prepare decision makers, operators, and affected population to alleviate socio-economic flooding consequences [1]. Many models have been introduced recently to provide more accurate flood forecasts with longer lead times. However, they rely highly on availability of input data which may contain missing values in measurement for one or more timesteps mainly due to wide range of reasons such as random/systematic errors and blunders. Hence, real-time early warning systems cannot be operated properly unless these missing data are properly infilled [2]. Despite data imputation techniques have been mainly employed in pre-processing step of historical data i.e., models training and validation, they have not been properly elaborated in real-time operation practically [3].This paper aims to propose a new event-based data imputation method for infilling rainfall and water level missing data appearing in real-time operation of flood early warning systems. Event identification is first used to divide the real-time data into the wet or dry weather conditions which then are used for selecting the best strategy of infilling missing data. Imputation decision framework takes advantage of various imputation techniques including t-copula, move-median, and kriging based on external available benchmarks and temporal location of missing data. Proposed methodology is tested in real-world case study of urban drainage system in London, UK. Conventional techniques such as linear regression, kriging, nearest neighbourhood, t-copula, inverse distance, and similar calendar are first compared together and best techniques are then tested with proposed methodology in three real-time scenarios as (1) missing rainfall intensity, (2) missing water level, (3) missing both rainfall and water level. Recurrent neural network model used for flood forecasting and results are demonstrated for the next 3hr-ahead predictions.Results show the proposed method can reduce root mean square error (RMSE) from 55% to 13%, 43% to 12%, and 97% to 17% for the above scenarios, respectively. Furthermore, using external benchmark data resources, i.e. other near rainfall/water level stations, shows very efficient when missing data appears at early steps of rainfall events where selected conventional techniques suffer from predicting rainfall pattern. Finally, when both water level and rainfall intensity were missing, the proposed imputation method can reduce RMSE from 197mm to 117mm (RMSE was originally 100 for no missing data) for 3hr-ahead predictions. Generally, this study shows the proposed imputation method can better infill the missing data, especially those in the flood event by using correlated data in other weather/gauging stations and flexibility in applying different methods.

Unsupervised classification of satellite image time series (SITS) is prominent in numerous multitemporal remote sensing applications. However, when optical images are concerned, a missing value reconstruction task becomes pivotal, due to the impact of cloud cover on the input SITS. This task is usually addressed as a pre-processing step before feeding the time series to a clustering model. In this paper, we propose a pixelwise SITS clustering algorithm which integrates missing value imputation jointly with the clustering task. Moreover, the clustering process is designed to jointly perform both representation learning and cluster assignment, enabling the proposed model to simultaneously tackle three duties (missing value imputation, representation learning, and cluster assignment), while being trained in an end-to-end manner. The experimental results show that the proposed model performs well compared to other models that address time series imputation and clustering independently. Furthermore, a visualization analysis suggests that the proposed model learns both the imputation and clustering effectively despite being trained simultaneously.

Clustering algorithms can identify groups in large data sets, such as star catalogs and hyperspectral images. In general, clustering methods cannot analyze items that have missing data values. Common solutions either fill in the missing values (imputation) or ignore the missing data (marginalization). Imputed values are treated as being just as reliable as the observed data, but they are only as good as the assumptions used to create them. In contrast, we present a method for encoding partially observed features as a set of supplemental soft constraints and introduce the KSC algorithm, which incorporates constraints into the clustering process. In experiments on artificial data and data from the Sloan Digital Sky Survey, we show that soft constraints are an effective way to enable clustering with missing values.

This article introduces the Missingness-aware Self-Supervised Graph Neural Network (MissGNN), a novel model for tabular data predictions in scenarios with missing values. Addressing a significant challenge in fields like finance, healthcare, and environmental studies, MissGNN innovatively integrates feature imputation and label prediction within a unified framework. Utilizing a Tabular Bipartite Graph, it distinctively represents data points and features, enabling intricate modeling of their relationships. This approach is pivotal in contexts where missing data can significantly skew model training and predictions. MissGNN employs a Graph Neural Network to generate embeddings, capturing interactions between data points and features for accurate missing value imputation and label prediction. It features a dual-focus mechanism, comprising edge-wise feature imputation and node-wise label prediction. A pre-training phase for feature imputation enhances the model’s predictive ability for both features and labels. Additionally, MissGNN incorporates a multi-view learning strategy through feature subsetting, further enriching its representation learning. Experiments on 14 tabular datasets demonstrate MissGNN’s superiority over existing models in managing missing values across both regression and classification tasks. The results underscore its robustness and practical applicability, establishing it as a versatile tool for tackling tabular data prediction challenges with missing values. MissGNN’s contributions significantly redefine the approach to missing value imputation, aligning feature imputation and label prediction in a cohesive manner and underscoring the potential of Graph Neural Networks in processing tabular data.

The accuracy of a knowledge extraction algorithm in a large database depends on the quality of the data preprocessing and the methods used. The massive amounts of data that we collect every day are putting storage capacity at a premium. In reality, many databases are characterized by attributes with outliers, redundant, and even more missing values. Missing data and outliers are ubiquitous in our databases, and imputation techniques will help us mitigate their influence. To solve this problem, as well as the problem of data size, this paper proposes a data preprocessing approach based on the k-nearest neighbor (KNN) completion for imputation of missing data and principal component analysis (PCA) for processing redundant data, thus reducing the data size by generating a significant quality sample after imputation of missing and outlier data. A rigorous comparison is made between our approach and two others. The dissolved gas data from Rio Tinto Alcan’s transformer T0001 were imputed by KNN, where k equals 5. For 6 imputed gases, the average percentage error is about 2%, 17.5% after average imputation, and 23.65% after multiple imputations. For data compression, 2 axes were selected based on the elbow rule and the Kaiser threshold.

In various fields of information examination, for example, AI, profoundly getting the hang of missing information is a typical issue. Missing qualities should be tended to since they can adversely affect the exactness and adequacy of prescient models. This research investigates how data discretization affects deep learning methods for filling the missing values in datasets with continuous features. They provide a unique method for imputing missing values using deep neural networks (DNNs) called extravagant expectation maximization-deep neural network (EEM-DNN). This approach discretizes continuous features into separate intervals initially. This is justified by treating the issue of missing value imputation as a classification work, with the missing values being considered a distinct class. A DNN, designed explicitly for imputation, is then trained using the discretized data. The expectation maximization concepts are incorporated into the network architecture, and as a result, the network iteratively improves its imputation predictions. They run comprehensive experiments on several datasets from different fields to gauge the efficacy of the suggested strategy. The effectiveness of EEM-DNN is compared to that of other imputation approaches, such as traditional imputation techniques and deep learning methods without data discretization. Our findings show that data discretization significantly enhances imputation accuracy. In terms of imputation accuracy and prediction performance on downstream tasks, the EEM-DNN method regularly performs better than alternative methods. It also examines if various discretization techniques affect the overall imputation process. They find that the trade-off between bias and variance in imputed data depends on the discretization method selected. This highlights the significance of choosing a suitable discretization approach depending on the unique properties of the dataset.

Despite considerable advances in high throughput technology over the last decade, new challenges have emerged related to the analysis, interpretation, and integration of high-dimensional data. The arrival of omics datasets has contributed to the rapid improvement of systems biology, which seeks the understanding of complex biological systems. Metabolomics is an emerging omics field, where mass spectrometry technologies generate high dimensional datasets. As advances in this area are progressing, the need for better analysis methods to provide correct and adequate results are required. While in other omics sectors such as genomics or proteomics there has and continues to be critical understanding and concern in developing appropriate methods to handle missing values, handling of missing values in metabolomics has been an undervalued step. Missing data are a common issue in all types of medical research and handling missing data has always been a challenge. Since many downstream analyses such as classification methods, clustering methods, and dimension reduction methods require complete datasets, imputation of missing data is a critical and crucial step. The standard approach used is to remove features with one or more missing values or to substitute them with a value such as mean or half minimum substitution. One of the major issues from the missing data in metabolomics is due to a limit of detection, and thus sophisticated methods are needed to incorporate different origins of missingness. This dissertation contributes to the knowledge of missing value imputation methods with three separate but related research projects. The first project consists of a novel missing value imputation method based on a modification of the k nearest neighbor method which accounts for truncation at the minimum value/limit of detection. The approach assumes that the data follows a truncated normal distribution with the truncation point at the detection limit. The aim of the second project arises from the limitation in the first project. While the novel approach is useful, estimation of the truncated mean and standard deviation is problematic in small sample sizes (N < 10). In this project, we develop a Bayesian model for imputing missing values with small sample sizes. The Bayesian paradigm has generally been utilized in the omics field as it exploits the data accessible from related components to acquire data to stabilize parameter estimation. The third project is based on the motivation to determine the impact of missing value imputation on down-stream analyses and whether ranking of imputation methods