Data discretization impact on deep learning for missing value imputation of continuous data

In this research, deep neural networks (CNNs) were used to estimate missing values ​​in a univariate time series dataset and compared with DES/Holt and (SVR) estimation methods to determine their accuracy in handling missing values ​​in the dataset. These missing values ​​directly affect the process of building mathematical and statistical models, affecting the accuracy of the final results that enable us to make the right decisions in the future. The presence of missing values ​​in the dataset is due to problems that occur during sampling, such as a malfunction in the measuring device, security attacks, or communication errors. The results indicated that the CNN model outperformed other methods by using a simulation approach that generated data randomly. Three different sample sizes (60, 100, 300) were selected with missing data percentages of (10%, 15%, 20%) for each, meeting the "missing at random" (MAR) condition. The Box-Jenkins MA (1) model was applied once with β = 0.5 and again with β = 0.9. The accuracy of the estimation methods was evaluated using the accuracy criteria mean sum of squared error (MSE) and root mean squared error (RMSE).

Multiply imputing missing values in data sets with mixed measurement scales using a sequence of generalised linear models

The data-dependent acquisition in mass spectrometry-based proteomics combined with quantitative analysis using isobaric labeling (iTRAQ and TMT) inevitably introduces missing values in proteomic experiments where a number of LC-runs are combined, especially in the growing field of shotgun clinical proteomics, where the protein profiles from the proteomics analysis of several hundred patient samples are compared and correlated to clinical traits such as a specific disease or disease treatment in order to link specific outcomes to one or more proteins. In the context of clinical research it is evident that missing values in such datasets reduce the power of the downstream statistical analysis therefore may hampers the linking of the expression of disease traits to the expression of specific proteins that may be useful for prognostic, diagnostic, or predictive purposes. In our study, we tested three data imputation approaches initially developed for microarray data for the imputation of missing values in datasets that are generated by several runs of shotgun proteomic experiments and where the data were relative protein abundances based on isobaric tags (iTRAQ and TMT). Our conclusion is that imputation methods based on k Nearest Neighbors successfully impute missing values in datasets with up to 50% missing values.

Background . Properly constructed decision support systems (DSS) for modelling and forecasting behaviour of dynamic systems provide a possibility for taking into consideration uncertainties of probabilistic, statistical and structural types what results in higher quality of developed models and estimated forecasts. Objective . To consider general reasons for loosing (missing) data in statistical data analysis; to provide categorization of missing data into several groups, and identify the reasons for missing measurements; to provide stepwise system methodology for uncertainty analysis and selection of data imputation techniques; to give an insight into some popular missing values imputation techniques regarding their possible applications. Methods . To solve the problems mentioned the following methods have been used: data categorization approach from business or practical point of view that is necessary for discovering the reasons for availability of systemic and/or random missing values; the modern systemic methodology was hired for analysis of uncertainty causes and missing values imputation; the decision tree based imputation procedures; EM algorithm and regression model approach to forecasting missing data using forecasting functions. Results . The main results of the study are in categorization of the missing data into groups; development of systemic methodology for analysis of uncertainty causes and missing values imputation; providing an analysis for possibilities of missing values imputation with decision trees, EM algorithm and regression models; development of multistep forecasting functions on the basis of autoregression models; illustration of application of some selected perspective methods for missing data imputation. Conclusions . We proposed the six steps system methodology for data imputation which stresses that selection of correct method for imputation is tightly connected with the step-by-step analysis of the gaps causes and finding an appropriate technique for their imputation. The results of imputation sometimes are rather far from the existing data and should be smoothed or even broken from the sample due to their incorrectness. For such cases it should be proposed a new probabilistic-regression method which allows define parameters of the probability interval for the regression aiming missing data imputation. A series of computing experiments performed with EM algorithm, forecast regression based imputation technique and some other approaches shows that it is possible to reach high quality results regarding correct processing of data with missing values.

High-throughput biotechnologies, such as gene expression microarrays or mass-spectrometry-based proteomic assays, suffer from frequent missing values due to various experimental reasons. Since the missing data points can hinder downstream analyses, there exists a wide variety of ways in which to deal with missing values in large-scale data sets. Nowadays, it has become routine to estimate (or impute) the missing values prior to the actual data analysis. After nearly a decade since the publication of the first missing value imputation methods for gene expression microarray data, new imputation approaches are still being developed at an increasing rate. However, what is lagging behind is a systematic and objective evaluation of the strengths and weaknesses of the different approaches when faced with different types of data sets and experimental questions. In this review, the present strategies for missing value imputation and the measures for evaluating their performance are described. The imputation methods are first reviewed in the context of gene expression microarray data, since most of the methods have been developed for estimating gene expression levels; then, we turn to other large-scale data sets that also suffer from the problems posed by missing values, together with pointers to possible imputation approaches in these settings. Along with a description of the basic principles behind the different imputation approaches, the review tries to provide practical guidance for the users of high-throughput technologies on how to choose the imputation tool for their data and questions, and some additional research directions for the developers of imputation methodologies.

Currently, many databases contain missing values, especially in medical data. Statistical and data mining approaches often require complete data conditions, where these two approaches will not provide adequate performance if the data contains missing values. Several techniques have been made to overcome missing values, one of which is by deleting data containing missing values. However, this approach will omit a lot of information if the data found includes many missing values. This study used an imputation approach (filling in the missing attributes) with a clustering approach. One of the most common clustering approaches is K-Means Clustering. In K-means clustering, the value of the centroid gets from the closest observed value. In this study, we propose updating the centroid value based on the harmonic average of the distance across all observations per centroid. This method is known as K-Harmonic Means Clustering (KHM). We proposed a new program approach for a mixed dataset on three scenarios for missing values of 10%, 20%, and 30%. From the experiments conducted on experimental data sets containing missing values, we get a small proportion of missing values (10%) with a small number of clusters or K, which gives a smaller RMSE value compared to other scenarios.

Missing values in datasets form a very relevant and often overlooked problem in many fields. Most algorithms are not able to handle missing values for training a predictive model or analyzing a dataset. For this reason, records with missing values are either rejected or repaired. However, both repairing and rejecting affects the dataset and the final results, creating bias and uncertainty. Therefore, knowledge about the nature of missing values and the underlying mechanisms behind them are of vital importance. To gain more in-depth insight into the underlying structures and patterns of missing values, the concept of Monotone Mixture Patterns is introduced and used to analyze the patterns of missing values in datasets. Several visualization methods are proposed to present the “patterns of missingness” in an informative way. Finally, an algorithm to generate missing values in datasets is provided to form the basis of a benchmarking tool. This algorithm can generate a large variety of missing value patterns for testing and comparing different algorithms that handle missing values.

Surveys are mainly conducted to obtain valuable information on some criteria from a specified population. But, the survey results often become biased due to non-response of the subjects under study for highly significant attributes. Such non-ignorable missingness need to be treated and the actual values should be retrieved. Many methods have already been proposed for handling missing values in either discrete or continuous attributes. But, there exists a large gap in handling non-ignorable missing values in datasets with mixed attributes. With the intent of addressing this gap, this paper proposes a methodology called as Bayesian Genetic Algorithm (BAGEL) with hybridized Bayesian and Genetic Algorithm principles. In BAGEL, the initial population is generated using Bayesian model and fitness values of the chromosomes are evaluated using Bayesian principles. BAGEL is implemented in real datasets for imputing both discrete and continuous missing values and the imputation accuracy is observed. The experimental results show the superior performance of BAGEL than other standard imputation techniques. Statistical tests conducted to validate the experimental results also prove that BAGEL outperforms at all missing rates from 5% to 50%.

Prediction model is used to forecast or predict value from dataset. But one of the most common problems in training prediction model is there are missing values in datasets. Problem is usually managed by two methods for solving this problem. First is ignoring, but it reduces the predictive model's performance because of the data that was cut off may be important. Another method is replacing the missing values or data imputation. Benefit of imputation is it still keep all of data. It means an important data will not loss. Therefore, most researchers offer an imputation method for solving this problem. In the past most researches are proposed algorithm that trying to recover the original data, but main object of using prediction model is accuracy of prediction. Algorithm is based on Genetics Algorithm and Multiple Linear Regression is create for improving performance of prediction model.

Excessive withdrawal of groundwater for agricultural irrigation can cause seawater intrusion into coastal aquifers. Such a case will in turn results in deterioration of irrigation water quality. Determination of irrigation water quality with traditional methods is a time-consuming and costly process. However, machine learning algorithms can be useful tools for modeling and estimating groundwater quality used for irrigation water purposes. In this study, TDS, PS, SAR, and Cl parameters of groundwater were estimated with models based on EC and pH variables. For this purpose, prediction performances of two different deep learning methods (convolutional neural network (CNN) and deep neural network (DNN)) and two different classical machine learning (Random Forest (RF) and extreme gradient boosting (XGBoost)) methods were compared. In addition, predictive uncertainty of the models was determined by quantile regression (QR) analysis. Performance criteria and results of uncertainty analysis revealed that CNN (in testing phase, NSE = 0.95 for TDS, NSE = 0.96 for PS, NSE = 0.67 for SAR and NSE = 0.93 for CI) and DNN (in testing phase, NSE = 0.91 for TDS, NSE = 0.91 for PS, NSE = 0.57 for SAR and NSE = 0.94 for Cl) models had quite a close performance in estimation of TDS, PS, SAR, and Cl parameters and higher than the other two classical machine learning methods. As a result, the CNN model can be considered the best performing model in estimating all quality parameters due to the highest NSE and lowest RMSE values. In addition, the Taylor diagram showed that the values estimated using the CNN model had the highest correlation with the measured data. It was determined that the model with the lowest uncertainty based on the PICP statistics was DNN, followed by the CNN model. However, the CNN model has predicted outliers more accurately. Present findings proved that deep learning models could offer efficient tools for predicting irrigation water quality parameters.

This article introduces the Missingness-aware Self-Supervised Graph Neural Network (MissGNN), a novel model for tabular data predictions in scenarios with missing values. Addressing a significant challenge in fields like finance, healthcare, and environmental studies, MissGNN innovatively integrates feature imputation and label prediction within a unified framework. Utilizing a Tabular Bipartite Graph, it distinctively represents data points and features, enabling intricate modeling of their relationships. This approach is pivotal in contexts where missing data can significantly skew model training and predictions. MissGNN employs a Graph Neural Network to generate embeddings, capturing interactions between data points and features for accurate missing value imputation and label prediction. It features a dual-focus mechanism, comprising edge-wise feature imputation and node-wise label prediction. A pre-training phase for feature imputation enhances the model’s predictive ability for both features and labels. Additionally, MissGNN incorporates a multi-view learning strategy through feature subsetting, further enriching its representation learning. Experiments on 14 tabular datasets demonstrate MissGNN’s superiority over existing models in managing missing values across both regression and classification tasks. The results underscore its robustness and practical applicability, establishing it as a versatile tool for tackling tabular data prediction challenges with missing values. MissGNN’s contributions significantly redefine the approach to missing value imputation, aligning feature imputation and label prediction in a cohesive manner and underscoring the potential of Graph Neural Networks in processing tabular data.

Targeted proteomics utilizing antibody-based proximity extension assays provides sensitive and highly specific quantifications of plasma protein levels. Multivariate analysis of this data is hampered by frequent missing values (random or left censored), calling for imputation approaches. While appropriate missing-value imputation methods exist, benchmarks of their performance in targeted proteomics data are lacking. Here, we assessed the performance of two methods for imputation of values missing completely at random, the previously top-benchmarked ‘missForest’ and the recently published ‘GSimp’ method. Evaluation was accomplished by comparing imputed with remeasured relative concentrations of 91 inflammation related circulating proteins in 86 samples from a cohort of 645 patients with venous thromboembolism. The median Pearson correlation between imputed and remeasured protein expression values was 69.0% for missForest and 71.6% for GSimp (p = 5.8e-4). Imputation with missForest resulted in stronger reduction of variance compared to GSimp (median relative variance of 25.3% vs. 68.6%, p = 2.4e-16) and undesired larger bias in downstream analyses. Irrespective of the imputation method used, the 91 imputed proteins revealed large variations in imputation accuracy, driven by differences in signal to noise ratio and information overlap between proteins. In summary, GSimp outperformed missForest, while both methods show good overall imputation accuracy with large variations between proteins.

Traffic data is a fundamental component for applications and researches in transportation systems. However, real traffic data collected from loop detectors or other channels often include missing data which affects the relative applications and researches. This paper proposes an approach based on deep learning to impute the missing traffic data. The proposed approach treats the traffic data including observed data and missing data as a whole data item and restores the complete data with the deep structural network. The deep learning approach can discover the correlations contained in the data structure by a layer-wise pre-training and improve the imputation accuracy by conducting a fine-tuning afterwards. The authors analyze the imputation patterns that can be realized with the proposed approach and conduct a series of experiments. The results show that the proposed approach can keep a stable error under different traffic data missing rate. Deep learning is promising in the field of traffic data imputation.

Summary With the constantly increasing quantity of data recorded in the oil and gas industry, data analytics and data-driven algorithms are gaining popularity. Meanwhile, they are highly sensitive to employed data management methods. Using drilling data as an example, the methods for data quality improvement play an ever increasing role in the data preparation phase. One of the most common data issues, especially in real-time sensing systems, is missing data (gaps between measured points). Various data imputation approaches (forward filling, interpolation, regression modeling, etc.) have been used to fill gaps to complete data sets as standard data processing procedures. The metrics, such as coefficient of determination (R2), mean absolute error, root mean square error, and mean absolute percentage error, are common ways to evaluate the imputation approaches, assuming that the ground truth data are at hand. In reality, the ground truth of missing data is not available for quantitative method evaluation. Especially when data are received occasionally, the ground truth is impossibly or inaccurately estimated, leading to tricky situations. For instance, how to evaluate infilling methods quantitatively or how to compare the methods’ behaviors? To the best of the authors’ knowledge, there are so far few existing methods to quantitatively estimate the accuracy of data imputation without the ground truth. To some extent, one may lack proven confidence in using imputed data for following up data analysis and data-driven modeling. In this study, a novel approach has been developed to quantitatively evaluate the data imputation quality. The presented method is built on an analytical estimation of the R2 bounds in the context of linear interpolation. The reasons for choosing linear regression as a benchmarking method are twofold. First, the bounds evaluation of R2 can be determined analytically for data imputation in the context of linear regression. Second, linear regression is a relatively simple imputation method, and we assume that more sophisticated methods, such as random forest (RF), k-nearest neighbors (KNN), and other iterative imputation techniques, can achieve higher accuracy with proper parameter tuning. The evaluation method is demonstrated by examples using real-time drilling data logs. Moreover, the application to drilling logs also enables informed decision-making in relation to gap filling and data resampling workflows.

This review paper examines the current state of lung disease diagnosis based on deep learning (DL) methods. Lung diseases, such as Pneumonia, TB, Covid-19, and lung cancer, are significant causes of morbidity and mortality worldwide. Accurate and timely diagnosis of these diseases is essential for effective treatment and improved patient outcomes. DL methods, which utilize artificial neural networks to extract features from medical images automatically, have shown great promise in improving the accuracy and efficiency of lung disease diagnosis. This review discusses the various DL methods that have been developed for lung disease diagnosis, including convolutional neural networks (CNNs), deep neural networks (DNNs), and generative adversarial networks (GANs). The advantages and limitations of each method are discussed, along with the types of medical imaging techniques used, such as X-ray and computed tomography (CT). In addition, the review discusses the most commonly used performance metrics for evaluating the performance of DL for lung disease diagnosis: the area under the curve (AUC), sensitivity, specificity, F1-score, accuracy, precision, and the receiver operator characteristic curve (ROC). Moreover, the challenges and limitations of using DL for lung disease diagnosis, including the limited availability of annotated data, the variability in imaging techniques and disease presentation, and the interpretability and generalizability of DL models, are highlighted in this paper. Furthermore, strategies to overcome these challenges, such as transfer learning, data augmentation, and explainable AI, are also discussed. The review concludes with a call for further research to address the remaining challenges and realize DL's full potential for improving lung disease diagnosis and treatment.