Missing value imputation in high-dimensional phenomic data: imputable or not, and how?

The absence of some data values in any observed dataset has been a real hindrance to achieving valid results in statistical research. This paper aimed at the missing data widespread problem faced by analysts and statisticians in academia and professional environments. Some data-driven methods were studied to obtain accurate data. Projects that highly rely on data face this missing data problem. And since machine learning models are only as good as the data used to train them, the missing data problem has a real impact on the solutions developed for real-world problems. Therefore, in this dissertation, there is an attempt to solve this problem using different mechanisms. This is done by testing the effectiveness of both traditional and modern data imputation techniques by determining the loss of statistical power when these different approaches are used to tackle the missing data problem. At the end of this research dissertation, it should be easy to establish which methods are the best when handling the research problem. It is recommended that using Multivariate Imputation by Chained Equations (MICE) for MAR missingness is the best approach to dealing with missing data.

Given the logistical and financial complications in routine inspection of water pipelines, data-driven forecasting models play a crucial role in prioritizing inspection and maintenance tasks for pipes with a higher probability of failure. These models rely on extensive data sets that include pipe characteristics, external factors, and historical failure records. Due to several reasons, including the age of water networks and inadequate documentation systems, pipeline data sets often contain missing or incomplete records. These gaps may significantly reduce the predictive power and reliability of forecasting models. As a result, robust data imputation methods that estimate missing values and reconstruct data sets are essential. However, the few existing solutions that address this challenge face critical limitations, including low accuracy and an inability to handle heterogeneous data that involve both numerical and categorical variables. To overcome these issues, this study introduces a novel progressive method for multivariate heterogeneous data imputation tailored to water pipeline data sets. The proposed method uses a deep learning–based progressive forecasting approach using a gated recurrent unit (GRU) autoencoder and an extreme gradient boosting (XGBoost) to estimate missing values by capturing complex associations and interdependencies among variables. To evaluate the effectiveness of the proposed method, it was applied to real-world data from Calgary, Canada, to estimate missing values for three key pipe attributes: pipe diameter, pipe material, and pipe installation year. The results demonstrated that the proposed method achieves strong predictive performance and significantly outperforms benchmark methods, such as k-nearest neighbors (KNN), multivariate imputation by chained equations (MICE), and MissForest.

The problem of incomplete data and its implications for drawing valid conclusions from statistical analyses is not related to any particular scientific domain, it arises in economics, sociology, education, behavioural sciences or medicine. Almost all standard statistical methods presume that every object has information on every variable to be included in the analysis and the typical approach to missing data is simply to delete them. However, this leads to ineffective and biased analysis results and is not recommended in the literature. The state of the art technique for handling missing data is multiple imputation. In the paper, some selected multiple imputation methods were taken into account. Special attention was paid to using principal components analysis (PCA) as an imputation method. The goal of the study was to assess the quality of PCA‑based imputations as compared to two other multiple imputation techniques: multivariate imputation by chained equations (MICE) and missForest. The comparison was made by artificially simulating different proportions (10–50%) and mechanisms of missing data using 10 complete data sets from the UCI repository of machine learning databases. Then, missing values were imputed with the use of MICE, missForest and the PCA‑based method (MIPCA). The normalised root mean square error (NRMSE) was calculated as a measure of imputation accuracy. On the basis of the conducted analyses, missForest can be recommended as a multiple imputation method providing the lowest rates of imputation errors for all types of missingness. PCA‑based imputation does not perform well in terms of accuracy.

The problem of missing data in a database is something that causes frequent difficulties for its processing and analysis. This research presents a new missing data methodology based on multivariate adaptive regression splines (MARS) for missing data imputation. The performance of the proposed method is checked using as input information a database created from the hourly records of environmental stations located in the city of Madrid (Spain). Data analyzed corresponds to hourly measurements from 10th February 2004 to 31st May 2010. The proposed methodology has three variants. The first of these makes use of all the available information in order to calculate different MARS models with the ability to predict missing information based on the available data. In the second case, the MARS models are trained after the removal of 1% of the most extreme cases according to Mahalanobis’ distances, as they are considered outliers. Finally, the third model proposed makes use of the information corresponding only to the previous month in order to calculate the MARS models for the missing data prediction. The results obtained outperformed those given by multivariate imputation by chained equations (MICE) when applied to the same data sets. For a data set with 20% of its information missing, the proposed algorithm outperforms MICE in RMSE values at least in 65.5% of cases, MAE in 75.2% and MAPE in 76%.KeywordsMissing Data ImputationMultivariate Adaptive Regression Splines (MARS)Multiple Imputation by Chained Equations (MICE)Pollutants

In the field of higher education, predicting students At-Risk of failing is crucial since they could then be recommended for various interventions. These predictions are made on real world datasets that most likely have various missing data. Addressing this missing data could have substantial affects on the eventual At-Risk predictions. In this study we address the missing data problem with recently developed missing data imputation techniques not currently seen in the relevant literature. These techniques include multivariate imputation by chained equations (MICE) and missForest. We found that MICE does not perform better than simple listwise-deletion. However, missForest is shown to substantially improve predictive performance. This is important since it implies that any subsequent machine learning predictions on students At-Risk of failing are potentially much better.

International large-scale assessments (ILSAs) like TIMSS, PIRLS, and PISA use planned missingness completely at random (MCAR) but suffer from non-response bias with missing data not at random (MNAR), especially in home questionnaires where parental omissions may correlate with item content, compromising cross-country comparisons. This study evaluated whether PIRLS 2021’s Home Socioeconomic Status (HSES) could be reliably imputed using Multivariate Imputation by Chained Equations (MICE) based on student and school data. Six methods—Predictive Mean Matching (PMM), Weighted PMM, CART, Random Forests, Bayesian Linear Regression, and Lasso—were evaluated on 5,500 students with 10–60% simulated missingness. Imputation accuracy declined from 36% explained variance (correlation ∼0.6) at 10% missingness to 16% (correlation <0.4) at 60% missingness. PMM performed best but exhibited shrinkage, overestimating low HSES and underestimating high HSES. Results underscore the critical need to address MNAR in ILSAs to ensure equitable and reliable international educational comparisons.

Reconstructing aquifer dynamics with machine learning: Linking irrigation expansion to groundwater decline in a data-scarce hyper-arid region

High-throughput biotechnologies, such as gene expression microarrays or mass-spectrometry-based proteomic assays, suffer from frequent missing values due to various experimental reasons. Since the missing data points can hinder downstream analyses, there exists a wide variety of ways in which to deal with missing values in large-scale data sets. Nowadays, it has become routine to estimate (or impute) the missing values prior to the actual data analysis. After nearly a decade since the publication of the first missing value imputation methods for gene expression microarray data, new imputation approaches are still being developed at an increasing rate. However, what is lagging behind is a systematic and objective evaluation of the strengths and weaknesses of the different approaches when faced with different types of data sets and experimental questions. In this review, the present strategies for missing value imputation and the measures for evaluating their performance are described. The imputation methods are first reviewed in the context of gene expression microarray data, since most of the methods have been developed for estimating gene expression levels; then, we turn to other large-scale data sets that also suffer from the problems posed by missing values, together with pointers to possible imputation approaches in these settings. Along with a description of the basic principles behind the different imputation approaches, the review tries to provide practical guidance for the users of high-throughput technologies on how to choose the imputation tool for their data and questions, and some additional research directions for the developers of imputation methodologies.

LINE-1 methylation mediates the inverse association between body mass index and breast cancer risk: A pilot study in the Lebanese population

BackgroundMissing data are a common problem in prospective studies with a long follow-up, and the volume, pattern and reasons for missing data may be relevant when estimating the cost of illness. We aimed to evaluate the effects of different methods for dealing with missing longitudinal cost data and for costing caregiver time on total societal costs in Alzheimer’s disease (AD).MethodsGERAS is an 18-month observational study of costs associated with AD. Total societal costs included patient health and social care costs, and caregiver health and informal care costs. Missing data were classified as missing completely at random (MCAR), missing at random (MAR) or missing not at random (MNAR). Simulation datasets were generated from baseline data with 10–40 % missing total cost data for each missing data mechanism. Datasets were also simulated to reflect the missing cost data pattern at 18 months using MAR and MNAR assumptions. Naïve and multiple imputation (MI) methods were applied to each dataset and results compared with complete GERAS 18-month cost data. Opportunity and replacement cost approaches were used for caregiver time, which was costed with and without supervision included and with time for working caregivers only being costed.ResultsTotal costs were available for 99.4 % of 1497 patients at baseline. For MCAR datasets, naïve methods performed as well as MI methods. For MAR, MI methods performed better than naïve methods. All imputation approaches were poor for MNAR data. For all approaches, percentage bias increased with missing data volume. For datasets reflecting 18-month patterns, a combination of imputation methods provided more accurate cost estimates (e.g. bias: −1 % vs −6 % for single MI method), although different approaches to costing caregiver time had a greater impact on estimated costs (29–43 % increase over base case estimate).ConclusionsMethods used to impute missing cost data in AD will impact on accuracy of cost estimates although varying approaches to costing informal caregiver time has the greatest impact on total costs. Tailoring imputation methods to the reason for missing data will further our understanding of the best analytical approach for studies involving cost outcomes.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-016-0188-1) contains supplementary material, which is available to authorized users.

(1) Background: Gene-expression data usually contain missing values (MVs). Numerous methods focused on how to estimate MVs have been proposed in the past few years. Recent studies show that those imputation algorithms made little difference in classification. Thus, some scholars believe that how to select the informative genes for downstream classification is more important than how to impute MVs. However, most feature-selection (FS) algorithms need beforehand imputation, and the impact of beforehand MV imputation on downstream FS performance is seldom considered. (2) Method: A modified chi-square test-based FS is introduced for gene-expression data. To deal with the challenge of a small sample size of gene-expression data, a heuristic method called recursive element aggregation is proposed in this study. Our approach can directly handle incomplete data without any imputation methods or missing-data assumptions. The most informative genes can be selected through a threshold. After that, the best-first search strategy is utilized to find optimal feature subsets for classification. (3) Results: We compare our method with several FS algorithms. Evaluation is performed on twelve original incomplete cancer gene-expression datasets. We demonstrate that MV imputation on an incomplete dataset impacts subsequent FS in terms of classification tasks. Through directly conducting FS on incomplete data, our method can avoid potential disturbances on subsequent FS procedures caused by MV imputation. An experiment on small, round blue cell tumor (SRBCT) dataset showed that our method found additional genes besides many common genes with the two compared existing methods.

Recently, data-driven decision-making has attracted great interest; this requires high-quality datasets. However, real-world datasets often feature missing values for unknown or intentional reasons, rendering data-driven decision-making inaccurate. If a machine learning model is trained using incomplete datasets with missing values, the inferred results may be biased. In this case, a commonly used technique is the missing value imputation (MVI), which fills missing data with possible values estimated based on observed values. Various data imputation methods using machine learning, statistical inference, and relational database theories have been developed. Among them, conventional machine learning based imputation methods that handle tabular data can deal with only numerical columns or are time-consuming and cumbersome because they create an individualized predictive model for each column. Therefore, we have developed a novel imputational neural network that we term the Denoising Self-Attention Network (DSAN). Our proposed DSAN can deal with tabular datasets containing both numerical and categorical columns; it considers discretized numerical values as categorical values for embedding and self-attention layers. Furthermore, the DSAN learns robust feature expression vectors by combining self-attention and denoising techniques, and can predict multiple, appropriate substituted values simultaneously (via multi-task learning). To verify the validity of the method, we performed data imputation experiments after arbitrarily generating missing values for several real-world tabular datasets. We evaluated both imputational and downstream task performances, and we have seen that the DSAN outperformed the other models, especially in terms of category variable imputation.

Imputing data is a critical issue for machine learning practitioners, including in the life sciences domain, where missing clinical data is a typical situation and the reliability of the imputation is of great importance. Currently, there is no canonical approach for imputation of clinical data and widely used algorithms introduce variance in the downstream classification. Here we propose novel imputation methods based on determinantal point processes (DPP) that enhance popular techniques such as the multivariate imputation by chained equations and MissForest. Their advantages are twofold: improving the quality of the imputed data demonstrated by increased accuracy of the downstream classification and providing deterministic and reliable imputations that remove the variance from the classification results. We experimentally demonstrate the advantages of our methods by performing extensive imputations on synthetic and real clinical data. We also perform quantum hardware experiments by applying the quantum circuits for DPP sampling since such quantum algorithms provide a computational advantage with respect to classical ones. We demonstrate competitive results with up to 10 qubits for small-scale imputation tasks on a state-of-the-art IBM quantum processor. Our classical and quantum methods improve the effectiveness and robustness of clinical data prediction modeling by providing better and more reliable data imputations. These improvements can add significant value in settings demanding high precision, such as in pharmaceutical drug trials where our approach can provide higher confidence in the predictions made.

Background: Heart failure poses a significant global health challenge, with high rates of readmission and mortality. Accurate models to predict these outcomes are essential for effective patient management. This study investigates the impact of data pre-processing techniques on XGBoost model performance in predicting all-cause readmission and mortality among heart failure patients. Methods: A dataset of 168 features from 2008 heart failure patients was used. Pre-processing included handling missing values, categorical encoding, and standardization. Four imputation techniques were compared: Mean, Multivariate Imputation by Chained Equations (MICEs), k-nearest Neighbors (kNNs), and Random Forest (RF). XGBoost models were evaluated using accuracy, recall, F1-score, and Area Under the Curve (AUC). Robustness was assessed through 10-fold cross-validation. Results: The XGBoost model with kNN imputation, one-hot encoding, and standardization outperformed others, with an accuracy of 0.614, recall of 0.551, and F1-score of 0.476. The MICE-based model achieved the highest AUC (0.647) and mean AUC (0.65 ± 0.04) in cross-validation. All pre-processed models outperformed the default XGBoost model (AUC: 0.60). Conclusions: Data pre-processing, especially MICE with one-hot encoding and standardization, improves XGBoost performance in heart failure prediction. However, moderate AUC scores suggest further steps are needed to enhance predictive accuracy.

This research launches a novel Competency-Quality Temporal-Graph Network (CompQual-TGNet) which serves as a hybrid deep learning (DL) framework to study oil and gas business operations through competency metrics and quality indicators. The system combines Graph Neural Network (GNN) and Temporal Convolutional Network (TCN) technology to analyze multiple operational datasets and make accurate future performance estimates. The preprocessing system builds strong data connections by standardizing values with Z-score normalization while handling missing values via Multivariate Imputation by Chained Equations (MICE), finding outliers through Interquartile Range (IQR) and matching temporal data patterns across different inputs. The system also enhances outputs with advanced feature engineering methods including feature extraction and fusion. The proposed framework uses Multi-View Matrix Factorization (MVMF) to extract shared latent features, and Cross-Dataset Attention Mechanism (CDAM) to identify correlations. Moreover, Dynamic Feature Importance Network (DFIN) to learn optimal feature combinations across datasets, and Cross-Attention Fusion Layer to create context-aware feature representations are employed for fusion. Besides, this development brings together GNN for mapping interdependencies among operational components and personnel competencies and TCN for monitoring operational history and long-term temporal patterns. The adaptive fusion layer uses weighting and attention modules to unite these models to make clear predictions based on domain-relevant predictions. The proposed framework shows improvement through updating graphs during operations, adjusting loss functions to match target requirements, and transferring learned models for constant updating. Using these data sources, the developed CompQual-TGNet predicts operating efficiency and detects quality problems with 94.2% accuracy and outran existing systems.