Advancing Software Project Effort Estimation: Leveraging a NIVIM for Enhanced Preprocessing

Imputing data is a critical issue for machine learning practitioners, including in the life sciences domain, where missing clinical data is a typical situation and the reliability of the imputation is of great importance. Currently, there is no canonical approach for imputation of clinical data and widely used algorithms introduce variance in the downstream classification. Here we propose novel imputation methods based on determinantal point processes (DPP) that enhance popular techniques such as the multivariate imputation by chained equations and MissForest. Their advantages are twofold: improving the quality of the imputed data demonstrated by increased accuracy of the downstream classification and providing deterministic and reliable imputations that remove the variance from the classification results. We experimentally demonstrate the advantages of our methods by performing extensive imputations on synthetic and real clinical data. We also perform quantum hardware experiments by applying the quantum circuits for DPP sampling since such quantum algorithms provide a computational advantage with respect to classical ones. We demonstrate competitive results with up to 10 qubits for small-scale imputation tasks on a state-of-the-art IBM quantum processor. Our classical and quantum methods improve the effectiveness and robustness of clinical data prediction modeling by providing better and more reliable data imputations. These improvements can add significant value in settings demanding high precision, such as in pharmaceutical drug trials where our approach can provide higher confidence in the predictions made.

Given the logistical and financial complications in routine inspection of water pipelines, data-driven forecasting models play a crucial role in prioritizing inspection and maintenance tasks for pipes with a higher probability of failure. These models rely on extensive data sets that include pipe characteristics, external factors, and historical failure records. Due to several reasons, including the age of water networks and inadequate documentation systems, pipeline data sets often contain missing or incomplete records. These gaps may significantly reduce the predictive power and reliability of forecasting models. As a result, robust data imputation methods that estimate missing values and reconstruct data sets are essential. However, the few existing solutions that address this challenge face critical limitations, including low accuracy and an inability to handle heterogeneous data that involve both numerical and categorical variables. To overcome these issues, this study introduces a novel progressive method for multivariate heterogeneous data imputation tailored to water pipeline data sets. The proposed method uses a deep learning–based progressive forecasting approach using a gated recurrent unit (GRU) autoencoder and an extreme gradient boosting (XGBoost) to estimate missing values by capturing complex associations and interdependencies among variables. To evaluate the effectiveness of the proposed method, it was applied to real-world data from Calgary, Canada, to estimate missing values for three key pipe attributes: pipe diameter, pipe material, and pipe installation year. The results demonstrated that the proposed method achieves strong predictive performance and significantly outperforms benchmark methods, such as k-nearest neighbors (KNN), multivariate imputation by chained equations (MICE), and MissForest.

International large-scale assessments (ILSAs) like TIMSS, PIRLS, and PISA use planned missingness completely at random (MCAR) but suffer from non-response bias with missing data not at random (MNAR), especially in home questionnaires where parental omissions may correlate with item content, compromising cross-country comparisons. This study evaluated whether PIRLS 2021’s Home Socioeconomic Status (HSES) could be reliably imputed using Multivariate Imputation by Chained Equations (MICE) based on student and school data. Six methods—Predictive Mean Matching (PMM), Weighted PMM, CART, Random Forests, Bayesian Linear Regression, and Lasso—were evaluated on 5,500 students with 10–60% simulated missingness. Imputation accuracy declined from 36% explained variance (correlation ∼0.6) at 10% missingness to 16% (correlation <0.4) at 60% missingness. PMM performed best but exhibited shrinkage, overestimating low HSES and underestimating high HSES. Results underscore the critical need to address MNAR in ILSAs to ensure equitable and reliable international educational comparisons.

The problem of missing data in a database is something that causes frequent difficulties for its processing and analysis. This research presents a new missing data methodology based on multivariate adaptive regression splines (MARS) for missing data imputation. The performance of the proposed method is checked using as input information a database created from the hourly records of environmental stations located in the city of Madrid (Spain). Data analyzed corresponds to hourly measurements from 10th February 2004 to 31st May 2010. The proposed methodology has three variants. The first of these makes use of all the available information in order to calculate different MARS models with the ability to predict missing information based on the available data. In the second case, the MARS models are trained after the removal of 1% of the most extreme cases according to Mahalanobis’ distances, as they are considered outliers. Finally, the third model proposed makes use of the information corresponding only to the previous month in order to calculate the MARS models for the missing data prediction. The results obtained outperformed those given by multivariate imputation by chained equations (MICE) when applied to the same data sets. For a data set with 20% of its information missing, the proposed algorithm outperforms MICE in RMSE values at least in 65.5% of cases, MAE in 75.2% and MAPE in 76%.KeywordsMissing Data ImputationMultivariate Adaptive Regression Splines (MARS)Multiple Imputation by Chained Equations (MICE)Pollutants

Reconstructing aquifer dynamics with machine learning: Linking irrigation expansion to groundwater decline in a data-scarce hyper-arid region

BackgroundIn modern biomedical research of complex diseases, a large number of demographic and clinical variables, herein called phenomic data, are often collected and missing values (MVs) are inevitable in the data collection process. Since many downstream statistical and bioinformatics methods require complete data matrix, imputation is a common and practical solution. In high-throughput experiments such as microarray experiments, continuous intensities are measured and many mature missing value imputation methods have been developed and widely applied. Numerous methods for missing data imputation of microarray data have been developed. Large phenomic data, however, contain continuous, nominal, binary and ordinal data types, which void application of most methods. Though several methods have been developed in the past few years, not a single complete guideline is proposed with respect to phenomic missing data imputation.ResultsIn this paper, we investigated existing imputation methods for phenomic data, proposed a self-training selection (STS) scheme to select the best imputation method and provide a practical guideline for general applications. We introduced a novel concept of “imputability measure” (IM) to identify missing values that are fundamentally inadequate to impute. In addition, we also developed four variations of K-nearest-neighbor (KNN) methods and compared with two existing methods, multivariate imputation by chained equations (MICE) and missForest. The four variations are imputation by variables (KNN-V), by subjects (KNN-S), their weighted hybrid (KNN-H) and an adaptively weighted hybrid (KNN-A). We performed simulations and applied different imputation methods and the STS scheme to three lung disease phenomic datasets to evaluate the methods. An R package “phenomeImpute” is made publicly available.ConclusionsSimulations and applications to real datasets showed that MICE often did not perform well; KNN-A, KNN-H and random forest were among the top performers although no method universally performed the best. Imputation of missing values with low imputability measures increased imputation errors greatly and could potentially deteriorate downstream analyses. The STS scheme was accurate in selecting the optimal method by evaluating methods in a second layer of missingness simulation. All source files for the simulation and the real data analyses are available on the author’s publication website.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-014-0346-6) contains supplementary material, which is available to authorized users.

Survey data play a crucial role in various research fields, including economics, education, and healthcare, by providing insights into human behavior and opinions. However, item non-response, where respondents fail to answer specific questions, presents a significant challenge by creating incomplete datasets that undermine data integrity and can hinder or even prevent accurate analysis. Traditional methods for addressing missing data, such as statistical imputation techniques and deep learning models, often fall short when dealing with the rich linguistic content of survey data. These approaches are also hampered by high time complexity for training and the need for extensive preprocessing or feature selection. In this paper, we introduce an approach that leverages Large Language Models (LLMs) through prompt engineering for predicting item non-responses in survey data. Our method combines the strengths of both traditional imputation techniques and deep learning methods with the advanced linguistic understanding of LLMs. By integrating respondent similarities, question relevance, and linguistic semantics, our approach enhances the accuracy and comprehensiveness of survey data analysis. The proposed method bypasses the need for complex preprocessing and additional training, making it adaptable, scalable, and capable of generating explainable predictions in natural language. We evaluated the effectiveness of our LLM-based approach through a series of experiments, demonstrating its competitive performance against established methods such as Multivariate Imputation by Chained Equations (MICE), MissForest, and deep learning models like TabTransformer. The results show that our approach not only matches but, in some cases, exceeds the performance of these methods while significantly reducing the time required for data processing.

Accurate quality prediction can find and eliminate quality hazards. It is difficult to construct an accurate quality mathematical model for the production of small samples with high dimensionality due to the influence of quality characteristics and the complex mechanism of action. In addition, overfitting scenarios are prone to occur in high-dimensional, small-sample industrial product quality prediction. This paper proposes an ensemble learning and measurement model based on stacking and selects eight algorithms as the base learning model. The maximal information coefficient (MIC) is used to obtain the correlation between the base learning models. Models with low correlation and strong predictive power were chosen to build stacking ensemble models, which effectively avoids overfitting and obtains better predictive performance. To improve the prediction performance as the optimization goal, in the data preprocessing stage, boxplots, ordinary least squares (OLS), and multivariate imputation by chained equations (MICE) are used to detect and replace outliers. The CatBoost algorithm is used to construct combined features. Strong combination features were selected to construct a new feature set. Concrete slump data from the University of California Irvine (UCI) machine learning library were used to conduct comprehensive verification experiments. The experimental results show that, compared with the optimal single model, the minimum correlation stacking ensemble learning model has higher precision and stronger robustness, and a new method is provided to guarantee the accuracy of final product quality prediction.

The problem of incomplete data and its implications for drawing valid conclusions from statistical analyses is not related to any particular scientific domain, it arises in economics, sociology, education, behavioural sciences or medicine. Almost all standard statistical methods presume that every object has information on every variable to be included in the analysis and the typical approach to missing data is simply to delete them. However, this leads to ineffective and biased analysis results and is not recommended in the literature. The state of the art technique for handling missing data is multiple imputation. In the paper, some selected multiple imputation methods were taken into account. Special attention was paid to using principal components analysis (PCA) as an imputation method. The goal of the study was to assess the quality of PCA‑based imputations as compared to two other multiple imputation techniques: multivariate imputation by chained equations (MICE) and missForest. The comparison was made by artificially simulating different proportions (10–50%) and mechanisms of missing data using 10 complete data sets from the UCI repository of machine learning databases. Then, missing values were imputed with the use of MICE, missForest and the PCA‑based method (MIPCA). The normalised root mean square error (NRMSE) was calculated as a measure of imputation accuracy. On the basis of the conducted analyses, missForest can be recommended as a multiple imputation method providing the lowest rates of imputation errors for all types of missingness. PCA‑based imputation does not perform well in terms of accuracy.

LINE-1 methylation mediates the inverse association between body mass index and breast cancer risk: A pilot study in the Lebanese population

The absence of some data values in any observed dataset has been a real hindrance to achieving valid results in statistical research. This paper aimed at the missing data widespread problem faced by analysts and statisticians in academia and professional environments. Some data-driven methods were studied to obtain accurate data. Projects that highly rely on data face this missing data problem. And since machine learning models are only as good as the data used to train them, the missing data problem has a real impact on the solutions developed for real-world problems. Therefore, in this dissertation, there is an attempt to solve this problem using different mechanisms. This is done by testing the effectiveness of both traditional and modern data imputation techniques by determining the loss of statistical power when these different approaches are used to tackle the missing data problem. At the end of this research dissertation, it should be easy to establish which methods are the best when handling the research problem. It is recommended that using Multivariate Imputation by Chained Equations (MICE) for MAR missingness is the best approach to dealing with missing data.

Background and ObjectivesSovateltide (Tycamzzi™), an endothelin-B (ET-B) receptor agonist, increases cerebral blood flow, has anti-apoptotic activity, and promotes neural repair following cerebral ischaemic stroke. The objectives of this study were to evaluate the efficacy and safety of sovateltide in adult participants with acute cerebral ischaemic stroke. MethodsThis was a randomised, double-blind, placebo-controlled, multicentre, Phase III clinical trial of sovateltide in participants with cerebral ischaemic stroke receiving standard of care (SOC) in India. Patients aged 18–78 years presenting up to 24 h after the onset of symptoms with radiologic confirmation of ischaemic stroke and a National Institutes of Health Stroke Scale score (NIHSS) of ≥ 6 were enrolled. Patients with recurrent stroke, receiving endovascular therapy, or with intracranial haemorrhage were excluded. The study drug (saline or sovateltide [0.3 µg/kg] was administered intravenously in three doses at 3 ± 1 h intervals on Days 1, 3, and 6, and follow-up was 90 days). The Multivariate Imputation by Chained Equations (MICE) was used to impute the missing assessments on the endpoints. An unpaired t-test, two-way analysis of variance with Tukey's multiple comparison test, and the Chi-square test were used for the statistical analysis. The objective was to determine at Day 90 (1) the number of patients with a modified Rankin Scale score (mRS) 0–2, and (2) the number of patients with an NIHSS 0–5 at 90 days.ResultsPatients were randomised with 80 patients in the sovateltide and 78 in the control group. Patients received the investigational drug at about 18 h of stroke onset in both control and sovateltide groups. The median NIHSS at randomisation was 10.00 (95% CI 9.99–11.65) in the control group and 9.00 (95% CI 9.11–10.46) in the sovateltide group. Seventy patients completed the 90-day follow-up in the control group and 67 in the sovateltide group. The proportion of intention-to-treat (ITT) patients with mRS 0–2 score at Day 90 post-randomisation was 22.67% higher (odds ratio [OR] 2.75, 95% CI 1.37–5.57); similarly, the proportion of patients with NIHSS score of 0–5 at Day 90 was 17.05% more (OR 2.67, 95% CI 1.27–5.90) in the sovateltide group than in the control group. An improvement of ≥ 2 points on the mRS was observed in 51.28% and 72.50% of patients in the control and sovateltide groups, respectively (OR 2.50, 95% CI 1.29–4.81). Seven of 78 patients (8.97%) in the control group and 7 of 80 (8.75%) in the sovateltide group developed intracranial haemorrhage (ICH). The adverse events were not related to sovateltide.ConclusionsThe sovateltide group had a greater number of cerebral ischaemic stroke patients with lower mRS and NIHSS scores at 90 days post-treatment than the control group. This trial supported the regulatory approval of sovateltide in India, but a multinational RESPECT-ETB trial will be conducted for US approval.Trial RegistrationClinical Trials Registry, India (CTRI/2019/09/021373) and the United States National Library of Medicine, ClinicalTrials.gov (NCT04047563).Supplementary InformationThe online version contains supplementary material available at 10.1007/s40265-024-02121-5.

Background: Heart failure poses a significant global health challenge, with high rates of readmission and mortality. Accurate models to predict these outcomes are essential for effective patient management. This study investigates the impact of data pre-processing techniques on XGBoost model performance in predicting all-cause readmission and mortality among heart failure patients. Methods: A dataset of 168 features from 2008 heart failure patients was used. Pre-processing included handling missing values, categorical encoding, and standardization. Four imputation techniques were compared: Mean, Multivariate Imputation by Chained Equations (MICEs), k-nearest Neighbors (kNNs), and Random Forest (RF). XGBoost models were evaluated using accuracy, recall, F1-score, and Area Under the Curve (AUC). Robustness was assessed through 10-fold cross-validation. Results: The XGBoost model with kNN imputation, one-hot encoding, and standardization outperformed others, with an accuracy of 0.614, recall of 0.551, and F1-score of 0.476. The MICE-based model achieved the highest AUC (0.647) and mean AUC (0.65 ± 0.04) in cross-validation. All pre-processed models outperformed the default XGBoost model (AUC: 0.60). Conclusions: Data pre-processing, especially MICE with one-hot encoding and standardization, improves XGBoost performance in heart failure prediction. However, moderate AUC scores suggest further steps are needed to enhance predictive accuracy.

This research launches a novel Competency-Quality Temporal-Graph Network (CompQual-TGNet) which serves as a hybrid deep learning (DL) framework to study oil and gas business operations through competency metrics and quality indicators. The system combines Graph Neural Network (GNN) and Temporal Convolutional Network (TCN) technology to analyze multiple operational datasets and make accurate future performance estimates. The preprocessing system builds strong data connections by standardizing values with Z-score normalization while handling missing values via Multivariate Imputation by Chained Equations (MICE), finding outliers through Interquartile Range (IQR) and matching temporal data patterns across different inputs. The system also enhances outputs with advanced feature engineering methods including feature extraction and fusion. The proposed framework uses Multi-View Matrix Factorization (MVMF) to extract shared latent features, and Cross-Dataset Attention Mechanism (CDAM) to identify correlations. Moreover, Dynamic Feature Importance Network (DFIN) to learn optimal feature combinations across datasets, and Cross-Attention Fusion Layer to create context-aware feature representations are employed for fusion. Besides, this development brings together GNN for mapping interdependencies among operational components and personnel competencies and TCN for monitoring operational history and long-term temporal patterns. The adaptive fusion layer uses weighting and attention modules to unite these models to make clear predictions based on domain-relevant predictions. The proposed framework shows improvement through updating graphs during operations, adjusting loss functions to match target requirements, and transferring learned models for constant updating. Using these data sources, the developed CompQual-TGNet predicts operating efficiency and detects quality problems with 94.2% accuracy and outran existing systems.

The data in real-world scenario often consists of missing values which leads to difficulty in analysis. Though there has been an emergence of various algorithms handling the issue, it is, in fact, troublesome to implement from industry perspective. Based on this issue, the ‘R’-software with various available packages for missing data handling can be a fruitful solution, which is hardly reported in any previous study. Hence, the availability of such packages demands analysis to compare their performances and check their suitability for a given dataset. A comparative study is performed using the ‘R’-packages, namely missForest, Multivariate imputation by chained equations (MICE), and AMELIA-II. Two classifiers, support vector machine (SVM) and logistic regression (LR), are used for prediction. The packages are compared with regard to imputation time, effects on variance, and the efficiency. The experimental results reveal that the performances depend on the dataset size and the percentage of missing values in data.