Data imputation methods for missing U-values of building envelopes in building performance database

In this paper, a data imputation method, which can achieve robust data collection and also extend network lifetime for Wireless Sensor Networks, is proposed. Due to the lossy nature of the wireless medium, it is common for packets to be lost. In order to provide some reliability for the application layer, a data imputation method which predicts the data values contained in missing packets is proposed. The method is able to maximize the use of both temporal and spatial predictors. The proposed imputation method improves accuracy by up to 73% when compared with a previously proposed imputation method. The method can also be used to reduce the number of retransmissions in the network by up to 12% at the cost of a small increase in error, thus increasing the network lifetime of the wireless sensor network.

The article describes a new missing data imputation method. It is based on the use of a high-speed neural-like structure of the Successive Geometric Transformations Model. The importance of the research is based on the analysis of disadvantages of the known methods for missing data processing. Various simple and complex algorithms are analyzed, among which are the arithmetic mean algorithm, regression modeling, etc. It is shown that the above-mentioned imputation methods in data monitoring of air pollution do not allow to obtain reliable results due to the low prediction accuracy. An effective method for processing data imputation through SGTM neural-like structure is proposed. An example of filling data by forecasting CO, NO and NO2 missed parameters in data monitoring of air pollution is given. A comparison of the proposed method with the arithmetic mean algorithm is carried out. Accuracies of the data imputation by developed method and by arithmetic mean algorithm are based on calculated evaluation criteria: Root mean squared errors. Experimentally established that the data imputation method through SGTM neural-like structure has a three times higher accuracy of the data imputation than the arithmetic mean algorithm. The proposed approach can be used in various areas such as medicine, materials science, economics, science services, etc.

BackgroundIn modern biomedical research of complex diseases, a large number of demographic and clinical variables, herein called phenomic data, are often collected and missing values (MVs) are inevitable in the data collection process. Since many downstream statistical and bioinformatics methods require complete data matrix, imputation is a common and practical solution. In high-throughput experiments such as microarray experiments, continuous intensities are measured and many mature missing value imputation methods have been developed and widely applied. Numerous methods for missing data imputation of microarray data have been developed. Large phenomic data, however, contain continuous, nominal, binary and ordinal data types, which void application of most methods. Though several methods have been developed in the past few years, not a single complete guideline is proposed with respect to phenomic missing data imputation.ResultsIn this paper, we investigated existing imputation methods for phenomic data, proposed a self-training selection (STS) scheme to select the best imputation method and provide a practical guideline for general applications. We introduced a novel concept of “imputability measure” (IM) to identify missing values that are fundamentally inadequate to impute. In addition, we also developed four variations of K-nearest-neighbor (KNN) methods and compared with two existing methods, multivariate imputation by chained equations (MICE) and missForest. The four variations are imputation by variables (KNN-V), by subjects (KNN-S), their weighted hybrid (KNN-H) and an adaptively weighted hybrid (KNN-A). We performed simulations and applied different imputation methods and the STS scheme to three lung disease phenomic datasets to evaluate the methods. An R package “phenomeImpute” is made publicly available.ConclusionsSimulations and applications to real datasets showed that MICE often did not perform well; KNN-A, KNN-H and random forest were among the top performers although no method universally performed the best. Imputation of missing values with low imputability measures increased imputation errors greatly and could potentially deteriorate downstream analyses. The STS scheme was accurate in selecting the optimal method by evaluating methods in a second layer of missingness simulation. All source files for the simulation and the real data analyses are available on the author’s publication website.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-014-0346-6) contains supplementary material, which is available to authorized users.

Recently, data-driven decision-making has attracted great interest; this requires high-quality datasets. However, real-world datasets often feature missing values for unknown or intentional reasons, rendering data-driven decision-making inaccurate. If a machine learning model is trained using incomplete datasets with missing values, the inferred results may be biased. In this case, a commonly used technique is the missing value imputation (MVI), which fills missing data with possible values estimated based on observed values. Various data imputation methods using machine learning, statistical inference, and relational database theories have been developed. Among them, conventional machine learning based imputation methods that handle tabular data can deal with only numerical columns or are time-consuming and cumbersome because they create an individualized predictive model for each column. Therefore, we have developed a novel imputational neural network that we term the Denoising Self-Attention Network (DSAN). Our proposed DSAN can deal with tabular datasets containing both numerical and categorical columns; it considers discretized numerical values as categorical values for embedding and self-attention layers. Furthermore, the DSAN learns robust feature expression vectors by combining self-attention and denoising techniques, and can predict multiple, appropriate substituted values simultaneously (via multi-task learning). To verify the validity of the method, we performed data imputation experiments after arbitrarily generating missing values for several real-world tabular datasets. We evaluated both imputational and downstream task performances, and we have seen that the DSAN outperformed the other models, especially in terms of category variable imputation.

Determining cell types by single-cell transcriptomics data is fundamental for downstream analysis. However, cell clustering and data imputation still face the computation challenges, due to the high dropout rate, sparsity and dimensionality of single-cell data. Although some deep learning based solutions have been proposed to handle these challenges, they still can not leverage gene attribute information and cell topology in a sensible way to explore the consistent clustering. In this paper, we present scDeepFC, a deep information fusion-based single-cell data clustering method for cell clustering and data imputation. Specifically, scDeepFC uses a deep auto-encoder (DAE) network and a deep graph convolution network to embed high-dimensional gene attribute information and high-order cell-cell topological information into different low-dimensional representations, and then fuses them to generate a more comprehensive and accurate consensus representation via a deep information fusion network. In addition, scDeepFC integrates the zero-inflated negative binomial (ZINB) into DAE to model the dropout events. By jointly optimizing the ZINB loss and cell graph reconstruction loss, scDeepFC generates a salient embedding representation for clustering cells and imputing missing data. Extensive experiments on real single-cell datasets prove that scDeepFC outperforms other popular single-cell analysis methods. Both the gene attribute and cell topology information can improve the cell clustering.

Missing values are a common problem of time series such as sensor-generated data due to issues of data collection, transmission, and storage. To deal with this problem, various data imputation methods have been proposed. Among these methods, variants of recurrent neural networks (RNNs) have attracted a lot of attention as they provide a natural representation of temporal dynamics. Though such methods have been shown to achieve higher imputation accuracy, they mostly focus on the modeling of temporal dynamics from a short-term perspective while ignoring the long-term features such as trend and seasonal information embedded in time series data. In this paper, we investigate different paradigms of time series imputation methods to capture temporal dynamics from both long-term and short-term perspectives. In particular, we propose a new learning paradigm called MuSDRI which combines the time series decomposition method Seasonal Trend decomposition using Loess (STL) with RNNs. Specifically, STL is used to decompose long sequences of time series into trend, seasonal, and remainder components such that long-term dynamics can be revealed; RNNs are used to capture local patterns from the remainder component extracted by STL such that short-term dynamics can be refined. Moreover, MuSDRI is able to handle time series with multiple seasonal patterns by learning their contributions with respect to the imputation task. We evaluate the proposed imputation methods on three real-world datasets. Experimental results show that the imputation methods considering both the long-term and short-term dynamics achieve higher imputation accuracy, and among them MuSDRI in general achieves the best performance.

Chapter 11 - Treatment of Missing Values

The increasing prevalence of marine pollution during the past few decades motivated recent research to help ease the situation. Typical water quality assessment requires continuous monitoring of water and sediments at remote locations with labour-intensive laboratory tests to determine the degree of pollution. We propose an automated water quality assessment framework where we formalise a predictive model using machine learning to infer the water quality and level of pollution using collected water and sediments samples. Firstly, due to the sparsity of sample collection locations, the amount of sediment samples of water is limited, and the dataset is incomplete. Therefore, after an extensive investigation on various data imputation methods’ performance in water and sediment datasets with different missing data rates, we chose the best imputation method to process the missing data. Afterwards, the water sediment sample will be tagged as one of four levels of pollution based on some guidelines and then the machine learning model will use a specific technique named classification to find the relationship between the data and the final result. After that, the result of prediction can be compared to the real result so that it can be checked whether the model is good and whether the prediction is accurate. Finally, the research gave improvement advice based on the result obtained from the model building part. Empirically, we show that our best model archives an accuracy of 75% after accounting for 57% of missing data. Experimentally, we show that our model would assist in automatically assessing water quality screening based on possibly incomplete real-world data.

Buildings are subject to faults in their heating, ventilation and air-conditioning (HVAC) systems that can lead to excessive energy wastage, poor indoor climate, equipment failures and high maintenance costs. Field studies have shown that employing fault detection, diagnosis and prognosis (FDDP) tools followed up with equipment services and corrections can help achieve up to 40% of energy savings within the HVAC system and improve indoor climate, increase equipment lifecycle and reduce maintenance costs. The increasing adoption of building automation systems (BAS), Internet of Things (IoT) and other smart technologies in recent years have allowed large amounts of data to be continuously collected from building systems. This data-rich environment, along with the surge in data analytics and machine learning tools, has made cost-effective data-driven FDDP strategies possible. Compared to purely physics-based methods, data-driven methods require less explicit knowledge of the underlying physical system, thus are often easier to develop, and can learn certain intricate relationships that exist among data. Within the reported data-driven FDDP methods, there exists a few research gaps: 1) data imputation methods that leverage mutual information from correlated measurements to defy poor data quality from BAS have not been utilized efficiently; 2) there lacks a systematic and scalable fault diagnosis framework that incorporates probabilistic temporal relationships to track fault evolution; 3) existing fault diagnosis strategies typically focus on traditional rule-based control strategies and their scalability for advanced control strategies such as Guideline 36 have not been explored yet; 4) active threats information such as cyber-attacks, are typically not incorporated in an FDDP framework; 5) fault prognosis strategies to preemptively identify gradual faults for predictive maintenance have rarely been studied. This research attempts to address the above-mentioned research gaps through the following: Data Imputation: reported data imputation methods that are suitable for handling and repairing multi-source BAS data are evaluated. Data collected from a medium-sized, mixed-use institution building situated in Stockholm, Sweden and a small commercial building simulated in a laboratory setup is used to evaluate five different data imputation methods. Results demonstrate that incorporating time-lagged cross-correlations within the k-Nearest Neighbor (kNN) method helps to significantly improve the imputation accuracy and minimize the impact of repaired data on data-driven algorithms. Dynamic Bayesian Network (DBN)-based Framework for Cyber-Physical Fault Diagnosis: a DBN framework with discretized conditional probabilities parameters to represent the temporal relationships among building measurements is developed. Both domain knowledge and machine learning methods are used to develop the DBN structure and parameter model. The developed framework is evaluated for both traditional rule-based and Guideline 36 controls using datasets from a real building, a laboratory building, and a virtual testbed. Results show that the developed DBN framework is effective in diagnosing and isolating faults in systems even with different control strategies. The framework also successfully distinguishes whether system abnormalities originate from cyber-attacks or naturally occurring physical faults. Potential future direction to improve fault isolation using modified DBN topological structure is also reported in this study. DBN-based Framework for Fault Prognosis: an extension of the DBN framework in conjunction with Robust Multivariate Temporal (RMT) variate selection is proposed for fault prognosis. The RMT variate selection is used to extract localized temporal features from high dimensional datasets to determine the best inputs for training forecasting models. The expected fault-free behavior of multiple target variates, selected using domain knowledge, is forecasted using incoming data. The prediction errors generated from the forecasting phase are used as evidences in the DBN inference to estimate future fault probabilities. Gradual faults simulated in the virtual testbed are used to evaluate the prognosis framework. Results show that the developed framework is effective in prognosing gradual faults by leveraging the trending growth on the prediction errors. The research presented in this thesis contributes to the overall objective of developing a robust and cost-effective DBN-based framework for fault diagnosis and prognosis of building HVAC systems. Potential solutions to other existing challenges of implementing data-driven FDDP strategies, such as obtaining high-quality datasets, handling and repairing missing data, establishing a baseline model for detecting abnormalities despite other disturbances such as weather and internal conditions changes, and extracting temporal features from timeseries data are also examined.

Real-world datasets often have missing values, which hinders the use of a large number of machine learning (ML) estimators. To overcome this limitation in a data analysis pipeline, data points may be deleted in a data preprocessing stage. However, an alternative better solution is data imputation.Several methods based on Artificial Neural Networks (ANN) have been recently proposed as successful alternatives to classical discriminative imputation methods. Amongst those ANN imputation methods are the ones that rely on Generative Adversarial Networks (GAN).This paper presents three data imputation methods based on GAN: SGAIN, WSGAIN-CP and WSGAIN-GP. These methods were tested on datasets with different settings of missing values probabilities, where the values are missing completely at random (MCAR). The evaluation of the newly developed methods shows that they are equivalent or outperform competitive state-of-the-art imputation methods in different ways, either in terms of response time, the data imputation quality, or the accuracy of post-imputation tasks (e.g., prediction or classification).

Recent advances in spatial transcriptomics allow spatially resolved gene expression measurements with cellular or even sub-cellular resolution, directly characterizing the complex spatiotemporal gene expression landscape and cell-to-cell interactions in their native microenvironments. Due to technology limitations, most spatial transcriptomic technologies still yield incomplete expression measurements with excessive missing values. Therefore, gene imputation is critical to filling in missing data, enhancing resolution, and improving overall interpretability. However, existing methods either require additional matched single-cell RNA-seq data, which is rarely available, or ignore spatial proximity or expression similarity information. To address these issues, we introduce Impeller, a path-based heterogeneous graph learning method for spatial transcriptomic data imputation. Impeller has two unique characteristics distinct from existing approaches. First, it builds a heterogeneous graph with two types of edges representing spatial proximity and expression similarity. Therefore, Impeller can simultaneously model smooth gene expression changes across spatial dimensions and capture similar gene expression signatures of faraway cells from the same type. Moreover, Impeller incorporates both short- and long-range cell-to-cell interactions (e.g. via paracrine and endocrine) by stacking multiple GNN layers. We use a learnable path operator in Impeller to avoid the over-smoothing issue of the traditional Laplacian matrices. Extensive experiments on diverse datasets from three popular platforms and two species demonstrate the superiority of Impeller over various state-of-the-art imputation methods. The code and preprocessed data used in this study are available at https://github.com/aicb-ZhangLabs/Impeller and https://zenodo.org/records/11212604.

The absence of some data values in any observed dataset has been a real hindrance to achieving valid results in statistical research. This paper aimed at the missing data widespread problem faced by analysts and statisticians in academia and professional environments. Some data-driven methods were studied to obtain accurate data. Projects that highly rely on data face this missing data problem. And since machine learning models are only as good as the data used to train them, the missing data problem has a real impact on the solutions developed for real-world problems. Therefore, in this dissertation, there is an attempt to solve this problem using different mechanisms. This is done by testing the effectiveness of both traditional and modern data imputation techniques by determining the loss of statistical power when these different approaches are used to tackle the missing data problem. At the end of this research dissertation, it should be easy to establish which methods are the best when handling the research problem. It is recommended that using Multivariate Imputation by Chained Equations (MICE) for MAR missingness is the best approach to dealing with missing data.

Application of a novel hybrid method for spatiotemporal data imputation: A case study of the Minqin County groundwater level

The quality of traffic data is crucial for modern transportation planning and operations. However, data could be missing for various reasons. Hence, the data imputation approaches which aim at predicting/replacing the missing data or bad data have been considered very important. The traditional traffic data imputation approaches mainly focus on using different probability models or regression methods to impute data, and they only take limited temporal or spatial information as inputs. Thus, they are not very accurate especially for data with a high missing ratio. To overcome the weaknesses of previous approaches, this study proposes an innovative traffic data imputation method, which first transforms the raw data into spatial‐temporal images and then implements a deep‐learning method on the images. The key idea of this approach is developing a convolutional neural network (CNN)‐based context encoder to reconstruct the complete image from the missing source. To the best of the authors’ knowledge, this is the first time a CNN method has been incorporated for traffic data imputation. Experiments are conducted on three months of data from 256 loop detectors. Through comparison with two state‐of‐the‐art approaches, the results indicate that this new approach increases the imputation accuracy greatly and has a stable error distribution.

Given the logistical and financial complications in routine inspection of water pipelines, data-driven forecasting models play a crucial role in prioritizing inspection and maintenance tasks for pipes with a higher probability of failure. These models rely on extensive data sets that include pipe characteristics, external factors, and historical failure records. Due to several reasons, including the age of water networks and inadequate documentation systems, pipeline data sets often contain missing or incomplete records. These gaps may significantly reduce the predictive power and reliability of forecasting models. As a result, robust data imputation methods that estimate missing values and reconstruct data sets are essential. However, the few existing solutions that address this challenge face critical limitations, including low accuracy and an inability to handle heterogeneous data that involve both numerical and categorical variables. To overcome these issues, this study introduces a novel progressive method for multivariate heterogeneous data imputation tailored to water pipeline data sets. The proposed method uses a deep learning–based progressive forecasting approach using a gated recurrent unit (GRU) autoencoder and an extreme gradient boosting (XGBoost) to estimate missing values by capturing complex associations and interdependencies among variables. To evaluate the effectiveness of the proposed method, it was applied to real-world data from Calgary, Canada, to estimate missing values for three key pipe attributes: pipe diameter, pipe material, and pipe installation year. The results demonstrated that the proposed method achieves strong predictive performance and significantly outperforms benchmark methods, such as k-nearest neighbors (KNN), multivariate imputation by chained equations (MICE), and MissForest.