System identification with missing data via nuclear norm regularization

Missing data problem is ubiquitous in many real life situations. Information Technology researchers have explored and tried to address this problem in different settings. In this thesis, we undertake research to address missing data problem associated with order book information in stock markets. This is an in-depth and large-scale study with systematic and comprehensive framework to address missing data problem in the finance literature. Orders placed by traders and the corresponding order imbalance (OIB) is informative to predict future stock returns, however, stock exchange rules do not reveal price sensitive complete order book data for traders. Hence, return prediction using the revealed, incomplete trade book data (that contains only matched buy and sell orders and deletes the unmatched orders), does not let traders to completely exploit possible short term trading opportunities. Hence, this can be considered as a classical missing data problem for predicting future returns, by using the information content of order book. This thesis addresses the missing data problem by developing an integrated theoretical framework applied in stock market trading environment. We use relational Markov networks theory and build an empirically testable Algorithm for Imputed Complete Order Book (AICOB). The thesis contributes by developing a new theoretical advancement of information technology research relating to missing data problem and applying it to financial markets. First, the thesis presents the missing data problem as a Missing at Random (MAR) data and builds a systematic framework to estimate single as well as joint log likelihood functions. The thesis demonstrates that estimating by using incomplete records, improves the accuracy of the parameter estimates. Second, the thesis proposes a Relational Markov Network Model for estimation of the joint distribution function of orders, order characteristics and their interactions. Later, the Expectation Maximization Algorithm is proposed to address the missing data problem during the joint estimation procedure. All pooled regression results follow Fama and MacBeth (1973) and Generalized Methods of Moments (GMM) methods. These methods control the cross sectional and time series correlations between the observations and across the pooled stocks. The proposed novel methodology overcomes the estimation problem in the context of missing order book data. Third, the thesis develops an objective evaluation strategy for AICOB, based on efficiency, accuracy and adaptability dimensions. The thesis uses Australian stock market data, which provides not only trade book data but also historical order book data for cross validating the results. This unique setting allows validating the accuracy of AICOB methodology by comparing with complete order book data. The main contribution of the thesis is to show that AICOB based predictions match with the complete order book data. Whereas, trade book based predictions are quite inconsistent to the complete order book data. The AICOB based results are also consistent with the theoretical predictions proposed in finance literature that OIB predicts better as the firm size increases. The results show that large firms, with higher trading activity and more competition for order flows, report more significant OIB prediction of future returns. Trade Book based OIB estimates, which suffer from missing data problem, fail to predict future returns for stock portfolios. Hence, addressing the missing data problem is important before implementing OIB based trading strategies. Overall, the thesis finds that machine learning applications, similar to AICOB, can be helpful in implementing the trading strategies in financial markets. The imputation of missing data, through systematic procedures, based on theoretical distributional properties of the financial variables, can be informative for more accurate prediction of future returns. The thesis contributes towards establishing a common ground for cross-disciplinary research in Finance and Information Technology (IT) by applying the advances in IT research to solve research and corresponding implementation problems in finance. Further, the thesis contributes towards advancing the order imbalance literature by showing that missing data can play a critical role in the predictive ability of order imbalance.

Researchers are commonly faced with the problem of missing data. This article presents theoretical and empirical information for the selection and application of approaches for handling missing data on a single variable. An actual data set of 492 cases with no missing values was used to create a simulated yet realistic data set with missing at random (MAR) data. The authors compare and contrast five approaches (listwise deletion, mean substitution, simple regression, regression with an error term, and the expectation maximization [EM] algorithm) for dealing with missing data, and compare the effects of each method on descriptive statistics and correlation coefficients for the imputed data (n = 96) and the entire sample (n = 492) when imputed data are inculded. All methods had limitations, although our findings suggest that mean substitution was the least effective and that regression with an error term and the EM algorithm produced estimates closest to those of the original variables.

The absence of some data values in any observed dataset has been a real hindrance to achieving valid results in statistical research. This paper aimed at the missing data widespread problem faced by analysts and statisticians in academia and professional environments. Some data-driven methods were studied to obtain accurate data. Projects that highly rely on data face this missing data problem. And since machine learning models are only as good as the data used to train them, the missing data problem has a real impact on the solutions developed for real-world problems. Therefore, in this dissertation, there is an attempt to solve this problem using different mechanisms. This is done by testing the effectiveness of both traditional and modern data imputation techniques by determining the loss of statistical power when these different approaches are used to tackle the missing data problem. At the end of this research dissertation, it should be easy to establish which methods are the best when handling the research problem. It is recommended that using Multivariate Imputation by Chained Equations (MICE) for MAR missingness is the best approach to dealing with missing data.

Missing data result in less precise and possibly biased effect estimates in single studies. Bias arising from studies with incomplete outcome data is naturally propagated in a meta‐analysis. Conventional analysis using only individuals with available data is adequate when the meta‐analyst can be confident that the data are missing at random (MAR) in every study—that is, that the probability of missing data does not depend on unobserved variables, conditional on observed variables. Usually, such confidence is unjustified as participants may drop out due to lack of improvement or adverse effects. The MAR assumption cannot be tested, and a sensitivity analysis to assess how robust results are to reasonable deviations from the MAR assumption is important. Two methods may be used based on plausible alternative assumptions about the missing data. Firstly, the distribution of reasons for missing data may be used to impute the missing values. Secondly, the analyst may specify the magnitude and uncertainty of possible departures from the missing at random assumption, and these may be used to correct bias and reweight the studies. This is achieved by employing a pattern mixture model and describing how the outcome in the missing participants is related to the outcome in the completers. Ideally, this relationship is informed using expert opinion. The methods are illustrated in two examples with binary and continuous outcomes. We provide recommendations on what trial investigators and systematic reviewers should do to minimize the problem of missing outcome data in meta‐analysis.

The problem of missing data in a database is something that causes frequent difficulties for its processing and analysis. This research presents a new missing data methodology based on multivariate adaptive regression splines (MARS) for missing data imputation. The performance of the proposed method is checked using as input information a database created from the hourly records of environmental stations located in the city of Madrid (Spain). Data analyzed corresponds to hourly measurements from 10th February 2004 to 31st May 2010. The proposed methodology has three variants. The first of these makes use of all the available information in order to calculate different MARS models with the ability to predict missing information based on the available data. In the second case, the MARS models are trained after the removal of 1% of the most extreme cases according to Mahalanobis’ distances, as they are considered outliers. Finally, the third model proposed makes use of the information corresponding only to the previous month in order to calculate the MARS models for the missing data prediction. The results obtained outperformed those given by multivariate imputation by chained equations (MICE) when applied to the same data sets. For a data set with 20% of its information missing, the proposed algorithm outperforms MICE in RMSE values at least in 65.5% of cases, MAE in 75.2% and MAPE in 76%.KeywordsMissing Data ImputationMultivariate Adaptive Regression Splines (MARS)Multiple Imputation by Chained Equations (MICE)Pollutants

Machine learning-based imputation soft computing approach for large missing scale and non-reference data imputation

The problem of missing data is particularly present in archaeological research where, because of the fragmentariness of the finds, only a part of the characteristics of the whole object can be observed. The performance of various dissimilarity indices differently weighting missing values is studied on archaeological data via a simulation. An alternative solution consisting in randomly substituting missing values with character sets is also examined. Gower's dissimilarity coefficient seems to be the least biased one either with 25% missing values and 49%; it has not however a constant behaviour as to the sign of the bias. The simulation experiment has also shown that when average linkage cluster analysis is performed on an incomplete data set either using Gower's index or randomly substituting missing values gives satisfactory results while the modified indices fail to detect the cluster structure

Multiple Imputation

Real-time flood warning systems as part of digital and innovative non-structural solutions have been widely used to prepare decision makers, operators, and affected population to alleviate socio-economic flooding consequences [1]. Many models have been introduced recently to provide more accurate flood forecasts with longer lead times. However, they rely highly on availability of input data which may contain missing values in measurement for one or more timesteps mainly due to wide range of reasons such as random/systematic errors and blunders. Hence, real-time early warning systems cannot be operated properly unless these missing data are properly infilled [2]. Despite data imputation techniques have been mainly employed in pre-processing step of historical data i.e., models training and validation, they have not been properly elaborated in real-time operation practically [3].This paper aims to propose a new event-based data imputation method for infilling rainfall and water level missing data appearing in real-time operation of flood early warning systems. Event identification is first used to divide the real-time data into the wet or dry weather conditions which then are used for selecting the best strategy of infilling missing data. Imputation decision framework takes advantage of various imputation techniques including t-copula, move-median, and kriging based on external available benchmarks and temporal location of missing data. Proposed methodology is tested in real-world case study of urban drainage system in London, UK. Conventional techniques such as linear regression, kriging, nearest neighbourhood, t-copula, inverse distance, and similar calendar are first compared together and best techniques are then tested with proposed methodology in three real-time scenarios as (1) missing rainfall intensity, (2) missing water level, (3) missing both rainfall and water level. Recurrent neural network model used for flood forecasting and results are demonstrated for the next 3hr-ahead predictions.Results show the proposed method can reduce root mean square error (RMSE) from 55% to 13%, 43% to 12%, and 97% to 17% for the above scenarios, respectively. Furthermore, using external benchmark data resources, i.e. other near rainfall/water level stations, shows very efficient when missing data appears at early steps of rainfall events where selected conventional techniques suffer from predicting rainfall pattern. Finally, when both water level and rainfall intensity were missing, the proposed imputation method can reduce RMSE from 197mm to 117mm (RMSE was originally 100 for no missing data) for 3hr-ahead predictions. Generally, this study shows the proposed imputation method can better infill the missing data, especially those in the flood event by using correlated data in other weather/gauging stations and flexibility in applying different methods.

Electronic health records (EHRs) are the records containing the patients' clinic information. The EHRs have been widely used in disease diagnosis and therapy due to the numerous and valuable medical information in them. However, the missing data problem of EHRs hinders the usage. Replacing the missing data with mean values is an approach of data imputation. But, that method weakens the feature importance. In this study, we use the expectation-maximization (EM) algorithm to impute the missing data in EHRs. Some machine learning models, including artificial neural network, logistic regression, support vector machine, and random forests are used to evaluate the effectiveness of data imputation. The experimental results show that the prediction accuracies of cancers by using those models on the EHRs imputed by EM algorithm are higher than those by mean values, which indicates the EM algorithm is able to provide accurate estimations in data imputation of EHRs.

Real-time and accurate prediction about current and future traffic conditions is one of the effective ways to alleviate traffic problems. However, missing data problem is inevitable for various reasons when obtaining real-time traffic flow information. Incomplete traffic information may seriously affect the prediction accuracy. To address this problem, in this paper, we propose a method that predicts the traffic flow in real time under missing data. Considering the spatio-temporal characteristics of traffic flows and the spatial location of road segments, we first evaluate the importance of traffic flows using a spatio-temporal correlation function to analyze the correlations of traffic flows. Then, we present a PPCA-based minimum data imputation optimization (P-MDIO) algorithm to reduce computation time of data imputation. Finally, we utilize the complete traffic data and relevant road segments sequences to predict real-time traffic flows. The experimental data are obtained from the real-time traffic data collected by loop detectors in Taipei, Taiwan. Our experimental results show the performance and validity of the proposed approach, particularly in large-scale prediction.

Background . Properly constructed decision support systems (DSS) for modelling and forecasting behaviour of dynamic systems provide a possibility for taking into consideration uncertainties of probabilistic, statistical and structural types what results in higher quality of developed models and estimated forecasts. Objective . To consider general reasons for loosing (missing) data in statistical data analysis; to provide categorization of missing data into several groups, and identify the reasons for missing measurements; to provide stepwise system methodology for uncertainty analysis and selection of data imputation techniques; to give an insight into some popular missing values imputation techniques regarding their possible applications. Methods . To solve the problems mentioned the following methods have been used: data categorization approach from business or practical point of view that is necessary for discovering the reasons for availability of systemic and/or random missing values; the modern systemic methodology was hired for analysis of uncertainty causes and missing values imputation; the decision tree based imputation procedures; EM algorithm and regression model approach to forecasting missing data using forecasting functions. Results . The main results of the study are in categorization of the missing data into groups; development of systemic methodology for analysis of uncertainty causes and missing values imputation; providing an analysis for possibilities of missing values imputation with decision trees, EM algorithm and regression models; development of multistep forecasting functions on the basis of autoregression models; illustration of application of some selected perspective methods for missing data imputation. Conclusions . We proposed the six steps system methodology for data imputation which stresses that selection of correct method for imputation is tightly connected with the step-by-step analysis of the gaps causes and finding an appropriate technique for their imputation. The results of imputation sometimes are rather far from the existing data and should be smoothed or even broken from the sample due to their incorrectness. For such cases it should be proposed a new probabilistic-regression method which allows define parameters of the probability interval for the regression aiming missing data imputation. A series of computing experiments performed with EM algorithm, forecast regression based imputation technique and some other approaches shows that it is possible to reach high quality results regarding correct processing of data with missing values.

Extreme learning machine (ELM) is a successful machine learning approach for its extremely fast training speed and good generalization performance. The sparse Bayesian ELM (SBELM) approach, which is a variant of ELM, can result in a more accurate and compact model. However, SBELM can not deal with the missing data problem in its standard form. To solve this problem, we design two novel methods, additive models for missing data (AMMD) and self-adjusting neuron state for missing data (SANSMD), by adjusting the calculation of outputs of the hidden layers in SBELM. Experimental results on several data sets from the UCI repository indicate that the proposed modified SBELM methods have significant advantages: high accuracy and good generalization performance compared with several other existing methods. Moreover, the proposed methods enrich ELM with new tools to solve missing data problem for multi-class classification even with up to 50% of the features missing in the input data.

The estimation of statistical parameters for multivariate data leads to waste in the information if the missing values are neglected, which will subsequently lead to inaccurate estimates. Therefore, the incomplete data must be estimated using one of the statistical estimation methods to obtain accurate results and thus obtaining good estimates for the parameters.The aim of this paper is to estimate the missing values for the multivariate skew normal distribution function using the Expectation Conditional Maximization (ECM) algorithm. After estimating the missing values, the parameters are estimated using Maximum Likelihood Estimation (MLE) with the Newton-Raphson algorithm, as well as using the Genetic Algorithm (GA). Using simulation, the Mean Squared Error (MSE) was calculated to find out which method is the best for estimation by comparing the two methods using different sample sizes (400, 600, and 800). The (GA) that is based on the (ECM) algorithm to estimate the missing values proved to be better and more efficient than the (MLE) method in terms of the results.

Trial Quality in Nephrology: How Are We Measuring Up?