Missing value imputation using a fuzzy clustering-based EM approach

Preprocessing is one of the steps in Data Mining, which involves Noise removal, Identification of outlier, Normalization, Data transformation, Handling missing values, etc. Missing value is a common problem in large datasets. Most frequently used method to handle missing values by statistical is discarding the instances with missing values. Sometime deletion of instances with missing values cause loss of essential information, which affects the performance of statistical and machine learning algorithms. This paper focuses on handling missing values using unsupervised learning techniques. Rough K-Means based missing value imputation was proposed and compared with K-Means, Fuzzy C-Means based imputation methods. The experimental analysis is carried out on two data sets Lung Cancer and Cleveland Heart data sets. The proposed method achieves the best accuracy for some of the datasets.

A Comparison of Multiple Imputation and Optimal Estimation for Missing and Uncertain Urban Air Toxics Data

Summary With the constantly increasing quantity of data recorded in the oil and gas industry, data analytics and data-driven algorithms are gaining popularity. Meanwhile, they are highly sensitive to employed data management methods. Using drilling data as an example, the methods for data quality improvement play an ever increasing role in the data preparation phase. One of the most common data issues, especially in real-time sensing systems, is missing data (gaps between measured points). Various data imputation approaches (forward filling, interpolation, regression modeling, etc.) have been used to fill gaps to complete data sets as standard data processing procedures. The metrics, such as coefficient of determination (R2), mean absolute error, root mean square error, and mean absolute percentage error, are common ways to evaluate the imputation approaches, assuming that the ground truth data are at hand. In reality, the ground truth of missing data is not available for quantitative method evaluation. Especially when data are received occasionally, the ground truth is impossibly or inaccurately estimated, leading to tricky situations. For instance, how to evaluate infilling methods quantitatively or how to compare the methods’ behaviors? To the best of the authors’ knowledge, there are so far few existing methods to quantitatively estimate the accuracy of data imputation without the ground truth. To some extent, one may lack proven confidence in using imputed data for following up data analysis and data-driven modeling. In this study, a novel approach has been developed to quantitatively evaluate the data imputation quality. The presented method is built on an analytical estimation of the R2 bounds in the context of linear interpolation. The reasons for choosing linear regression as a benchmarking method are twofold. First, the bounds evaluation of R2 can be determined analytically for data imputation in the context of linear regression. Second, linear regression is a relatively simple imputation method, and we assume that more sophisticated methods, such as random forest (RF), k-nearest neighbors (KNN), and other iterative imputation techniques, can achieve higher accuracy with proper parameter tuning. The evaluation method is demonstrated by examples using real-time drilling data logs. Moreover, the application to drilling logs also enables informed decision-making in relation to gap filling and data resampling workflows.

 Electrocardiogram signal analysis is very difficult to classify cardiac arrhythmia using machine learning methods. The ECG datasets normally come with multiple missing values. The reason for the missing values is the faults or distortion. When performing data mining, missing value imputation is the biggest task for data preprocessing. This problem could arise due to incomplete medical datasets if the incomplete missing values and cases were removed from the original database. To produce a good quality dataset for better analyzing the clinical trials, the suitable missing value imputation method is used. In this paper, we explore the different machine-learning techniques for the computed missing value in the electrocardiogram dataset. To estimate the missing imputation values, the collected data contains feature dimensions with their attributes. The experiments to compute the missing values in the dataset are carried out by using the four feature selection methods and imputation methods. The implemented results are shown by combined features using IG (information gain), GA (genetic algorithm) and the different machine learning classifiers such as NB (naïve bayes), KNN (K-nearest neighbor), MLP (Multilayer perception), and RF (Random forest). The GA (genetic algorithm) and IG (information gain) are the best suitable methods for obtaining the results on lower dimensional datasets with RMSE (Root mean square error. It efficiently calculates the best results for missing values. These four classifiers are used to analyze the impact of imputation methods. The best results for missing rate 10% to 40% are obtained by NB that is 0.657, 0.6541, 0.66, 0.657, and 0.657, as computed by RMSE (Root mean Square error). It means that error will efficiently reduced by naïve bayes classifier.

Commodity markets, such as crude oil and precious metals, play a strategic role in the economic development of nations, with crude oil prices influencing geopolitical relations and the global economy. Moreover, gold and silver are argued to hedge the stock and cryptocurrency markets during market downsides. Therefore, accurate forecasting of crude oil and precious metals prices is critical. Nevertheless, due to the nonlinear nature, substantial fluctuations, and irregular cycles of crude oil and precious metals, predicting their prices is a challenging task. Our study contributes to the commodity market price forecasting literature by implementing and comparing advanced deep-learning models. We address this gap by including silver alongside gold in our analysis, offering a more comprehensive understanding of the precious metal markets. This research expands existing knowledge and provides valuable insights into predicting commodity prices. In this study, we implemented 16 deep- and machine-learning models to forecast the daily price of the West Texas Intermediate (WTI), Brent, gold, and silver markets. The employed deep-learning models are long short-term memory (LSTM), BiLSTM, gated recurrent unit (GRU), bidirectional gated recurrent units (BiGRU), T2V-BiLSTM, T2V-BiGRU, convolutional neural networks (CNN), CNN-BiLSTM, CNN-BiGRU, temporal convolutional network (TCN), TCN-BiLSTM, and TCN-BiGRU. We compared the forecasting performance of deep-learning models with the baseline random forest, LightGBM, support vector regression, and k-nearest neighborhood models using mean absolute error (MAE), mean absolute percentage error, and root mean squared error as evaluation criteria. By considering different sliding window lengths, we examine the forecasting performance of our models. Our results reveal that the TCN model outperforms the others for WTI, Brent, and silver, achieving the lowest MAE values of 1.444, 1.295, and 0.346, respectively. The BiGRU model performs best for gold, with an MAE of 15.188 using a 30-day input sequence. Furthermore, LightGBM exhibits comparable performance to TCN and is the best-performing machine-learning model overall. These findings are critical for investors, policymakers, mining companies, and governmental agencies to effectively anticipate market trends, mitigate risk, manage uncertainty, and make timely decisions and strategies regarding crude oil, gold, and silver markets.

The effects of air pollution, amongst the greatest problems facing our planet, are felt throughout all spheres of society, including the transportation sector. Its impacts can range from increased risk of illness to rising temperatures. One of the essential situations for improving inner-city general health and assisting in the creation of a sustainable environment is the ability to forecast air pollution concentrations with accuracy and effectiveness. The backpropagation model is employed in this research to predict the future concentration levels of PM2.5. Data on air quality are gathered and used in the experiment. Empirical Wavelet Transform (EWT) is used to break down the air quality data, which is then utilized to train and evaluate the model. 30% of the data collected is utilized during testing, and 70% was used to train the BP model. The evaluation criteria are applied after testing to determine the model's correctness. The Mean Absolute Error (MAE), Mean Absolute Percent Error (MAPE), and Root Mean Square Error (RMSE) are the evaluation criteria used and their values were 0.0896, 0.8112, and 0.1162, respectively.

Estimation of missing data is essential in the meteorological, climatologically and hydrology analyses. This study employed the arithmetic mean method, normal ratio method, the modified normal ratio method, and correlation coefficient weighting method. The performance of these methods are then compared using correlation coefficient, the S-index, the root mean squared error and mean absolute error methods. The objective of this study is to determine the best estimation method for missing data for four precipitation stations in Makassar city. The results show that the modified usual ratio method is suitable to estimate the missing precipitation data in Makassar city. This study result could be useful information for climate research to complete the missing precipitation data, especially for rain gauge stations in Makassar city.

Globally, wind energy if properly harnessed, could serve as a source of energy generation in Africa. This study compared the performance of two Machine Learning (ML) algorithms (Linear regression and Random Forest) in predicting wind speed in five major cities in Africa (Yaoundé, Pretoria, Nairobi, Cairo and Abuja). Wind data were collected between January 1, 2000, and December 31, 2022, using the Solar Radiation Data Archive. The data preprocessing was carried out with 80% of the data used for training and 20% for validation. The performance of these ML algorithms was evaluated using Mean Square Error (MSE), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE) and coefficient of determination (R2). The result shows that Nairobi (3.814795 m/s) closely followed by Cairo (3.606453 m/s) has the highest mean wind speed while Yaoundé (1.090512 m/s) has the lowest. Based on the performance metrics used, the two Machine Learning algorithms were competitive. Still, the Linear Regression (LR) algorithm outperformed the Random Forest Algorithm in predicting wind speed in all the selected major African cities. In Yaoundé (RMSE = 0.3892, MAE= 0.3001, MAPE =0.5030), Pretoria (RMSE=1.2339, MAE=0.9480, MAPE=0.7450) Nairobi (RMSE= 0.4223, MAE =0.6499, MAPE =0.1872), Nairobi (RMSE=0.6499, MAE=0.5171, MAPE =0.1872), Cairo (RMSE =1.0909, MAE =0.8544, MAPE =0.3541) and Abuja (RMSE = 0.70245, MAE =0.5441, MAPE= 0.4515) the Linear regression algorithms was found to outperformed Random Forest Regression. Therefore, the Linear regression algorithm is more reliable in predicting wind speed compared with the Random Forest regression.

Missing data reconstruction in attitude for quadrotor unmanned aerial vehicle based on deep regression model with different sensor failures

Improving forecasts, particularly the accuracy, efficiency, and precision of time-series forecasts, is becoming critical for authorities to predict, monitor, and prevent the spread of the Coronavirus disease. However, the results obtained from the predictive models are imprecise and inefficient because the dataset contains linear and non-linear patterns, respectively. Linear models such as autoregressive integrated moving average cannot be used effectively to predict complex time series, so nonlinear approaches are better suited for such a purpose. Therefore, to achieve a more accurate and efficient predictive value of COVID-19 that is closer to the true value of COVID-19, a hybrid approach was implemented. Therefore, the objectives of this study are twofold. The first objective is to propose intelligence-based prediction methods to achieve better prediction results called autoregressive integrated moving average-least-squares support vector machine. The second objective is to investigate the performance of these proposed models by comparing them with the autoregressive integrated moving average, support vector machine, least-squares support vector machine, and autoregressive integrated moving average-support vector machine. Our investigation is based on three COVID-19 real datasets, i.e., daily new cases data, daily new death cases data, and daily new recovered cases data. Then, statistical measures such as mean square error, root mean square error, mean absolute error, and mean absolute percentage error were performed to verify that the proposed models are better than the autoregressive integrated moving average, support vector machine model, least-squares support vector machine, and autoregressive integrated moving average-support vector machine. Empirical results using three recent datasets of known the Coronavirus Disease-19 cases in Malaysia show that the proposed model generates the smallest mean square error, root mean square error, mean absolute error, and mean absolute percentage error values for training and testing datasets compared to the autoregressive integrated moving average, support vector machine, least-squares support vector machine, and autoregressive integrated moving average-support vector machine models. This means that the predicted value of the proposed model is closer to the true value. These results demonstrate that the proposed model can generate estimates more accurately and efficiently. Compared to the autoregressive integrated moving average, support vector machine, least-squares support vector machine, and autoregressive integrated moving average-support vector machine models, our proposed models perform much better in terms of percent error reduction for both training and testing all datasets. Therefore, the proposed model is possibly the most efficient and effective way to improve prediction for future pandemic performance with a higher level of accuracy and efficiency.

While remote camera surveys have the potential to improve the accuracy of recreational fishing estimates, missing data are common and require robust analytical techniques to impute. Time-lapse cameras are being used in Western Australia to monitor recreational boating activities, but outages have occurred. Generalized linear mixed effect models formulated in a fully conditional specification multiple imputation framework were used to reconstruct missing data, with climatic and some temporal classifications as covariates. Using a complete 12-month camera record of hourly counts of recreational powerboat retrievals, data were simulated based on ten observed camera outage patterns, with a missing proportion of between 0.06 and 0.61. Nine models were evaluated, including Poisson and negative binomial models, and their associated zero-inflated variants. The imputed values were cross-validated against actual observations using percent bias, mean absolute error, root mean square error, and skill score as performance measures. In 90% of the cases, 95% confidence intervals for the total imputed estimates from at least one of the models contained the total actual counts. With no systematic trends in performance among the models, zero-inflated Poisson and its bootstrapping variant models consistently ranked among the top 3 models and possessed the narrowest confidence intervals. The robustness and generality of the imputation framework were demonstrated using other camera datasets with distinct characteristics. The results provide reliable estimates of the number of boat retrievals for subsequent estimates of fishing effort and provide time series data on boat-based activity.

Background: PIn air quality studies, it is very often to have missing data due to reasons such as machine failure or human error. The approach used in dealing with such missing data can affect the results of the analysis. The main aim of this study was to review the types of missing mechanism, imputation methods, application of some of them in imputation of missing of PM10 and O3 in Tabriz, and compare their efficiency. Methods: Methods of mean, EM algorithm, regression, classification and regression tree, predictive mean matching (PMM), interpolation, moving average, and K-nearest neighbor (KNN) were used. PMM was investigated by considering the spatial and temporal dependencies in the model. Missing data were randomly simulated with 10, 20, and 30% missing values. The efficiency of methods was compared using coefficient of determination (R2 ), mean absolute error (MAE) and root mean square error (RMSE). Results: Based on the results for all indicators, interpolation, moving average, and KNN had the best performance, respectively. PMM did not perform well with and without spatio-temporal information. Conclusion: Given that the nature of pollution data always depends on next and previous information, methods that their computational nature is based on before and after information indicated better performance than others, so in the case of pollutant data, it is recommended to use these methods.

Rainfall data can be regarded as the most essential input for various applications in hydrological sciences. Continuous rainfall data with adequate length is the main requirement to solve complex hydrological problems. Mostly in developing countries hydrologists are still facing problems of missing rainfall data with inadequate length. Researchers have been applying a number of statistical and data driven approaches to overcome this insufficiency. This study is an application of neuro-fuzzy system to infill the missing rainfall data for Klang River catchment. Pettitt test, standard normal homogeneity test (SNHT) and Von Neumann Ratio (VNR) tests were performed to check the homogeneity of rainfall data. The neuro-fuzzy model performances were assessed both in calibration and validation stages based on statistical measures such as coefficient of determination (R2), Root Mean Square Error (RMSE) and Mean Absolute Error (MAE). To evaluate the performance of the neuro-fuzzy system model, it was compared with a traditional modeling technique known as autoregressive model with exogenous inputs (ARX). The neuro-fuzzy system model gave better performances in both stages for the best input combinations. The missing rainfall data was predicted using the input combination with best performances. The results of this study showed the effectiveness of the neuro-fuzzy systems and it is recommended as a prominent tool for filling the missing data.

Gradient Boosted Tree Approaches for Mapping European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 Onto 5-Level Version of EQ-5D Index for Patients With Cancer

This paper proposes a new technique for simultaneous estimation of missing values using Weighted kNN approach whose weight is computed by the product of Grey relational grade and weight vector of SVM. A mixture kernel is used which is a hybridization of Polynomial and Radial Basis Kernel. Applying this proposed technique to fisher iris dataset, we find that the mixture kernel gives better result. Mean Absolute Error, Mean Absolute Percentage Error and Root Mean Square Error are used as the evaluation criteria.