Selective Imputation for Multivariate Time Series Datasets With Missing Values

This work presents a novel approach to multivariate time series classification. The method exploits the multivariate structure of the time series and the possibilities of the stacking ensemble method. The basics of the method may be described in three steps: first, decomposing the multivariate time series on its constituent univariate time series; second, inducing a classifier for each univariate time series plus and additional multivariate classifier for the whole time series; third, creating the final multivariate time series classifier stacking the previous classifiers. The ensemble obtained has the potential to improve the accuracy of the single multivariate time series classifier. Several configurations of the stacking method have been tested on seven multivariate time series data sets. In five out of seven data sets, the proposed method obtains the smallest error rate. Moreover, in two out of seven data sets, stacking only the univariate time series classifiers provides the best results. The experimental results show that when a multivariate time series method does not produce an accurate classifier, stacking it with univariate time series classifiers is an alternative worthy of consideration.

Local similarity analysis of biological time series data helps elucidate the varying dynamics of biological systems. However, its applications to large scale high-throughput data are limited by slow permutation procedures for statistical significance evaluation. We developed a theoretical approach to approximate the statistical significance of local similarity analysis based on the approximate tail distribution of the maximum partial sum of independent identically distributed (i.i.d.) random variables. Simulations show that the derived formula approximates the tail distribution reasonably well (starting at time points > 10 with no delay and > 20 with delay) and provides P-values comparable with those from permutations. The new approach enables efficient calculation of statistical significance for pairwise local similarity analysis, making possible all-to-all local association studies otherwise prohibitive. As a demonstration, local similarity analysis of human microbiome time series shows that core operational taxonomic units (OTUs) are highly synergetic and some of the associations are body-site specific across samples. The new approach is implemented in our eLSA package, which now provides pipelines for faster local similarity analysis of time series data. The tool is freely available from eLSA's website: http://meta.usc.edu/softs/lsa. Supplementary data are available at Bioinformatics online. fsun@usc.edu.

Multivariate time series (MTS) arise due to multisensor data collection in manufacturing. These data are complex in the sense that attributes have a varying scale, volitivity, continuity, and so on, and interattribute dependence also appears, which can mask the inherent information about system health status. Conventional machine learning-based process monitoring techniques are inefficient in predicting faults with MTS—their detection capability heavily relies on the input features, yet the classification power of MTS attributes is weakened by the complexity of multisensor data. Effective feature extraction is, therefore, necessary to facilitate fault prediction with MTS. This study proposes a fault prediction framework for MTS based on bag-of-words (BOW) feature extraction, statistical feature selection, and classification analysis. BOW models are for the first time adopted in a manufacturing context. Their superior capability in information preservation, local pattern recognition, and temporal effect accommodation has overcome the major limitations in current manufacturing practices with MTS. A comprehensive case study demonstrates the desirable performance of this framework on two MTS data sets from paper manufacturing and automotive manufacturing, as well as its superiority over conventional machine learning-based fault prediction. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Process monitoring in a multisensor environment has been a vital interest in manufacturing. The difficulty lies in the lack of detection power in many conventional techniques, e.g., control chart and logistic regression. A critical reason for such failure is the neglect of time effect in MTS—patterns associated with system faults tend to stretch a period, but a conventional control chart or classifier inspects each time stamp separately. Fault detection based on time series sequences is, therefore, essential. However, how to effectively extract features from MTS becomes a challenge. The framework proposed in this study adopts BOW models, specifically symbolic aggregate approximation (SAX), to extract features from MTS sequences, thus substantially improves the detection power against local patterns over time. Many manufacturing multisensor data are subject to such local patterns that point to the root cause of fault, so the proposed framework has a wide application in manufacturing.

Background . Properly constructed decision support systems (DSS) for modelling and forecasting behaviour of dynamic systems provide a possibility for taking into consideration uncertainties of probabilistic, statistical and structural types what results in higher quality of developed models and estimated forecasts. Objective . To consider general reasons for loosing (missing) data in statistical data analysis; to provide categorization of missing data into several groups, and identify the reasons for missing measurements; to provide stepwise system methodology for uncertainty analysis and selection of data imputation techniques; to give an insight into some popular missing values imputation techniques regarding their possible applications. Methods . To solve the problems mentioned the following methods have been used: data categorization approach from business or practical point of view that is necessary for discovering the reasons for availability of systemic and/or random missing values; the modern systemic methodology was hired for analysis of uncertainty causes and missing values imputation; the decision tree based imputation procedures; EM algorithm and regression model approach to forecasting missing data using forecasting functions. Results . The main results of the study are in categorization of the missing data into groups; development of systemic methodology for analysis of uncertainty causes and missing values imputation; providing an analysis for possibilities of missing values imputation with decision trees, EM algorithm and regression models; development of multistep forecasting functions on the basis of autoregression models; illustration of application of some selected perspective methods for missing data imputation. Conclusions . We proposed the six steps system methodology for data imputation which stresses that selection of correct method for imputation is tightly connected with the step-by-step analysis of the gaps causes and finding an appropriate technique for their imputation. The results of imputation sometimes are rather far from the existing data and should be smoothed or even broken from the sample due to their incorrectness. For such cases it should be proposed a new probabilistic-regression method which allows define parameters of the probability interval for the regression aiming missing data imputation. A series of computing experiments performed with EM algorithm, forecast regression based imputation technique and some other approaches shows that it is possible to reach high quality results regarding correct processing of data with missing values.

Manufacturers are struggling to use data from multiple products production lines to predict rare events. Improving the quality of training data is a common way to improve the performance of algorithms. However, there is little research about how to select training data quantitatively. In this study, a training data selection method is proposed to improve the performance of deep learning models. The proposed method can represent different time length multivariate time series spilt by categorical variables and measure the (dis)similarities by the distance matrix and clustering method. The contributions are: (1) The proposed method can find the changes to the training data caused by categorical variables in a multivariate time series dataset; (2) according to the proposed method, the multivariate time series data from the production line can be clustered into many small training datasets; and (3) same structure but different parameters prediction models are built instead of one model which is different from the traditional way. In practice, the proposed method is applied in a real multiple products production line dataset and the result shows it can not only significantly improve the performance of the reconstruction model but it can also quantitively measure the (dis)similarities of the production behaviors.

Evaluating the distribution patterns of surface temperature data at Very Long Baseline Interferometry (VLBI)/Global Positioning System (GPS) co-located sites w.r.t. normality is one of the most important issues in modeling surface temperature data over long periods. Such evaluation can generate algorithms for filling in missing data at measurement sites. Some algorithms in the literature, such as those in the study of Cho et al. J Coast Res 65. doi: 10. 2112/SI65-321. 1, (2013), require trend, harmonic, and residual components to fill in the missing data. Trend and harmonic components estimate an optimal model that can be used to assist such algorithms when filling in missing data. The present study is based on the investigation of the normal distribution of the residuals of a surface temperature time series at VLBI/GPS co-located sites, after removing the trend and seasonal effects through harmonic components (inter-daily variations). This study uses surface temperature data collected from the VLBI/GPS co-located sites of two different regions in Europe: Matera (Italy) and Wettzell (Germany). The data collected from these sites form a time series, and time series analyses and conventional k-sigma outlier detection are implemented on these data sets before subjecting them to goodness of fit tests for normality. The residual components of the time series are acquired through a decomposing trend and signal effect from the original time series, assuming that the residuals of the time series are normally distributed. In testing the hypothesis that an observed frequency distribution fits the normal distribution, the following tests are used: Pearson χ2, Kolmogorov-Smirnov, Anderson-Darling, Shapiro-Wilk or Shapiro-Francia, D’Agostino, Jarque-Bera, skewness, and kurtosis tests. Some graphical methods are also applied to support the results of the goodness of fit tests for normality. Some proposals on the application of the goodness of fit tests are put forward, such as the evaluation of the estimation model for trend and harmonic components by considering the properties of the implemented goodness of fit tests. The results of this study can be used to determine the optimal model for estimating trend and harmonic components. The output of the present study is expected to have an important role in modeling surface temperature distributions at co-located VLBI/GPS sites for filling in missing data. Above all, meteorological data, such as temperature, pressure, and humidity, are of specific interest for modeling tropospheric delay, the main error factor in positioning in space geodesy, which in turn makes investigations on the distribution of meteorological data more attractive in geoscience.

Rivers act as the primary harmonizer of the critical zone, reflecting the confluence of processes that regulate nutrient cycling, water quality, and energy flow within ecosystems. Understanding water quality dynamics and their relationship with hydrological and biogeochemical processes warrants the study of river chemistry at a detailed temporal resolution, but this is often hindered by logistical challenges in the acquisition of such data. Recent efforts in automated data acquisition, such as the RiverLab “lab-on-a-field” setup (Floury et al. 2017) have equipped us with a toolbox to peek into riverine systems at unprecedentedly fine temporal scales. This assemblage consists of a primary circuit that continuously samples water from the middle of a stream, and measures various physico-chemical parameters using probes. A portion of this water is also routed through a filtration system towards two ion chromatographs housed in the setup, allowing the measurement of dissolved ions. This system thus enables sub-hourly measurement of major ions and near-continuous monitoring of pH, Temperature, Conductivity, etc. However, despite these advances in data acquisition, technical challenges remain. The datasets often contain gaps due to practical limitations like sensor drift and malfunction, power outages, etc. These induce gaps in the multivariate time series datasets, and can impede the reliable application of statistical techniques, introduce bias in model outputs and obscure periodicities. Collectively, these pose a risk of misconstruing the true water quality patterns. To address these challenges and for an efficient understanding of high-frequency stream chemistry, here we present a novel two-pronged approach: Reconstruction and forecasting of stream chemistry using machine learning (ML) techniques; and Understanding trends and seasonalities in stream chemistry using Singular Spectrum Analysis (SSA). Reconstruction and forecasting of stream chemistry using machine learning (ML) techniques; and Understanding trends and seasonalities in stream chemistry using Singular Spectrum Analysis (SSA). We used one year of high-frequency stream chemistry measurements from the RiverLab at Orgeval Critical Zone Observatory (Loumagne and Tallec 2013), a constituent site of the OZCAR network (Gaillardet et al. 2018) of French Critical Zone Observatories. By incorporating the temporal structure and physico-chemical parameters (Discharge, Temperature, Conductivity, pH) as inputs, multiple ML-based models were developed and applied to predict the dissolved major ions in the stream. Preliminary results hold considerable promise, with the mean absolute percentage error (MAPE) on Cl- prediction remaining below 5% for forecasts extending up to one year. This underscores the potential of ML-assisted approaches in complementing cost-effective and accessible water quality monitoring, owing to the lower operating costs and near-continuous data output provided by the probes. Following this, we discuss the applications of Singular Spectrum Analysis (SSA) on the reconstructed RiverLab data. SSA is a non-parametric technique that combines aspects of classical time series analysis and multivariate statistics (e.g. Principal Component Analysis) to decompose the time series into trends, seasonalities and residual components without relying on the assumption of stationarity (Vautard and Ghil 1989). This study marks one of the first applications of SSA in hydrogeochemistry, and is proposed as a novel tool for exploring concentration-discharge (C-Q) relationships, water quality forecasting, anomaly detection and interpretation of periodic signals in streams.

A dual-stage advanced deep learning algorithm for long-term and long-sequence prediction for multivariate financial time series

The high dimension of multivariate time series (MTS) is one of the major factors that impact on the efficiency and effectiveness of data mining. It has two kinds of dimensions, time-based dimensionality, and variable-based dimensionality. They often cause most of the algorithms and techniques applied to the field of MTS data mining to be a failure. In view of the importance of the correlation between any two variables in an MTS, the covariances between any two variables are applied to analyze the extraction of the features for every MTS. In this way, a covariance sequence can be constructed to represent the characteristic of the MTS. Furthermore, an excellent method of dimensionality reduction, principal component analysis (PCA), is used to extract the features of the covariance sequences that derived from an MTS dataset. Thus Euclidean distance is suitable to measure the similarity between the features fast. The experimental results demonstrate that the proposed method not only can handle multivariate time series with different lengths but also is more efficient and effective than the existing methods for the MTS data mining.

Multivariate time series (MTS) datasets broadly exist in numerous fields, including health care, multimedia, finance, and biometrics. How to classify MTS accurately has become a hot research topic since it is an important element in many computer vision and pattern recognition applications. In this paper, we propose a Mahalanobis distance-based dynamic time warping (DTW) measure for MTS classification. The Mahalanobis distance builds an accurate relationship between each variable and its corresponding category. It is utilized to calculate the local distance between vectors in MTS. Then we use DTW to align those MTS which are out of synchronization or with different lengths. After that, how to learn an accurate Mahalanobis distance function becomes another key problem. This paper establishes a LogDet divergence-based metric learning with triplet constraint model which can learn Mahalanobis matrix with high precision and robustness. Furthermore, the proposed method is applied on nine MTS datasets selected from the University of California, Irvine machine learning repository and Robert T. Olszewski's homepage, and the results demonstrate the improved performance of the proposed approach.

Time-series forecasting is the core of the prognostics and health management (PHM) of turbomachinery. However, missing data often occurs due to several reasons, such as the failure of sensors. These partially missing and irregular data greatly affect the quality of time-series modeling and prediction as most time-series models assume that the data are sampled uniformly over time. Meanwhile, the training process of models requires a large number of samples and time. Due to various reasons, it is difficult to obtain a significant amount of monitoring data, and the PHM of turbomachinery has high timeliness and accuracy requirements. To fix these problems, we propose a multi-task Gaussian process (MTGP)-based data imputation method that leverages knowledge transfer across multiple sensors and even equipment. Thereafter, we adopt long short-term memory (LSTM) neural networks to build time-series forecasting models based on the imputed data. In addition, the model integrates the methods of denoising and dimensionality reduction. The superiority of this integrated time-series forecasting framework, termed MT-LSTM, has been verified in various data imputation scenarios of a synthetic dataset and a real turbomachinery case.

Many datasets include missing values in their attributes. Data mining techniques are not applicable in the presence of missing values. So an important step in preprocessing of a data mining task is missing value management. One of the most important categories in missing value management techniques is missing value imputation. This paper presents a new imputation technique. The proposed imputation technique is based on statistical measurements. The suggested imputation technique employs an ensemble of the estimators built to estimate the missing values based on positive and negative correlated observed attributes separately. Each estimator guesses a value for a missed value based on the average and variance of that feature. The average and variance of the feature are estimated from the non-missed values of that feature. The final consensus value for a missed value is the weighted aggregation of the values estimated by different estimators. The chief weight is attribute correlation, and the slight weight is dependent to kernel function such as kurtosis, skewness, number of involved samples and composition of them. The missing values are deliberately produced randomly at different levels. The experimentations indicate that the suggested technique has a good accuracy in comparison with the classical methods.

Multivariate time-series forecasting is one of the crucial and persistent challenges in time-series forecasting tasks. As a kind of data with multivariate correlation and volatility, multivariate time series impose highly nonlinear time characteristics on the forecasting model. In this paper, a new multivariate time-series forecasting model, multivariate temporal convolutional attention network (MTCAN), based on a self-attentive mechanism is proposed. MTCAN is based on the Convolution Neural Network (CNN) model, using 1D dilated convolution as the basic unit to construct asymmetric blocks, and then, the feature extraction is performed by the self-attention mechanism to finally obtain the prediction results. The input and output lengths of this network can be determined flexibly. The validation of the method is carried out with three different multivariate time-series datasets. The reliability and accuracy of the prediction results are compared with Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Long Short-Term Memory (ConvLSTM), and Temporal Convolutional Network (TCN). The prediction results show that the model proposed in this paper has significantly improved prediction accuracy and generalization.

Real-world time series often present missing values due to sensor malfunctions or human errors. Traditionally, missing values are simply omitted or reconstructed through imputation or interpolation methods. Omitting missing values may cause temporal discontinuity. Reconstruction methods, on the other hand, alter in some way the original time series. In this paper, we consider an application in the field of meteorological variables that exploits end-to-end machine learning. The idea is to entrust the task of dealing with missing values to a suitably trained recurrent neural network that completely by-passes the phase of reconstruction of missing values. A difficult case of reproduction of a rainfall field from five rain gauges in Northern Italy is used as an example, and the results are compared to those computed by more traditional methods. The proposed methodology is general-purpose and can be easily applied to every kind of spatial time series prediction problem, quite common in many environmental studies.

Transfer condition assessment of gas turbines via double multi-task Gaussian process