Feature Representation and Similarity Measure Based on Covariance Sequence for Multivariate Time Series

In terms of existing time series clustering method based on Euclidean distance metric, with the increasing dimension of time series, the time complexity of the algorithm will be increased too; and this method can also lead to incorrect clustering result because of it unable to recognize the abnormal values in time series.Principal component analysis retains large variance and contains more information by linear transformation; it can effectively reduce the dimension of the time series and identify outliers.This paper proposes the idea of time series clustering analysis method based on principal component analysis.Firstly, applying principal component analysis to time series dataset, by way of dimension reduction, obtained the corresponding coefficient matrix and eigenvalues.Secondly, using clustering method based on Euclidean distance on the calculated coefficient matrix, the clustering result of coefficient matrix is consistent with time series dataset.Using simulation data and meteorological data to validate this method, the experimental results show that time complexity of time series clustering method proposed in this paper is significantly better than the algorithm based on Euclidean distance, especially for time series dataset which has linear correlation.

Multivariate time series (MTS) arise due to multisensor data collection in manufacturing. These data are complex in the sense that attributes have a varying scale, volitivity, continuity, and so on, and interattribute dependence also appears, which can mask the inherent information about system health status. Conventional machine learning-based process monitoring techniques are inefficient in predicting faults with MTS—their detection capability heavily relies on the input features, yet the classification power of MTS attributes is weakened by the complexity of multisensor data. Effective feature extraction is, therefore, necessary to facilitate fault prediction with MTS. This study proposes a fault prediction framework for MTS based on bag-of-words (BOW) feature extraction, statistical feature selection, and classification analysis. BOW models are for the first time adopted in a manufacturing context. Their superior capability in information preservation, local pattern recognition, and temporal effect accommodation has overcome the major limitations in current manufacturing practices with MTS. A comprehensive case study demonstrates the desirable performance of this framework on two MTS data sets from paper manufacturing and automotive manufacturing, as well as its superiority over conventional machine learning-based fault prediction. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Process monitoring in a multisensor environment has been a vital interest in manufacturing. The difficulty lies in the lack of detection power in many conventional techniques, e.g., control chart and logistic regression. A critical reason for such failure is the neglect of time effect in MTS—patterns associated with system faults tend to stretch a period, but a conventional control chart or classifier inspects each time stamp separately. Fault detection based on time series sequences is, therefore, essential. However, how to effectively extract features from MTS becomes a challenge. The framework proposed in this study adopts BOW models, specifically symbolic aggregate approximation (SAX), to extract features from MTS sequences, thus substantially improves the detection power against local patterns over time. Many manufacturing multisensor data are subject to such local patterns that point to the root cause of fault, so the proposed framework has a wide application in manufacturing.

Several spatio-temporal data collected in many applications, such as fMRI data in medical applications, can be represented as a Multivariate Time Series (MTS) matrix with m rows (capturing the spatial features) and n columns (capturing the temporal observations). Any data mining task such as clustering or classification on MTS datasets are usually hindered by the large size (i.e., dimensions) of these MTS items. In order to reduce the dimensions without losing the useful discriminative features of the dataset, feature selection techniques are usually preferred by domain experts since the relation of the selected subset of features to the originally acquired features is maintained. In this paper, we propose a new feature selection technique for MTS datasets where their spatial features (i.e., number of rows) are much larger than their temporal observations (i.e., number of columns), or m /ge n. Our approach is based on Principal Component Analysis, Recursive Feature Elimination and Support Vector Machines. Our empirical results on real-world datasets show that our technique significantly outperforms the closest competitor technique.

The article will present the methodology for holistic time series analysis, based on time series transformation model REFII (REFII is an acronym for Raise-Equal-Fall and the model version is II or 2) Patel et al. (Mining motifs in massive time series databases, 2002) [1], Perng and Parker (SQL/LPP: a time series extension of SQL based on limited patience patterns, 1999) [2], Popivanov and Miller (Similarity search over time series data using wavelets, 2002) [3]. The main purpose of REFII model is to automate time series analysis through a unique transformation model of time series. The advantage of this approach to a time series analysis is the linkage of different methods for time series analysis linking traditional data mining tools in time series, and constructing new algorithms for analyzing time series. REFII model is not a closed system, which means that there is a finite set of methods. This is primarily a model used for transformation of values of time series, which prepares data used by different sets of methods based on the same model of transformation in the domain of problem space. REFII model gives a new approach in time series analysis based on a unique model of transformation, which is a base for all kind of time series analyses. In combination with elements of other methods, such as self-organizing maps or frequent-pattern trees, REFII models can make new hybrid methods for efficient time temporal data mining. Similar principle of hybridization could be used as a tool for time series temporal pattern recognition. The article describes real case study illustrating practical application of described methodology.

This work presents a novel approach to multivariate time series classification. The method exploits the multivariate structure of the time series and the possibilities of the stacking ensemble method. The basics of the method may be described in three steps: first, decomposing the multivariate time series on its constituent univariate time series; second, inducing a classifier for each univariate time series plus and additional multivariate classifier for the whole time series; third, creating the final multivariate time series classifier stacking the previous classifiers. The ensemble obtained has the potential to improve the accuracy of the single multivariate time series classifier. Several configurations of the stacking method have been tested on seven multivariate time series data sets. In five out of seven data sets, the proposed method obtains the smallest error rate. Moreover, in two out of seven data sets, stacking only the univariate time series classifiers provides the best results. The experimental results show that when a multivariate time series method does not produce an accurate classifier, stacking it with univariate time series classifiers is an alternative worthy of consideration.

Rivers act as the primary harmonizer of the critical zone, reflecting the confluence of processes that regulate nutrient cycling, water quality, and energy flow within ecosystems. Understanding water quality dynamics and their relationship with hydrological and biogeochemical processes warrants the study of river chemistry at a detailed temporal resolution, but this is often hindered by logistical challenges in the acquisition of such data. Recent efforts in automated data acquisition, such as the RiverLab “lab-on-a-field” setup (Floury et al. 2017) have equipped us with a toolbox to peek into riverine systems at unprecedentedly fine temporal scales. This assemblage consists of a primary circuit that continuously samples water from the middle of a stream, and measures various physico-chemical parameters using probes. A portion of this water is also routed through a filtration system towards two ion chromatographs housed in the setup, allowing the measurement of dissolved ions. This system thus enables sub-hourly measurement of major ions and near-continuous monitoring of pH, Temperature, Conductivity, etc. However, despite these advances in data acquisition, technical challenges remain. The datasets often contain gaps due to practical limitations like sensor drift and malfunction, power outages, etc. These induce gaps in the multivariate time series datasets, and can impede the reliable application of statistical techniques, introduce bias in model outputs and obscure periodicities. Collectively, these pose a risk of misconstruing the true water quality patterns. To address these challenges and for an efficient understanding of high-frequency stream chemistry, here we present a novel two-pronged approach: Reconstruction and forecasting of stream chemistry using machine learning (ML) techniques; and Understanding trends and seasonalities in stream chemistry using Singular Spectrum Analysis (SSA). Reconstruction and forecasting of stream chemistry using machine learning (ML) techniques; and Understanding trends and seasonalities in stream chemistry using Singular Spectrum Analysis (SSA). We used one year of high-frequency stream chemistry measurements from the RiverLab at Orgeval Critical Zone Observatory (Loumagne and Tallec 2013), a constituent site of the OZCAR network (Gaillardet et al. 2018) of French Critical Zone Observatories. By incorporating the temporal structure and physico-chemical parameters (Discharge, Temperature, Conductivity, pH) as inputs, multiple ML-based models were developed and applied to predict the dissolved major ions in the stream. Preliminary results hold considerable promise, with the mean absolute percentage error (MAPE) on Cl- prediction remaining below 5% for forecasts extending up to one year. This underscores the potential of ML-assisted approaches in complementing cost-effective and accessible water quality monitoring, owing to the lower operating costs and near-continuous data output provided by the probes. Following this, we discuss the applications of Singular Spectrum Analysis (SSA) on the reconstructed RiverLab data. SSA is a non-parametric technique that combines aspects of classical time series analysis and multivariate statistics (e.g. Principal Component Analysis) to decompose the time series into trends, seasonalities and residual components without relying on the assumption of stationarity (Vautard and Ghil 1989). This study marks one of the first applications of SSA in hydrogeochemistry, and is proposed as a novel tool for exploring concentration-discharge (C-Q) relationships, water quality forecasting, anomaly detection and interpretation of periodic signals in streams.

The time series clustering is one of the important research contents in the time series data mining. Since the dimension of time series is common high, the performance of direct raw time series data clustering is not ideal. How to improve the clustering performance of time series is the main research point of this paper. Firstly, use Locality Preserving Projection (LPP) to time series samples for dimensionality reduction; secondly, carry out clustering ensemble to the lowdimension data; finally, compare the clustering performance with the existing methods such as Principal Component Analysis (PCA) and Piecewise Aggregate Approximation (PAA). Experiment results show that the proposed method is superior to the compared methods.

A dual-stage advanced deep learning algorithm for long-term and long-sequence prediction for multivariate financial time series

Multivariate time series (MTS) datasets broadly exist in numerous fields, including health care, multimedia, finance, and biometrics. How to classify MTS accurately has become a hot research topic since it is an important element in many computer vision and pattern recognition applications. In this paper, we propose a Mahalanobis distance-based dynamic time warping (DTW) measure for MTS classification. The Mahalanobis distance builds an accurate relationship between each variable and its corresponding category. It is utilized to calculate the local distance between vectors in MTS. Then we use DTW to align those MTS which are out of synchronization or with different lengths. After that, how to learn an accurate Mahalanobis distance function becomes another key problem. This paper establishes a LogDet divergence-based metric learning with triplet constraint model which can learn Mahalanobis matrix with high precision and robustness. Furthermore, the proposed method is applied on nine MTS datasets selected from the University of California, Irvine machine learning repository and Robert T. Olszewski's homepage, and the results demonstrate the improved performance of the proposed approach.

Recent advances in computing, communications, digital storage technologies, and high-throughput data-acquisition technologies, make it possible to gather and store incredible volumes of data. It creates unprecedented opportunities for large-scale knowledge discovery from huge database. Data mining (DM) technology has emerged as a means of performing this discovery. There are countless researchers working on designing efficient data mining techniques, methods, and algorithms. Many data mining methods and algorithms have been developed and applied in a lot of application fields [1]. Unfortunately, most data mining researchers pay much attention to technique problems for developing data mining models and methods, while little to basic issues of data mining [2]. In this talk, some basic issues of data mining are addressed. What is data mining? What is the product of a data mining process? What are we doing in a data mining process? What is the rule we should obey in a data mining process? Through analyzing existing data mining methods, and domain-driven (or user-driven) data mining models [3-5], we find that we should take a data mining process as a process of knowledge transformation. Based on this understanding of data mining, a conceptual data mining model of domain-oriented data-driven data mining (3DM) is proposed [2]. The relationship between traditional domain-driven (or user-driven) data mining models and our proposed 3DM model is also analyzed. Some domain-oriented data-driven data mining algorithms for mining such knowledge as default rule [6], decision tree [7], and concept lattice [8] from database are proposed. The experiment results for these algorithms are also shown to illustrate the efficiency and performance of the knowledge acquired by our 3DM data mining algorithms.

Multivariate time series often contain missing values for reasons such as failures in data collection mechanisms. Since these missing values can complicate the analysis of time series data, imputation techniques are typically used to deal with this issue. However, the quality of the imputation directly affects the performance of downstream tasks. In this paper, we propose a selective imputation method that identifies a subset of timesteps with missing values to impute in a multivariate time series dataset. This selection, which will result in shorter and simpler time series, is based on both reducing the uncertainty of the imputations and representing the original time series as good as possible. In particular, the method uses multi-objective optimization techniques to select the optimal set of points, and in this selection process, we leverage the beneficial properties of the Multi-task Gaussian Process (MGP). The method is applied to different datasets to analyze the quality of the imputations and the performance obtained in downstream tasks, such as classification or anomaly detection. The results show that much shorter and simpler time series are able to maintain or even improve both the quality of the imputations and the performance of the downstream tasks.

Forecasting Foreign Exchange Rate Movements with k-Nearest-Neighbour, Ridge Regression and Feed-Forward Neural Networks

Manufacturers are struggling to use data from multiple products production lines to predict rare events. Improving the quality of training data is a common way to improve the performance of algorithms. However, there is little research about how to select training data quantitatively. In this study, a training data selection method is proposed to improve the performance of deep learning models. The proposed method can represent different time length multivariate time series spilt by categorical variables and measure the (dis)similarities by the distance matrix and clustering method. The contributions are: (1) The proposed method can find the changes to the training data caused by categorical variables in a multivariate time series dataset; (2) according to the proposed method, the multivariate time series data from the production line can be clustered into many small training datasets; and (3) same structure but different parameters prediction models are built instead of one model which is different from the traditional way. In practice, the proposed method is applied in a real multiple products production line dataset and the result shows it can not only significantly improve the performance of the reconstruction model but it can also quantitively measure the (dis)similarities of the production behaviors.

Multivariate time series (MTS) are collected for different variables in studying scientific phenomena or monitoring system health where each time series records the values of one variable for a time period. Among the different variables, it is common that only a few variables contribute significantly to a specific phenomenon. Furthermore, the variables contributing significantly to different phenomena are often different. We denote the different variables that contribute to the occurrences of different phenomena as Phenomenon-specific Variables (PVs) . In this paper, we formulate a novel problem of identifying significant PVs from MTS datasets. To analyze MTS data, feature extraction techniques have been extensively studied. However, most of them identify important global features for one dataset and do not utilize the temporal order of time series. To solve the newly introduced problem, we propose a solution framework, CNN $_{mts}$ m t s -X , which is a new variant of the Convolutional Neural Networks ( CNN ) and can embed other feature extraction techniques (as X ). Furthermore, we design a CNN $_{mts}$ m t s -LR method that implements a new feature identification approach ( LR ) as X in the CNN $_{mts}$ m t s -X framework. The LR method leverages both Linear Discriminant Analysis ( LDA ) and Random Forest ( RF ). Our extensive experiments on five real datasets show that the CNN $_{mts}$ m t s -LR method has exhibited much better performance than several other baseline methods. Using 30 percent of the PVs discovered from the CNN $_{mts}$ m t s -LR , classifications can achieve better or similar performance than using all the variables.

Recently, multivariate time series (MTS) clustering has gained lots of attention. However, state-of-the-art algorithms suffer from two major issues. First, few existing studies consider correlations and redundancies between variables of MTS data. Second, since different clusters usually exist in different intrinsic variables, how to efficiently enhance the performance by mining the intrinsic variables of a cluster is challenging work. To deal with these issues, we first propose a variable-weighted K-medoids clustering algorithm (VWKM) based on the importance of a variable for a cluster. In VWKM, the proposed variable weighting scheme could identify the important variables for a cluster, which can also provide knowledge and experience to related experts. Then, a Reverse nearest neighborhood-based density Peaks approach (RP) is proposed to handle the problem of initialization sensitivity of VWKM. Next, based on VWKM and the density peaks approach, an ensemble Clustering framework (SSEC) is advanced to further enhance the clustering performance. Experimental results on ten MTS datasets show that our method works well on MTS datasets and outperforms the state-of-the-art clustering ensemble approaches.