Non-Parametric Classification of Time Series Using Permutation Ordinal Statistics

Shannon Entropy is the preeminent tool for measuring the level of uncertainty (and conversely, information content) in a random variable. In the field of communications, entropy can be used to express the information content of given signals (represented as time series) by considering random variables which sample from specified subsequences. In this paper, we will discuss how an entropy variant, the \textit{permutation entropy} can be used to study and classify radio frequency signals in a noisy environment. The permutation entropy is the entropy of the random variable which samples occurrences of permutation patterns from time series given a fixed window length, making it a function of the distribution of permutation patterns. Since the permutation entropy is a function of the relative order of data, it is (global) amplitude agnostic and thus allows for comparison between signals at different scales. This article is intended to describe a permutation patterns approach to a data driven problem in radio frequency communications research, and includes a primer on all non-permutation pattern specific background. An empirical analysis of the methods herein on radio frequency data is included. No prior knowledge of signals analysis is assumed, and permutation pattern specific notation will be included. This article serves as a self-contained introduction to the relationship between permutation patterns, entropy, and signals analysis for studying radio frequency signals and includes results on a classification task.

Stationarity is a crucial yet rarely questioned assumption in the analysis of time series of magneto- (MEG) or electroencephalography (EEG). One key drawback of the commonly used tests for stationarity of encephalographic time series is the fact that conclusions on stationarity are only indirectly inferred either from the Gaussianity (e.g. the Shapiro-Wilk test or Kolmogorov-Smirnov test) or the randomness of the time series and the absence of trend using very simple time-series models (e.g. the sign and trend tests by Bendat and Piersol). We present a novel approach to the analysis of the stationarity of MEG and EEG time series by applying modern statistical methods which were specifically developed in econometrics to verify the hypothesis that a time series is stationary. We report our findings of the application of three different tests of stationarity--the Kwiatkowski-Phillips-Schmidt-Schin (KPSS) test for trend or mean stationarity, the Phillips-Perron (PP) test for the presence of a unit root and the White test for homoscedasticity--on an illustrative set of MEG data. For five stimulation sessions, we found already for short epochs of duration of 250 and 500 ms that, although the majority of the studied epochs of single MEG trials were usually mean-stationary (KPSS test and PP test), they were classified as nonstationary due to their heteroscedasticity (White test). We also observed that the presence of external auditory stimulation did not significantly affect the findings regarding the stationarity of the data. We conclude that the combination of these tests allows a refined analysis of the stationarity of MEG and EEG time series.

This work presents a novel approach to multivariate time series classification. The method exploits the multivariate structure of the time series and the possibilities of the stacking ensemble method. The basics of the method may be described in three steps: first, decomposing the multivariate time series on its constituent univariate time series; second, inducing a classifier for each univariate time series plus and additional multivariate classifier for the whole time series; third, creating the final multivariate time series classifier stacking the previous classifiers. The ensemble obtained has the potential to improve the accuracy of the single multivariate time series classifier. Several configurations of the stacking method have been tested on seven multivariate time series data sets. In five out of seven data sets, the proposed method obtains the smallest error rate. Moreover, in two out of seven data sets, stacking only the univariate time series classifiers provides the best results. The experimental results show that when a multivariate time series method does not produce an accurate classifier, stacking it with univariate time series classifiers is an alternative worthy of consideration.

Classification of time series is an essential requirement of various applications that demand continuous monitoring of dynamical systems such as industrial process and health care monitoring. Feature extraction plays a vital role in deciding performance of the time series models. In recent years deep learning techniques have shown an excellent performance to extract highly discriminating features for the classification of the time series. A Convolutional Neural Network (CNN) is a unified framework that performs the feature learning and classification tasks simultaneously. Using the CNN to perform the classification of the multivariate time series is still a challenging task. In this paper, we propose an attention-based multivariate convolutional neural network (AT-MVCNN) that consists of the attention feature-based input tensor scheme to encode informations across the multiple time stamps. The method is capable of learning the temporal characteristics of the multivariate time series. The efficacy of the proposed method is tested on Human Activity Recognition (HAR) and Occupancy Detection datasets. The experiments and results show the proposed method outperforms the other deep learning and traditional machine learning models.

Time series analysis comprises a wide repertoire of methods for extracting information from data sets. Despite great advances in time series analysis, identifying and quantifying the strength of nonlinear temporal correlations remain a challenge. We have recently proposed a new method based on training a machine learning algorithm to predict the temporal correlation parameter, , of flicker noise (FN) time series. The algorithm is trained using as input features the probabilities of ordinal patterns computed from FN time series, , generated with different values of . Then, the ordinal probabilities computed from the time series of interest, , are used as input features to the trained algorithm and that returns a value, , that contains meaningful information about the temporal correlations present in . We have also shown that the difference, , of the permutation entropy (PE) of the time series of interest, , and the PE of a FN time series generated with , , allows the identification of the underlying determinism in . Here, we apply our methodology to different datasets and analyze how and correlate with well-known quantifiers of chaos and complexity. We also discuss the limitations for identifying determinism in highly chaotic time series and in periodic time series contaminated by noise. The open source algorithm is available on Github.

In 2002, in a seminal article, Bandt and Pompe proposed a new methodology for the analysis of complex time series, now known as Ordinal Analysis. The ordinal methodology is based on the computation of symbols (known as ordinal patters) which are defined in terms of the temporal ordering of data points in a time series, and whose probabilities are known as ordinal probabilities. With the ordinal probabilities the Shannon entropy can be calculated, which is the permutation entropy. Since it was proposed, the ordinal method has found applications in fields as diverse as biomedicine and climatology. However, some properties of ordinal probabilities are still not fully understood, and how to combine the ordinal approach of feature extraction with machine learning techniques for model identification, time series classification or forecasting, remains a challenge. The objective of this perspective article is to present some recent advances and to discuss some open problems.

The study is concerned with a concept and a design of granular time series and granular classifiers. In contrast to the plethora of various models of time series, which are predominantly numeric, we propose to effectively exploit the idea of information granules in the description and classification of time series. The numeric (optimization-oriented) and interpretation abilities of granular time series and their classifiers are highlighted and quantified. A general topology of the granular classifier involving a formation of a granular feature space and the usage of the framework of relational structures (relational equations) in the realization of the classifiers is presented. A detailed design process is elaborated on along with a discussion of the pertinent optimization mechanisms. A series of experiments is covered leading to a quantitative assessment of the granular classifiers and their parametric analysis.

Time series classification is one of the most active research topics in time series data mining, because it covers a broad range of applications in many different domains. Representation for time series is a technique that converts time series to feature vectors representing the characteristics of time series. The performance of classifying time series depends on this representation. Chaotic time series analyses have been well-studied. Moreover, recurrence plotting underlying chaos theory is one of the most robust representation for time series. In this study, we propose a new time series representation utilising the recurrence plot technique. Moving average convergence divergence (MACD) histogram is the acceleration of time that can represent local-variation in time series. Therefore, a recurrence plot that is made from MACD histogram, which is called a MACD-histogram-based recurrence plot (MHRP), can handle time series very well. Recurrence plots are referred to as grey-scale images and we utilise stacked auto-encoders as a classifier for MHRPs. To evaluate the performance of the proposed classifier, experiments using the UCR time series classification archive was conducted. The experimental results showed that the proposed classifier outperforms other methods.

Time series classification (TSC) is a challenging task that attracted many researchers in the last few years. One main challenge in TSC is the diversity of domains where time series data come from. Thus, there is no “one model that fits all” in TSC. Some algorithms are very accurate in classifying a specific type of time series when the whole series is considered, while some only target the existence/non-existence of specific patterns/shapelets. Yet other techniques focus on the frequency of occurrences of discriminating patterns/features. This paper presents a new classification technique that addresses the inherent diversity problem in TSC using a nature-inspired method. The technique is stimulated by how flies look at the world through “compound eyes” that are made up of thousands of lenses, called ommatidia. Each ommatidium is an eye with its own lens, and thousands of them together create a broad field of vision. The developed technique similarly uses different lenses and representations to look at the time series, and then combines them for broader visibility. These lenses have been created through hyper-parameterisation of symbolic representations (Piecewise Aggregate and Fourier approximations). The algorithm builds a random forest for each lens, then performs soft dynamic voting for classifying new instances using the most confident eyes, i.e., forests. We evaluate the new technique, coined Co-eye, using the recently released extended version of UCR archive, containing more than 100 datasets across a wide range of domains. The results show the benefits of bringing together different perspectives reflecting on the accuracy and robustness of Co-eye in comparison to other state-of-the-art techniques.

Time Series Classification (TSC) is an important and challenging problem in data mining. With the increase of time series data availability, hundreds of TSC algorithms have been proposed. Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task. This is surprising as deep learning has seen very successful applications in the last years. DNNs have indeed revolutionized the field of computer vision especially with the advent of novel deeper architectures such as Residual and Convolutional Neural Networks. Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition. In this article, we study the current state-of-the-art performance of deep learning algorithms for TSC by presenting an empirical study of the most recent DNN architectures for TSC. We give an overview of the most successful deep learning applications in various time series domains under a unified taxonomy of DNNs for TSC. We also provide an open source deep learning framework to the TSC community where we implemented each of the compared approaches and evaluated them on a univariate TSC benchmark (the UCR/UEA archive) and 12 multivariate time series datasets. By training 8,730 deep learning models on 97 time series datasets, we propose the most exhaustive study of DNNs for TSC to date.

In recent years, permutation entropy is widely used to characterize the complexity of EEG time series and can be applied to predict the onset of serious brain diseases, such as the epileptic seizure. In many practical situations, the number of EEG time series that need to be analyzed simultaneously is very large, so the computation of the permutation entropy is time-consuming and should be accelerated so that the real-time analysis is possible. Noting that mathematical operations can be sped up effectively with hardware implementation, we design a parallel FPGA platform consisting of 128 reconfigurable pipelines, which are used to calculate the permutation entropy for a single EEG time series. When the platform works at 150MHz and the embedding dimension is 5, an average speedup of 5553 for different window sizes is achieved compared with C codes running on a 3GHz Intel(R) Core(TM) i5-2320 CPU. Meanwhile, the hardware cost is very low.

Decision templates for the classification of bioacoustic time series

Traditional works on time series classification usually use all of data in time series without distinction. However, that will swamp the discriminative information and decrease the correctness of classification. In this paper, a feature segment based time series classification algorithm was proposed, which only selects some highly discriminative time series data for classification. Firstly, an adaptive time series segmentation method was proposed. Then, a large margin based feature segment selection method was given. Based on these two methods, a time series classification framework was established after representing the time series with the optimal segments. By exploring the discriminative temporal patterns hidden in subsequences of time series and giving them more emphasize, the algorithm proposed in this paper can improve the time series classification performance greatly. Extensive experimental results showed that the proposed algorithm can achieve a good classification performance.

In the past decade, time series data have been generated from various fields at a rapid speed, which offers a huge opportunity for mining valuable knowledge. As a typical task of time series mining, Time Series Classification (TSC) has attracted lots of attention from both researchers and domain experts due to its broad applications ranging from human activity recognition to smart city governance. Specifically, there is an increasing requirement for performing classification tasks on diverse types of time series data in a timely manner without costly hand-crafting feature engineering. Therefore, in this paper, we propose a framework named Edge4TSC that allows time series to be processed in the edge environment, so that the classification results can be instantly returned to the end-users. Meanwhile, to get rid of the costly hand-crafting feature engineering process, deep learning techniques are applied for automatic feature extraction, which shows competitive or even superior performance compared to state-of-the-art TSC solutions. However, because time series presents complex patterns, even deep learning models are not capable of achieving satisfactory classification accuracy, which motivated us to explore new time series representation methods to help classifiers further improve the classification accuracy. In the proposed framework Edge4TSC, by building the binary distribution tree, a new time series representation method was designed for addressing the classification accuracy concern in TSC tasks. By conducting comprehensive experiments on six challenging time series datasets in the edge environment, the potential of the proposed framework for its generalization ability and classification accuracy improvement is firmly validated with a number of helpful insights.

Training deep neural networks often requires careful hyper-parameter tuning and significant computational resources. In this paper, we propose ConvTimeNet (CTN): an off-the-shelf deep convolutional neural network (CNN) trained on diverse univariate time series classification (TSC) source tasks. Once trained, CTN can be easily adapted to new TSC target tasks via a small amount of fine-tuning using labeled instances from the target tasks. We note that the length of convolutional filters is a key aspect when building a pre-trained model that can generalize to time series of different lengths across datasets. To achieve this, we incorporate filters of multiple lengths in all convolutional layers of CTN to capture temporal features at multiple time scales. We consider all 65 datasets with time series of lengths up to 512 points from the UCR TSC Benchmark for training and testing transferability of CTN: We train CTN on a randomly chosen subset of 24 datasets using a multi-head approach with a different softmax layer for each training dataset, and study generalizability and transferability of the learned filters on the remaining 41 TSC datasets. We observe significant gains in classification accuracy as well as computational efficiency when using pre-trained CTN as a starting point for subsequent task-specific fine-tuning compared to existing state-of-the-art TSC approaches. We also provide qualitative insights into the working of CTN by: i) analyzing the activations and filters of first convolution layer suggesting the filters in CTN are generically useful, ii) analyzing the impact of the design decision to incorporate multiple length decisions, and iii) finding regions of time series that affect the final classification decision via occlusion sensitivity analysis.