Segmentation algorithm for long time series analysis

This scientific article addresses two critical tasks in data analysis—time series classification and clustering, particularly focusing on heart sound recordings. One of the main challenges in analyzing time series lies in the difficulty of comparing different series due to their variability in length, shape, and amplitude. Various algorithms were employed to tackle these tasks, including the Long Short-Term Memory (LSTM), KNN, recurrent neural network for classification and the K-means and DBSCAN methods for clustering. The study emphasizes the effectiveness of these methods in solving classification and clustering problems involving time series data containing heart sound recordings. The results indicate that LSTM is a powerful tool for time series classification due to its ability to retain contextual information over time. In contrast, KNN demonstrated high accuracy and speed in classification, though its limitations became apparent with larger datasets. For clustering tasks, the K-means method proved to be more effective than DBSCAN, showing higher clustering quality based on metrics such as silhouette score, Rand score, and others. The data used in this research were obtained from the UCR Time Series Archive, which includes heart sound recordings from various categories: normal sounds, murmurs, additional heart sounds, artifacts, and extra systolic rhythms. The analysis of results demonstrated that the chosen classification and clustering methods could be effectively used for diagnosing heart diseases. Furthermore, this research opens up new opportunities for further improvement in data processing and analysis methods, particularly in developing new medical diagnostic tools. Thus, this work illustrates the effectiveness of machine learning algorithms for time series analysis and their significance in improving cardiovascular disease diagnosis.

With the diversification of applications and the diversity of users, the era of big data is always producing various new types of data, such as graph data, time series data, and spatial data. When predicting the future trend of the system, we always need to learn from the historical working state, and time series data plays a huge value in this aspect. Hardware resources are becoming cheaper, computing power has made major breakthroughs, and multi-machine and multi-core scenarios have become commonplace. At present, the focus of large-scale time series data processing is either focused on data collection and storage, or on single-machine processing, which has been unable to meet the existing needs. In response to this problem, this paper designs and implements a distributed large-scale time series processing and analysis system based on the Spark platform. The system framework is mainly divided into storage layer, operator layer and algorithm layer. At the storage layer, the system completes the organization and indexing of large-scale time series data based on HDFS and Hive. At the operator layer, the system provides users with basic operations commonly used for time series data on the Spark platform, and allows users to directly use these operators to implement custom time series related processing algorithms. At the algorithm level, the system implements some commonly used time series analysis algorithms in the Spark platform, including time series similarity query, clustering and prediction, so that users can directly use these algorithms to analyze time series. Finally, the feasibility and practicability of this system are verified by testing the performance and function of the system.

Time series analysis of Mexico City subsidence constrained by radar interferometry

CUDA-based parallelization of time-weighted dynamic time warping algorithm for time series analysis of remote sensing data

The article considers the problem of detecting anomalies in the time series of media content metrics obtained from Telegram channels. The task of detecting anomalies is relevant in the context of combating disinformation and analyzing the dynamics of content distribution in social networks. Anomalies in metrics, such as the number of views, shares, comments, and reactions, may indicate manipulative actions, including the use of bots, falsification of reach, or the spread of disinformation. A dataset obtained through the official Telegram API was used for the analysis. A feature of the data is the lack of retrospective values of metrics, which complicates the analysis of their dynamics. This problem was partially solved by collecting data at fixed time intervals after the publication of each post. The collected data was grouped by the channels of origin of the posts and time intervals after publication to ensure the accuracy of the analysis. Since the data were unlabeled, manual processing was used to remove outliers and ensure the reliability of the modeling. The article analyzes the functionality of five Python libraries for detecting anomalies in time series: PyOD, TODS, PySAD, Darts, and Prophet. Their compliance with the requirements was assessed, in particular, with respect to working with time series, processing incomplete data, real-time support, seasonality, and computational efficiency. A comparison was made based on tables and graphs that demonstrate the results of using each library. In particular, the PyOD library is a well-known tool for detecting anomalies, but does not support direct work with time series. TODS has the potential to detect anomalies in streaming data, but its development has been discontinued. PySAD specializes in streaming data analysis, but requires a fixed frequency of input data, which limits its application. The Darts library offers a wide set of algorithms for time series analysis, but requires pre-filling of missing values, which creates an additional load on the model. The best results were achieved using the Prophet model, which is able to work with irregular time series without the need for additional data augmentation. The experiments conducted showed that Prophet provides the best balance between forecast accuracy and computational efficiency. The choice of the amount of historical data for modeling is crucial, since excessive data increases processing time, and insufficient data reduces forecast accuracy.The results of the study indicate that none of the considered libraries is a universal solution for all tasks.However, Prophet showed the greatest potential for detecting anomalies in the time series of Telegram metrics, which makes it the best candidate for further development and adaptation in media content monitoring tasks.

The article will present the methodology for holistic time series analysis, based on time series transformation model REFII (REFII is an acronym for Raise-Equal-Fall and the model version is II or 2) Patel et al. (Mining motifs in massive time series databases, 2002) [1], Perng and Parker (SQL/LPP: a time series extension of SQL based on limited patience patterns, 1999) [2], Popivanov and Miller (Similarity search over time series data using wavelets, 2002) [3]. The main purpose of REFII model is to automate time series analysis through a unique transformation model of time series. The advantage of this approach to a time series analysis is the linkage of different methods for time series analysis linking traditional data mining tools in time series, and constructing new algorithms for analyzing time series. REFII model is not a closed system, which means that there is a finite set of methods. This is primarily a model used for transformation of values of time series, which prepares data used by different sets of methods based on the same model of transformation in the domain of problem space. REFII model gives a new approach in time series analysis based on a unique model of transformation, which is a base for all kind of time series analyses. In combination with elements of other methods, such as self-organizing maps or frequent-pattern trees, REFII models can make new hybrid methods for efficient time temporal data mining. Similar principle of hybridization could be used as a tool for time series temporal pattern recognition. The article describes real case study illustrating practical application of described methodology.

The goal of this work is to generate large statistically representative data sets to train machine learning models for disruption prediction provided by data from few existing discharges. Such a comprehensive training database is important to achieve satisfying and reliable prediction results in artificial neural network classifiers. Here, we aim for a robust augmentation of the training database for multivariate time series data using Student $t$ process regression. We apply Student $t$ process regression in a state space formulation via Bayesian filtering to tackle challenges imposed by outliers and noise in the training data set and to reduce the computational complexity. Thus, the method can also be used if the time resolution is high. We use an uncorrelated model for each dimension and impose correlations afterwards via colouring transformations. We demonstrate the efficacy of our approach on plasma diagnostics data of three different disruption classes from the DIII-D tokamak. To evaluate if the distribution of the generated data is similar to the training data, we additionally perform statistical analyses using methods from time series analysis, descriptive statistics and classic machine learning clustering algorithms.

In the paper the algorithm of singular structural analysis and forecasting of a multidimensional series by the MSSA method is used. The program was developed, in which the steps of the method for the selection of singular decomposition components were implemented, the analysis and forecast of real time series was carried out. Singular spectrum analysis (SSA) is increasingly used in the study of time series. Unlike other methods of statistical research of time series, this method is used to study the structure, selection of individual components and forecast of both stationary and non-stationary time series. It does not require an analytical model of the series. In fact, this approach is based on the method of principal components. It is based on the transformation of a series into a matrix and its singular decomposition. After identifying the components of the singular decomposition, their grouping takes place, which leads to the decomposition of the original series into additive components, such as trend, oscillations (periodics), and noise. The SSA method allows you to continue the structure of the time series, thus building a forecast (continuation). An important direction of the development of the SSA method as a method of time series analysis is its generalization for the analysis of multidimensional time series. The method is known as MSSA (Multi-Channel SSA) or E-EOFs (Extended Empirical Orthogonal Functions). In this case, the expected result is the simultaneous decomposition of several series into interpreted components. However, a sufficiently complete theory for MSSA does not exist.

The goal of this paper is to analyze the COVID-19 policies of 10 European countries, including Sweden, Finland, Norway, Italy, France, Germany, Poland, Belgium, the Netherlands, and Hungary, with a time-series policy analysis tool. The results of the COVID-19 policy analysis are based on a single time-series indicator, or daily population mortality rate: the number of COVID-19 daily cumulative deaths divided by the population in millions. The lower the score, the better the policy. Although many experts believe that the COVID-19 policy outcome analysis is premature, time series analysis is an excellent analysis that can provide information on the progress and transition of policy outcomes. In other words, the proposed time series analysis tool allows policymakers to identify and quantify when mistakes were made during the on-going COVID-19 pandemic. The COVID-19 policy analysis discovered many useful facts. Sweden failed due to the herd immunity approach. Hungary made a fundamental mistake in COVID-19 tactics. Countries such as Sweden, Hungary, Belgium, and Poland showed time-series changes that differed from the others. Public health interventions can play a key role in mitigating the COVID-19 pandemic. The proposed policy analysis tool, hiscovid demonstrated the effectiveness of the time-series score behavior for discovering when policymakers made mistakes.

Sentinel-2/MSI and Landsat-8/OLI sensors enable the mapping of ocean color-related bio-optical parameters of surface coastal and inland waters. While many algorithms have been developed to estimate the Chlorophyll-a concentration, Chl-a, and the suspended particulate matter, SPM, from OLI and MSI data, the absorption by colored dissolved organic matter, acdom, a key parameter to monitor the concentration of dissolved organic matter, has received less attention. Herein we present an inverse model (hereafter referred to as AquaCDOM) for estimating acdom at the wavelength 412 nm (acdom (412)), within the surface layer of coastal waters, from measurements of ocean remote sensing reflectance, Rrs (λ), for these two high spatial resolution (around 20 m) sensors. Combined with a water class-based approach, several empirical algorithms were tested on a mixed dataset of synthetic and in situ data collected from global coastal waters. The selection of the final algorithms was performed with an independent validation dataset, using in situ, synthetic, and satellite Rrs (λ) measurements, but also by testing their respective sensitivity to typical noise introduced by atmospheric correction algorithms. It was found that the proposed algorithms could estimate acdom (412) with a median absolute percentage difference of ~30% and a median bias of 0.002 m−1 from the in situ and synthetic datasets. While similar performances have been shown with two other algorithms based on different methodological developments, we have shown that AquaCDOM is much less sensitive to atmospheric correction uncertainties, mainly due to the use of band ratios in its formulation. After the application of the top-of-atmosphere gains and of the same atmospheric correction algorithm, excellent agreement has been found between the OLI- and MSI-derived acdom (412) values for various coastal areas, enabling the application of these algorithms for time series analysis. An example application of our algorithms for the time series analysis of acdom (412) is provided for a coastal transect in the south of Vietnam.

Time series (TS) analysis is an active application area for Artificial Intelligence (AI) methods, where the objective is to analyze numeric quantities indexed by time for tasks like classification, forecasting, and abnormality detection. In health, TS manifests as biosignals like the electroencephalogram (EEG), where electrical signals from the brain are analyzed. AI and health communities can tremendously benefit each other in TS, with the former offering advanced analytical methods while the latter provides complex data sets and trust-sensitive use cases. But the communities also need to overcome confusing terminologies, hidden assumptions, and a lack of necessary domain contexts for result evaluation and interpretation. In this paper, we attempt to bridge the gap using the problem of channel selection in EEG. We outline challenges in working with EEG data, demonstrate via two experiments how simple explainable AI (XAI) methods can be quite effective for channel selection, irrespective of the EEG tasks/paradigms, and argue that recent TS trends in AI, like LLMs and XAI methods, can benefit health as well. We hope that this work will bring researchers working on TS problems at the intersection of AI and health, closer to work in AI trustworthiness so that they can better leverage results from their respective areas to overcome common challenges. All code and resources are released on GitHub to help others replicate.

The ice core time series from Vostok Station in Antarctica and the North Greenland Ice Core Project have seasonal variation corresponding to the Milankovitch cycles. After removing these cycles, and interpolating to equal time intervals, stationary time series models are fitted. The series show clear directionality and this feature is modelled by either non-Gaussian errors or non-linear time series models. Threshold autoregressive models are fitted by penalized least squares and compared with non-threshold autoregressive models. Since both ice core time series are reasonably modelled as first order autoregressive series with parameters close to one, directionality will arise from non-symmetric error distributions. However, two regime threshold autoregressive models, of order one and two for Greenland and Vostok, respectively, give an improved match to the observed directionality and a reduced sum of squared residuals. Realizations from the threshold autoregressive models are noticeably different from the non-threshold models. Since the non-threshold models are a restricted case of the threshold models, and the threshold models are a better fit to the observed time series, threshold models should provide more realistic realizations. References C. Chatfield. The Analysis of Time Series: An Introduction . CRC Press, 2004. https://www.crcpress.com/The-Analysis-of-Time-Series-An-Introduction-Sixth-Edition/Chatfield/p/book/9781584883173 A. J. Lawrance. Directionality and reversibility in time series. Int. Stat. Rev. 59(1):67–79, 1991. doi:10.2307/1403575 M. M. Mansor, M. E. Glonek, D. A. Green and A. V. Metcalfe. Threshold autoregressive models for directional time series. In I. Rojas and H. Pomares (Eds.), Time Series Analysis and Forecasting Selected Contributions from the ITISE Conference (ITISE 2015) . pp. 13–25, 2016. doi:10.1007/978-3-319-28725-6 M. M. Mansor, M. E. Glonek, D. A. Green and A. V. Metcalfe. Modelling directionality in stationary geophysical time series. International work-conference on Time Series (ITISE 2015) . http://www.researchgate.net/publication/281835075 M. M. Mansor, D. A. Green and A. V. Metcalfe. Modelling and simulation of directional financial time series. Proceedings of the 21st International Congress on Modelling and Simulation (MODSIM 2015) , pp. 1022–1028, 2015. http://www.mssanz.org.au/modsim2015/E4/mansor.pdf M. M. Mansor, D. A. Green and A. V. Metcalfe. Directionality and volatility in electroencephalogram time series. Proceedings of the 2nd International Conference on Mathematical Sciences and Statistics (ICMSS 2016), AIP Conf. Proc. 1739:020080, 2016. doi:10.1063/1.4952560 North Greenland Ice Core Project members. High-resolution record of Northern Hemisphere climate extending into the last interglacial period. Nature , 431:147–151, 2004. doi:10.1038/nature02805 J. R. Petit, J. Jouzel, D. Raynaud, N. I. Barkov, J.-M. Barnola, I. Basile, M. Bender, J. Chappellaz, M. Davis, G. Delaygue, M. Delmotte, V. M. Kotlyakov, M. Legrand, V. Y. Lipenkov, C. Lorius, L. Pepin, C. Ritz, E. Saltzman and M. Stievenard. Climate and atmospheric history of the past 420,000 years from the Vostok ice core, Antarctica. Nature , 399:429–436, 1999. doi:10.1038/20859 S. Soubeyrand, C. E. Morris and E. K. Bigg. Analysis of fragmented time directionality in time series to elucidate feedbacks in climate data. Environ. Modell. Softw. 61:78–86, 2014. doi:10.1016/j.envsoft.2014.07.003

Inferring linear dependence between time series is central to our understanding of natural and artificial systems. Unfortunately, the hypothesis tests that are used to determine statistically significant directed or multivariate relationships from time-series data often yield spurious associations (Type I errors) or omit causal relationships (Type II errors). This is due to the autocorrelation present in the analysed time series -- a property that is ubiquitous across diverse applications, from brain dynamics to climate change. Here we show that, for limited data, this issue cannot be mediated by fitting a time-series model alone (e.g., in Granger causality or prewhitening approaches), and instead that the degrees of freedom in statistical tests should be altered to account for the effective sample size induced by cross-correlations in the observations. This insight enabled us to derive modified hypothesis tests for any multivariate correlation-based measures of linear dependence between covariance-stationary time series, including Granger causality and mutual information with Gaussian marginals. We use both numerical simulations (generated by autoregressive models and digital filtering) as well as recorded fMRI-neuroimaging data to show that our tests are unbiased for a variety of stationary time series. Our experiments demonstrate that the commonly used $F$- and $\chi^2$-tests can induce significant false-positive rates of up to $100\%$ for both measures, with and without prewhitening of the signals. These findings suggest that many dependencies reported in the scientific literature may have been, and may continue to be, spuriously reported or missed if modified hypothesis tests are not used when analysing time series.

We present algorithms for time-series gene expression analysis that permit the principled estimation of unobserved time-points, clustering, and dataset alignment. Each expression profile is modeled as a cubic spline (piecewise polynomial) that is estimated from the observed data and every time point influences the overall smooth expression curve. We constrain the spline coefficients of genes in the same class to have similar expression patterns, while also allowing for gene specific parameters. We show that unobserved time-points can be reconstructed using our method with 10-15% less error when compared to previous best methods. Our clustering algorithm operates directly on the continuous representations of gene expression profiles, and we demonstrate that this is particularly effective when applied to non-uniformly sampled data. Our continuous alignment algorithm also avoids difficulties encountered by discrete approaches. In particular, our method allows for control of the number of degrees of freedom of the warp through the specification of parameterized functions, which helps to avoid overfitting. We demonstrate that our algorithm produces stable low-error alignments on real expression data and further show a specific application to yeast knockout data that produces biologically meaningful results.

Has the Feldstein-Horioka puzzle waned? Evidence from time series and dynamic panel data analysis