Time Series Forecasting Using Sequence-to-Sequence Deep Learning Framework

Multivariate time series forecasting is a crucial task that predicts the future states based on historical inputs. Although current deep learning-based methods have made significant advancements, they still face the criticism of lacking interpretability. The rise of the Kolmogorov-Arnold Network (KAN) provides a new perspective to implement an efficient and interpretable deep learning-based method for forecasting time series. However, we find there are two main challenges in the application of KAN in time series forecasting: how to select the appropriate one from various KAN variants and how to train the deep KAN-based network. To this end, we propose the multi-layer mixture-of-KAN network, which achieves excellent performance while retaining KAN's ability to be transformed into a combination of symbolic functions. The core module is the mixture-of-KAN layer, which uses a mixture-of-experts structure to assign variables to best-matched KAN experts. Then, we analyze the shortcomings of parameter initialization in the original KAN and provide an effective initialization method to alleviate training instability. Extensive experimental results demonstrate that our proposed method is effective in multivariate time series forecasting. Codes are released in https://github.com/2448845600/EasyTSF.

This work explores the application of state-of-the-art techniques of time series forecasting to the delignification process in the pulp and paper industry, aiming to enhance sustainability and efficiency. While traditional machine learning models, such as Multilayer Perceptron (MLP) and Long Short-Term Memory (LSTM), have been widely used, recent advancements in time series architectures provide significant improvements in prediction accuracy. This study adopts cutting-edge time series architectures and integrates them with explainability techniques (Explainable Artificial Intelligence, XAI) to analyze critical features and their temporal saliency, providing insights into the most influential variables in the delignification process. For a sequence length of 96, Crossformer attains the lowest error of 0.625±0.032, while for a sequence length of 24, LMS-AutoTSF achieves the lowest error of 0.281±0.0001. Additionally, we perform simulation analyses using the identified important features to evaluate the effects of input parameter changes–such as temperature or H-factor adjustments–on correlated variables and the target Kappa number. To model these interdependencies and generate realistic input scenarios, we employ a Conditional Variational Autoencoder (CVAE), which enables the adjustment of one input feature while automatically adapting correlated features in a coherent and interpretable manner. This allows for counterfactual simulations that help operators understand the dynamic impact of process modifications on the overall system behavior. By leveraging advanced forecasting models, XAI-driven feature analysis, and CVAE-based simulation studies, we aim to improve prediction accuracy, optimize resource usage, and enhance operational efficiency. This work underscores the potential of combining modern time series forecasting, explainability techniques, and generative modeling to advance delignification processes, contributing to a more sustainable future for the pulp and paper industry. We further extend our experimental analysis to another industrial pulp dataset comprising a large number of instances, where Crossformer achieves the lowest error of 0.283 ± 0.112, followed by LMS-AutoTSF with the second-lowest error of 0.367 ± 0.136, across average prediction lengths of 12, 24, and 48.

In the present work we extend the traditional C4.5 decision tree method for regression and forecasting of multivariate time series. In the proposed method, time series data is first fuzzified leading to a fuzzy time series (FTS) representation of the data. A fuzzy decision tree (FDT) based on C4.5 is employed to form the knowledge base of the FTS model. The method can deal with high-order and multivariate fuzzy time series, offering an explainable model. The FDT-FTS method is tested with data from IBOVESPA stock market index, which tracks the performance of around 50 most liquid stocks traded on the Sao Paulo Stock Exchange in Brazil. The method is applied to the IBOVESPA mini future contract time series in order to forecast future values using a mix of historical values and technical analysis indicators. This method is compared with Support Vector Regression (SVR) and Random Forest Regression (RFR), both methods implemented in the Scikit-Learn open-source library. The FDT-FTS model was implemented in Python programming language in the open-source pyFTS library. Although all three methods have similar performance, according to the MAPE, SMAPE, RMSE, NRMSE and MAE metrics, the proposed method is computationally faster and explainable.

Time series forecasting is often confronted with multivariate data, but few model is available in this situation. Besides, data distortion aggravates the difficulty to predict multivariate time series. To tackle such problems, we propose an approach based on convolutional neural network with a feature extraction layer added before convolution layer to extract multivariate features and handle multivariate time series data, as well as decreases the effect of distortion by transforming the sample into a denser representation with both its information and the information of its temporal neighbors. A full connection layer then fuses these extracted features and gets the final result. Given that events in the world are always related, using both the target time series and other related time series to forecast the future changes of the target dimension would achieve a better prediction. The proposed approach can process multivariate time series data and is robust to the number of samples, numeric ranges of data etc. Extensive experiments validate the effectiveness of the approach in accomplishing multivariate time series forecasting.

Continual learning of multivariate time series (MTS) forecasting is critical for process industries where working condition drift is frequent due to the variation in the feed properties and other factors. However, existing continual learning methods struggle with dynamic input dimension changes, and the lack of symmetry-aware feature and dimension regulation further exacerbates the interference of irrelevant variables and dimensional inconsistency. To overcome this problem, G-MLoRA, a continual learning method based on dynamic redundancy clustering and multiple low-rank adapters, is proposed in this paper. This method can effectively enhance the network’s capability for prediction of multivariate time series under dynamic input dimensions. First, it groups MTS via Wasserstein distance-K-means clustering to reduce irrelevant variable interference. Second, each group is assigned to an exclusive LoRA adapter, with pre-trained backbone weights frozen during fine-tuning to lower complexity and mitigate catastrophic forgetting. Third, mini-batch gradient accumulation enables reuse of inconsistent-dimensional historical knowledge. Extensive experiments on two real grinding classification datasets show G-MLoRA outperforms baselines in new/historical knowledge compatibility, especially under dynamic dimensions.

Existing multivariate time series forecasting (MTSF) methods have begun to explore architectures built entirely upon Graph Neural Networks (GNNs). Directly connecting variables across different timestamps with a fully connected adjacency matrix introduces unnecessary noise and ignores the inherent temporal ordering between nodes. Besides this, the limited depth of Spatial graph convolutional layers restricts the ability of the model to capture long-term dependencies. We propose <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{S}^{\text{2}}$</tex>-<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{GNN}$</tex>, a multi-scale adaptive graph neural network method, which fusion spatial and spectral graph convolutions. An adaptive mechanism is used to capture the unidirectional relationship between nodes at different timestamps, which enables the construction of a graph structure that preserves the temporal order. Multi-order neighbor information and original node features are fused at each layer of spatial graph convolution to improve the local feature extraction capability of the spatial graph convolution. <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{S}^{\text{2}}$</tex>-GNN introduces spectral graph convolution and applies singular value decomposition (SVD) in the frequency domain to decompose the adjacency matrix and implement spectral graph convolution, which enables the extraction of key structural information and provides a compact representation of global spatio-temporal dependencies. We conduct extensive experiments on 7 public datasets for time series forecasting. The experimental results show that the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{S}^{\text{2}}$</tex>-GNN outperforms typical baseline models in time series forecasting tasks.

LAVARNET: Neural network modeling of causal variable relationships for multivariate time series forecasting

Training Deep Learning (DL) models with missing labels is a challenge in diverse engineering applications. Missing value imputation methods have been proposed to try to address this problem, but their performance is affected with Massive Proportion of Missing Labels (MPML). This paper presents a approach for handling MPML in Multivariate Long-Term Time Series Forecasting. It is an two-step process where interpolation (using Gaussian Processes Regression (GPR) and domain knowledge from experts) and prediction model are separated to enable the integration of prior domain knowledge. First, a set of samples of the possible interpolation of the missing outputs are generated by the GPR based on the domain knowledge. Second, the observed input sensor data and interpolated labels from GPR are used to train the prediction model. We evaluated our approach with the development of a soft-sensor with one real datasets to forecast the biomass during recombinant adeno-associated virus (rAAV) production in bioreactors. Our experimental results demonstrate the potential of the approach through quantitative evaluation of the generated forecasts in a case that would be extremely difficult to train a DL model due to MPML.

Time Series Forecasting plays a critical role in wide range of physical applications, from finance to weather forecasting, climate science, trend analysis and digital communication systems etc. Accurate Forecasting and classification of Data is very crucial for decision making, system development and strategy planning. However, forecasting in these domains is relatively complicated due to the presence of imbalanced datasets like when rare and extreme events must be predicted with precision. These challenges are compounded by interconnection between multiple variables where dependencies across multiple time series can cause complexity that cannot be predicted by simple calculations. Therefore need for an enhanced approach is necessary particularly ones that can handle such intricacies of complex dependencies and imbalance in dataset. This study proposes deep learning algorithms such as LSTM (Long Short Term Memory) networks, Vector AutoRegressive (VAR) Model and block resampling in joint predictor forecast space. The objective is to evaluate these techniques with datasets, analysis and compare their results to significantly advance accuracy for time series forecasting for dynamic and interconnected systems.

Discussion of “Bayesian forecasting of multivariate time series: scalability, structure uncertainty and decisions”

Forecasting of multivariate time-series is an important problem that has applications in traffic management, cellular network configuration, and quantitative finance. A special case of the problem arises when there is a graph available that captures the relationships between the time-series. In this paper we propose a novel learning architecture that achieves performance competitive with or better than the best existing algorithms, without requiring knowledge of the graph. The key element of our proposed architecture is the learnable fully connected hard graph gating mechanism that enables the use of the state-of-the-art and highly computationally efficient fully connected time-series forecasting architecture in traffic forecasting applications. Experimental results for two public traffic network datasets illustrate the value of our approach, and ablation studies confirm the importance of each element of the architecture. The code is available here: https://github.com/boreshkinai/fc-gaga.

Introduction: This study addresses the challenge of concept drift in multivariate time series (MTS) forecasting, where data distributions evolve, degrading model performance. The objective is to propose an adaptive hybrid model that combines Long Short-Term Memory (LSTM) networks with the ADaptive WINdowing (ADWIN) algorithm for effective drift detection and adaptive forecasting in real-time environments. Methods: The proposed ADWIN-LSTM framework integrates Bi-LSTM layers for temporal sequence modelling and ADWIN for monitoring prediction residuals to detect concept drift. A dynamic sliding window mechanism adjusts the training data scope based on drift type. Data preprocessing includes normalization, STL-based detrending, and PCA for dimensionality reduction. Hyperparameters are optimized using grid search. Results: Experiments on real-world and synthetic MTS datasets demonstrate that the proposed model outperforms baseline models (LSTM, GRU, CNN-LSTM, and ADWIN-RF) across RMSE, MAE, MAPE, and R² metrics. The model detects both abrupt and gradual drifts with minimal false positives and low detection delay (≤ 5 steps). Post-drift adaptation significantly improves forecasting accuracy. Discussion: The adaptive retraining strategy triggered by drift detection ensures computational efficiency and robustness in volatile environments. The dynamic integration of forecasting and drift detection enhances model adaptability to evolving data distributions. Conclusion: The ADWIN-LSTM framework effectively combines predictive learning and realtime drift adaptation, making it suitable for dynamic, high-stakes environments such as energy, traffic, and environmental systems. Future work includes online optimization and deployment on streaming platforms.

A multi-granularity hierarchical network for long- and short-term forecasting on multivariate time series data

This article proposes a lightweight deep learning model based on residual thinking, RAR-LSTM (Residual and Regime aware LSTM), to address the difficult issues of high noise, non stationarity, and state switching in economic and financial time series forecasting. This method uses interpretable naive models as baselines (such as seasonal naive method for quarterly forecasting and random walk method for financial daily frequency forecasting), and only learns the residual part through small-scale LSTM, thus balancing stability and reproducibility. At the same time, introducing minimalist state features (such as quarterly dummy variables, high and low rolling volatility states) helps the model perceive changes in the macro or market environment; In terms of loss function, Pinball Loss is supported for quantile prediction to meet the demand for asymmetric underestimation and overestimation costs in practical business. This article validates the generality of RAR-LSTM in two typical tasks: the macro quarterly task is to predict the US unemployment rate, and the financial daily frequency task is to predict the volatility of S&P 500 days. The experimental results show that RAR-LSTM significantly outperforms methods such as naive baseline, traditional LSTM, and SARIMA in terms of prediction accuracy (MAE, RMSE, etc.), especially in high volatility states where it performs more robustly. This model has the characteristics of simple structure, low computational cost, and easy reproducibility, and is suitable for economic and financial forecasting tasks in teaching, competition, and practical scenarios. Beyond naive baselines, we benchmark against stronger statistical models (ARIMA and ETS) and show that RAR-LSTM remains robust across baselines. We also report quantile forecasts (τ=0.1/0.5/0.9) using Pinball Loss, demonstrating calibrated coverage and practical benefits for risk-aware decisions.

This paper introduces a new integrated framework for short-term time series forecasting used as decision logic support in Energy Management Systems (EMS). The proposed framework defines data preprocessing, forecast model training, and adaptation techniques. For data preprocessing, Gaussian and Kalman filtering were used in offline learning on historical data and online (real-time) forecasting. Optimal parametrisation for both filters was selected based on the forecast accuracy. Next, a multiple-input, single-output forecast model based on the Long Short-Term Memory (LSTM) neural network was tuned for each device. The forecast horizon is single-step, and chaining of predictions was used for longer horizons. Finally, the best model was adapted to the changes in the time series data. A significant drop was observed in the forecast accuracy on the past data after adapting the model to the new data using a naive approach. Due to this drop, a novel sample rehearsal selection strategy for adaptation of the model is proposed and compared to other different strategies. All the techniques are demonstrated based on the examples of forecasting multivariate time series for different household devices: a water heater, a refrigerator and an HVAC device. The use of data preprocessing improved the forecast accuracy by up to 56% compared to no preprocessing. Moreover, by introducing a sample rehearsal, knowledge retention was improved by up to 76.6% compared to the naive approach, and by up to 62% compared to the random selection of samples.