Interpretable Multivariate Time Series Forecasting with Temporal Attention Convolutional Neural Networks

Accurate load forecasting is essential for ensuring safe, stable, and economical operation of energy internet. Temporal convolutional networks (TCNs) have demonstrated superior performance, when compared to recurrent neural network models, since their introduction in electrical load forecasting. However, the current TCN‐based models are unable to obtain a large receptive field and strong long‐time feature extraction capability owing to the specific kernel size of the 1D convolution structure. This paper proposes a temporal inception convolutional network based on multi‐head attention (TICN‐Att) for ultra‐short‐term load prediction. By introducing an inception structure into the TCN, the proposed model can extract multi‐dimensional information from the input features, through the multiple hidden convolutional kernels of different scales, without stacking layers depth‐wise. Simultaneously, by introducing a multi‐head attention mechanism, the TICN‐Att model has a long time‐dependent extraction capability, similar to that of the long short‐term memory network models. The generalization and validity of the model are tested using the global energy forecasting competition (GEFCOM2014) dataset, electrical load data of a city in Jiangsu (China), and PJM power system dataset. The experimental results demonstrated that the proposed model has the best prediction effect, compared to the other state‐of‐the‐art models.

Research on smartphone-based human activity recognition (HAR) is prevalent in the field of healthcare, especially for elderly activity monitoring. Researchers usually propose to use of accelerometers, gyroscopes or magnetometers that are equipped in smartphones as an individual sensing modality for human activity recognition. However, any of these alone is limited in capturing comprehensive movement information for accurate human activity analysis. Thus, we propose a smartphone-based HAR approach by leveraging the inertial signals captured by these three sensors to classify human activities. These heterogeneous sensors deliver information on various aspects of nature, motion and orientation, offering a richer set of features for more accurate representations of the activities. Hence, a deep learning approach that amalgamates long short-term memory (LSTM) in temporal convolutional network (TCN) is proposed. We use independent temporal convolutional networks, coined as temporal convolutional streams, to independently analyse the temporal data of each sensing modality. We name this architecture multi-stream TC-LSTM. The performance of multi-stream TC-LSTM is assessed on the self-collected elderly activity database. Empirical results exhibit that multi-stream TC-LSTM outperforms the existing machine learning and deep learning models, with an F1 score of 98.3 %

Multivariate time-series forecasting is one of the crucial and persistent challenges in time-series forecasting tasks. As a kind of data with multivariate correlation and volatility, multivariate time series impose highly nonlinear time characteristics on the forecasting model. In this paper, a new multivariate time-series forecasting model, multivariate temporal convolutional attention network (MTCAN), based on a self-attentive mechanism is proposed. MTCAN is based on the Convolution Neural Network (CNN) model, using 1D dilated convolution as the basic unit to construct asymmetric blocks, and then, the feature extraction is performed by the self-attention mechanism to finally obtain the prediction results. The input and output lengths of this network can be determined flexibly. The validation of the method is carried out with three different multivariate time-series datasets. The reliability and accuracy of the prediction results are compared with Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Long Short-Term Memory (ConvLSTM), and Temporal Convolutional Network (TCN). The prediction results show that the model proposed in this paper has significantly improved prediction accuracy and generalization.

Locomotion mode recognition has been shown to substantially contribute to the precise control of robotic lower-limb prostheses under different walking conditions. In this study, we proposed a temporal convolutional capsule network (TCCN) which integrates the spatial-temporal-based, dilation-convolution-based, dyna- mic routing and vector-based features for recognizing locomotion mode recognition with small data rather than big-data-based neural networks for robotic prostheses. TCCN proposed in this study has four characteristics, which extracts the (1) spatial-temporal information in the data and then makes (2) dilated convolution to deal with small data, and uses (3) dynamic routing, which produces some similarities to the human brain to process the data as a (4) vector, which is different from other scalar-based networks, such as convolutional neural network (CNN). By comparison with a traditional machine learning, e.g., support vector machine(SVM) and big-data-driven neural networks, e.g., CNN, recurrent neural network(RNN), temporal convolutional network(TCN) and capsule network(CN). The accuracy of TCCN is 4.1% higher than CNN under 5-fold cross-validation of three-locomotion-mode and 5.2% higher under the 5-fold cross-validation of five-locomotion modes. The main confusion we found appears in the transition state. The results indicate that TCCN may handle small data balancing global and local information which is closer to the way how the human brain works, and the capsule layer allows for better processing vector information and retains not only magnitude information, but also direction information.

BackgroundThe prediction of coronavirus disease in 2019 (COVID-19) in broader regions has been widely researched, but for specific areas such as urban areas the predictive models were rarely studied. It may be inaccurate to apply predictive models from a broad region directly to a small area. This paper builds a prediction approach for small size COVID-19 time series in a city.MethodsNumbers of COVID-19 daily confirmed cases were collected from November 1, 2022 to November 16, 2023 in Xuzhou city of China. Classical deep learning models including recurrent neural network (RNN), long and short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN) are initially trained, then RNN, LSTM and GRU are integrated with a new attention mechanism and transfer learning to improve the performance. Ten times ablation experiments are conducted to show the robustness of the performance in prediction. The performances among the models are compared by the mean absolute error, root mean square error and coefficient of determination.ResultsLSTM outperforms than others, and TCN has the worst generalization ability. Thus, LSTM is integrated with the new attention mechanism to construct an LSTMATT model, which improves the performance. LSTMATT is trained on the smoothed time series curve through frequency domain convolution augmentation, then transfer learning is adopted to transfer the learned features back to the original time series resulting in a TLLA model that further improves the performance. RNN and GRU are also integrated with the attention mechanism and transfer learning and their performances are also improved, but TLLA still performs best.ConclusionsThe TLLA model has the best prediction performance for the time series of COVID-19 daily confirmed cases, and the new attention mechanism and transfer learning contribute to improve the prediction performance in the flatten part and the jagged part, respectively.

AC arc faults are one of the most important causes of residential electrical wiring fires, which may produce extremely high temperatures and easily ignite surrounding combustible materials. The global interest in machine learning-based methods for arc fault diagnosis applications is increasing due to continuous challenges in efficiency and accuracy. In this paper, a temporal domain visualization convolutional neural network (TDV-CNN) methodology is proposed. The current transformer and high-speed data acquisition system are used to collect the current of a series of arc faults, then the signal is filtered by a digital filter and converted into a gray image in time sequence before being fed into TDV-CNN. Five different electric loads were selected for experimental validation with various signal characteristics, including vacuum cleaner, fluorescent lamp, dimmer, heater, and desktop computer. The experimental results confirm that the classification accuracy of the five loads’ work states in the ten categories could reach 98.7% or even higher by adjusting parameters perfectly. The methodology is believed to be reliable for series arc detection with relatively high accuracy and also has important potential applications in other fault diagnosis fields.

BackgroundThe dynamic tracking of tumors with radiation beams in radiation therapy requires the prediction of real-time target locations prior to beam delivery, as treatment involving radiation beams and gating tracking results in time latency.ObjectiveIn this study, a deep learning model that was based on a temporal convolutional neural network was developed to predict internal target locations by using multiple external markers.MethodsRespiratory signals from 69 treatment fractions of 21 patients with cancer who were treated with the CyberKnife Synchrony device (Accuray Incorporated) were used to train and test the model. The reported model’s performance was evaluated by comparing the model to a long short-term memory model in terms of the root mean square errors (RMSEs) of real and predicted respiratory signals. The effect of the number of external markers was also investigated.ResultsThe average RMSEs of predicted (ahead time=400 ms) respiratory motion in the superior-inferior, anterior-posterior, and left-right directions and in 3D space were 0.49 mm, 0.28 mm, 0.25 mm, and 0.67 mm, respectively.ConclusionsThe experiment results demonstrated that the temporal convolutional neural network–based respiratory prediction model could predict respiratory signals with submillimeter accuracy.

The accuracy of short-term electricity price forecasting has profound influence on the business decisions made by the participants in the electricity market. Thus, we propose a short-term electricity price combination prediction model based on Complete Ensemble Empirical Mode Decomposition(CEEMD) as well as Temporal Convolutional Attention-based Network(TCAN). First, we defined the correlations among the features according to the Pearson Correlation Analysis and selected the desired features for our model. We then defined the single prediction period to be 24 hours based on the Unit Root test. In addition, we decomposed and denoised the data based on CEEMD and built the attention mechanism into the temporal convolution network. Using the expansion cascade network to extract the relationship between the inputs, combined with the self-attention mechanism to extract the internal information and learn the position dependence, effectively improve the model training speed and prediction accuracy. In order to verify the validity of the model, the actual electricity price data of DK1 in the Nordic electricity market from April to May 2020 were forecasted. The results show that the proposed method can effectively improve the accuracy and reliability of electricity price forecasting, and can provide reliable decision-making basis.

The short term and small-scale sea surface temperature prediction using deep learning has achieved good results. The long-term sea surface temperature prediction technology in large-scale sea area is limited by the large and complex data. So how to use deep learning to select more valuable data and realize high precision of sea surface temperature prediction is an important problem. In this paper, attention mechanism and Temporal convolutional network (TCN) are used to predict the Indian Ocean 40°E–110°E and –25°S–25°N from 2015 to 2018 with 1° × 1° spatial resolution. The attention mechanism is used to distinguish the importance of the data, and the prediction models of full-feature (81 dimensions) and partial-feature (66 dimensions) are constructed. The experimental results show that the fitting degree of partial-feature models to sea surface temperature time series does not decrease significantly. The method proposed in this paper uses less data to ensure that the experimental accuracy does not decline significantly, and improves the long-term sea surface temperature prediction technology in large-scale sea area.

Spoofing speech detection aims to differentiate spoofing speech from natural speech. Frame-based features are usually used in most of previous works. Although multiple frames or dynamic features are used to form a super-vector to represent the temporal information, the time span covered by these features are not sufficient. Most of the systems failed to detect the non-vocoder or unit selection based spoofing attacks. In this work, we propose to use a temporal convolutional neural network (CNN) based classifier for spoofing speech detection. The temporal CNN first convolves the feature trajectories with a set of filters, then extract the maximum responses of these filters within a time window using a max-pooling layer. Due to the use of max-pooling, we can extract useful information from a long temporal span without concatenating a large number of neighbouring frames, as in feedforward deep neural network (DNN). Five types of feature are employed to access the performance of proposed classifier. Experimental results on ASVspoof 2015 corpus show that the temporal CNN based classifier is effective for synthetic speech detection. Specifically, the proposed method brings a significant performance boost for the unit selection based spoofing speech detection.

Bearings play a crucial role in rotating machinery, and their failure significantly impacts performance and safety. Early fault detection is challenging due to weak signals, especially under variable operating conditions. Deep learning techniques, such as Convolutional Neural Networks (CNN) and attention mechanisms, have shown great potential in improving fault detection accuracy in recent years. However, handling complex fault patterns and extracting early fault features remain challenging. To address these issues, this paper proposes a new method that combines Temporal Convolutional Networks (TCN), CNN, and Multi-Head Attention. TCN captures short-term dependencies, CNN extracts long-term features, and Multi-Head Attention enhances focus on key fault-related features. The model was tested on the Case Western Reserve University (CWRU) bearing fault dataset and compared with baseline models, showing a significant improvement in accuracy, achieving 99.3%. This study provides valuable insights for handling complex and multi-fault scenarios, offering essential implications for industrial applications.

In addressing the problem of low prediction accuracy in the remaining useful life (RUL) prediction of rolling bearings, caused by noise interference and insufficient extraction of sensitive features by deep learning models, this paper presents a life prediction method based on signal reconstruction and dual-channel network fusion. First, addressing the issue of extracting weak features from rolling bearing vibration signals, an optimized combination of variational mode decomposition (VMD) and Teager–Kaiser energy operator (TKEO) for signal reconstruction is proposed. TKEO is used to track the energy of high-frequency impulses in the original signal. The sparrow search algorithm is employed for optimizing VMD to perform high-frequency filtering, resulting in an optimized impulse energy signal. This signal is then multiplied with the original signal to enhance weak impulse features under noisy background. Next, a dual-channel network model for RUL prediction is constructed based on temporal convolutional network (TCN) and convolutional neural networks (CNNs). The one-dimensional time-series signal obtained after reconstruction and denoising serves as the input for the TCN network channel. Simultaneously, the signal obtained after reconstruction and denoising undergoes synchrosqueezed wavelet transforms to generate a two-dimensional time–frequency representation as input for the CNN network channel. This setup allows TCN and CNN to respectively extract temporal features from the vibration signal and time–frequency features from the spectrogram. By incorporating self-attention mechanisms, internal correlations between different features are fully explored, thereby enhancing prediction accuracy.

This paper presents an accurate state of health (SOH) estimation algorithm using a temporal convolutional neural network (TCN) for lithium-ion batteries (LIB). With its self-learning ability, this data-driven approach can model the highly non-linear behaviour of LIB due to changes of environment and working conditions all along the battery lifetime. The precise SOH predictions of the TCN are especially needed to ensure a safe and efficient usage of retired electric vehicle batteries within second-life applications. The provided network is trained and tested with data gathered from commercial industry applications in the domain of energy storage. It is shown, that even for dynamic load profiles, the TCN achieves a mean squared error (MSE) of less than 1.0 %. Using this approach, the uncertainty of the heterogeneous performances and characteristics of retired electric vehicle batteries can be drastically reduced.

The prediction of the remaining useful life (RUL) of bearings is of great significance for reducing cost and increasing efficiency of mechanical equipment and ensuring healthy operation. Most of the traditional RUL prediction methods based on deep learning only analyze single sensor data, which makes the prediction accuracy fluctuate greatly and the reliability is low, and the contributions of data collected by different sensors to RUL prediction are inconsistent, which makes the low utilization of data lead to poor prediction results. To solve the above problems, a new RUL prediction framework with convolutional attention mechanism and temporal convolutional network (CAMTCN) is proposed. Firstly, the vibration signals collected by multi-sensor are normalized and processed directly as the input of the network. Secondly, an Efficient Adaptive Shrinkage (EAS) model is designed to adaptively eliminate noise interference and improve the accuracy of RUL prediction. Finally, we use the attention mechanism to design the convolutional attention subnet and combine it with the TCN to obtain the contribution degree of bearing degradation features collected by different sensors. Adaptive weight allocation captures more important degradation features to achieve end-to-end RUL prediction. Experiments were conducted on IEEE PHM 2012 dataset and XJTU-SY dataset respectively, and compared with some advanced in-depth learning methods. The experimental results show that this method can predict RUL more accurately on different bearing data sets, and has good generalization.

Abstract. The present work proposes a prediction model of significant wave height (SWH) and average wave period (APD) based on variational mode decomposition (VMD), temporal convolutional networks (TCNs), and long short-term memory (LSTM) networks. The wave sequence features were obtained using VMD technology based on the wave data from the National Data Buoy Center. Then the SWH and APD prediction models were established using TCNs, LSTM, and Bayesian hyperparameter optimization. The VMD–TCN–LSTM model was compared with the VMD–LSTM (without TCN cells) and LSTM (without VMD and TCN cells) models. The VMD–TCN–LSTM model has significant superiority and shows robustness and generality in different buoy prediction experiments. In the 3 h wave forecasts, VMD primarily improved the model performance, while the TCN had less of an influence. In the 12, 24, and 48 h wave forecasts, both VMD and TCNs improved the model performance. The contribution of the TCN to the improvement of the prediction result determination coefficient gradually increased as the forecasting length increased. In the 48 h SWH forecasts, the VMD and TCN improved the determination coefficient by 132.5 % and 36.8 %, respectively. In the 48 h APD forecasts, the VMD and TCN improved the determination coefficient by 119.7 % and 40.9 %, respectively.