A study of Knowledge Distillation in Fully Convolutional Network for Time Series Classification

Time Series Classification (TSC) problems are encountered in many real life data mining tasks ranging from medicine and security to human activity recognition and food safety. With the recent success of deep neural networks in various domains such as computer vision and natural language processing, researchers started adopting these techniques for solving time series data mining problems. However, to the best of our knowledge, no previous work has considered the vulnerability of deep learning models to adversarial time series examples, which could potentially make them unreliable in situations where the decision taken by the classifier is crucial such as in medicine and security. For computer vision problems, such attacks have been shown to be very easy to perform by altering the image and adding an imperceptible amount of noise to trick the network into wrongly classifying the input image. Following this line of work, we propose to leverage existing adversarial attack mechanisms to add a special noise to the input time series in order to decrease the network's confidence when classifying instances at test time. Our results reveal that current state-of-the-art deep learning time series classifiers are vulnerable to adversarial attacks which can have major consequences in multiple domains such as food safety and quality assurance.

Training deep neural networks often requires careful hyper-parameter tuning and significant computational resources. In this paper, we propose ConvTimeNet (CTN): an off-the-shelf deep convolutional neural network (CNN) trained on diverse univariate time series classification (TSC) source tasks. Once trained, CTN can be easily adapted to new TSC target tasks via a small amount of fine-tuning using labeled instances from the target tasks. We note that the length of convolutional filters is a key aspect when building a pre-trained model that can generalize to time series of different lengths across datasets. To achieve this, we incorporate filters of multiple lengths in all convolutional layers of CTN to capture temporal features at multiple time scales. We consider all 65 datasets with time series of lengths up to 512 points from the UCR TSC Benchmark for training and testing transferability of CTN: We train CTN on a randomly chosen subset of 24 datasets using a multi-head approach with a different softmax layer for each training dataset, and study generalizability and transferability of the learned filters on the remaining 41 TSC datasets. We observe significant gains in classification accuracy as well as computational efficiency when using pre-trained CTN as a starting point for subsequent task-specific fine-tuning compared to existing state-of-the-art TSC approaches. We also provide qualitative insights into the working of CTN by: i) analyzing the activations and filters of first convolution layer suggesting the filters in CTN are generically useful, ii) analyzing the impact of the design decision to incorporate multiple length decisions, and iii) finding regions of time series that affect the final classification decision via occlusion sensitivity analysis.

Time Series Classification (TSC) is an important and challenging problem in data mining. With the increase of time series data availability, hundreds of TSC algorithms have been proposed. Among these methods, only a few have considered Deep Neural Networks (DNNs) to perform this task. This is surprising as deep learning has seen very successful applications in the last years. DNNs have indeed revolutionized the field of computer vision especially with the advent of novel deeper architectures such as Residual and Convolutional Neural Networks. Apart from images, sequential data such as text and audio can also be processed with DNNs to reach state-of-the-art performance for document classification and speech recognition. In this article, we study the current state-of-the-art performance of deep learning algorithms for TSC by presenting an empirical study of the most recent DNN architectures for TSC. We give an overview of the most successful deep learning applications in various time series domains under a unified taxonomy of DNNs for TSC. We also provide an open source deep learning framework to the TSC community where we implemented each of the compared approaches and evaluated them on a univariate TSC benchmark (the UCR/UEA archive) and 12 multivariate time series datasets. By training 8,730 deep learning models on 97 time series datasets, we propose the most exhaustive study of DNNs for TSC to date.

Deep learning is a complex machine learning algorithm that can learn the deep inner rules of a sample. In recent years, deep learning has achieved good results in the field of image processing. The essence of deep learning is to build a multi-layered machine learning architecture that trains on large-scale data. With the development of computer technology, deep learning has achieved good results in image recognition and natural language processing. Deep learning is also becoming more and more widely used in industry. This paper introduces the applications of deep learning in the industrial field, and finally discusses development and challenges of deep learning in industrial applications in the future.

As a fundamental task in modern data mining, time series classification is powering mission-critical tasks including stock price prediction and network traffic analysis. Due to the non-linear structure of deep neural networks (DNN), deep learning has established as a promising solution to time series classification. However, the excessive learning capacity of DNNs may make them prone to threats of backdoor attacks, where an attacker embeds hidden functionalities (i.e., backdoor) to DNNs and activates the backdoor by specially-designed inputs (i.e., triggers). Despite extensive studies concerning backdoor attacks on image and text domains, there is little known about the vulnerability of DNN based time series classifiers against backdoor attacks. Due to the unique characteristics of time series data, most existing backdoor attack techniques fail to threaten time series classifiers. In this paper, through analyzing the key factors which influence the effectiveness of a backdoor, we systematize a list of practical principles for designing triggers on time series data. In this light, we propose a novel framework called TimeTrojan, which aims to learn to form the trigger pattern through a constrained multi-objective optimization. To solve the hereafter challenging optimization issue, we further design an iterative learning algorithm. Remarkably, the proposed framework is agnostic to a wide range of DNN classifiers. Extensive empirical results on 6 representative DNN classifiers and 6 real-world datasets validate the effectiveness of the proposed attack framework. In most cases, TimeTrojan successfully injects backdoors with 100% attack success rate without affecting the model accuracy on clean samples, which implies the complete control of the behavior of the DNN classifiers by the adversary.

Time series classification (TSC) is a crucial task across various domains, and its performance heavily depends on the quality of input representations. Among various representations, the recurrence plot (RP) effectively captures topological recurrence, the unique property of time series data. However, conventional convolutional neural networks (CNNs) cannot fully exploit this property since they treat the RP as grid-like data. In this study, we propose RP-GCN, a novel approach that uses a graph convolutional network (GCN) to exploit topological recurrence inherent in the RP, thereby improving TSC performance. Our method transforms a multivariate time series into graphs where state matrices act as node feature matrices and RPs serve as adjacency matrices, enabling graph convolution to utilize recurrence relationships. We evaluated RP-GCN on 35 benchmark multivariate time series classification datasets and demonstrated superior accuracy and efficient inference time compared to existing methods.

Classification of time series is an essential requirement of various applications that demand continuous monitoring of dynamical systems such as industrial process and health care monitoring. Feature extraction plays a vital role in deciding performance of the time series models. In recent years deep learning techniques have shown an excellent performance to extract highly discriminating features for the classification of the time series. A Convolutional Neural Network (CNN) is a unified framework that performs the feature learning and classification tasks simultaneously. Using the CNN to perform the classification of the multivariate time series is still a challenging task. In this paper, we propose an attention-based multivariate convolutional neural network (AT-MVCNN) that consists of the attention feature-based input tensor scheme to encode informations across the multiple time stamps. The method is capable of learning the temporal characteristics of the multivariate time series. The efficacy of the proposed method is tested on Human Activity Recognition (HAR) and Occupancy Detection datasets. The experiments and results show the proposed method outperforms the other deep learning and traditional machine learning models.

In recent years Neural Networks and especially Deep Neural Networks (DNN) have seen a rise in popularity. DNNs have increased the overall performance of algorithms in applications such as image recognition and classification, 2D and 3D pose detection, natural language processing, and time series forecasting. Especially in image classification and recognition, so-called Convolutional Neural Network (CNN) have gained high interest as they can reach a high accuracy, which makes them a viable solution in cancer detection or autonomous driving. As CNNs are widely used in image tasks, different visualizations techniques have bee developed to show how their internals are working. Apart from image tasks, CNNs are applicable to other problems, e.g., time series classification, time series forecasting, or natural language processing. CNNs in those contexts behave similarly, allowing for the same visualization techniques to make them more interpretable. In this chapter, we adapt image visualization algorithms to time series problems, allowing us to build granular, intuitively interpretable feature hierarchies to make a time series forecast as understandable as an image recognition task. We do so by using our previous work on power time series forecasting using CNN Auto Encoders (AEs) and applying typical CNN visualization techniques to it. Thus, we guid computer scientists to provide better interpretable figures for a time series forecasting task to application domain experts.KeywordsVisualizationConvolutional neural networksTime series data

Deep learning techniques have recently demonstrated remarkable success in numerous domains. Typically, the success of these deep learning models is measured in terms of performance metrics such as accuracy and mean average precision (mAP). Generally, a model's high performance is highly valued, but it frequently comes at the expense of substantial energy costs and carbon footprint emissions during the model building step. Massive emission of CO2 has a deleterious impact on life on earth in general and is a serious ethical concern that is largely ignored in deep learning research. In this article, we mainly focus on environmental costs and the means of mitigating carbon footprints in deep learning models, with a particular focus on models created using knowledge distillation (KD). Deep learning models typically contain a large number of parameters, resulting in a 'heavy' model. A heavy model scores high on performance metrics but is incompatible with mobile and edge computing devices. Model compression techniques such as knowledge distillation enable the creation of lightweight, deployable models for these low-resource devices. KD generates lighter models and typically performs with slightly less accuracy than the heavier teacher model (model accuracy by the teacher model on CIFAR 10, CIFAR 100, and TinyImageNet is 95.04%, 76.03%, and 63.39%; model accuracy by KD is 91.78%, 69.7%, and 60.49%). Although the distillation process makes models deployable on low-resource devices, they were found to consume an exorbitant amount of energy and have a substantial carbon footprint (15.8, 17.9, and 13.5 times more carbon compared to the corresponding teacher model). The enormous environmental cost is primarily attributable to the tuning of the hyperparameter, Temperature (τ). In this article, we propose measuring the environmental costs of deep learning work (in terms of GFLOPS in millions, energy consumption in kWh, and CO2 equivalent in grams). In order to create lightweight models with low environmental costs, we propose a straightforward yet effective method for selecting a hyperparameter (τ) using a stochastic approach for each training batch fed into the models. We applied knowledge distillation (including its data-free variant) to problems involving image classification and object detection. To evaluate the robustness of our method, we ran experiments on various datasets (CIFAR 10, CIFAR 100, Tiny ImageNet, and PASCAL VOC) and models (ResNet18, MobileNetV2, Wrn-40-2). Our novel approach reduces the environmental costs by a large margin by eliminating the requirement of expensive hyperparameter tuning without sacrificing performance. Empirical results on the CIFAR 10 dataset show that the stochastic technique achieves an accuracy of 91.67%, whereas tuning achieves an accuracy of 91.78%-however, the stochastic approach reduces the energy consumption and CO2 equivalent each by a factor of 19. Similar results have been obtained with CIFAR 100 and TinyImageNet dataset. This pattern is also observed in object detection classification on the PASCAL VOC dataset, where the tuning technique performs similarly to the stochastic technique, with a difference of 0.03% mAP favoring the stochastic technique while reducing the energy consumptions and CO2 emission each by a factor of 18.5.

In recent years, plant leaf diseases has become a widespread problem for which an accurate research and rapid application of deep learning in plant disease classification is required, tomato is also one of the most important plants and seeds which are used worldwide for cooking in either dried or fresh form, tomato are a great source of protein that offer many health benefits, but there are a lot of diseases associated with tomato leaf which hinder its production. Thus, an accurate classification of tomato leaf diseases is needed to solve the problem in the early stage. A deep learning approach is proposed to identify and classify leaf disease by using public dataset of leaf image and CNN model with the open source library TensorFlow. In this project, we proposed a method to classify tomato leaf disease and to find and describe the efficient network architecture (hyper parameters and optimization methods). Moreover, after applying each architecture separately, we compared their obtained results to find out the best architecture configuration for classifying tomato leaf diseases and their results. Furthermore, to satisfy the classification requirements, the model was trained using CNN architecture check if we could get faster training times, higher accuracy and easier retraining. Deep learning is a branch of artificial intelligence. In recent years, with the advantages of automatic learning and feature extraction, it has been widely concerned by academic and industrial circles. It has been widely used in image and video processing, voice processing, and natural language processing. At the same time, it has also become a research hotspot in the field of agricultural plant protection, such as plant disease recognition. The application of deep learning in plant disease recognition can avoid the disadvantages caused by artificial selection of disease spot features, make plant disease feature extraction more objective, and improve the research efficiency and technology transformation speed. This review provides the research progress of deep learning technology in the field of crop leaf disease identification in recent years. In this project, we present the current trends and challenges for the detection of plant leaf disease using deep learning and advanced imaging techniques. We hope that this project will be a valuable resource for researchers who study the detection of plant diseases. At the same time, we also discussed some of the current challenges and problems that need to be resolved.

With the explosive growth of computer science and big data, quantitative finance is undergoing a huge shift from traditional econometrics to data-driven Artificial Intelligence (AI). This study aims to review the applications of Machine Learning (ML), Deep Learning (DL), and Reinforcement Learning (RL) in vital areas such as asset pricing, algorithmic trading and risk management. Research shows that Deep Neural Networks (DNNs) capture non-linear market patterns, Natural Language Processing (NLP) analyses unstructured data while providing superior sentiment signals compared to generic alternatives. Deep Reinforcement Learning (DRL) can effectively automate execution in dynamic market environments, which optimizes trading decisions. Similarly, Temporal Fusion Transformers (TFT) have emerged as a dominant architecture for multi-horizon time series forecasting, offering superior interpretability and accuracy over standard recurrent networks. These ML models greatly outperform traditional methods like CAPM and ARIMA in prediction accuracy and handling complex data. However, there are still challenges about the lack of interpretability ("black box") and overfitting. In the future, quantitative finance lies in combining "Explainable AI" (XAI).

Deep learning has been successfully explored in addressing different multimedia topics recent years, ranging from object detection, semantic classification, entity annotation, to multimedia captioning, multimedia question answering and storytelling. Open source libraries and platforms such as Tensorflow, Caffe, MXnet significantly help promote the wide deployment of deep learning in solving real-world applications. On one hand, deep learning practitioners, while not necessary to understand the involved math behind, are able to set up and make use of a complex deep network. One recent deep learning tool based on Keras even provides the graphical interface to enable straightforward 'drag and drop' operation for deep learning programming. On the other hand, however, some general theoretical problems of learning such as the interpretation and generalization, have only achieved limited progress. Most deep learning papers published these days follow the pipeline of designing/modifying network structures - tuning parameters - reporting performance improvement in specific applications. We have even seen many deep learning application papers without one single equation. Theoretical interpretation and the science behind the study are largely ignored. While excited about the successful application of deep learning in classical and novel problems, we multimedia researchers are responsible to think and solve the fundamental topics in deep learning science. Prof. Guanrong Chen recently wrote an editorial note titled 'Science and Technology, not SciTech' [1]. This panel falls into similar discussion and aims to invite prestigious multimedia researchers and active deep learning practitioners to discuss the positioning of deep learning research now and in the future. Specifically, each panelist is asked to present their opinions on the following five questions: 1)How do you think the current phenomenon that deep learning applications are explosively growing, while the general theoretical problems remain slow progress? 2)Do you agree that deployment of deep learning techniques is getting easy (with a low barrier), while deep learning research is difficult (with a high barrier) 3)What do you think are the core problems for deep learning techniques? 4)What do you think are the core problems for deep learning science? 5)What's your suggestion on the multimedia research in the post-deep learning era?

Most neurodegenerative diseases such as Alzheimer's and Parkinson's are life-threatening, critical, and incurable affecting mainly the elderly population. Early diagnosis is challenging as disease phenotype is very crucial for predicting, preventing the progression, and effective drug discovery. In the last few years, Deep learning (DL) based neural networks are the state-of-the-art models deployed in industries and academics across different areas like natural language processing, image analysis, speech recognition, audio classification, and many more. It has been slowly realized that they have a high potential in medical image analysis and diagnostics and medical management in general. As this field is vast and expanding rapidly, we have put focused on existing DL-based models to detect Alzheimer's and Parkinson's in particular. This study gives a summary of related medical examinations for these diseases. Frameworks and applications of many deep learning models have been discussed. We have given precise notes on pre-processing techniques used by various studies for MRI image analysis. An overview of the application of DL-based models in different stages of medical image analysis has been conferred. It has been realized from the review that more studies are focused on Alzheimer's compared to Parkinson's disease. Additionally, we have tabulated the various public datasets available for these diseases. We have highlighted the potential use of a novel biomarker for the early diagnosis of these disorders. Also, some challenges and issues in implementing deep learning techniques for the detection of these diseases have been addressed. Finally, we concluded with some directions for future research regarding deep learning in the diagnosis of these diseases.

In the past decade, time series data have been generated from various fields at a rapid speed, which offers a huge opportunity for mining valuable knowledge. As a typical task of time series mining, Time Series Classification (TSC) has attracted lots of attention from both researchers and domain experts due to its broad applications ranging from human activity recognition to smart city governance. Specifically, there is an increasing requirement for performing classification tasks on diverse types of time series data in a timely manner without costly hand-crafting feature engineering. Therefore, in this paper, we propose a framework named Edge4TSC that allows time series to be processed in the edge environment, so that the classification results can be instantly returned to the end-users. Meanwhile, to get rid of the costly hand-crafting feature engineering process, deep learning techniques are applied for automatic feature extraction, which shows competitive or even superior performance compared to state-of-the-art TSC solutions. However, because time series presents complex patterns, even deep learning models are not capable of achieving satisfactory classification accuracy, which motivated us to explore new time series representation methods to help classifiers further improve the classification accuracy. In the proposed framework Edge4TSC, by building the binary distribution tree, a new time series representation method was designed for addressing the classification accuracy concern in TSC tasks. By conducting comprehensive experiments on six challenging time series datasets in the edge environment, the potential of the proposed framework for its generalization ability and classification accuracy improvement is firmly validated with a number of helpful insights.

The well-being of residents is a top priority for megacities, which is why urban design and sustainable development are crucial topics. Quality of Life (QoL) is used as an effective key performance index (KPI) to measure the efficiency of a city plan’s quantity and quality factors. For city dwellers, QoL for pedestrians is also significant. The walkability concept evaluates and analyzes the QoL in a walking scene. However, the traditional questionnaire survey approach is costly, time-consuming, and limited in its evaluation area. To overcome these limitations, the paper proposes using artificial intelligence (AI) technology to evaluate walkability data collected through a questionnaire survey using virtual reality (VR) tools. The proposed method involves knowledge extraction using deep convolutional neural networks (DCNNs) for information extraction and deep learning (DL) models to infer QoL scores. Knowledge distillation (KD) is also applied to reduce the model size and improve real-time performance. The experiment results demonstrate that the proposed approach is practical and can be considered an alternative method for acquiring QoL.