Do Transformer Interpretability Methods Transfer to RNNs?

In this article, we propose a new behavioral thermal modeling method for fast building performance analysis, which is critical for energy-efficient smart building control and management. The new approach is based on two recurrent neutral network architecture to obtain the compact nonlinear thermal models for complicated building. We start with a more realistic building simulation program, EnergyPlus, from Department of Energy, to model some practical buildings such as office buildings and data centers. EnergyPlus can model the various time-series inputs to a building such as ambient temperature, heating, ventilation, and air-conditioning (HVAC) inputs, power consumption of electronic equipment, lighting and number of occupants in a room sampled in each hour and produce resulting temperature traces of zones (rooms). In this work, we apply two recurrent neural network (RNN) architectures to build the non-linear compact thermal model of the building: one is non-linear state-space RNN architecture (NLSS), which has global feedbacks, and the other one is Elman's RNN architecture (ELNN), which has local feedbacks in each layer. We give a simple formula to calculate the RNN layer number, layer size to configure RNN architecture to avoid overfitting and underfitting problems. A cross-validation based training technique is further applied to improve predictable accuracy of models. Experimental results from a case study of three buildings show that ELNN and NLSS can both build very accurate building thermal models for the 2-zone and 5-zone building cases: both of them have average errors from around 1% to 1.5% for the two buildings. For the more complex 6-zone building case, ELNN outperforms NLSS with maximum errors 16% against 23%. But both methods have 2.2% average errors.

This paper examines and evaluates the effectiveness of recurrent neural network (RNN) architectures for natural language knowledge. The paper starts by imparting historical data on recurrent neural networks and their software to various responsibilities. It then goes on to talk about diverse RNN architectures, such as lengthy quick-time period reminiscence (LSTM), Gated Recurrent Unit (GRU) and Bonior Recurrent Neural network (BONN). The paper then focuses on the efficacy of each RNN structure for textual content expertise obligations. A sequence of experiments is carried out to degree the overall performance of the one-of-a-kind fashions, the use of 3 preferred datasets-Wiki text, Squad and IMDB, as well as two newly created datasets. The paper analyses the empirical consequences of the various RNN architectures and finds that LSTM fashions are normally satisfactory performers, although BONN architectures can also be aggressive in some cases. The paper additionally looks at numerous techniques to enhance performance, which include pre-processing and hyper-parameter tuning. The paper concludes that, whilst RNN architectures are powerful for herbal language know-how responsibilities, the exact architecture and parameters can also need to be chosen cautiously, relying on the project and dataset to hand.

Recurrent neural networks (RNN) are ubiquitous computing systems for sequences and multivariate time-series data. While several robust RNN architectures are known, it is unclear how to relate RNN initialization, architecture, and other hyperparameters with accuracy for a given task. In this work, we propose treating RNN as dynamical systems and correlating hyperparameters with accuracy through Lyapunov spectral analysis, a methodology designed explicitly for nonlinear dynamical systems. To address the fact that RNN features go beyond the existing Lyapunov spectral analysis, we propose to infer relevant features from the Lyapunov spectrum with an Autoencoder and an embedding of its Latent representation (AeLLE). Our studies of various RNN architectures show that AeLLE successfully correlates RNN Lyapunov spectrum with accuracy. Furthermore, the Latent representation learned by AeLLE is generalizable to novel inputs from the same task and is formed early in the process of RNN training. The latter property allows for predicting the accuracy to which RNN would converge when training is complete. We conclude that the representation of RNN through the Lyapunov spectrum, along with AeLLE, provides a novel method for the organization and interpretation of variants of RNN architectures.

The Open AI language model is a powerful tool for generating AI content. A Large amount of text data is trained through the language model, which can generate new text that is similar in style and tone to the training data. This language model can assist writers in generating high-quality content by offering suggestions and insights for improving language usage, sentence structure, and overall readability. This study represents the development of a content generation tool based on the open AI language model by utilising GPT 3 in the backend as an API to generate the necessary information for the model. With the help of this tool, businesses and individuals can produce high-quality, engaging content more efficiently than ever before. This content generation tool uses a recurrent neural network (RNN) architecture, which enables it to make more accurate predictions than rule-based chatbots. All the features, like Facebook ads, LinkedIn posts, Amazon product descriptions, blogs, company bios, chat bots, and so on, will be presented in the dashboard. This tool is powered by advanced machine learning algorithms that can analyse and understand natural language, allowing them to produce content that is grammatically correct, free of errors, and tailored to specific audiences. They can also help optimize content for search engines, ensuring that it reaches a wider audience and generates more traffic with fine-tuning templates.

Quantifying Gross Primary Production (GPP) is fundamental for understanding terrestrial carbon dynamics, particularly in forests. The overarching question we address here is whether integrating remote sensing (RS) with deep learning (DL) methodologies can enhance the estimation of daily forest GPP on a European scale.The Eddy Covariance (EC) method, although widely used to infer ecosystem-scale estimates of GPP from in situ CO2 exchange measurements, suffers from limited global coverage. When EC data are not available, RS data are often employed to estimate GPP by establishing statistical relationships with in situ observations. Recently, Machine Learning (ML) strategies, particularly involving RS and meteorological inputs, have been used for estimating GPP continuously in space and time.However, the potential of DL techniques, particularly those exploiting the sequential characteristics of time series data (i.e. recurrent neural network architectures) in estimating daily forest GPP has not been comprehensively explored. This includes their performance during photosynthetic downregulation (e.g. during climate extremes such as droughts) and an in-depth examination of the importance of the utilised features.This study presents a comparative analysis of three recurrent neural network architectures—Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs), and Long-Short Term Memory (LSTMs)—for daily forest GPP estimation across ICOS ecosystem stations. We assess their performance throughout the entire year, during the growing season, and under photosynthetic downregulation events. This assessment utilises RS inputs —optical data from Sentinel-2, Land Surface Temperature (LST) from MODIS, and radar data from Sentinel-1—, combined with potential radiation data. Furthermore, we analysed the importance of these features to provide insights into the complex interactions and dependencies when estimating GPP.Our results indicate that all three architectures yield similar accuracy for full period and growing season GPP estimations, with mean NRMSE values of 0.136 and 0.170, respectively. All models exhibit increased errors while estimating GPP during photosynthetic downregulation events and their performance varies notably, with LSTMs showing the best results (NRMSE=0.202), followed by RNNs (NRMSE=0.214), and GRUs exhibiting the least efficacy (NRMSE=0.276). Additionally, our study underscores the importance of potential radiation as a critical feature influencing the GPP response, with LST data proving particularly valuable when estimating GPP during photosynthetic downregulation events, more so than optical or radar data.These findings suggest that recurrent neural network architectures, especially LSTMs, are effective in daily GPP estimations, with slight performance decreases during climate extreme conditions. The study also reveals the efficacy of combining RS data with potential radiation for accurate forest GPP estimation, with LST data being especially crucial when estimating GPP during photosynthetic downregulation events.Our research paves the way for further exploration into recurrent models and innovative architectures, with an emphasis on overcoming the challenges in modelling climate-induced GPP extremes.

In recent years artificial neural networks achieved performance close to or better than humans in several domains: tasks that were previously human prerogatives, such as language processing, have witnessed remarkable improvements in state of the art models. One advantage of this technological boost is to facilitate comparison between different neural networks and human performance, in order to deepen our understanding of human cognition. Here, we investigate which neural network architecture (feed-forward vs. recurrent) matches human behavior in artificial grammar learning, a crucial aspect of language acquisition. Prior experimental studies proved that artificial grammars can be learnt by human subjects after little exposure and often without explicit knowledge of the underlying rules. We tested four grammars with different complexity levels both in humans and in feedforward and recurrent networks. Our results show that both architectures can 'learn' (via error back-propagation) the grammars after the same number of training sequences as humans do, but recurrent networks perform closer to humans than feedforward ones, irrespective of the grammar complexity level. Moreover, similar to visual processing, in which feedforward and recurrent architectures have been related to unconscious and conscious processes, our results suggest that explicit learning is best modeled by recurrent architectures, whereas feedforward networks better capture the dynamics involved in implicit learning.

We present novel methods for analyzing the activation patterns of recurrent neural networks from a linguistic point of view and explore the types of linguistic structure they learn. As a case study, we use a standard standalone language model, and a multi-task gated recurrent network architecture consisting of two parallel pathways with shared word embeddings: The Visual pathway is trained on predicting the representations of the visual scene corresponding to an input sentence, and the Textual pathway is trained to predict the next word in the same sentence. We propose a method for estimating the amount of contribution of individual tokens in the input to the final prediction of the networks. Using this method, we show that the Visual pathway pays selective attention to lexical categories and grammatical functions that carry semantic information, and learns to treat word types differently depending on their grammatical function and their position in the sequential structure of the sentence. In contrast, the language models are comparatively more sensitive to words with a syntactic function. Further analysis of the most informative n-gram contexts for each model shows that in comparison with the Visual pathway, the language models react more strongly to abstract contexts that represent syntactic constructions.

Analysis of surface ozone using a recurrent neural network

In this report, a deep learning network architecture, which combined with the architecture of convolutional neural network, the architecture of recurrent neural network is proposed and implemented to do the text classification with the pre-trained word vectors. The proposed method set the word vector be as a static lookup table without updating, and the network still can ignore the noise which caused by missing words. The experimental results show that the accuracy of this study is consistent with the accuracy of other studies. It is shown the feasibility of this architecture. And has the following advantages: the accuracy rate of this architecture is higher than that of recurrent neural network only; compared with the convolutional neural network, the accuracy results are more stable; and less epoch is used to get stable results. But the shortcoming of this proposed architecture is that the training time will consumed much time.

Recurrent neural networks are particularly useful for processing time sequences and simulating dynamical systems. However, methods for building recurrent architectures have been hindered by the fact that available training algorithms are considerably more complex than those for feedforward networks. In this paper, we present a new method to build recurrent neural networks based on evolutionary computation, which combines a linear chromosome with a two-dimensional representation inspired by Parallel Distributed Genetic Programming (a form of genetic programming for the evolution of graph-like programs) to evolve the architecture and the weights simultaneously. Our method can evolve general asymmetric recurrent architectures as well as specialized recurrent architectures. This paper describes the method and reports on results of its application.

Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models.

Applications like smart homes, security surveillance, healthcare, and workplace safety now depend heavily on Human Activity Recognition (HAR) and monitoring. Automated systems can respond quickly, stop dangerous situations, and provide individualized services when they can precisely identify and categorize human activity. The intricate temporal dependencies present in human motion are frequently difficult to capture by conventional HAR techniques that rely on hand-crafted features and shallow learning models [1]. This paper investigates the effectiveness of a pure Recurrent Neural Network (RNN) architecture for activity recognition using smartphone sensor data, despite the fact that deep learning models like CNNs and LSTMs have dominated recent HAR research. Without the need for manually created features, we suggest a simplified RNN-based framework that can identify temporal dependencies in unprocessed accelerometer and gyroscope data. Pure RNNs can attain competitive accuracy, as shown by experiments on the UCI HAR and WISDM datasets, which yielded results of 92.4% and 90.7%, respectively. The results imply that, when properly optimized, RNNs despite their simplicity remain feasible for HAR tasks.

Funding Information: V. CONCLUSIONS This work proposed four classification models based on LSTM RNNs to classify SIP dialogs. The detection probability was evaluated based on experimental data. To detect abnormal SIP dialogs, we have adopted classification features computed from the output of the LSTM RNN model and two different classification schemes were proposed. A semi-supervised scheme is shown to reach higher performance, achieving a detection probability of 99.45%, thus confirming the effective utility of the proposed methodology to detect abnormal SIP sequences in a short period of time. ACKNOWLEDGEMENTS This work was funded by Fundac¸ão para a Ciência e Tecnologia, under the projects InfoCent-IoT (PTDC/EEI-TEL/30433/2017), CoSHARE (PTDC/EEI-TEL/30709/2017), and RFSense (UIDB/50008/2020).

We explore the architecture of recurrent neural networks (RNNs) by studying the complexity of string sequences it is able to memorize. Symbolic sequences of different complexity are generated to simulate RNN training and study parameter configurations with a view to the network's capability of learning and inference. We compare Long Short-Term Memory (LSTM) networks and gated recurrent units (GRUs). We find that an increase in RNN depth does not necessarily result in better memorization capability when the training time is constrained. Our results also indicate that the learning rate and the number of units per layer are among the most important hyper-parameters to be tuned. Generally, GRUs outperform LSTM networks on low-complexity sequences while on high-complexity sequences LSTMs perform better.

A Contextual Recurrent Collaborative Filtering framework for modelling sequences of venue checkins