Comprehensive Deep Recurrent Artificial Neural Network (CDRANN): Evolutionary Model for Future Prediction

Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mechanism concept can help RNNs remember and forget previous information. Representing the hidden layers of an RNN with more expressive operations (i.e., tensor products) helps it learn a more complex relationship between the current input and the previous hidden layer information. These ideas can generally improve RNN performances. In this paper, we proposed a novel RNN architecture that combine the concepts of gating mechanism and the tensor product into a single model. By combining these two concepts into a single RNN, our proposed models learn long-term dependencies by modeling with gating units and obtain more expressive and direct interaction between input and hidden layers using a tensor product on 3-dimensional array (tensor) weight parameters. We use Long Short Term Memory (LSTM) RNN and Gated Recurrent Unit (GRU) RNN and combine them with a tensor product inside their formulations. Our proposed RNNs, which are called a Long-Short Term Memory Recurrent Neural Tensor Network (LSTMRNTN) and Gated Recurrent Unit Recurrent Neural Tensor Network (GRURNTN), are made by combining the LSTM and GRU RNN models with the tensor product. We conducted experiments with our proposed models on word-level and character-level language modeling tasks and revealed that our proposed models significantly improved their performance compared to our baseline models.

MOOCs (Massive Open Online Courses) are definitely one of the best approach to support the international agenda about inclusive and equitable education and lifelong learning opportunities for all (SDG4) [1]. A great deal universities and institutions offer valuable free courses to their numerous students and to people around the word through MOOC platforms. However, because of huge number of learners and data generated, learner’s behaviour in those platforms remain a kind of black box for learners themselves and for courses instructors who are supposed to tutor or monitor learners in the learning process. Therefore, learner do not receive sufficient support from instructors and from their peers, during the learning process [2]. This is one the main reasons that lead to high dropout, low completion and success rate observed in the MOOCs. Many research work have suggested descriptive, predictive and prescriptive models to address this issue, but most of these models focus on predicting dropout, completion and/or success, and do not generally pay enough attention to one of the key step (learner behaviour), that comes before, and can explain dropping out and failure. Our research aims to develop a deep learning model to predict learner behaviour (learner interactions) in the learning process, in order to equip learners and course instructors with insight understanding of the learner behaviour in the learning process. This specific paper will focus on analysing relevant algorithms to develop such model. For this analysis, we used data from UNESCO-IICBA (International Institute for Capacity Building in Africa) MOOC platform, designed for teacher training in Africa, and then we examine many types of features: geographical, social behavioural and learning behavioural features. Learner’s behaviour being a time series Big data, we built the predictive model using Deep Learning algorithms for better performance and accuracy (Thanks to the power of deep learning) compared to baseline Machine learning algorithms. Time series data is best handled by recurrent neural networks (RNN) [3], so, we choose RNN and implemented/tested the three main architectures of RNN: Simple RNNs, GRU (Gated Recurrent Unit) RNNs and LSTM (Long short-term memory) RNNs. The models were trained using L2 Regularization, based on the predictions results, we unexpectedly found model with simple RNNs produced the best performance and accuracy on the dataset used than the other RNN architectures. We had couple of observations, example: we saw a correlation between video viewing and quiz behaviour and the participation of the learner to the learning process. This observation could allow teachers to provide meaningful support and guidance to at risk or less active students. We also observed that, the shorter the video or the quiz, the more the viewer. We conclude that we could use learner video or quiz viewing behaviour to predict his behaviour concerning other MOOC contents. These results suggest the need of deeper research on educational video and educational quiz designing for MOOCs.

In this study, we investigate the effectiveness of various deep learning algorithms for customer segmentation in the banking sector, aiming to enhance targeted service delivery and customer experience. We employ a comprehensive pipeline encompassing data collection, preprocessing, feature selection, feature extraction, model development, and rigorous evaluation. Our dataset, derived from real-world banking customer profiles, was processed using normalization, encoding, and dimensionality reduction techniques. We implemented and compared eight models: Artificial Neural Network (ANN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), CNN-LSTM Hybrid, Autoencoder-Based Network, and Deep Neural Network (DNN). Among them, the Autoencoder-Based model achieved the highest accuracy of 91.56%, outperforming others in terms of segmentation clarity and computational efficiency. These findings suggest that deep learning methods, particularly Autoencoder-Based architectures, offer robust solutions for real-time banking customer segmentation, enabling institutions to tailor products and services more effectively.

Capturing long-range interactions between structural but not sequence neighbors of proteins is a long-standing challenging problem in bioinformatics. Recently, long short-term memory (LSTM) networks have significantly improved the accuracy of speech and image classification problems by remembering useful past information in long sequential events. Here, we have implemented deep bidirectional LSTM recurrent neural networks in the problem of protein intrinsic disorder prediction. The new method, named SPOT-Disorder, has steadily improved over a similar method using a traditional, window-based neural network (SPINE-D) in all datasets tested without separate training on short and long disordered regions. Independent tests on four other datasets including the datasets from critical assessment of structure prediction (CASP) techniques and >10 000 annotated proteins from MobiDB, confirmed SPOT-Disorder as one of the best methods in disorder prediction. Moreover, initial studies indicate that the method is more accurate in predicting functional sites in disordered regions. These results highlight the usefulness combining LSTM with deep bidirectional recurrent neural networks in capturing non-local, long-range interactions for bioinformatics applications. SPOT-disorder is available as a web server and as a standalone program at: http://sparks-lab.org/server/SPOT-disorder/index.php . j.hanson@griffith.edu.au or yuedong.yang@griffith.edu.au or yaoqi.zhou@griffith.edu.au. Supplementary data is available at Bioinformatics online.

<p dir="ltr">Recurrent Neural Networks (RNNs) are a major class of Artificial Neural Networks (ANNs). Suitable network architecture is vital for RNNs to achieve high performance. However, designing the architecture of deep RNNs (DRNNs) is a complicated and time-consuming process. Each layer of a DRNN contains numerous hyper-parameters. It is challenging to optimize the hyper-parameters of all layers jointly to achieve the best possible learning performance. Trial-and-error methods for designing DRNNs have been proven to be laborious and highly costly in practice. Therefore, an efficient automatic architecture search technique is needed to design DRNNs.</p><p dir="ltr">Genetic Algorithm (GA) is a popular evolutionary computation based approach. Existing research works already explored the use of GAs in designing ANNs. But, most of the existing approaches focus on the design of fixed-depth ANNs. However, it is not easy to fix the depth of DRNNs in advance, since different problems need DRNNs of different depths. Hence, the potential of GA has to be further explored to evolve the architecture of DRNNs of varying depths.</p><p dir="ltr">The primary goal of this thesis is to develop advanced GA approaches for designing Long Short Term Memory (LSTM) based DRNN architectures of varying depths effectively and efficiently.</p><p dir="ltr">Firstly, this thesis proposes a GA-based algorithm called Two-Stage Surrogate-Assisted GA (TS-SA-GA) with a progressive incremental strategy to design the architecture of LSTM networks of varying depths. The progressive approach can effectively extend well-designed shallow networks to high performing deep networks. Moreover, the new GA-based algorithm adopts newly designed knowledge-driven crossover and mutation operators to identify and repair LSTM network designs affected by inappropriate use of activation functions, thereby significantly reducing the chances for GA to evolve hard-to-train LSTM network architectures. Furthermore, this thesis proposes a two-stage surrogate method to predict the trainability and fitness of LSTM networks. This improves the efficiency and effectiveness of the GA-based algorithm to evolve LSTM networks.</p><p dir="ltr">Secondly, this thesis proposes a new algorithm called Evolutionary Design of LSTM Ensembles (ED-LSTM-Ensemble) based on multi-objective optimization techniques to evolve LSTM networks and ensembles simultaneously to directly utilize the ensemble performance to drive the evolution of base LSTM networks. Additionally, this thesis proposes a connection weight inheritance strategy to evolve LSTM networks and ensembles efficiently and effectively.</p><p dir="ltr">Thirdly, this thesis proposes a new algorithm called Skip Connections through Evolutionary and Differential Architecture search (SCEDA) to design LSTM networks together with appropriate skip connections automatically. Designing LSTM networks together with skip connections is highly challenging. This thesis hybridize gradient-based DAS and evolutionary architecture search to optimize the architecture of LSTM networks with suitable skip connections in a single evolutionary process to improve the efficiency and effectiveness of the search process.</p><p dir="ltr">Finally, this thesis conducts an in-depth empirical analysis of the impact of activation functions on the trainability of LSTM networks based on the concept of Edge Of Chaos. The analysis reveals the strong interrelation between the activation functions across multiple layers of an LSTM network and the trainability of the network. The analysis also highlights the importance of controlling activation functions to avoid generating untrainable LSTM networks. On the basis of the analysis, this thesis proposes a machine learning model to guide the use of activation functions in LSTM networks.</p>

This article explores the suitability of a long short-term memory recurrent neural network (LSTM-RNN) and artificial intelligence (AI) method for low-flow time series forecasting. The long short-term memory works on the sequential framework which considers all of the predecessor data. This forecasting method used daily discharged data collected from the Basantapur gauging station located on the Mahanadi River basin, India. Different metrics [root-mean-square error (RMSE), Nash–Sutcliffe efficiency (ENS), correlation coefficient (R) and mean absolute error] were selected to assess the performance of the model. Additionally, recurrent neural network (RNN) model is also used to compare the adaptability of LSTM-RNN over RNN and naive method. The results conclude that the LSTM-RNN model (R = 0.943, ENS = 0.878, RMSE = 0.487) outperformed RNN model (R = 0.935, ENS = 0.843, RMSE = 0.516) and naive method (R = 0.866, ENS = 0.704, RMSE = 0.793). The finding of this research concludes that LSTM-RNN can be used as new reliable AI technique for low-flow forecasting.

Trajectory-level fog detection based on in-vehicle video camera with TensorFlow deep learning utilizing SHRP2 naturalistic driving data

While the growing accessibility of technology improves usability, it also creates more opportunities for cybercriminals to exploit vulnerabilities, significantly accelerating the proliferation of cybersecurity attacks. Deep learning (DL) approaches present significant advancements over conventional machine learning (ML) techniques by automating feature selection and extraction while minimizing external dependencies. This study proposes a deep learning-based model to enhance cyberattack detection and ensure that data security goals are achieved. This study employs a quantitative research design, utilizing simulation and modeling as the primary analytical tools. The dataset used is the Canadian Institute for Cybersecurity Intrusion Detection System (CIC-IDS-2017) dataset. Three distinct DL algorithms are used to design the detection models, namely, Deep Neural Network (DNN), Long Short-Term Memory (LSTM), and Recurrent Neural Network (RNN). The performance comparison metrics are F1-score, accuracy, sensitivity, false positive rate (FPR), specificity, positive predictive value (PPV), false negative rate (FNR), and negative predictive value (NPV). Optimization concepts are integrated to enhance the detection efficiency in web-based systems, including loss functions, gradient-based optimization, and efficient model generalization techniques. The results of k-fold cross-validation show LSTM’s higher scores for F1-score (94.6%), recall (94.7%), accuracy (94.8%), and precision (94.6%). LSTM outperformed RNN and DNN, achieving the highest accuracy, precision, specificity, and sensitivity at 94.7%, 94.3%, 98.9%, and 94.7% respectively, validating LSTM's superior generalizability for cyberattack detection tasks.

The reliability of Industrial Control Systems (ICS) is crucial given their substantial importance in society and business. These systems are susceptible to physical and cyber-attacks, which can seriously affect human life and the economy. Given the growing prevalence of Internet of Things (IoT) technologies and the possibility of cyber warfare, it is essential to prioritize safeguarding Industrial Control Systems (ICS) from cyberattacks. Previous studies revealed an imbalance in the ICS datasets. As a result, models didn't do well in minority classes but did well in majority classes, which made the intrusion detection system (IDS) less useful. The key objective is to provide insights into the normal functioning of the system and the disruptions produced by cyberattacks. In this study, we introduced an interdisciplinary framework that aims to enhance network intrusion detection systems (NIDSs). In this framework, we introduced an IDS via feature selection and feature reduction technique(s) with the attention-driven lightweight deep neural networks: Deep Recurrent Neural Networks (RNN), Deep Long Short-Term Memory (LSTM), and Deep Bi-directional Long Short-Term Memory (Bi-LSTM). Several feature selection techniques exclude features that fail to match the specified criteria. We employed Sparse Principal Component Analysis (SPCA) to extract higher-order features. We conducted experiments on the three datasets: the Secure Water Treatment System (SWaT), the Water Distribution (WADI), and the Gas Heating Loop (GHL).Among the models used in the framework, the attention-driven Deep LSTM model produced better results than the others, revealing lower training and testing times for the three datasets. In terms of precision, recall, F1-score, computational speed, and ability to work with larger datasets and different ICSs, the proposed framework is better than previous methods in detecting cyberattacks. This highlights how useful it is in the real world.

Internet of Things (IoT) technology has been incorporated into the majority of people’s everyday lives and places of employment due to the quick development in information technology. Modern agricultural techniques increasingly use the well-known and superior approach of managing a farm known as “smart farming”. Utilizing a variety of information and agricultural technologies, crops are observed for their general health and productivity. This requires monitoring the condition of field crops and looking at many other indicators. The goal of smart agriculture is to reduce the amount of money spent on agricultural inputs while keeping the quality of the final product constant. The Internet of Things (IoT) has made smart agriculture possible through data collection and storage techniques. For example, modern irrigation systems use effective sensor networks to collect field data for the best plant irrigation. Smart agriculture will become more susceptible to cyber-attacks as its reliance on the IoT ecosystem grows, because IoT networks have a large number of nodes but limited resources, which makes security a difficult issue. Hence, it is crucial to have an intrusion detection system (IDS) that can address such challenges. In this manuscript, an IoT-based privacy-preserving anomaly detection model for smart agriculture has been proposed. The motivation behind this work is twofold. Firstly, ensuring data privacy in IoT-based agriculture is of the utmost importance due to the large volumes of sensitive information collected by IoT devices, including on environmental conditions, crop health, and resource utilization data. Secondly, the timely detection of anomalies in smart agriculture systems is critical to enable proactive interventions, such as preventing crop damage, optimizing resource allocation, and ensuring sustainable farming practices. In this paper, we propose a privacy-encoding-based enhanced deep learning framework for the difficulty of data encryption and intrusion detection. In terms of data encoding, a novel method of a sparse capsule-auto encoder (SCAE) is proposed along with feature selection, feature mapping, and feature normalization. An SCAE is used to convert information into a new encrypted format in order to prevent deduction attacks. An attention-based gated recurrent unit neural network model is proposed to detect the intrusion. An AGRU is an advanced version of a GRU which is enhanced by an attention mechanism. In the results section, the proposed model is compared with existing deep learning models using two public datasets. Parameters such as recall, precision, accuracy, and F1-score are considered. The proposed model has accuracy, recall, precision, and F1-score of 99.9%, 99.7%, 99.9%, and 99.8%, respectively. The proposed method is compared using a variety of machine learning techniques such as the deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), and long short-term memory (LSTM).

Abstract: Stock market forecasting remains a major concern to investors, financial institutions, and traders. Most classical forecasting techniques such as statistical modelling and technical analysis generally fail to account for the complexity and non-linearity of financial markets. Recently, neural networks, a new-age branch of machine learning, have emerged as a powerful weapon for better forecasting in the stock market. Neural networks, especially deep learning models such as Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM), are most suited to learning complex patterns from the historical data and reveal hidden correlations or relationships between different variables which have not been captured by the traditional models. The article discusses the role of neural networks in predicting stock market trends and the applications of these models in financial forecasting. The fundamentals of neural networks are explored, their special applications in stock market prediction, and the advantages they confer over traditional techniques. The objective of this article is to also provide insights into key obstacles, including data quality, overfitting, and interpretability, which can impinge upon the effective use of neural networks in finance. Case studies and simulated situations help present neural networks' use, where they are positioned to enhance stock market prediction output with accuracy and reliability. It seems that future directions that research in this field will take would involve integration with different alternative data sources, with quantum computing anticipated to change the landscape of financial prediction models. The insights provided... Keywords: Neural Networks, Stock Market Predictions, Machine Learning, Financial Forecasting, Trend Analysis, Deep Learning, Predictive Analytics, Data Sciences, and Algorithmic Trading.

Prediction of dissolved oxygen in a fishery pond based on gated recurrent unit (GRU)

Deep learning based multi-labelled soil classification and empirical estimation toward sustainable agriculture

This study investigates the application of advanced deep learning models to forecast fossil energy prices, a critical factor influencing global economic stability. Unlike previous research, this study conducts a comparative analysis of Gated Recurrent Unit (GRU), Recurrent Neural Network (RNN), Bidirectional Long Short-Term Memory (Bi-LSTM), Long Short-Term Memory (LSTM), Convolutional Neural Network (CNN), and Deep Neural Network (DNN) models. The evaluation metrics employed include Root Mean Squared Error (RMSE) and Mean Absolute Percentage Error (MAPE). The results reveal that recurrent architectures, particularly GRU, LSTM, and Bi-LSTM, consistently outperform feedforward and convolutional models, demonstrating superior ability to capture temporal dependencies and nonlinear dynamics in energy markets. In contrast, the RNN and DNN show relatively weaker generalization capabilities. Additionally, visualizations of actual versus predicted prices for each model further emphasize superior forecasting accuracy of recurrent models. The results highlight the potential of deep learning in enhancing investment and policy decisions. Additionally, the results provide significant implications for policymakers and investors by emphasizing the value of accurate energy price forecasting in mitigating market volatility, improving portfolio management, and supporting evidence-based energy policies.

Historical information in the Canadian Maritime Judiciary increases with time because of the need to archive data to be utilized in case references and for later application when determining verdicts for similar cases. However, such data are typically stored in multiple systems, making its reachability technical. Utilizing technologies like deep learning and sentiment analysis provides chances to facilitate faster access to court records. Such practice enhances impartial verdicts, minimizing workloads for court employees, and decreases the time used in legal proceedings for claims during maritime contracts such as shipping disputes between parties. This paper seeks to develop a sentiment analysis framework that uses deep learning, distributed learning, and machine learning to improve access to statutes, laws, and cases used by maritime judges in making judgments to back their claims. The suggested approach uses deep learning models, including convolutional neural networks (CNNs), deep neural networks, long short-term memory (LSTM), and recurrent neural networks. It extracts court records having crucial sentiments or statements for maritime court verdicts. The suggested approach has been used successfully during sentiment analysis by emphasizing feature selection from a legal repository. The LSTM + CNN model has shown promising results in obtaining sentiments and records from multiple devices and sufficiently proposing practical guidance to judicial personnel regarding the regulations applicable to various situations.