Comparative Analysis on variants of Neural Networks: An Experimental Study

The Sixth International Symposium on Neural Networks (ISNN 2009)

Deep learning (DL) has shown explosive growth in its application to bioinformatics and has demonstrated thrillingly promising power to mine the complex relationship hidden in large-scale biological and biomedical data. A number of comprehensive reviews have been published on such applications, ranging from high-level reviews with future perspectives to those mainly serving as tutorials. These reviews have provided an excellent introduction to and guideline for applications of DL in bioinformatics, covering multiple types of machine learning (ML) problems, different DL architectures, and ranges of biological/biomedical problems. However, most of these reviews have focused on previous research, whereas current trends in the principled DL field and perspectives on their future developments and potential new applications to biology and biomedicine are still scarce. We will focus on modern DL, the ongoing trends and future directions of the principled DL field, and postulate new and major applications in bioinformatics.

Machine learning is the study of algorithms and models for computing systems to do tasks based on pattern identification and inference. When it is difficult or infeasible to develop an algorithm to do a particular task, machine learning algorithms can provide an output based on previous training data. A well-known machine learning model is deep learning. The most recent deep learning models are based on artificial neural networks (ANN). There exist several types of artificial neural networks including the feedforward neural network, the Kohonen self-organizing neural network, the recurrent neural network, the convolutional neural network, the modular neural network, among others. This article focuses on convolutional neural networks with a description of the model, the training and inference processes and its applicability. It will also give an overview of the most used CNN models and what to expect from the next generation of CNN models.

Deep learning (also known as deep structured learning, hierarchical learning, or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and nonlinear transformations. Deep learning has been characterized as a class of machine learning algorithms with the following characteristics [257]: • They use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised). • They are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher-level features are derived from lower-level features to form a hierarchical representation. • They are part of the broader machine learning field of learning representations of data. • They learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts. These definitions have in common: multiple layers of nonlinear processing units and the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks (DBN), and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition, and bioinformatics where they have been shown to produce state-of-the-art results on various tasks. In this chapter we start in Section 7.1 with an introduction, giving a brief history of this field, the relevant literature, and its applications. Then we study some basic concepts of deep learning such as convolutional neural networks, recurrent neural networks, backpropagation algorithm, restricted Boltzmann machines, and deep learning networks in Section 7.2. Then we illustrate three examples for Apache Spark implementation for mobile big data (MBD), user moving pattern extraction, and combination with nonparametric Bayesian learning, respectively, in Sections 7.3 through 7.5. Finally, we have summary in Section 7.6.

The study presents the results of the development and testing of deep learning neural network architectures, which demonstrate high accuracy rates in classifying neurophysiological data, in particular hemodynamic brain activation patterns obtained by functional near-infrared spectroscopy, during solving mathematical problems on spatial-numerical associations. The analyzed signal represents a multidimensional time series of oxyhemoglobin and deoxyhemoglobin dynamics. Taking the specificity of the fNIRS signal into account, a comparative analysis of 2 types of neural network architectures was carried out: (1) architectures based on recurrent neural networks: recurrent neural network with long short-term memory, recurrent neural network with long short-term memory with fully connected layers, bidirectional recurrent neural network with long short-term memory, convolutional recurrent neural network with long short-term memory; (2) architectures based on convolutional neural networks with 1D convolutions: convolutional neural network, fully convolutional neural network, residual neural network. Trained long short-term memory recurrent neural network architectures showed worse results in accuracy in comparison with 1D convolutional neural network architectures. Residual neural network (model_Resnet) demonstrated the highest accuracy rates in three experimental conditions more than 88% in detecting age-related differences in brain activation during spatial-numerical association tasks considering the individual characteristics of the respondents’ signal.

The adoption of wind energy has grown significantly in recent years. New, cost-effective technologies have been developed, led by customer awareness of green technologies and a legal framework proposed at the European Union level. The stochastic nature of wind speed is transferred to wind turbine output, making wind energy difficult to predict. The main scope of predicting wind energy production is to be proactive in balancing and reserving energy to meet demand. When the prediction identifies a potential gap between supply and demand, additional energy from other sources must be generated and supplied. Creating a synergy of physical devices through advanced sensing capabilities, software, storage and analytics capabilities, the Industrial Internet of Things is enabling the effective transition to wind energy through automation by removing many of the disadvantages in a way that has recently become accessible. This research focuses on the data analytics, proposing a fast univariate network-based approach for wind energy prediction, using Feed Forward Neural Networks, Recurrent Neural Networks, Long-Short Term Memory, Gated Recurrent Unit, and Convolutional Neural Networks. Moreover, by introducing the theoretical fundamentals, the implementation method and the hyperparameters of the final models, this article becomes unique in the context of wind energy. At the time of this study, no prior research studies have presented a direct comparison between feedforward, recurrent, and convolutional neural networks ? these being the most important in the field of supervised learning.

Deep learning approaches for bias correction in WRF model outputs for enhanced solar and wind energy estimation: A case study in East and West Malaysia

In the field of big data analytics, deep learning has garnered a lot of interest due to the fact that it is capable of doing feature extraction and classification. The categorization of big data has traditionally been carried out by academics via the use of Machine Learning methods; nonetheless, the process of feature extraction has traditionally been carried out manually. Deep learning is a method developed by researchers that allows for the extraction of features via the use of various algorithms. This research made use of an algorithm known as "Convolutional Neural Network," or CNN for short, in order to get a deeper comprehension of the dependability of big data analytics. Primary research has been conducted in order to get an understanding of how hidden nodes and layers influence the accuracy of this neural network. The "Recurrent Neural Network" (also known as RNN) and the "Artificial Neural Network" (also known as ANN) are the two other neural network algorithms that have been selected for comparison with CNN in order to evaluate whether or not CNN is more effective than the other algorithms. In the first research, the reliability of CNN as well as other methods was analyzed in relation to the number of nodes, the number of hidden layers, the amount of training time, and the amount of validating time. Control variables were employed in both correlation and regression investigations. These variables included Hidden Nodes, Training time, Valid time, and Secret Layers. CNN, ANN, and RNN were significant variables. According to the findings, CNN is more dependable than other neural networks for analyzing huge amounts of data, scoring 92%, whereas other neural networks scored less than 90%. The inclusion of hidden nodes results in a significant increase in CNN's reliability.

Electroencephalography (EEG) datasets are often small and high dimensional, owing to cumbersome recording processes. In these conditions, powerful machine learning techniques are essential to deal with the large amount of information and overcome the curse of dimensionality. Artificial Neural Networks (ANNs) have achieved promising performance in EEG-based Brain-Computer Interface (BCI) applications, but they involve computationally intensive training algorithms and hyperparameter optimization methods. Thus, an awareness of the quality-cost trade-off, although usually overlooked, is highly beneficial. In this paper, we apply a hyperparameter optimization procedure based on Genetic Algorithms to Convolutional Neural Networks (CNNs), Feed-Forward Neural Networks (FFNNs), and Recurrent Neural Networks (RNNs), all of them purposely shallow. We compare their relative quality and energy-time cost, but we also analyze the variability in the structural complexity of networks of the same type with similar accuracies. The experimental results show that the optimization procedure improves accuracy in all models, and that CNN models with only one hidden convolutional layer can equal or slightly outperform a 6-layer Deep Belief Network. FFNN and RNN were not able to reach the same quality, although the cost was significantly lower. The results also highlight the fact that size within the same type of network is not necessarily correlated with accuracy, as smaller models can and do match, or even surpass, bigger ones in performance. In this regard, overfitting is likely a contributing factor since deep learning approaches struggle with limited training examples.

Predicting the category of tumors and the types of the cancer in its early stage remains as a very essential process to identify depth of the disease and treatment available for it. The neural network that functions similar to the human nervous system is widely utilized in the tumor investigation and the cancer prediction. The paper presents the analysis of the performance of the neural networks such as the, FNN (Feed Forward Neural Networks), RNN (Recurrent Neural Networks) and the CNN (Convolutional Neural Network) investigating the tumors and predicting the cancer. The results obtained by evaluating the neural networks on the breast cancer Wisconsin original data set shows that the CNN provides 43 % better prediction than the FNN and 25% better prediction than the RNN.

Recent experience in applying recurrent neural networks (RNNs) to interpreting Permannt Downhole Gauge (PDG) records has highlighted the potential utility of machine learning algorithms to learn reservoir behavior from data. The power of the RNN resides in its ability to retain information in a form of memory of previous patterns and information contained in the previous behavior of phenomena being modeled. This memory plays a role of informing the decision at the present time by using what happened in the past. This property suggests RNNs as a suitable choice to model sequences of reservoir information, even when the reservoir modeler is faced with incomplete knowledge of the underlying physical system. Convolutional Neural Networks (CNNs) are another variant of machine learning algorithms that have shown promise in sequence modeling domains, such as audio synthesis and machine translation. In this study, RNN and CNN were applied to tasks that traditionally would be modeled by a reservoir simulator. This was achieved by formulating the relationship between physical quantities of interest from subsurface reservoirs as a sequence mapping problem. In addition, the performance of a CNN layer as compared to RNN was evaluated systematically to investigate their capabilities in a variety of tasks of interest to the reservoir engineer. Preliminary results suggest that CNNs, with specific design modifications, are as capable as RNNs in modeling sequences of information and as reliable when making inferences to cases that has not been seen by the algorithm during training. Design details and reasons pertaining to the way these two seemingly different architectures process information, and handle memory are also discussed.

A hybrid model based on neural networks for biomedical relation extraction

Recent approaches based on artificial neural networks (ANNs) have shown promising results for short-text classification. However, many short texts occur in sequences (e.g., sentences in a document or utterances in a dialog), and most existing ANN-based systems do not leverage the preceding short texts when classifying a subsequent one. In this work, we present a model based on recurrent neural networks and convolutional neural networks that incorporates the preceding short texts. Our model achieves state-of-the-art results on three different datasets for dialog act prediction.

Early detection and treatment are crucial for recovery from cancer. Due to rapid advancement in medical imaging techniques, today doctors can scan various body parts from different perspectives to diagnose cancer or see the progress during a course of treatment. Such techniques are less painful and require less preparation and recovery. The recent evolution of AI, particularly deep learning algorithms have shown potential in computer-aided diagnosis of cancer. These include analysis of cancer images for detection of tumor region, prognosis, skin tissue inspection, etc. Convolutional Neural Network (CNN) based image processing techniques are showing promising results in the diagnosis of breast cancer, skin cancer, prostate cancer, lung cancer, etc. Literature shows many other CNN-based models such as feed-forward neural network, Deep CNN, Multi-Crop CNN, fully convolutional networks (FCNs), Deep Fully Convolutional Network (DFCNet), Recurrent neural networks, etc. The CNN focus solely on the image-specific features and hence require a lesser number of input parameters. A CNN model reduces the size of the input image vector without losing the features critical for making an accurate prediction. This makes it easier to process and reduces computational complexity. CNNs are also not sensitive to the object positioning in an image since the approximate position of image motifs relative to others is known. CNN deals with the variations in images through augmentation and other pre-processing techniques. This makes them robust to various straining conditions in images. This chapter presents an overview of a generic CNN architecture followed by a discussion on major advances in CNN models and their applications for cancer image analysis.

In this study, we investigate the effectiveness of various deep learning algorithms for customer segmentation in the banking sector, aiming to enhance targeted service delivery and customer experience. We employ a comprehensive pipeline encompassing data collection, preprocessing, feature selection, feature extraction, model development, and rigorous evaluation. Our dataset, derived from real-world banking customer profiles, was processed using normalization, encoding, and dimensionality reduction techniques. We implemented and compared eight models: Artificial Neural Network (ANN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), CNN-LSTM Hybrid, Autoencoder-Based Network, and Deep Neural Network (DNN). Among them, the Autoencoder-Based model achieved the highest accuracy of 91.56%, outperforming others in terms of segmentation clarity and computational efficiency. These findings suggest that deep learning methods, particularly Autoencoder-Based architectures, offer robust solutions for real-time banking customer segmentation, enabling institutions to tailor products and services more effectively.