Nonlinear classification and regression in underwater sonar applications using convex-complex neural networks

Extreme weather modeling is a challenge for modeling experts in Indonesia and the world. Extreme weather prediction is a complex problem because the chances of it happening are very small, so the developed models often have a low level of accuracy. The purpose of this research is to combine the classic model, Autoregressive Integrated Moving Average (ARIMA), recurrent neural network (RNN) model using Adam and SGD estimation (RNN-Adam and RNN-SGD) with the reLU, tanh, sigmoid and gaussian activation functions. In addition, the ARIMA-RNN mix model was also demonstrated in this study. These models are applied to monthly period extreme weather data obtained from the Meteorology, Climatology and Geophysics Agency (BMKG) of West Sulawesi Province which are converted into training data and test data. The RMSE value is used to see the level of prediction accuracy in both training data and test data. Based on the research results, the best model obtained for modeling Indonesia’s extreme weather is the ARIMA-RNN-Adam mix model with the reLU activation function based on the RMSE value on the training and test data. At n = 50, the smallest RMSE and MSE values of the third model are the ARIMA-RNN-Adam model which is 0.23212 using the reLU activation function, then the ARIMA-RNN-SGD model which is 0.25432 with the same activation function, while the ARIMA value is 0.3270. At n=100 it can be seen that the smallest RMSE and MSE values of the three models are the ARIMA-RNN-Adam model which is equal to 0.25149 using the reLU activation function, then the ARIMA-RNN-SGD model which is equal to 0.25256 with the same activation function, while the ARIMA value is 0.2644.

Artificial neural networks (ANN) have been used in many applications such like handwriting recognition and speech recognition. It is well-known that learning rate is a crucial value in the training procedure for artificial neural networks. It is shown that the initial value of learning rate can confoundedly affect the final result and this value is always set manually in practice. A new parameter called beta stabilizer has been introduced to reduce the sensitivity of the initial learning rate. But this method has only been proposed for deep neural network (DNN) with sigmoid activation function. In this paper we extended beta stabilizer to long short-term memory (LSTM) and investigated the effects of beta stabilizer parameters on different models, including LSTM and DNN with relu activation function. It is concluded that beta stabilizer parameters can reduce the sensitivity of learning rate with almost the same performance on DNN with relu activation function and LSTM. However, it is shown that the effects of beta stabilizer on DNN with relu activation function and LSTM are fewer than the effects on DNN with sigmoid activation function.

Solving mixed-integer optimization problems with embedded neural networks with ReLU activation functions is challenging. Big-M coefficients that arise in relaxing binary decisions related to these functions grow exponentially with the number of layers. We survey and propose different approaches to analyze and improve the run time behavior of mixed-integer programming solvers in this context. Among them are clipped variants of the activation function and regularization techniques applied during training as well as optimization-based bound tightening and a novel scaling for given ReLU networks. We numerically compare these approaches for three benchmark problems from the literature and discuss cost-benefit tradeoffs of the post-processing methods. We use the number of linear regions, the percentage of stable neurons, and overall computational effort as indicators. As a major takeaway we observe and quantify a trade-off between the often desired redundancy of neural network models versus the computational costs for solving related optimization problems.

این مطالعه به منظور بررسی پیش بینی شاخص های عملکرد تولیدی در مرغان تخم گذار با استفاده از شبکه های عصبی مصنوعی و رگرسیون غیرخطی چندگانه انجام شد. بررسی بر روی اطلاعات چهار دوره متوالی پرورش در یک واحد پرورش مرغ تخم گذار صورت گرفت. روش های داده-کاوی شامل رگرسیون خطی و غیر خطی، شبکه عصبی پرسپترون سه لایه، شبکه عصبی پرسپترون چهار لایه و شبکه عصبی با تابع پایه ای شعاعی بود. در این مدل ها از متغیرهای سن گله، میزان خوراک مصرفی و فصل تولید به عنوان متغیر پیشگو و شاخص های عملکرد تولیدی شامل درصد تخم-گذاری، وزن توده ای تخم مرغ تولیدی و ضریب تبدیل غذایی به عنوان متغیر پاسخ استفاده شد. نتایج نهایی رگرسیون های خطی نشان داد که برای تمامی متغیرهای وابسته مورد مطالعه متغیر مستقل سن گله معنی دار می باشد. بنابراین رگرسیون غیر خطی شاخص های عملکرد تولیدی در مقابل سن برای مقایسه با شبکه های عصبی مختلف مورد بررسی قرار گرفت و برای مقایسه کلیه مدل ها از ضریب تعیین (R2) و میانگین قدر مطلق خطا (MAE) استفاده شد. نتایج نشان داد بین شبکه های عصبی مصنوعی مختلف مورد مطالعه، شبکه با تابع پایه ای شعاعی بهتر از سایر مدل های در پیش‌بینی شاخص های عملکرد تولیدی مرغان تخم گذار عمل می کند.

This benchmark suite presents a detailed description of a series of closed-loop control systems with artificial neural network controllers. In many applications, feed-forward neural networks are heavily involved in the implementation of controllers by learning and representing control laws through several methods such as model predictive control (MPC) and reinforcement learning (RL). The type of networks that we consider in this manuscript are feed-forward neural networks consisting of multiple hidden layers with ReLU activation functions and a linear activation function in the output layer. While neural network con- trollers have been able to achieve desirable performance in many contexts, they also present a unique challenge in that it is difficult to provide any guarantees about the correctness of their behavior or reason about the stability a system that employs their use. Thus, from a controls perspective, it is necessary to verify them in conjunction with their corresponding plants in closed-loop. While there have been a handful of works proposed towards the verification of closed-loop systems with feed-forward neural network controllers, this area still lacks attention and a unified set of benchmark examples on which verification techniques can be evaluated and compared. Thus, to this end, we present a range of closed-loop control systems ranging from two to six state variables, and a range of controllers with sizes in the range of eleven neurons to a few hundred neurons in more complex systems.

Convolution neural network takes in input image, assign importance (learnable weights and biases) to various aspects/objects in the image and be able to differentiate one from the other. At each layer, a linear transformation is carried out on the received data. Aiming for solving non-linear problems, a mandatory unit named activation function is used by neural networks. The activation function greatly influences the success of training deep neural networks. In this paper, a new rectified non linear function unit that we call Rectified non-linear unit (ReNU) is proposed. The presented activation function returns x − log(x + 1) for positive values, and zero for negative ones. The ReNU multiplies the received gradient by values between 0 and 1 depending on the importance of neurons ( 1 for the biggest neurons and 0 for the smaller ones) unlike the ReLU that returns the same received gradient for all positive values in its back propagation. Using the proposed activation function the CNN performance remarkably raises. The ReNU has been tested on MNIST and CIFAR-10 data sets and compared to the ReLU and ELU activation functions. The experimental results are satisfactory when compared to known activation functions in terms of convergence speed and CNN accuracy.

SummaryThe present existing weather prediction systems deployed by weather forecasting agencies predict the anomalous events using numerical simulation models developed based on geophysics and oceanographic theories. The major disadvantage with existing models is that are explicit simulation models. So, we try to build a real time prediction system that will automatically predict the intensity of any marine natural disaster events and alert the authorities to take necessary actions using a satellite and observational real time data set including moored met ocean data. We propose a novel hybrid model which is a combination of Convolutional Neural Network (CNN) and Long Short‐Term Memory (LSTM). The paper analyzes various other models such as Gated Recurrent Unit (GRU), Bidirectional LSTM (Bi‐LSTM) Recurrent Neural Network (RNN), and Extreme Learning Machine. In addition the other standard CNN architectures are also analyzed. Since the residual CNN network is the most effective architecture, the paper proposes a modification of the residual CNN network by combining the residual CNN network with LSTM/GRU RNN network along with a proposed new activation function which was experimentally proved to be more effective than the deep learning community standard activation function ReLU. The proposed new activation function overcomes the dead neuron propagation problem in ReLU activation function and hence the model's accuracy is increased by more than 1%. The proposed CNN + LSTM/GRU model is benchmarked with UCR time series benchmark dataset and it is experimentally seen that the model performs better than all existing algorithms.

In order to solve the deep self-coding neural network training process, the Sigmoid function back-propagation gradient is easy to disappear, a method based on ReLU activation function is proposed for training the self coding neural network. This paper analyzes the performance of different activation functions and comparing ReLU with traditional Tanh and Sigmoid activation function and in Reuters-21578 standard for experiments on the test set. The experimental results show that using ReLU as the activation function, not only can improve the network convergence speed, and can also improve the accuracy.

Infrared (IR) images derived from cloudy skies are always too spatially varied for tiny targets to be detected, especially for single-frame detection. Using the neural networks (NN) nonlinear regression, discrimination, and self-leaning capability, an NN-based method is proposed for tiny point target detection in single-frame IR images with high background clutter. First, the background was estimated by an improved NN-based morphologic filter, the structure element of which was optimized by a two-layer NN. Second, noise characteristics were well studied, and thus a two-level segmentation is presented to delete noises as well as to further remove remaining background components. Last, images with several potential targets were fed to a BP NN that predicted the identity of the input, which was either a target or a pseudo-target. It is these two neural networks that separate target from background and pseudo-target, respectively, with different training destinations, thus avoiding the over-training problem. Results on real data indicate that, given the false alarm probability, the detection probability by this method reaches 98.3%, which is improved by 11.02% compared to the traditional approach with fixed SE and without trained NN.

Although for neural networks with locally Lipschitz continuous activation functions the classical derivative exists almost everywhere, the standard chain rule is in general not applicable. We will consider a way of introducing a derivative for neural networks that admits a chain rule, which is both rigorous and easy to work with. In addition we will present a method of converting approximation results on bounded domains to global (pointwise) estimates. This can be used to extend known neural network approximation theory to include the study of regularity properties. Of particular interest is the application to neural networks with ReLU activation function, where it contributes to the understanding of the success of deep learning methods for high-dimensional partial differential equations.

Measuring the similarity or difference of two strings is a fundamental task in computer science and engineering. Among the various measures, Hamming distance and Levenshtein distance have been widely utilized. In this paper, we present neural networks for computing these distances, employing the Rectified Linear Unit function as the activation function. We show that the number of units in the networks is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(n)</i> for the Hamming distance between two strings of length <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i>, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(mn)</i> for the Levenshtein distance between two strings of lengths <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i>. These complexities align with those of traditional algorithms. While the depth of the latter network is large (O(m+n)), we show that the depth can be significantly reduced at the expense of employing a larger number of units. Furthermore, we show that the networks can be modified for learning a string from a set of strings along with their respective distances to the target string. We performed computational experiments to evaluate the potential usufulness of our proposed networks. The results suggest that these networks can efficiently compute distances in a time proportional to the network size, and can proficiently learn a string when utilizing the Hamming distance.

Deep Learning and Beyond: AI’s New Horizons” is an expansive monograph that delves into the dynamic and rapidly evolving field of deep learning, a key area of artificial intelligence. This comprehensive work provides an insightful exploration of the revolutionary developments, foundational concepts, and future directions of deep learning, positioning itself as a crucial resource for understanding this transformative technology. The monograph commences with an introductory chapter that lays the groundwork by elucidating the background of machine learning and chronicling the ascent of deep learning. This foundational perspective is vital for comprehending the underlying principles and methodologies that have propelled deep learning to the forefront of technological innovation. Subsequent chapters delve into the “Historical Evolution” of AI, tracing the origins of neural networks from their early inception to the Renaissance period of deep learning. This historical analysis is pivotal in understanding the field’s trajectory and significant milestones. The “Foundations of Deep Learning” chapter presents an in-depth analysis of neural networks, covering their basic architecture, activation functions, layers, and the critical process of backpropagation and gradient descent. This is followed by examining “Advanced Neural Network Architectures,” where the focus shifts to state-of-the-art models such as convolutional neural networks, recurrent neural networks, and groundbreaking transformers and attention mechanisms. In addressing the “Training Deep Models,” the monograph explores regularization techniques, optimization strategies, and the increasingly significant role of transfer learning and pre-trained models. The application-oriented chapter, “Applications of Deep Learning,” showcases real-world implementations in domains like image recognition, natural language viii processing, and autonomous systems, illustrating deep learning technologies’ vast potential and versatility. The monograph also tackles the “Challenges in Deep Learning,” including overfitting, interpretability, and the ethical considerations surrounding AI deployment. It then ventures into the “Future of Deep Learning,” contemplating emerging trends such as integrating quantum computing, neural architecture search, and the convergence of neuroscience and deep learning. “Case Studies” provide practical insights and empirical evidence of the impact of deep learning. At the same time, the concluding chapter synthesizes the current state of deep learning, offers predictions for the future, and provides reflective thoughts on this transformative field. “Deep Learning and Beyond: AI’s New Horizons” educates and informs readers about the vast potential and future possibilities of deep learning and artificial intelligence. Keywords: Deep Learning, Artificial Intelligence, Neural Networks, Machine Learning, Convolutional Neural Networks, Recurrent Neural Networks, Transformers, Quantum Computing, Ethical AI, Neural Architecture Search, Natural Language Processing, Autonomous Systems, AI in Healthcare, Future of AI, AI Applications, AI Challenges.

Neural networks have emerged as a cornerstone in the field of machine learning, driving significant advancements across various domains such as computer vision, natural language processing, and autonomous systems. This paper explores the fundamental principles of neural networks, including their structural design, activation functions, and training algorithms. Key architectures such as feedforward neural networks, convolutional neural networks, recurrent neural networks, and generative adversarial networks are discussed in detail, highlighting their unique capabilities and applications. The training process of neural networks, involving techniques like backpropagation and gradient descent, is examined alongside methods to enhance performance and prevent overfitting, such as regularization and optimization strategies. This paper also reviews major applications of neural networks, showcasing their impact on image and speech recognition, language translation, and autonomous vehicles. Recent advancements in the field, including the rise of deep learning, improved model explainability, and the development of specialized hardware accelerators, are analyzed to provide insights into current trends and future prospects. The ongoing research in areas like unsupervised learning, few-shot learning, and the integration of neural networks with other AI paradigms is highlighted as a promising avenue for further innovation. By providing a comprehensive overview of neural networks and their applications, this paper underscores their transformative role in advancing machine learning technologies and anticipates future developments that will continue to shape the landscape of artificial intelligence.

A tunable approximate piecewise linear regression model [A. Zaknich, Y. Attikiouzel, December 2000] has been previously developed, which can be adjusted by a single smoothing parameter continuously from the best piecewise linear model in each sub-space to the best approximate piecewise linear model over the whole data space. A suitable value in between ensures that all neighbouring piecewise linear models merge together smoothly at their boundaries. This model was developed by making relatively minor changes to the form of the modified probabilistic neural network [A. Zaknich, July 1998], a network used for general nonlinear regression. The special modified probabilistic neural network (MPNN) structure allows it to be easily used to model a process by appropriately weighting piecewise linear models associated with each piecewise linear models associated with each of the network's radial basis functions, which together cover the data space. The model has now bee further extended to allow each piecewise linear model section to be adapted separately as new data flows through it. By doing this, the proposed adaptive sub-space filter (ASF) model represents a learning/filtering method for nonlinear processes that provides a solution to the stability/plasticity dilemma associated with standard adaptive filters.

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher–student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.