Introduction to Neural Networks

An artificial neural network(ANN) is a type of computing system that mimics the functioning of the neural networks in a biological brain. In this paper, an ANN has been simulated on an Field Programmable Gate Array (FPGA) for the application of handwritten digit recognition using the sigmoid and hyperbolic tangent(tanh) activation functions. Both software and hardware simulations have been carried out, using python and Verilog HDL respectively. The two activation functions were implemented on an Artix-7 FPGA. A comparison has been made between the sigmoid and tanh activation functions based on speed, accuracy, and hardware required, and it has been inferred that the tanh function is best for the application of handwritten digit recognition as it has 3% higher accuracy and uses 5 less LUTs than the sigmoid activation function.

In recurrent neural networks such as the long short-term memory (LSTM), the sigmoid and hyperbolic tangent functions are commonly used as activation functions in the network units. Other activation functions developed for the neural networks are not thoroughly analyzed in LSTMs. While many researchers have adopted LSTM networks for classification tasks, no comprehensive study is available on the choice of activation functions for the gates in these networks. In this paper, we compare 23 different kinds of activation functions in a basic LSTM network with a single hidden layer. Performance of different activation functions and different number of LSTM blocks in the hidden layer are analyzed for classification of records in the IMDB, Movie Review, and MNIST data sets. The quantitative results on all data sets demonstrate that the least average error is achieved with the Elliott activation function and its modifications. Specifically, this family of functions exhibits better results than the sigmoid activation function which is popular in LSTM networks.

This research mainly based on multilayer perceptron (MLP) neural networks technique of data mining to forecast the wheat crop yield at the district level. There are many statistical and simulation models available, but the proposed algorithm with new activation function provides promising results in a shorter time with more accuracy. Sigmoid and hyperbolic tangent activation functions are widely used in the neural network. The activation functions play an important role in the neural network learning algorithm. The main objective of the proposed work is to develop an amended MLP neural network with new activation function and revised random weights and bias values for crop yield estimation by using the different weather parameter datasets. MLP model has been tested by existing activation functions and newly created activation functions with different cases including weights and bias values. In this research study, we evaluate the result of different activation functions and recommend some new simple activation functions, named DharaSig, DharaSigm and SHBSig, to improve the performance of neural networks and accurate results. Also, three new activation functions created with little variations in the DharaSig function named DharaSig1, DharaSig2 and DharaSig3. In this research study, variable numbers of hidden layers are tested with the variable number of neurons per hidden layer for the agriculture dataset. Variable values of momentum, seed and learning rate are also used in this study. Experiments show that newly created activation functions provide better results compared to ‘sigmoid’ default neural network activation function for agriculture datasets.

This paper presents a FPGA based approach for a modular architecture of Fuzzy Neural Networks (FNN) to embed with easily different topologies set up. The project is based on a Takagi - Hayashi (T-H) method for the construction and tuning of fuzzy rules, this is commonly referred as neural network driven fuzzy reasoning. The proposed architecture approach consists of two main configurable modules: a Multilayer Perceptron - MLP with sigmoidal activation function that composes the first module to determine a Fuzzy membership function; the second employs an MLP with pure linear activation function to define the consequents. The DSPBuilder® software along the Simulink® is used to connect, set and synthesize the Fuzzy Neural Network desired. Other hardware components employed in the architecture proposed cooperate to the system modularity. The system was tested and validated through a control problem and an interpolation problem. Several papers proposed different hardware architecture to implement hybrid systems by using Fuzzy logic and Neural Network. However, there is no approach with this specific neural network driven fuzzy reasoning by T-H method and the aim to be embedded. The Self-Organizing Map (SOM) and Levenberg-Marquardt backpropagation were used to train the FNN proposed off-line.

An Artificial Neural Network (ANN) works by creating connections between different processing elements (artificial neurons). ANNs have been extensively used for Data Mining, which extracts hidden patterns and valuable information from large databases. This paper introduces a new adaptive Higher Order Neural Network (HONN) model and applies it in data mining tasks such as determining breast cancer recurrences and predicting incomes base on census data. An adaptive hyperbolic tangent function is used as the neuron activation function for the new adaptive HONN model. The paper compares the new HONN model against a Multi-Layer Perceptron (MLP) with the sigmoid activation function, an RBF Neural Network with the Gaussian activation function, and a Recurrent Neural Network (RNN) with the sigmoid activation function. Experimental results show that the new adaptive HONN model offers several advantages over conventional ANN models such as better generalisation capabilities as well as abilities in handling missing values in a dataset.

Multilevel image segmentation with adaptive image context based thresholding

Polynomial activation neural networks: Modeling, stability analysis and coverage BP-training

This paper shows that the performance of the Hopfield network for solving optimization problems can be improved by using a new activation (output) function. The effects of the activation function on the performance of the Hopfield network are analyzed. It is shown that the sigmoid activation function in the Hopfield network is sensitive to noise of neurons. The reason is that the sigmoid function is most sensitive in the range where noise is most predominant. A new activation function that is more robust against noise is proposed. The new activation function has the capability of amplifying the signals between neurons while suppressing noise. The performance of the new activation function is evaluated through simulation. Compared with the sigmoid function, the new activation function reduces the error rate of tour length by 30.6% and increases the percentage of valid tours by 38.6% during simulation on 200 randomly generated city distributions of the 10-city traveling salesman problem.

In this paper, a combined approach of Principal Component Analysis (PCA)-based Extreme Learning Machine (ELM) for boiler output forecasting in a thermal power plant is presented. The input used for this prediction model is taken from the boiler unit of the Yermarus Thermal Power Station (YTPS), India. Calculation of the accurate electrical output of a boiler in an operating system requires the knowledge of hundreds of operating parameters. The dimensionality of the input dataset is reduced by applying principal component analysis using IBM@SPSS Software. In the process of principal component analysis, a dataset of 232 parameters is standardized into 16 principal components. The total dataset collected is divided into training and testing datasets. The extreme learning machine is designed for various activation functions and the number of neurons. Sigmoid and hyperbolic tangent activation functions are studied here. Its generalization performance is examined in terms of the Mean Square Error (MSE), Mean Absolute Error (MAE), Root Mean Square (RMSE), and Mean Absolute Percentage Error (MAPE). ELM and PCA–ELM are compared. In both the ELM and PCA–ELM models, when the extreme learning machine was designed with a sigmoid activation function with 100 nodes in the hidden layer, RMSE was 5.026 and 4.730, respectively. Therefore, the developed combined approach of PCA–ELM proved as a promising technique in forecasting with reduced errors and reduced time.

Introduction. The article considers the solution of the problem of detecting fires at technological facilities in automatic mode. To solve this problem, an approach was chosen to create a convolutional neural network capable of operating on a real-time video stream. Aims and Purposes. The aim of this work is to create a neural network capable of detecting flames and smoke in the image from CCTV cameras. The purposes of the work include: selection of the optimal architecture of the neural network in accordance with the latest research in this field; speeding up the chosen architecture using quantization and filter thinning techniques. Methods. Different architectures of convolutional neural networks performing the task of detecting objects in an image are considered. Their performance and quality of work are compared. The YOLOv5 architecture, its target function, training methods and ways of speeding up work are considered. Results and discu ssion. The paper shows the results of training a convolutional neural network of the YOLOv5 architecture for the task of flame and smoke detection, as well as how the results change when applying neural network acceleration methods. It was determined that the use of such acceleration methods such as quantization and filter cleaning can significantly increase up the speed of the neural network, while almost no loss in accuracy of operation. Conclusions. As a result of the conducted work, the architecture of the neural network was determined for performing the task of detecting a fire source. Based on the chosen architecture, a neural network was trained to detect flames and smoke in the image. The speed of its work allows to process video stream in real-time without using graphic accelerator.

The design of neural network models involves numerous complexities, including the determination of input vectors, choosing the number of hidden layers and their computational units, and specifying activation functions for the latter. The combinatoric possibilities are daunting, yet experience has yielded informal guidelines that can be useful. Alternatively, current research on genetic algorithms (GA) suggests that they might be of practical use as a formal method of determining ‘good’ architectures for neural networks. In this paper, we use a genetic algorithm to find effective architectures for backpropagation neural networks (BP). We compare the performance of heuristically designed BP networks with that of GA-designed BP networks. Our test domains are sets of problems having compensatory, conjunctive, and mixed-decision structures. The results of our experiment suggest that heuristic methods produce architectures that are simpler and yet perform comparatively well. © 1998 John Wiley & Sons, Ltd.

Artificial neural networks (ANN) have been used in many applications such like handwriting recognition and speech recognition. It is well-known that learning rate is a crucial value in the training procedure for artificial neural networks. It is shown that the initial value of learning rate can confoundedly affect the final result and this value is always set manually in practice. A new parameter called beta stabilizer has been introduced to reduce the sensitivity of the initial learning rate. But this method has only been proposed for deep neural network (DNN) with sigmoid activation function. In this paper we extended beta stabilizer to long short-term memory (LSTM) and investigated the effects of beta stabilizer parameters on different models, including LSTM and DNN with relu activation function. It is concluded that beta stabilizer parameters can reduce the sensitivity of learning rate with almost the same performance on DNN with relu activation function and LSTM. However, it is shown that the effects of beta stabilizer on DNN with relu activation function and LSTM are fewer than the effects on DNN with sigmoid activation function.

Medical imaging is a technique to get images of the human body for medical science or clinical purposes. Segmentation of a medical image is a challenging task to isolate the suspicious region from the complex medical images. Genetic algorithms (GAs) are an effective tool to handle the problem of medical image segmentation. In this chapter, an application of color magnetic resonance (MR) brain image segmentation is presented by the parallel optimized multilevel sigmoidal (ParaOptiMUSIG) activation function with the parallel self-organizing neural network (PSONN) architecture. Not only confined within this approach, color MR brain image is also segmented by the NSGA-II-based ParaOptiMUSIG activation function to incorporate the multiple objective function-based scenario. These methods are compared with the process of color MR brain image segmentation by the MUSIG activation function with the PSONN architecture. All the methods are applied on a real-life color MR brain image and the quality of the segmented images are accessed by four standard objective functions. The comparison shows that the ParaOptiMUSIG activation function-based method and the NSGA-II-based ParaOptiMUSIG activation function-based method perform better than the MUSIG activation function-based method to segment the color MR brain image.

Neural networks have been widely applied in system dynamics modeling. However, traditional neural networks still encounter limitations in capturing long-term dynamics, nonlinear modeling, and interpretability. To address these challenges, this study proposes a novel neural network architecture, Deep Stacked State-observer based Neural Network (DSSO-NN). Firstly, the state-space representation is introduced, integrating discretized state-space equations into the neural network design to leverage both system state information and deep learning capabilities. Subsequently, two optimization measures are employed to enhance the network's nonlinear modeling ability with activation functions and the state observer, respectively. Finally, DSSO-NN is validated using the Case Western Reserve University bearing dataset. Experimental results demonstrate that activation functions provide minimal improvement to model performance. In contrast, the incorporation of the state observer significantly enhances the DSSO-NN's ability to capture system dynamics behaviors and improves modeling accuracy. DSSO-NN exhibits higher precision and greater stability, offering a novel perspective on using the state observer as an alternative to traditional activation functions.

In order to solve the deep self-coding neural network training process, the Sigmoid function back-propagation gradient is easy to disappear, a method based on ReLU activation function is proposed for training the self coding neural network. This paper analyzes the performance of different activation functions and comparing ReLU with traditional Tanh and Sigmoid activation function and in Reuters-21578 standard for experiments on the test set. The experimental results show that using ReLU as the activation function, not only can improve the network convergence speed, and can also improve the accuracy.