Neural Networks Based on a Learnable Two-Dimensional Separable Transform for Image Classification: Theory and Hardware Implementation on FPGA

Single-hidden-layer feed-forward quantum neural network based on Grover learning

We present a method to implement trained neural networks on digital programmable hardware. The neural network is trained off-line using a microprocessor system and suitable software. Thereafter the trained neural network is transferred to the hardware implementation. In order to reduce the costs, the network nodes are implemented as multiplexed structures requiring a certain number of clocks for the execution of each iteration. However, in order to increase the overall processing speed the pipelining structure can be also used. It only delays the response with a few microseconds, depending on the clock rate and network complexity. The hardware implementation of neural network using programmable logic devices allows boosting the processing speed, compared to the software implementation, to alternatively reduce the overall costs and provide similar flexibility as the software approach.

Typically, hardware implemented neural networks are trained before implementation. Extreme learning machine (ELM) is a noniterative training method for single-layer feed-forward (SLFF) neural networks well suited for hardware implementation. It provides fixed-time learning and simplifies retraining of a neural network once implemented, which is very important in applications demanding on-chip training. This study proposes the data flow of a software support tool in the design process of a hardware implementation of on-chip ELM learning for SLFF neural networks. The software tool allows the user to obtain the optimal definition of functional and hardware parameters for any application, and enables the user to interact throughout the design process. Combining in a transparent way for the user, simulation and Xilinx synthesis tools, the tool recommends the optimal configuration, generating, finally, a synthesizable IP-core. As application, the field-programmable gate array implementation for real-time detection of brain areas in electrode positioning during a deep brain stimulation surgery is described. The generated IP-core can execute a peak of 95 ELM trainings per second on a low-cost Spartan 6 device, making possible its real-time use in this application.

Electrical stimulation can be used to activate paralyzed muscles for the purpose of restoring motor functions such as stepping in individuals with spinal cord injury. Due to the variability observed in the responses of electrically stimulated muscles, several adaptive control techniques have been developed. The pattern shaper (PS) is an adaptive neural network that has been tested in a software implementation and has been shown to be capable of automatically generating cyclic patterns that are customized for an individual. The results of these tests are encouraging, but implementation of the real-time algorithm currently requires a dedicated PC. The purpose of this research is to develop a hardware implementation of a digital PS neural network to generate the electrical signals to stimulate muscles in individuals with spinal cord injury. We have implemented the pattern shaper algorithm in hardware by mapping the digital logic circuit to a field programmable gate array (FPGA), developed a user interface to input data to the FPGA from a computer, and constructed a wire-wrapped board to implement the PS in hardware for use in clinical tests. This hardware implementation is a step towards the development of low-power, portable, adaptive controller that can be used in electrical stimulation systems.

Among speech researchers, it is widely believed that Hidden Markov Models (HMMs) are the most successful modelling approaches for acoustic events in speech recognition. However, common assumptions limit the classification abilities of HMMs and these can been relaxed by introducing neural networks in the HMM framework. With today's advances in VLSI technology, artificial neural networks (ANNs) can be integrated into a single chip offering adequate circuit complexity required to attain both a high recognition accuracy and an improved learning time. Analogue implementations are considered due to the high processing speeds. The relative performance of different speech coding parameters for use with two different ANN architectures that lend themselves to analogue hardware implementations are investigated. In this case, the dynamic ranges of the different coefficients need to be taken into consideration since they will affect the performance of the analogue chip due to the scaling of the coefficients to voltage signals. The hardware requirements for implementing the two architectures are then discussed.

In this paper, we propose a new architecture of Pulse Mode Neural Network (PMNN) with very simple activation function. Pulse mode is gaining support in the field of hardware Neural Networks thanks to its higher density of integration. However, the complexity of the activation functions presents a drawback for hardware implementation of Neural Networks and limits its area of application. In this context, the main idea is to apply a new kind of activation function, simply generated by the product of two sigmoidal functions, which are very simple and already implemented in previous work. Details of important aspects concerning the hardware implementation are given. To verify the performance and capacity of the proposed design, we apply it for approximation of image denoising function. The filtered results are verified in terms of the Peak Signal to Noise Ratio (PSNR). Experimental results reveal that the proposed PMNN filter has a greater ability to recover the informative pixel intensities from the infected image with a recovery of 7.5 dB for Gaussian noise and 5.3 dB for Speckle noise. Besides, such results demonstrate the performance and efficiency of our Neural filter when compared to other conventional filtering techniques. The designed network is implemented on a field-programmable gate array (FPGA) platform and synthesis results are presented and discussed.

Artificial neural networks (ANNs, or simply NNs) are inspired by biological nervous systems and consist of simple processing units (artificial neurons) that are interconnected by weighted connections. Neural networks can be trained to solve problems that are difficult to solve by conventional computer algorithms. The usage of the FPGA (Field Programmable Gate Array) for neural Network implementation provides flexibility in programmable systems. For the neural network based instrument prototype in real time application, conventional specific VLSI neural chip design suffers the limitation in time and cost. With low precision artificial neural network design, FPGAs have higher speed and smaller size for real time application than the VLSI design. Several work show that FPGA are real opportunity for flexible hardware implementation of neural network and yet representation of standard neural network face some problem. The difficulty such as limit of size and architecture of neural network that can mapped on to FPGA. This paper discuss the usage of neural network implementations. Both assets and obstacles are described and various solution are outlined.

Learning rate scheduling (LRS) is a critical factor influencing the performance of neural networks by accelerating the convergence of learning algorithms and enhancing the generalization capabilities. The escalating computational demands in artificial intelligence (AI) necessitate advanced hardware solutions capable of supporting neural network training with LRS. This not only requires linear and symmetric analog programming capabilities but also the precise adjustment of channel conductance to achieve tunable slope in weight update behaviors. Here, a cascaded duplex organic vertical memory is proposed with the coupling of ferroelectric polarization effect and Schottky gate control on the same semiconducting channel, exhibiting adjustable‐slope conductance update with high linearity and symmetry. Therefore, in the chest X‐ray image detection, a fast‐to‐slow LRS is used for a bi‐layer ANN training, achieving a rapid, stable convergence behavior within only 15 epochs and a high recognition accuracy. Moreover, the proposed LRS training is also suitable for the Mackey Glass prediction task using long short‐term memory networks. This work integrates LRS into synaptic devices, enabling efficient hardware implementation of neural networks and thus enhancing AI performance in practical applications.

The evolution of smart water grids leads to new Big Data challenges boosting the development and application of Machine Learning techniques to support efficient and sustainable drinking water management. These powerful techniques rely on hyperparameters making the models’ tuning a tricky and crucial task. We hence propose an insightful analysis of the tuning of Artificial Neural Networks for drinking water demand forecasting. This study focuses on layers and nodes’ hyperparameters fitting of different Neural Network architectures through a grid search method by varying dataset, prediction horizon and set of inputs. In particular, the architectures involved are the Feed Forward Neural Network, the Long Short Term Memory, the Simple Recurrent Neural Network and the Gated Recurrent Unit, while the prediction interval ranges from 1 h to 1 week. To avoid the problem of the Neural Networks tuning stochasticity, we propose the selection of the median model among several repetitions for each hyperparameter’s configurations. The proposed iterative tuning procedure highlights the change of the required number of layers and nodes depending on Neural Network architectures, prediction horizon and dataset. Significant trends and considerations are pointed out to support Neural Network application in drinking water prediction.

This paper presents a hardware implementation of multilayer feedforward neural networks (NN) using reconfigurable field-programmable gate arrays (FPGAs). Despite improvements in FPGA densities, the numerous multipliers in an NN limit the size of the network that can be implemented using a single FPGA, thus making NN applications not viable commercially. The proposed implementation is aimed at reducing resource requirement, without much compromise on the speed, so that a larger NN can be realized on a single chip at a lower cost. The sequential processing of the layers in an NN has been exploited in this paper to implement large NNs using a method of layer multiplexing. Instead of realizing a complete network, only the single largest layer is implemented. The same layer behaves as different layers with the help of a control block. The control block ensures proper functioning by assigning the appropriate inputs, weights, biases, and excitation function of the layer that is currently being computed. Multilayer networks have been implemented using Xilinx FPGA "XCV400hq240". The concept used is shown to be very effective in reducing resource requirements at the cost of a moderate overhead on speed. This implementation is proposed to make NN applications viable in terms of cost and speed for online applications. An NN-based flux estimator is implemented in FPGA and the results obtained are presented.

The soft computing algorithms are being nowadays used for various multi input multi output complicated non linear control applications. This paper presented the development and implementation of back propagation of multilayer perceptron architecture developed in FPGA using VHDL. The usage of the FPGA (Field Programmable Gate Array) for neural network implementation provides flexibility in programmable systems. For the neural network based instrument prototype in real time application. The conventional specific VLSI neural chip design suffers the limitation in time and cost. With low precision artificial neural network design, FPGA have higher speed and smaller size for real time application than the VLSI design. The challenges are finding an architecture that minimizes the hardware cost, maximizing the performance, accuracy. The goal of this work is to realize the hardware implementation of neural network using FPGA. Digital system architecture is presented using Very High Speed Integrated Circuits Hardware Description Language (VHDL)and is implemented in FPGA chip. MATLAB ANN programming and tools are used for training the ANN. The trained weights are stored in different RAM, and is implemented in FPGA. The design was tested on a FPGA demo board. Keywords- Backpropagation, field programmable gate array (FPGA) hardware implementation, multilayer perceptron, pressure sensor, Xilinx FPGA.

Pathological brain detection has made notable stride in the past years, as a consequence many pathological brain detection systems (PBDSs) have been proposed. But, the accuracy of these systems still needs significant improvement in order to meet the necessity of real world diagnostic situations. In this paper, an efficient PBDS based on MR images is proposed that markedly improves the recent results. The proposed system makes use of contrast limited adaptive histogram equalization (CLAHE) to enhance the quality of the input MR images. Thereafter, two-dimensional PCA (2DPCA) strategy is employed to extract the features and subsequently, a PCA+LDA approach is used to generate a compact and discriminative feature set. Finally, a new learning algorithm called MDE-ELM is suggested that combines modified differential evolution (MDE) and extreme learning machine (ELM) for segregation of MR images as pathological or healthy. The MDE is utilized to optimize the input weights and hidden biases of single-hidden-layer feed-forward neural networks (SLFN), whereas an analytical method is used for determining the output weights. The proposed algorithm performs optimization based on both the root mean squared error (RMSE) and norm of the output weights of SLFNs. The suggested scheme is benchmarked on three standard datasets and the results are compared against other competent schemes. The experimental outcomes show that the proposed scheme offers superior results compared to its counterparts. Further, it has been noticed that the proposed MDE-ELM classifier obtains better accuracy with compact network architecture than conventional algorithms.

In the paper, Neural Network (NN) models for gas turbine diagnostics are studied and developed. The analyses carried out are aimed at the selection of the most appropriate NN structure for gas turbine diagnostics, in terms of computational time of the NN training phase, accuracy and robustness with respect to measurement uncertainty. In particular, feed-forward NNs with a single hidden layer trained by using a back-propagation learning algorithm are considered and tested. Moreover, Multi-Input/Multi-Output NN architectures (i.e. NNs calculating all the system outputs) are compared to Multi-Input/Single-Output NNs, each of them calculating a single output of the system. The results obtained show that NNs are robust with respect to measurement uncertainty, if a sufficient number of training patterns are used. Moreover, Multi-Input/Multi-Output NNs trained with data corrupted with measurement errors seem to be the best compromise between the computational time required for NN training phase and the NN accuracy in performing gas turbine diagnostics.

After the hype of the 1990s, where companies like Intel or Philips built commercial hardware systems based on neural networks, the approach quickly lost ground for multiple reasons: hardware neural networks were no match for software neural networks run on rapidly progressing general-purpose processors, their application scope was considered too limited, and even progress in machine-learning theory overshadowed neural networks. However, in the past few years, a remarkable convergence of trends and innovations is casting a new light on neural networks and could make them valuable components of future computing systems. Trends in technology call for architectures which can sustain a large number of defects, something neural networks are intrinsically capable of. Tends in applications, summarized in the recent RMS categorization, highlight a number of key algorithms which are eligible to neural networks implementations. At the same time, innovations in technology, such as the recent realization of a memristor, are creating the conditions for the efficient hardware implementation of neural networks. Innovations in machine learning, with the recent advent of Deep Networks, have revived interest in neural networks. Finally, recent findings in neurobiology carry even greater prospects, where detailed explanations of how complex functions, such as vision, can be implemented further open up the defect-tolerance and application potential of neural network architectures.

Neural networks (NNs) are used in numerous applications such as audio–video processing and image classification. NNs can address the limitations of the traditional computer algorithms. Recently, the research utilizing field-programmable gate arrays (FPGAs) to implement NNs is on the rise due to low power dissipation, easy and fast reconfigurability offered by these platforms. This is due to the research and development efforts put into the design optimization to improve the throughput of FPGAs. The integration of appropriate hardware with advanced artificial intelligence (AI) software presents many challenges. This paper offers a generalized model of NNs on a FPGA device. In addition, it presents the design process for the high-level synthesis (HLS) tools to implement the project on FPGA. This paper shows the results of latency, timing, pre- and post-synthesis, implementation, and hardware utilization on Xilinx FPGA target device. In this paper, NN for XOR logic gate operation is performed and verified by using HLS tool, and FPGA implementation is proposed. The NN and backpropagation algorithm are developed in high-level programming languages like Python, Java, C, C++. The training part of the NN has been done by using open-source software Dev C++, and tuned weights are taken to Xilinx Vitis HLS. Simulation, synthesis, and implementation are performed by using Xilinx Vitis HLS and Vivado 2020.1 electronic design automation (EDA) tools. NN architecture with tuned weights is implemented on Xilinx ZYNQ FPGA target device. This paper shows that we were able to achieve latency in one clock cycle interval through pipeline and array partition.KeywordsHigh-level synthesis (HLS)Neural networks (NNs)Register transfer level (RTL)Field-programmable gate arrays (FPGAs)Systems on chip (SOC)Artificial intelligence (AI)Graphic processing units (GPUs)Generalized NNs (GNN)Recurrent neural network (RNN)Feed-forward neural networks (FFNNs)