Memristor crossbar architectures for implementing deep neural networks

Convolution has been the core operation of modern deep neural networks. It is well known that convolutions can be implemented in the Fourier Transform domain. In this article, we propose to use binary block Walsh–Hadamard transform (WHT) instead of the Fourier transform. We use WHT-based binary layers to replace some of the regular convolution layers in deep neural networks. We utilize both one-dimensional (1D) and 2D binary WHTs in this article. In both 1D and 2D layers, we compute the binary WHT of the input feature map and denoise the WHT domain coefficients using a nonlinearity that is obtained by combining soft-thresholding with the tanh function. After denoising, we compute the inverse WHT. We use 1D-WHT to replace the 1 × 1 convolutional layers, and 2D-WHT layers can replace the 3 × 3 convolution layers and Squeeze-and-Excite layers. 2D-WHT layers with trainable weights can be also inserted before the Global Average Pooling layers to assist the dense layers. In this way, we can reduce the number of trainable parameters significantly with a slight decrease in trainable parameters. In this article, we implement the WHT layers into MobileNet-V2, MobileNet-V3-Large, and ResNet to reduce the number of parameters significantly with negligible accuracy loss. Moreover, according to our speed test, the 2D-FWHT layer runs about 24 times as fast as the regular 3 × 3 convolution with 19.51% less RAM usage in an NVIDIA Jetson Nano experiment.

Authorship attribution (AA) is the process of attempting to identify the likely authorship of a given document by analyzing previous works of the authors in question. This paper proposes deep neural network-based continuous skip-gram models for authorship attribution in Bengali Literature. We present a data set of 2400 Bengali blog posts from 6 authors of current time and compare the performances of traditional lexical n-gram based models to our proposed approaches. We achieve a best accuracy of more than 92% on the held-out dataset with a deep convolutional neural network with skipgram word embeddings by fastText as the feature, which outperforms the other traditional models examined in this paper on Bengali Language. The results provide a clear indication that extracting features with the use of hidden layers in deep neural networks from continuous word embeddings work better as a feature set for authorship attribution systems on Bengali Literature than sparse lexical n-gram based features and shallow classifiers.

Deep neural network is a technique of deep learning, where deep neural network model have multiple hidden layers with input and output layer, but artificial neural network have single hidden layer between input and output layer. The use of multiple hidden layers in deep neural network is to improve the performance of model and achieving the higher accuracy compare to machine learning models and their accuracy. The field of pattern recognition is mostly used by the researchers for their research work. There are lots of pattern are available in the field of pattern recognition like: handwritten digits, characters, images, faces, sound, speech etc. In this paper we have concentrated on handwritten digits classification and recognition. For handwritten digit datasets, we have used commonly known Arkiv Digital Sweden (ARDIS) [1] dataset and United State postal service (USPS) [7] dataset. ARDIS dataset is a collection of 7600 samples, where 6600 used as training samples and 1000 used as testing samples. USPS dataset is a collection of 10000 image samples where 7291 samples are used as training sample and 2007 samples are used as testing samples. In this paper we have implemented the proposed deep neural network technique for the classification and recognition of the ARDIS and USPS dataset. The proposed model has collection of 6 layers with relu and softmax activation function. After implementing model, 98.70% testing and 99.76% training accuracy for ARDIS samples achieved, which is higher than previous research accuracy. Also 98.22% training and 93.01%testing accuracy with USPS samples dataset has been achieved. The results represents the performance of deep neural networks have been outstanding compare to other previous techniques.

In this paper, we focus on visual complexity, an image attribute that humans can subjectively evaluate based on the level of details in the image. We explore unsupervised information extraction from intermediate convolutional layers of deep neural networks to measure visual complexity. We derive an activation energy metric that combines convolutional layer activations to quantify visual complexity. To show the effectiveness of our proposed metric for various applications, we introduce Savoias, a visual complexity dataset that compromises of more than 1,400 images from seven diverse image categories (e.g., advertisement and interior design). We demonstrate high correlations of our deep neural network-based measure of visual complexity with human-curated ground-truth (GT) scores on various widely used network architectures, e.g., VGG16, ResNet-v2-152, and EfficientNet, and in networks trained on two classification tasks (object and scene classification). This result reveals that intermediate convolutional layers of deep neural networks carry information about the complexity of images that is meaningful to people. Furthermore, we show that our method of measuring visual complexity outperforms traditional methods on Savoias and two other state-of-the-art benchmark datasets. Moreover, we perform extensive analysis on the performance difference between our unsupervised method and a supervised method trained on the feature map, and show that by supervision, we can improve the prediction. Finally, we demonstrate that, within the context of a category, visually more complex images are also more memorable to human observers.

Deep learning is one of the most recent development form of Artificial Neural Network (ANN) in machine learning. Deep Neural Network (DNN) algorithm is usually used in image and speech recognition applications. As the development of Artificial Neural Network, very possible there are so many hidden layers in Deep Neural Network. In DNN, the output of each node is a quadratic function of its inputs. The DNN training process is very difficult. In this paper, we try to optimizing the training process by slightly construct of the deep architecture and combines several existing algorithms. Output's error of each unit in the previous layer will be calculated. The weight of the unit with the smallest error will be maintained in the next iteration. This paper uses MNIST handwriting images as its data training and data test. After doing some tests, it can be concluded that the optimization by selecting any output in each hidden layer, the DNN training process will be faster approximately 8%.

In 5G networks, the Physical Random Access Channel (PRACH) significantly affects mobile network performance.The base station is responsible for detecting the preamble by calculating the correlation among predefined PRACH preamble sequences, which are essential for successful communication.However, this traditional method presents challenges, as it is computationally complex and resource-intensive.Additionally, inaccuracies in threshold settings can lead to increased energy consumption, primarily due to issues such as preamble collisions and the need for retransmissions.This paper explores the potential of replacing the current detection method with a deep-learning model to improve the efficiency of PRACH preamble detection.Experiments were conducted by varying the layers of Deep Neural Networks (DNN) and Convolutional Neural Networks (CNN) across four different SNR environments to evaluate their performance in detecting PRACH preambles.As a result, the DNN model, in particular, demonstrated high accuracy, averaging between 97.3% and 99.8%.Moreover, in low SNR environments, the DNN model exhibited a notable convergence speed reduction of up to 3.22 times with increased layers.Meanwhile, the CNN model demonstrated that employing fewer convolution layers led to a convergence speed that was 2.14 times faster in terms of loss values.These findings indicate that the design approaches for DNN and CNN models in PRACH preamble detection differ, potentially improving the initial access procedure through tailored strategies.

Computation of non-linear functions such as hyperbolic tangent and sigmoid is critical in machine learning. Furthermore, precision requirement varies across different layers in deep neural networks. Thus it is beneficial to design multi-precision function computation units. Since many function computation methods require lookup tables (LUT) along with simple arithmetic components, we propose a dual-precision architecture with shared LUT based on a recently proposed table-bound algorithm. Experimental results show that the proposed dual-precision designs has smaller area compared with the design with separate single-precision modules.

Influence functions are analytical tools from robust statistics that can help interpret the decisions of black-box machine learning models. Influence functions can be used to attribute changes in the loss function due to small perturbations in the input features. The current work on using influence functions is limited to the features available before the last layer of deep neural networks (DNNs). We extend the influence function approximation to DNNs by computing gradients in an end-to-end manner and relate changes in the loss function to individual input features using an efficient algorithm. We propose an accurate mortality prediction neural network and show the effectiveness of extended influence functions on the eICU dataset. The features chosen by proposed extended influence functions were more like those selected by human experts than those chosen by other traditional methods.

Visualizing features in deep neural networks (DNNs) can help understanding their computations. Many previous studies aimed to visualize the selectivity of individual units by finding meaningful images that maximize their activation. However, comparably little attention has been paid to visualizing to what image transformations units in DNNs are invariant. Here we propose a method to discover invariances in the responses of hidden layer units of deep neural networks. Our approach is based on simultaneously searching for a batch of images that strongly activate a unit while at the same time being as distinct from each other as possible. We find that even early convolutional layers in VGG-19 exhibit various forms of response invariance: near-perfect phase invariance in some units and invariance to local diffeomorphic transformations in others. At the same time, we uncover representational differences with ResNet-50 in its corresponding layers. We conclude that invariance transformations are a major computational component learned by DNNs and we provide a systematic method to study them.

Evolution of Abstraction Across Layers in Deep Learning Neural Networks

Learning in uncertain, noisy, or adversarial environments is a challenging task for deep neural networks (DNNs). We propose a new theoretically grounded and efficient approach for robust learning that builds upon Bayesian estimation and Variational Inference. We formulate the problem of density propagation through layers of a DNN and solve it using an Ensemble Density Propagation (EnDP) scheme. The EnDP approach allows us to propagate moments of the variational probability distribution across the layers of a Bayesian DNN, enabling the estimation of the mean and covariance of the predictive distribution at the output of the model. Our experiments using MNIST and CIFAR-10 datasets show a significant improvement in the robustness of the trained models to random noise and adversarial attacks.

Convolutional neural network-based wind pressure prediction on low-rise buildings

The analysis of customer pose is one of the most important topics for marketing. Using the customer pose information, the retailers can evaluate the customer interest level to the merchandise. However, the pose estimation is not easy because of the problems of occlusion and left-right similarity. To address these two problems, we propose an Integral Pose Network (IntePoseNet) which incorporates the body orientation and visibility mask. Firstly, benefiting from the simple gaits in retail store, the body orientation can give the global information of pose configuration. For example, if a person is facing to right, the body orientation indicates the occlusion of his or her left body. Similarly, if a person is facing to the camera, his or her right shoulder is probably at the left side of image. The body orientation is fused with local joint connections by a set of novel Orientational Message Passing (OMP) layers in Deep Neural Network. Secondly, the visibility mask models the occlusion state of each joint. It is tightly related to the body orientation because the body orientation is the main reason of self-occlusion. In addition, occluding object (e.g. shopping basket in retail store environment) detection can also give the clues of visibility mask prediction. Furthermore, the system models the temporal consistency by introducing the optical flow and Bi-directional Recurrent Neural Network. Therefore, the global body orientation, local joint connections, customer motion, occluding objects and temporal consistency are integrally considered in our system. At last, we conduct a series of comparison experiments to show the effectiveness of our system.

Efficient deep feature extraction and classification for identifying defective photovoltaic module cells in Electroluminescence images

Convolutional neural networks (CNNs) have demonstrated remarkable performance in many areas but require significant computation and storage resources. Quantization is an effective method to reduce CNN complexity and implementation. The main research objective is to develop a scalable quantization algorithm for CNN hardware design and model the performance metrics for the purpose of CNN implementation in resource-constrained devices (RCDs) and optimizing layers in deep neural networks (DNNs). The algorithm novelty is based on blending two quantization techniques to perform full model quantization with optimum accuracy, and without additional neurons. The algorithm is applied to a selected CNN model and implemented on an FPGA. Implementing CNN using broad data is not possible due to capacity issues. With the proposed quantization algorithm, we succeeded in implementing the model on the FPGA using 16-, 12-, and 8-bit quantization. Compared to the 16-bit design, the 8-bit design offers a 44% decrease in resource utilization, and achieves power and energy reductions of 41% and 42%, respectively. Models show that trading off one quantization bit yields savings of approximately 5.4K LUTs, 4% logic utilization, 46.9 mW power, and 147 μJ energy. The models were also used to estimate performance metrics for a sample DNN design.