Surface Defect Recognition of CSP Typical Slab Based on Improved ResNet

Enhancing deterministic prediction in unidirectional ocean waves using an Artificial Neural Network with exponential linear unit

Different parameters of a fully convolutional network (FCN) are experimented to evaluate which combination predicts sound velocity models from a single configuration of seismic modeling. The evaluation is made considering some fixed parameters of the deep learning model, such as number of epochs, batch size and loss function, but with variations of the optimizer and activation function. The considered optimizers were RMSprop, Adam and Adamax, whilst the activations functions were the Rectified Linear Unit (ReLU), Leaky ReLU, Exponential Linear Unit (ELU) and Parametric ReLU (PReLU). Five metrics were used to evaluate the model during the testing stage: R2, Pearson's r, factor of two, mean absolute error and mean squared error. To the extent of these experiments, it was found that the optimizers have much more influence than the activation functions when determining the resolution of the output model. The best combination was the one using the PReLU activation function with the Adamax optimizer.

Focusing on the problems of complex structure and low feature extraction efficiency that exist in some traditional neural network algorithms, an improved convolutional neural network, combined with a convolutional attention mechanism, is proposed to im- prove the network feature extraction efficiency and reduce the network complexity. First, the convolutional block attention module (CBAM) was introduced, which can strengthen the extraction of useful features from the data, thereby improving the network feature extraction efficiency. Second, the residual structure was optimized, and the original multi-layer convolution was simplified to a single-layer convolution, which reduces the complexity of the model and improves the diagnostic efficiency. Then, the Rectified Linear Unit (ReLU) activation function in the model was replaced with the better-performing Exponential Linear Unit (ELU) activation function. Finally, the overall structure of the ResNet model was optimized. The CBAM-ResNet network model was based on the traditional ResNet18 model and consists of a CBAM module and the optimized ResNet18 model. Finally, experiments were performed using sensor data collected from rolling bearing test benches. The experimental results show that the feature extraction capability and efficiency of the new model are better than those of several other classic deep network models and the original network model, and this reduces the time complexity and model size while also maintaining a high accuracy rate.

The scale of deer breeding has gradually increased in recent years and better information management is necessary, which requires the identification of individual deer. In this paper, a deer face dataset is produced using face images obtained from different angles, and an improved residual neural network (ResNet)-based recognition model is proposed to extract the features of deer faces, which have high similarity. The model is based on ResNet-50, which reduces the depth of the model, and the network depth is only 29 layers; the model connects Squeeze-and-Excitation (SE) modules at each of the four layers where the channel changes to improve the quality of features by compressing the feature information extracted through the entire layer. A maximum pooling layer is used in the ResBlock shortcut connection to reduce the information loss caused by messages passing through the ResBlock. The Rectified Linear Unit (ReLU) activation function in the network is replaced by the Exponential Linear Unit (ELU) activation function to reduce information loss during forward propagation of the network. The preprocessed 6864 sika deer face dataset was used to train the recognition model based on SE-Resnet, which is demonstrated to identify individuals accurately. By setting up comparative experiments under different structures, the model reduces the amount of parameters, ensures the accuracy of the model, and improves the calculation speed of the model. Using the improved method in this paper to compare with the classical model and facial recognition models of different animals, the results show that the recognition effect of this research method is the best, with an average recognition accuracy of 97.48%. The sika deer face recognition model proposed in this study is effective. The results contribute to the practical application of animal facial recognition technology in the breeding of sika deer and other animals with few distinct facial features.

Emotion recognition still remains a research issue that can be improved significantly. This paper worked on it using facial feature extraction followed by arousal level estimation. We extracted facial features via four Convolutional Neural Networks (CNNs) pre-trained on ImageNet: ResNet152V2, Xception, ResNeXt101, and InceptionResNetV2 based on low memory usage and modernity. For estimating arousal levels, Exponential Linear Unit (ELU) was compared with Rectified Linear Unit (ReLU) in a custom deep model framework. Wild facial expression image data was taken from the relatively new AffectNet database. Up to this point, recognition of emotions from AffectNet is largely treated as a classification problem, with arousal level estimation mostly unexplored. Our results showed that the custom deep model trained on multiple classes and fine-tuned on one class was more effective than that trained and tested on the same class. ResNet152V2 displayed the best performance among the CNNs, emphasizing its suitability for the CultureNet-based architecture from which this algorithm is based. Further research includes more activation functions, ResNet variants, and the application of a softmax layer.

Classical Convolutional Neural Networks, or ConvNets, have been the benchmarks for most object classification and face recognition tasks despite suffering from major limitations such as the inability to capture spatial co-locality between data points and favoring invariance over equivariance. Hence, a hierarchical routing layered architecture called Capsule Networks was proposed to overcome shortcomings of ConvNets. Capsules replace average or max pooling techniques of ConvNets with dynamic routing abilities between lower level and higher level neural units which better capture hierarchical relationships within the data and introduced reconstruction regularization mechanisms which deals with equivariance properties. By overcoming existing limitations, Capsules have proven themselves to be potential benchmarks in object segmentation, detection and reconstruction. Capsules have achieved state of the art results on the fundamental MNIST (Modified National Institute of Standards and Technology) handwritten digit dataset by reducing the ConvNets test error benchmark of 0.39% to 0.25%. In order to further augment this distinction, we experimented with five activation units such as sigmoid, e-Swish, Swish, variants of Rectified Linear Units (ReLU) like Parametric ReLU (PReLU), leaky ReLU (lReLU) and Scaled Exponential Linear Units (SELU), on two fundamental datasets - MNIST and cifar10. Based on these experimental results, we establish that e-Swish, and ReLU variants better optimize the Capsule architecture as compared to the currently used ReLU activation function.

Deep neural networks with Elastic Rectified Linear Units for object recognition

The convolutional neural networks (CNN), including AlexNet, GoogleNet, VGGNet, etc. extract features for many computer vision problems which are very discriminative. The trained CNN model over one dataset performs reasonably well whereas on another dataset of similar type the hand-designed feature descriptor outperforms the same trained CNN model. The Rectified Linear Unit (ReLU) layer discards some values in order to introduce the non-linearity. In this paper, it is proposed that the discriminative ability of deep image representation using trained model can be improved by Average Biased ReLU (AB-ReLU) at the last few layers. Basically, AB-ReLU improves the discriminative ability in two ways: 1) it exploits some of the discriminative and discarded negative information of ReLU and 2) it also neglects the irrelevant and positive information used in ReLU. The VGGFace model trained in MatConvNet over the VGG-Face dataset is used as the feature descriptor for face retrieval over other face datasets. The proposed approach is tested over six challenging, unconstrained and robust face datasets (PubFig, LFW, PaSC, AR, FERET and ExtYale) and also on a large scale face dataset (PolyUNIR) in retrieval framework. It is observed that the AB-ReLU outperforms the ReLU when used with a pre-trained VGGFace model over the face datasets. The validation error by training the network after replacing all ReLUs with AB-ReLUs is also observed to be favorable over each dataset. The AB-ReLU even outperforms the state-of-the-art activation functions, such as Sigmoid, ReLU, Leaky ReLU and Flexible ReLU over all seven face datasets.

Activation functions facilitate deep neural networks by introducing non-linearity to the learning process. The non-linearity feature gives the neural network the ability to learn complex patterns. Recently, the most widely used activation function is the Rectified Linear Unit (ReLU). Though, other various existing activation including hand-designed alternatives to ReLU have been proposed. However, none has succeeded in replacing ReLU due to their existing inconsistencies. In this work, activation function called ReLUMemristor-like Activation Function (RMAF) is proposed to leverage benefits of negative values in neural networks. RMAF introduces a constant parameter (α) and a threshold parameter (p) making the function smooth, non-monotonous, and introduces non-linearity in the network. Our experiments show that, the RMAF works better than ReLU and other activation functions on deeper models and across number of challenging datasets. Firstly, experiments are performed by training and classifying on multi-layer perceptron (MLP) over benchmark data such as the Wisconsin breast cancer, MNIST, Iris and Car evaluation. RMAF achieves high performance of 98.74%, 99.67%, 98.81% and 99.42% respectively, compared to Sigmoid, Tanh and ReLU. Secondly, experiments were performed on convolution neural network (ResNet) over MNIST, CIFAR-10 and CIFAR-100 data and observed the proposed activation function achieves higher performance accuracy of 99.73%, 98.77% and 79.82% respectively than Tanh, ReLU and Swish. Additionally, we experimented our work on deep networks i.e. squeeze network (SqueezeNet), Dense connected neural network (DenseNet121) and ImageNet dataset, which RMAF produced the best performance. We note that, the RMAF converges faster than the other functions and can replace ReLU in any neural network due to the efficiency, scalability and its similarity to both ReLU and Swish.

Neural networks with ReLU powers need less depth

Recently, Deep learning has made a great deal of success in processing images, audios, and natural languages and so on. The activation function is one of the key factors in Deep learning. In this paper, according to characteristics of biological neurons, an improved Leaky Single-Peaked Triangle Linear Unit (LSPTLU) activation function is presented for the right-hand response unbounded of Rectified Linear Unit (ReLU) and Leaky ReLU (LReLU). LSPTLU is more in line with the biological neuron essence and achieves the excellent performance of equivalent or beyond ReLU and LReLU on different datsets, e.g., MNIST, Fashion-MNIST, SVHN, IMAGENET, CALTECH101 and CIFAR10 datasets.

Recently, rectified linear units (ReLUs) have been used to solve the vanishing gradient problem. Their use has led to state-of-the-art results in various problems such as image classification. In this paper, we propose the hyperbolic linear units (HLUs) which not only speed up learning process in deep convolutional neural networks but also obtain better performance in image classification tasks. Unlike ReLUs, HLUs have inheriently negative values which could make mean unit outputs closer to zero. Mean unit outputs close to zero means we can speed up the learning process because they bring the normal gradient close to the natural gradient. Indeed, the difference called bias shift between natural gradient and the normal gradient is related to the mean activation of input units. Experiments with three popular CNN architectures, LeNet, Inception network and ResNet on various benchmarks including MNIST, CIFAR-10 and CIFAR-100 demonstrate that our proposed HLUs achieve significant improvement compared to other commonly used activation functions1.

Rectified linear units (ReLU) are known to be effective in many deep learning methods. Inspired by linear-mapping technique used in other super-resolution (SR) methods, we reinterpret ReLU into point-wise multiplication of an identity mapping and a switch, and finally present a novel nonlinear unit, called a selection unit (SU). While conventional ReLU has no direct control through which data is passed, the proposed SU optimizes this on-off switching control, and is therefore capable of better handling nonlinearity functionality than ReLU in a more flexible way. Our proposed deep network with SUs, called SelNet, was top-5th ranked in NTIRE2017 Challenge, which has a much lower computation complexity compared to the top-4 entries. Further experiment results show that our proposed SelNet outperforms our baseline only with ReLU (without SUs), and other state-of-the-art deep-learning-based SR methods.

Price movement predictions of financial instruments using traditional time-series models with a predefined mathematical structure are common, but this method restricts their ability to learn latent patterns in the data. In recent times, artificial neural networks (ANN) have been able to learn complex hidden patterns from financial datasets using a highly nonlinear architecture. However, most experiments with neural networks require a lot of time to search a suitable network and subsequently train the network. The authors have developed a deep multilayer perceptron (MLP) classifier with a zero-centered piecewise linear unit activation that yielded better classification performance according to their accuracy metric and required consistently less training time compared to a similar MLP network with leaky rectified linear unit and tanh activation functions. The authors illustrate their technique with a large high-frequency dataset on selected bank shares from the Indian stock market. The authors also discuss the theoretical properties and advantages of their proposal. <b>TOPICS:</b>Statistical methods, simulations, big data/machine learning <b>Key Findings</b> • Time-series models have been used extensively to understand stock price movements. They are constrained by a predefined structure. Artificial neural network (ANNs), due to their universal approximation properties, offer more flexibility; however, they suffer from long training times. • In this article, the authors introduce a zero-centered piecewise linear unit (PLU), a hybrid activation function derived from tanh and leaky rectified linear unit (ReLU) functions that exhibits the zero-centered property from tanh and the nonzero gradient property from leaky ReLU. • The authors developed a multilayer perceptron (MLP) classifier with PLU activation, which yielded better classification performance according to the accuracy metric and required consistently less training time compared to a similar MLP network with leaky ReLU and tanh activation.

Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a 7.28% relative error improvement on ImageNet with the NiN network, with only 0.0003% parameter increase. Our visual examination of the non-linear behaviors adopted by Vgg using PELU shows that the network took advantage of the added flexibility by learning different activations at different layers.