Parameter Distribution Balanced CNNs.

Using explainability to design physics-aware CNNs for solving subsurface inverse problems

The application of deep learning technology has increased rapidly in recent years. Technologies in deep learning increasingly emulate natural human abilities, such as knowledge learning, problem-solving, and decision-making. In general, deep learning can carry out self-training without repetitive programming by humans. Convolutional neural networks (CNNs) are deep learning algorithms commonly used in wide applications. CNN is often used for image classification, segmentation, object detection, video processing, natural language processing, and speech recognition. CNN has four layers: convolution layer, pooling layer, fully connected layer, and non-linear layer. The convolutional layer uses kernel filters to calculate the convolution of the input image by extracting the fundamental features. The pooling layer combines two successive convolutional layers. The third layer is the fully connected layer, commonly called the convolutional output layer. The activation function defines the output of a neural network, such as 'yes' or 'no'. The most common and popular CNN activation functions are Sigmoid, Tanh, ReLU, Leaky ReLU, Noisy ReLU, and Parametric Linear Units. The organization and function of the visual cortex greatly influence CNN architecture because it is designed to resemble the neuronal connections in the human brain. Some of the popular CNN architectures are LeNet, AlexNet and VGGNet.

Sound-event recognition often utilizes time-frequency analysis to produce an image-like spectrogram that provides a rich visual representation of original signal in time and frequency. Convolutional Neural Networks (CNN) with the ability of learning discriminative spectrogram patterns are suitable for sound-event recognition. However, there is relatively little effort that CNN makes full use of the important temporal information. In this paper, we propose MCRNN, a Convolutional Recurrent Neural Networks (CRNN) architecture for sound-event recognition, the letter “M” in the name “MCRNN” of our model denotes the multi-sized convolution filters. Richer features are extracted by using several different convolution filter sizes at the last convolution layer. In addition, cochleagram images are used as the input layer of the network, instead of the traditional spectrogram image of a sound signal. Experiments on the RWCP dataset shows that the recognition rate of the proposed method achieved 98.4% in clean conditions, and it robustly outperforms the existing methods, the recognition rate increased by 0.9%, 1.9% and 10.3% in 20 dB, 10 dB and 0 dB signal-to-noise ratios (SNR), respectively.

Automatic eyeglasses detection plays a major role in many facial analysis systems. To improve the robustness of these systems and cope with real-world applications, a high-speed eyeglasses detector that can achieve high accuracy is needed. Recent studies indicate that the features extracted from convolutional neural networks are compelling. Therefore, this paper presents an effective and efficient method for eyeglasses detection in facial images based on extracting deep features from a well-designed shallow convolutional neural network (CNN). The main contribution of this paper is to address the two essential aspects of CNN: (1) the size of the training dataset required and (2) the depth of the network architecture. To this end, we initialize the learning parameters of the shallow CNN by the parameters of a deep CNN which is fine-tuned on a small dataset. The depth of the neural network is then decreased by removing some convolutional layers after testing its performance on the validation dataset. As a result, a significantly more accurate shallow CNN architecture, Shallow-GlassNet, is obtained, which achieves not only high accuracy but also high speed in eyeglasses detection. Evaluation experiments have been conducted on two large unconstrained facial image databases, LFW and Celeb Faces. The results have demonstrated the superior performance of the proposed framework which achieves a mean accuracy of 99.73%.

Convolutional neural network (CNN) consists of a few pairs of both convolution layer and subsampling layer. Thus it has more hidden layers than multi-layer perceptron. With the increased layers, the size of convolution mask ultimately determines the total number of weights in CNN because the mask is shared among input images. It also is an important learning factor which makes or breaks CNNs learning. Therefore, this paper proposes the best method to choose the convolution size and the number of layers for learning CNN successfully. Through our face recognition with vast learning examples, we found that the best size of convolution mask is 5 by 5 and 7 by 7, regardless of the number of layers. In addition, the CNN with two pairs of both convolution and subsampling layer is found to make the best performance as if the multi-layer perceptron having two hidden layers does.

Sentiment Analysis was a process for identifying whether a source of text contains certain opinions, emotions, and polarity. Twitter Sentiment Analysis was a process for identifying sentiment and polarity on tweet. Twitter Sentiment Analysis provided a way for did survey about public sentiment to product, or particular even through collections of tweet. Main problem in sentiment identifying was how to determine classification model that gave high accuracy to classifying sentiment of tweet. One of the method for classifying sentiment of tweet was Deep Learning. Convolutional Neural Network (CNN) was special type of architecture from Deep Learning that its architecture had convolution layer. Convolution layer was important for extract relevant feature from text for classifying sentiment. The objective of this research was for found out the best CNN model for classifying sentiment of tweet. By using a dataset of tweets about public opinion on the Smartfren 4G network service, we searched the best CNN model using 6 combination parameters, that is the computational eficiency method, window size, and dimension of word embedding for parameters in Word2Vec Skip-gram model, then activation function in convolution layer, dropout rate, and pool size in pooling layer for parameters in CNN. The test is done using 10-fold cross validation for each search for the best parameter value and produced the best CNN model with an accuracy value of 88,21%.

In order to overcome the effects of posture, illumination, expression and other factors on face recognition, this paper proposes an algorithm which is based on local binary pattern (LBP) and convolutional neural network (CNN). LBP is a texture description method which describes the local texture features of an image. It has good robustness of illumination and posture. CNN can effectively extract the spatial features of images and reduce the dimensions of features. This paper combines the advantages of LBP and CNN to improve the accuracy of face recognition. The CNN in this algorithm has four convolution layers, two max-pooling layers, one activation layer, one fully connected layer, and one output layer. In order to optimize the network structure, batch normalization layer is added after the convolution layer. We get the local binary pattern coded images and put the images as the input of the CNN and train the network. Hence, we can use the well-trained CNN for classification and identification. The Experiments on the CMU-PIE face database show that our algorithm can effectively improve the rate of face recognition.

Recent research on fault diagnosis mainly focuses on how to improve diagnostic accuracy. For a given accuracy level, a variety of convolutional neural network architectures have been developed and available to achieve the specific accuracy level. With equivalent fault diagnosis accuracy, smaller convolutional neural network structure offers at least three advantages: (1) it can be deployed on hardware with limited memory, such as FPGAs; (2) the training of smaller convolutional neural networks can be faster under the same processor performance; (3) smaller network structures require less communication across severs during distributed training. So, in this paper, an improved convolutional neural network is constructed, and the following strategies are used to reduce network parameters and further make the convolutional neural networks with smaller structures: (1) equivalent replacement of large-scale convolution layers by multiple small-sized convolution layers; (2) avoid using a fully connected layer, and replace with a global pooling layer. Meanwhile, to ensure the model’s fault diagnosis accuracy and training effectiveness, inspired by the network in network, a 1*1 convolutional layer is inserted into a traditional convolutional layer to improve feature expression, and the batch-normalization layer is used to increase the training effectiveness.

As road traffic sign recognition is a crucial component for automatic driver assistance systems, it is a key problem in computer vision as well. Therefore, in this paper, we study on the problem of road traffic sign recognition utilising the computer vision technology. The main innovation of this paper is to propose an improved convolutional neural network, and then use it to tackle the road traffic sign recognition problem. Convolutional neural network can learn features from training data set, and a convolutional network contains alternating layers of convolution and pooling. Particularly, RGB traffic images are transformed to grey scale images, and then grey scale images are input to the improved convolutional neural network. Furthermore, the fixed layers are utilised to discover region of interests, and the learnable layers are used to extract features. In general, output information of the proposed two learnable layers are input to the classifier separately, and parameters of learnable layers and the classifier are trained at the same time. Finally, GTSDB data set is chosen to make performance evaluation, among which 600 images and 300 images are regarded as training and testing data set respectively. Experimental results demonstrate that the improved CNN-based traffic sign recognition performs better than the traditional CNN.

Deep convolutional neural networks (CNNs) are a popular choice for many image classification tasks due to their good performance. In general, there is a correlation between the generalization performance of a deep learning model, the number of training samples available, and the number of free parameters in the model. If there are too many free parameters in the model relative to the amount of training data, the model overfits the training data. Reducing the number of trainable parameters in CNNs while maintaining performance levels is an active area of research. This is important, especially in resource-constrained environments where either the size of the training data is insufficient or the memory of the hardware where the model is to be deployed is limited, such as in mobile vision applications. In this paper, kernel transposition is proposed as a method for the reduction of free parameters in the model’s convolutional layers. This allows kernel reuse, in which a model learns a given kernel only once but uses it twice. The learned kernels and their transposes can either be used in sequence or in parallel to one another. These form series and parallel convolutional modules respectively. The modules are used as replacements of the traditional convolutional layers in a CNN model. The use of the modules reduces the number of free parameters and computational costs associated with convolution operations in the model without necessarily compromising its generalization performance. The proposed method is generic and can be adapted to existing state-of-the-art network architectures as demonstrated in the experiments using established architectures. The proposed method was validated on the CIFAR-10 and CIFAR-100 standard datasets using a series of experiments with five model architectures: a basic 5-convolutional layer CNN model, the ResNet-56, small MobileNetV3, EfficientNetB0 and ConvNext model architectures; all trained from scratch. Models based on standard convolutional layers (for basic CNN and ResNet-56 architectures) and depth-wise convolutional layers (for MobileNetV3, EfficientNetB0 and ConvNext architectures) were compared against those based on the proposed convolutional modules. The models based on the standard and depth-wise convolutional layers were used as baselines. The comparisons were in terms of the number of parameters, computational efficiencies (Floating Point Operations / FLOPs) and classification accuracies of the models. For a kernel size k, the experiments showed that, despite improvements on parameter and computational efficiencies, models based on the proposed modules for the basic CNN (k = 3) architecture had lower accuracy levels compared to their baseline model. However, the huge saving on the number of parameters (> 88%) and FLOPs (> 84%) is significant compared to the drop in accuracy (< 4%). For the MobileNetV3 (k = 3 or k = 5) and EfficientNetB0 (k = 3 or k = 5), only the model based on the series module had lower accuracy than the baseline models. The models based on the parallel module were generally at par (within a margin of $$\pm$$ 1.5%) in terms of accuracy with their respective baselines. In relation to the ConvNext (k = 7) and the ResNet-56 (k = 3 or k = 11, or k = 19) architectures, other than the improved parameter and computational efficiencies, the classification accuracies of all the models based on the proposed modules were either at par or better than those of their baseline models. In all the above architectures, the use of the proposed modules reduced the number of convolutional parameters in the networks by at least 78% with fewer FLOPs relative to the baseline models

Optimizing convolutional neural networks on multi-core vector accelerator

A fingerprint is an impression left by the friction ridges of a human finger. A fingerprint classification system groups fingerprint according to their characteristics and therefore helps to match a fingerprint against an extensive database of fingerprints. The Henry classification system is widely used among fingerprint classification systems. Some researchers have used traditional machine learning or deep learning for fingerprint classification. Nevertheless, traditional algorithms cannot extract the depth features of the fingerprint, and most deep learning algorithms lack fingerprint image enhancement. So, this paper combined the Gabor Filter and Convolutional Neural Network to extract fingerprint features. The model has two channels, one is a Deep Convolutional Neural Network (DCNN), and the other is a Shallow Convolutional Neural Network (SCNN). The DCNN consists of a neural network with eight layers, which can extract deep features of the fingerprint. The SCNN consists of Gabor Filter and a neural network with two layers that can extract features from clear fingerprint images. This paper uses NIST Special Database 4 for experiments. Experimental results show that the model proposed in this paper has achieved 91.4% accuracy. Compared with other algorithms, this model has higher accuracy than others. It shows that combined with the Gabor Filter and Convolutional Neural Network can better extract the ridge features of fingerprint images.

We propose Absum, which is a regularization method for improving adversarial robustness of convolutional neural networks (CNNs). Although CNNs can accurately recognize images, recent studies have shown that the convolution operations in CNNs commonly have structural sensitivity to specific noise composed of Fourier basis functions. By exploiting this sensitivity, they proposed a simple black-box adversarial attack: Single Fourier attack. To reduce structural sensitivity, we can use regularization of convolution filter weights since the sensitivity of linear transform can be assessed by the norm of the weights. However, standard regularization methods can prevent minimization of the loss function because they impose a tight constraint for obtaining high robustness. To solve this problem, Absum imposes a loose constraint; it penalizes the absolute values of the summation of the parameters in the convolution layers. Absum can improve robustness against single Fourier attack while being as simple and efficient as standard regularization methods (e.g., weight decay and L1 regularization). Our experiments demonstrate that Absum improves robustness against single Fourier attack more than standard regularization methods. Furthermore, we reveal that robust CNNs with Absum are more robust against transferred attacks due to decreasing the common sensitivity and against high-frequency noise than standard regularization methods. We also reveal that Absum can improve robustness against gradient-based attacks (projected gradient descent) when used with adversarial training.

High-Efficiency Video Coding (HEVC) is the latest video coding standard developed by the Joint Collaborative Team on Video Coding (JCT-VC). An effective rate control mechanism plays a critical role in making the best possible use of the network resources. The coding performance can be maximised through the appropriate allocation of bits under the constraints of a total bit rate budget and the buffer size. Existing rate control strategies generally determine the rate control parameters according to previous encoding results. However, there is no priori-knowledge available when the first frame of a video sequence is encoded or scene changes occur. Consequently, it is challenging to choose appropriate rate control parameters in such cases. In order to address this issue, we investigated deep learning oriented methods and proposed a Convolutional Neural Network (CNN)-based rate control scheme for HEVC intra-coding. As the CNN has the advantage of local perception and the Coding Tree Unit's (CTU's) Quantisation Parameter (QP) value is determined during the CTU level rate control process, we applied the CNN at the CTU level as well to achieve an accurate QP prediction. An improved QP determination model (R-QP model) involving only two model coefficients (µ and v) was suggested to simplify the QP calculation process. A CNN was used to predict the model coefficients, namely µ and v. Our CNN has four convolution layers, two pooling layers and three fully connected layers. Each of the convolution layers contains a Rectified Linear Unit (ReLU). The last fully connected layer outputs the predicted values of µ and v. The images from the UCID dataset and the RAISE dataset were used to train the CNN. In order to obtain a universal model, none of the frames in the HEVC common test sequences were employed as training images. In order to achieve a reasonable bit budget allocation, a CNN-based framework was used to predict the bit consumption for each CTU according to the entire picture content that it contains. At the training stage, the network took the original images and the QP values as inputs, and the corresponding actual bit consumptions were used as the labels. Once the CNN coefficients have been obtained through training, the bit consumption for each CTU under different QP values can be directly predicted from the original images. The predicted bit consumption was subsequently used to readjust the bit budget for each CTU to achieve a more reasonable bit allocation. We implemented the proposed CNN-based rate control scheme in the HEVC reference software HM 16.9. As the priori-knowledge of the previous encoding results had not been considered in the proposed algorithm, only the first frame of each video sequence was encoded. The experimental results showed that our proposed scheme performs better than the default rate control implementation in the HEVC reference software, namely the BDBR is reduced by 1.33% on average while maintaining the same reconstructed picture quality. Compared with the state-of-the-art CNN-based rate control algorithm proposed by Li, our schemes achieved better coding efficiency. It can be concluded that the proposed R-QP model is effective in describing the rate-distortion relationship in HEVC, and the CNN-based bit allocation scheme is appropriate for ensuring that the best use of each bit is made.

Deep Convolutional Neural Networks for Large-scale Speech Tasks