Improving Convolutional Neural Networks Via Compacting Features

Deep learning has been successfully applied in classification of white blood cells (WBCs), however, accuracy and processing time are found to be less than optimal hindering it from getting its full potential. This is due to imbalanced dataset, intra-class compactness, inter-class separability and overfitting problems. The main research idea is to enhance the classification and prediction accuracy of blood images while lowering processing time through the use of deep convolutional neural network (DCNN) architecture by using the modified loss function. The proposed system consists of a deep neural convolution network (DCNN) that will improve the classification accuracy by using modified loss function along with regularization. Firstly, images are pre-processed and fed through DCNN that contains different layers with different activation function for the feature extraction and classification. Along with modified loss function with regularization, weight function aids in the classification of WBCs by considering weights of samples belonging to each class for compensating the error arising due to imbalanced dataset. The processing time will be counted by each image to check the time enhancement. The classification accuracy and processing time are achieved using the dataset-master. Our proposed solution obtains better classification performance in the given dataset comparing with other previous methods. The proposed system enhanced the classification accuracy of 98.92% from 96.1% and a decrease in processing time from 0.354 to 0.216 s. Less time will be required by our proposed solution for achieving the model convergence with 9 epochs against the current convergence time of 13.5 epochs on average, epoch is the formation white blood cells (WBCs) and the development of granular cells. The proposed solution modified loss function to solve the adverse effect caused due to imbalance dataset by considering weight and use regularization technique for overfitting problem.

Classification task is one of the most common tasks in machine learning. This supervised learning problem consists in assigning each input to one of a finite number of discrete categories. Classification task appears naturally in numerous applications, such as medical image processing, speech recognition, maintenance systems, accident detection, autonomous driving etc.In the last decade methods of deep learning have proven to be extremely efficient in multiple machine learning problems, including classification. Whereas the neural network architecture might depend a lot on data type and restrictions posed by the nature of the problem (for example, real-time applications), the process of its training (i.e. finding model’s parameters) is almost always presented as loss function optimization problem.Cross-entropy is a loss function often used for multiclass classification problems, as it allows to achieve high accuracy results.Here we propose to use a generalized version of this loss based on Renyi divergence and entropy. We remark that in case of binary labels proposed generalization is reduced to cross-entropy, thus we work in the context of soft labels. Specifically, we consider a problem of image classification being solved by application of convolution neural networks with mixup regularizer. The latter expands the training set by taking convex combination of pairs of data samples and corresponding labels. Consequently, labels are no longer binary (corresponding to single class), but have a form of vector of probabilities. In such settings cross-entropy and proposed generalization with Renyi divergence and entropy are distinct, and their comparison makes sense.To measure effectiveness of the proposed loss function we consider image classification problem on benchmark CIFAR-10 dataset. This dataset consists of 60000 images belonging to 10 classes, where images are color and have the size of 32×32. Training set consists of 50000 images, and the test set contains 10000 images.For the convolution neural network, we follow [1] where the same classification task was studied with respect to different loss functions and consider the same neural network architecture in order to obtain comparable results.Experiments demonstrate superiority of the proposed method over cross-entropy for loss function parameter value α < 1. For parameter value α > 1 proposed method shows worse results than cross-entropy loss function. Finally, parameter value α = 1 corresponds to cross-entropy.

Face recognition has made extraordinary progress owing to the advancement of deep convolutional neural networks (CNNs). The central task of face recognition, including face verification and identification, involves face feature discrimination. However, the traditional softmax loss of deep CNNs usually lacks the power of discrimination. To address this problem, recently several loss functions such as center loss, large margin softmax loss, and angular softmax loss have been proposed. All these improved losses share the same idea: maximizing inter-class variance and minimizing intra-class variance. In this paper, we propose a novel loss function, namely large margin cosine loss (LMCL), to realize this idea from a different perspective. More specifically, we reformulate the softmax loss as a cosine loss by L2 normalizing both features and weight vectors to remove radial variations, based on which a cosine margin term is introduced to further maximize the decision margin in the angular space. As a result, minimum intra-class variance and maximum inter-class variance are achieved by virtue of normalization and cosine decision margin maximization. We refer to our model trained with LMCL as CosFace. Extensive experimental evaluations are conducted on the most popular public-domain face recognition datasets such as MegaFace Challenge, Youtube Faces (YTF) and Labeled Face in the Wild (LFW). We achieve the state-of-the-art performance on these benchmarks, which confirms the effectiveness of our proposed approach.

Semantic-based image retrieval plays an important role in many practical applications, which aims to look for images with similar contents. Extracting discriminative representations of images is the real crux of this task. Directly utilizing the results of fully connected layers of convolutional neural networks (CNNs) is one of the best feature extraction methods. However, the fully connected layer is only supervised by the softmax loss, which only aims to maximize the accuracy of object classification and hardly pays attention to the spatial distribution of features, especially the intra-class and inter-class feature distances which are of great importance in semantic-based image retrieval. To compensate the performance degradation due to this reason, we try to address the spatial distribution of the features by two different loss functions. The first loss function jointly combines the softmax loss and the center loss in the training of the CNNs, which simultaneously ensures that the features of images with different contents are separable and the features of images in the same class are close. The second loss function is defined as an improved center loss which not only penalizes the distance between the obtained deep feature and the feature center of its own class, but also encourages a far distance between the feature and the feature centers of any other classes. We have conducted experiments on both ILSVRC data set and Caltech256 data set, demonstrated that the deep features got by our approaches can achieve better performance than other methods in semantic-based image retrieval, and the improved center loss can further outperform the joint supervision of the softmax loss and the center loss.

Facial Expression Recognition (FER) has demonstrated remarkable progress due to the advancement of deep Convolutional Neural Networks (CNNs). FER's goal as a visual recognition problem is to learn a mapping from the facial embedding space to a set of fixed expression categories using a supervised learning algorithm. Softmax loss as the de facto standard in practice fails to learn discriminative features for efficient learning. Center loss and its variants as promising solutions increase deep feature discriminability in the embedding space and enable efficient learning. They fundamentally aim to maximize intra-class similarity and inter-class separation in the embedding space. However, center loss and its variants ignore the underlying extreme class imbalance in challenging wild FER datasets. As a result, they lead to a separation bias toward majority classes and leave minority classes overlapped in the embedding space. In this paper, we propose a novel Discriminant Distribution-Agnostic loss (DDA loss) to optimize the embedding space for extreme class imbalance scenarios. Specifically, DDA loss enforces inter-class separation of deep features for both majority and minority classes. Any CNN model can be trained with the DDA loss to yield well separated deep feature clusters in the embedding space. We conduct experiments on two popular large-scale wild FER datasets (RAF-DB and AffectNet) to show the discriminative power of the proposed loss function.

As one of the indispensable components in convolutional neural network (CNN), loss function assists in updating parameters of CNN models during the training phase. Generally, different loss functions can assist convolutional neural network (CNN) to learn different feature representations, and different feature representations can be treated as different knowledge learned from objects. In this paper we introduce a novel training framework, namely Loss Transferring (LT), to improve the generalization ability of CNN. LT contains multiple training phases, and each training phase uses a different loss function. Under this framework, CNN models can combine different knowledge of objects by transferring the knowledge learned via one loss function to another. LT contains two components, i.e., loss function set and training strategy. In order to build appropriate loss function set, we establish two basic guides. And according to these basic guides, we design a new loss function in the last layer of CNN models (layer before softmax operation), namely Near Classifier Hyper-Plane (N-CHP) loss, which makes the learned object features belonging to the same category have the minimum intra-class distance and be near the classifier hyper-plane. Based on the two loss function set $\{\text {MSE, softmax}\}$ and $\{\text {N-CHP, softmax}\}$ , we setup two specific training methods, $LT_{\text {MSE, softmax}}$ and $LT_{\text {N-CHP, softmax}}$ , which can be universally applied to different CNN models with low additional computation cost. Meanwhile, two training strategies, multi-phase strategy 1 and multi-phase strategy 2, are further proposed to improve the training efficiency of LT. Extensive experimental results on shallow, moderate and deep models with four benchmark datasets, including MNIST, SVHN, CIFAR-10 and CIFAR-100, demonstrate that CNN models can bring obvious performance improvements when working with $LT_{\text {MSE, softmax}}$ and $LT_{\text {N-CHP, softmax}}$ , which verifies the effectiveness of LT and the proposed two basic guides.

Face recognition has witnessed significant progress due to the advances of deep convolutional neural networks (CNNs), the central task of which is how to improve the feature discrimination. To this end, several margin-based (e.g., angular, additive and additive angular margins) softmax loss functions have been proposed to increase the feature margin between different classes. However, despite great achievements have been made, they mainly suffer from three issues: 1) Obviously, they ignore the importance of informative features mining for discriminative learning; 2) They encourage the feature margin only from the ground truth class, without realizing the discriminability from other non-ground truth classes; 3) The feature margin between different classes is set to be same and fixed, which may not adapt the situations very well. To cope with these issues, this paper develops a novel loss function, which adaptively emphasizes the mis-classified feature vectors to guide the discriminative feature learning. Thus we can address all the above issues and achieve more discriminative face features. To the best of our knowledge, this is the first attempt to inherit the advantages of feature margin and feature mining into a unified loss function. Experimental results on several benchmarks have demonstrated the effectiveness of our method over state-of-the-art alternatives. Our code is available at http://www.cbsr.ia.ac.cn/users/xiaobowang/.

A balanced dataset is generally beneficial to underwater acoustic target recognition. However, the imbalanced class distribution is always meted out in a real scene. To address this, a weighted cross entropy loss function based on trigonometric function is proposed. Then, the proposed loss function is applied in a multi-scale residual convolutional neural network (named MR-CNN-A network) embedded with an attention mechanism for the recognition task. Firstly, a multi-scale convolution kernel is used to obtain multi-scale features. Then, an attention mechanism is used to fuse these multi-scale feature maps. Furthermore, a cosx-function-weighted cross-entropy loss function is used to deal with the class imbalance in underwater acoustic data. This function adjusts the loss ratio of each sample by adjusting the loss interval of every mini-batch based on cosx term to achieve a balanced total loss for each class. Two imbalanced underwater acoustic data sets, ShipsEar and autonomous underwater vehicle (self-collected data) are used to evaluate the proposed network. The experimental results show that the proposed network outperforms the support vector machine and a simple convolutional neural network. Compared with the other three loss functions, the proposed loss function achieves better stability and adaptability. The results strongly demonstrate the validity of the proposed loss function and the network.

In recent years, Convolutional neural networks (CNNs) have achieved start-of-art performance in the fault diagnosis field. If there is no available information on the unseen operating conditions, the model trained on the seen operating condition cannot perform well. One of the feasible strategies is to enhance the generalization ability of the network on various seen operating conditions. We introduce the center loss to the traditional CNN and build an end-to-end fault diagnosis framework (called CNN-C). By minimizing the intra-class variations, center loss cluster the learned features across various seen operating conditions. With the joint supervision of the center loss and the softmax loss, the learned features of the same class could minimize the domain difference across various seen operating conditions while the features of different classes are separable. The generalization ability of network is improved on unseen operating conditions. Compared with the shallow methods and traditional CNN, the proposed method is promising to deal with the fault diagnosis tasks of the bearing and gearbox.

With the development of deep learning technology, an enormous number of convolutional neural network (CNN) models have been proposed to address the challenging building extraction task from very high-resolution (VHR) remote sensing images. However, searching for better CNN architectures is time-consuming, and the robustness of a new CNN model cannot be guaranteed. In this paper, an improved boundary-aware perceptual (BP) loss is proposed to enhance the building extraction ability of CNN models. The proposed BP loss consists of a loss network and transfer loss functions. The usage of the boundary-aware perceptual loss has two stages. In the training stage, the loss network learns the structural information from circularly transferring between the building mask and the corresponding building boundary. In the refining stage, the learned structural information is embedded into the building extraction models via the transfer loss functions without additional parameters or postprocessing. We verify the effectiveness and efficiency of the proposed BP loss both on the challenging WHU aerial dataset and the INRIA dataset. Substantial performance improvements are observed within two representative CNN architectures: PSPNet and UNet, which are widely used on pixel-wise labelling tasks. With BP loss, UNet with ResNet101 achieves 90.78% and 76.62% on IoU (intersection over union) scores on the WHU aerial dataset and the INRIA dataset, respectively, which are 1.47% and 1.04% higher than those simply trained with the cross-entropy loss function. Additionally, similar improvements (0.64% on the WHU aerial dataset and 1.69% on the INRIA dataset) are also observed on PSPNet, which strongly supports the robustness of the proposed BP loss.

Improving metric-based few-shot learning with dynamically scaled softmax loss

While the deep convolutional neural network (DCNN) has achieved overwhelming success in various vision tasks, its heavy computational and storage overhead hinders the practical use of resource-constrained devices. Recently, compressing DCNN models has attracted increasing attention, where binarization-based schemes have generated great research popularity due to their high compression rate. In this article, we propose modulated convolutional networks (MCNs) to obtain binarized DCNNs with high performance. We lead a new architecture in MCNs to efficiently fuse the multiple features and achieve a similar performance as the full-precision model. The calculation of MCNs is theoretically reformulated as a discrete optimization problem to build binarized DCNNs, for the first time, which jointly consider the filter loss, center loss, and softmax loss in a unified framework. Our MCNs are generic and can decompose full-precision filters in DCNNs, e.g., conventional DCNNs, VGG, AlexNet, ResNets, or Wide-ResNets, into a compact set of binarized filters which are optimized based on a projection function and a new updated rule during the backpropagation. Moreover, we propose modulation filters (M-Filters) to recover filters from binarized ones, which lead to a specific architecture to calculate the network model. Our proposed MCNs substantially reduce the storage cost of convolutional filters by a factor of 32 with a comparable performance to the full-precision counterparts, achieving much better performance than other state-of-the-art binarized models.

Ship classification in synthetic aperture radar (SAR) images is essential in remote sensing but still full of challenges in the deep learning era. The unbalanced dataset and lack of models are two limitations. Upsampling with data augmentation and ratio batching are proposed to solve the first problem. Upsampling with data augmentation is upsampling by cropping and flipping. It can improve the diversity of the dataset. Ratio batching is realized by choosing the same amount of ships per class in each minibatch. It can make the model converge faster and better. To solve the second problem, a new loss function and convolutional neural network model are proposed. The new loss function can maximize the intraclass compactness and interclass separation simultaneously. Dense residual network has two submodules. One is the identity mapping through elementwise summation to reuse old features. The other is dense connection through concatenation to exploit new features. The designed architecture is suitable for the task of SAR ship classification. We use the confusion matrix and accuracy averaged on classes to measure the performance. From the experiments, we can find that the proposed ideas have excellent performance especially on the rare classes.

Convolutional neural networks (CNNs) have become the state-of-the-art in automatic feature learning which has led to outstanding performance on pattern recognition applications. On the other hand, methodologies for discriminant analysis on multiclass problems have been proposed to determine the discriminant contribution of each feature. In this paper, we combine the capabilities of CNN and multi-class discriminant analysis to propose a general data-driven methodology for feature learning and discriminability for texture classification. The whole pipeline has two main blocks: (a) An approach that understands intrinsic patterns in small image patches using CNNs customized for the focused problem; (b) A multi-class discriminant analysis technique, fed with the CNN output, to select the most discriminant features for classification tasks. Our experimental results have shown that the feature spaces generated by the combination of CNN and discriminant analysis allow higher recognition rates using very much less CNN features in five-class granite image analysis

Deep convolutional neural network (DCNN) is a powerful model for learning significant data at multiple levels of abstraction form an input image. However, training DCNN is often complicated because of parameter initialization, overfitting and convergence problems. Hence this work has been targeted to overcome the challenges of training DCNN with an optimized model. This chapter describes a deep learning framework for image classification with cifar-10 dataset. The model contains a set of convolutional layers with rectified linear unit activation function, max-pooling layers, and a fully-connected layer with softmax activation function. This model learns the features automatically and classifies the image without using the hand-crafted image based features. In this investigation, various optimizers have been applied in gradient descent technique for minimizing the loss function. Model with Adam optimizer constantly minimizes the objective function compared with other standard optimizers such as momentum, Rmsprop, and Adadelta. Dropout and batch normalization techniques are adapted to improve the model performance further by avoiding overfitting. Dropout function deactivates the insignificant node form the model after every epoch. The initialization of a large number of parameters in DCNN is regularized by batch normalization. Results obtained from the proposed model shows that batch normalization with dropout significantly improves the accuracy of the model with the tradeoff of computational complexity.