Багатоетапний метод глибинного навчання з попереднім самонавчанням для класифікаційного аналізу дефектів стічних труб

There have been various loss functions proposed to improve the training of neural networks with sigmoid activation output nodes. For neural networks with softmax activation output nodes, the cross-entropy loss function is commonly used for training and several attempts have been made to improve the performance of such networks by modifying the standard cross-entropy loss. However, rather than simply aiming to improve overall classification accuracy, it is often necessary to address misclassification costs differently depending on their real-world importance. In practice, the cost of errors can vary greatly across domains such as finance, security, insurance, and healthcare. From this perspective, this paper proposes a modified cross-entropy loss function designed to control the training of neural networks with softmax output nodes. The effectiveness of the proposed loss function is demonstrated through simulations on the CEDAR handwritten digit recognition task, showing that classification performance can be adjusted according to the order of the modified loss function. This approach will serve as the basis for designing a novel cost-sensitive learning method tailored to neural networks with softmax outputs.

With the development of deep learning technology, an enormous number of convolutional neural network (CNN) models have been proposed to address the challenging building extraction task from very high-resolution (VHR) remote sensing images. However, searching for better CNN architectures is time-consuming, and the robustness of a new CNN model cannot be guaranteed. In this paper, an improved boundary-aware perceptual (BP) loss is proposed to enhance the building extraction ability of CNN models. The proposed BP loss consists of a loss network and transfer loss functions. The usage of the boundary-aware perceptual loss has two stages. In the training stage, the loss network learns the structural information from circularly transferring between the building mask and the corresponding building boundary. In the refining stage, the learned structural information is embedded into the building extraction models via the transfer loss functions without additional parameters or postprocessing. We verify the effectiveness and efficiency of the proposed BP loss both on the challenging WHU aerial dataset and the INRIA dataset. Substantial performance improvements are observed within two representative CNN architectures: PSPNet and UNet, which are widely used on pixel-wise labelling tasks. With BP loss, UNet with ResNet101 achieves 90.78% and 76.62% on IoU (intersection over union) scores on the WHU aerial dataset and the INRIA dataset, respectively, which are 1.47% and 1.04% higher than those simply trained with the cross-entropy loss function. Additionally, similar improvements (0.64% on the WHU aerial dataset and 1.69% on the INRIA dataset) are also observed on PSPNet, which strongly supports the robustness of the proposed BP loss.

Classification task is one of the most common tasks in machine learning. This supervised learning problem consists in assigning each input to one of a finite number of discrete categories. Classification task appears naturally in numerous applications, such as medical image processing, speech recognition, maintenance systems, accident detection, autonomous driving etc.In the last decade methods of deep learning have proven to be extremely efficient in multiple machine learning problems, including classification. Whereas the neural network architecture might depend a lot on data type and restrictions posed by the nature of the problem (for example, real-time applications), the process of its training (i.e. finding model’s parameters) is almost always presented as loss function optimization problem.Cross-entropy is a loss function often used for multiclass classification problems, as it allows to achieve high accuracy results.Here we propose to use a generalized version of this loss based on Renyi divergence and entropy. We remark that in case of binary labels proposed generalization is reduced to cross-entropy, thus we work in the context of soft labels. Specifically, we consider a problem of image classification being solved by application of convolution neural networks with mixup regularizer. The latter expands the training set by taking convex combination of pairs of data samples and corresponding labels. Consequently, labels are no longer binary (corresponding to single class), but have a form of vector of probabilities. In such settings cross-entropy and proposed generalization with Renyi divergence and entropy are distinct, and their comparison makes sense.To measure effectiveness of the proposed loss function we consider image classification problem on benchmark CIFAR-10 dataset. This dataset consists of 60000 images belonging to 10 classes, where images are color and have the size of 32×32. Training set consists of 50000 images, and the test set contains 10000 images.For the convolution neural network, we follow [1] where the same classification task was studied with respect to different loss functions and consider the same neural network architecture in order to obtain comparable results.Experiments demonstrate superiority of the proposed method over cross-entropy for loss function parameter value α < 1. For parameter value α > 1 proposed method shows worse results than cross-entropy loss function. Finally, parameter value α = 1 corresponds to cross-entropy.

We replicated and extended a project by Dickinson and Burke (1996) that concerned human causal judgement. In a medical diagnostic setting, college students' ratings of the causal efficacy of target cues showed retrospective revaluation: relative to a proper control condition, ratings of target cues both increased ("recovery from overshadowing") and decreased ("backward blocking") during a second stage of training in which competing cues, but not target cues, were presented. These changes in causal judgements were exhibited only by subjects who had learned which target and competing cues were paired with one another during the first stage of training. These results cannot be explained by the Rescorla-Wagner (1972) model of associative learning, but they can be explained by the revised model of Van Hamme and Wasserman (1994); the revised model assigns non-zero salience to non-presented target stimuli whose memories or representations are retrieved by competing stimuli that had previously been paired with those target stimuli.

A neural network based compensation strategy for five-axis machine tool error correction has been developed. It is capable of significantly reducing geometric, kinematic and thermal errors between the tool tip and the workpiece. The error prediction ability of the neural network is demonstrated using a simulation of the compensation strategy. A two-stage training procedure for learning the weights of the neural network is proposed. The first stage uses a simulation model to train the dominant relationships between the errors and the sensors. The second stage of training would use actual measurements made on a five-axis machine tool to learn the remaining subtle relationships between the errors and the sensors. The simulation of the machine is based on Homogeneous Transformation Matrices (HTM). They model the position and orientation errors in each structural component. These error matrices are multiplied together to find the error between the tool and the workpiece. The coefficients required by the HTM model were selected so that the simulation model closely matched the measured performance of the five-axis machine tool. Thermocouples were used to measure temperature, and a laser interferometer, with the appropriate optics, was used for position and orientation error measurement. Preliminary simulation results based on an analysis of the spindle support arm and the Z column are available for the first stage of training. They indicate that a reduction in error from 100μm to 5μm is possible using a neural network model.

A balanced dataset is generally beneficial to underwater acoustic target recognition. However, the imbalanced class distribution is always meted out in a real scene. To address this, a weighted cross entropy loss function based on trigonometric function is proposed. Then, the proposed loss function is applied in a multi-scale residual convolutional neural network (named MR-CNN-A network) embedded with an attention mechanism for the recognition task. Firstly, a multi-scale convolution kernel is used to obtain multi-scale features. Then, an attention mechanism is used to fuse these multi-scale feature maps. Furthermore, a cosx-function-weighted cross-entropy loss function is used to deal with the class imbalance in underwater acoustic data. This function adjusts the loss ratio of each sample by adjusting the loss interval of every mini-batch based on cosx term to achieve a balanced total loss for each class. Two imbalanced underwater acoustic data sets, ShipsEar and autonomous underwater vehicle (self-collected data) are used to evaluate the proposed network. The experimental results show that the proposed network outperforms the support vector machine and a simple convolutional neural network. Compared with the other three loss functions, the proposed loss function achieves better stability and adaptability. The results strongly demonstrate the validity of the proposed loss function and the network.

Sleep staging is an indispensable indicator for measuring sleep quality and evaluating sleep disorders. Deep learning methods have been successfully applied to automatic sleep staging (ASS) based on EEG signals, achieving significant progress. However, previous studies have been limited to extracting local features of EEG signals while neglecting the importance of global features. To solve this problem, we propose a novel ASS model named LGSleepNet, which consists of asymmetric Siamese neural network (ASNN), Deep adaptive orthogonal fusion (DAOF) block, and weighted polynomial cross entropy (WPCE) loss function. Specifically, the ASNN is capable of simultaneously extracting local and global features from EEG signals, which provides diverse semantic representations for sleep staging. Moreover, a DAOF block is proposed to eliminate the information redundancy and semantic deviation among heterogeneous features by orthogonalization and adaptive fusion, which strengthens the correlation representation between local and global features. Ultimately, a weighted polynomial cross entropy (WPCE) loss function is designed to improve the decision-making ability of the classification head and alleviate the problem of sample imbalance. We evaluate the LGSleepNet on three publicly available datasets, namely Sleep-EDF-20, Sleep-EDF-78, and SVUH-UCD, which achieves macro F1-scores of 80.7%, 76.0%, and 75.1% and overall accuracy of 86.0%, 82.3%, and 76.3%, respectively. The experimental results indicate that the LGSleepNet performs at an advanced level compared to other state-of-the-art methods.

Training and credentialing in laparoscopic surgery — the need of the day

A new methodology was used to train two rats on a conditional discriminationtask that used multi-modal stimuli that differed along a number of sensory dimensions (i.e., visual, tactile and olfactory). Rats were trained across two stages to move from a multi-modal sample to a multi-modal comparison for food reward (i.e., given Wood as sample. move to the Rubber comparison [A1-B1], and given Foam as sample. move to the Carpet comparison [A2-B2]). Having reached criterion in the second stage of training. rats were then exposed to blocks of trials where symmetry probe test trials (B1-A1 and B2- A2) were interspersed with reinforced and non-reinforced baseline trials. The advantages of the methodology for facilitating multi-modal conditional discrimination responding in rats is discussed.

Convolutional neural networks (CNNs) have shown great advantages in computer vision fields, and loss functions are of great significance to their gradient descent algorithms. Softmax loss, a combination of cross-entropy loss and Softmax function, is the most commonly used one for CNNs. Hence, it can continuously increase the discernibility of sample features in classification tasks. Intuitively, to promote the discrimination of CNNs, the learned features are desirable when the inter-class separability and intra-class compactness are maximized simultaneously. Since Softmax loss hardly motivates this inter-class separability and intra-class compactness simultaneously and explicitly, we propose a new method to achieve this simultaneous maximization. This method minimizes the distance between features of homogeneous samples along with Softmax loss and thus improves CNNs' performance on vision-related tasks. Experiments on both visual classification and face verification datasets validate the effectiveness and advantages of our method.

This work is concerned with using deep neural networks for estimating binary masks within a speech enhancement framework. We first examine the effect of supplementing the audio features used in mask estimation with visual speech information. Visual speech is known to be robust to noise although not necessarily as discriminative as audio features, particularly at higher signal-to-noise ratios. Furthermore, most DNN approaches to mask estimate use the cross-entropy (CE) loss function which aims to maximise classification accuracy. However, we first propose a loss function that aims to maximise the hit minus false-alarm (HIT-FA) rate of the mask, which is known to correlate more closely to speech intelligibility than classification accuracy. We then extend this to a hybrid loss function that combines both the CE and HIT-FA loss functions to provide a balance between classification accuracy and HIT-FA rate of the resulting masks. Evaluations of the perceptually motivated loss functions are carried out using the GRID and larger RM-3000 datasets and show improvements to HIT-FA rate and ESTOI across all noises and SNRs tested. Tests also found that supplementing audio with visual information into a single bimodal audio-visual system gave best performance for all measures and conditions tested.

Corneal confocal microscopy (CCM) has been advocated as a non-invasive technique for objective diagnosis of very early neuropathy in patients by scanning the corneal subbasal nerve plexus. The obtained images provide a range of research opportunities to be explored. Current research revolves around providing automated solutions for nerve segmentation in CCM images. In this paper, we address the problem of low sensitivity of nerves in automatic segmentation caused by imbalanced pixel distribution in the CCM images. We evaluate three loss functions with varying parameters in the deep learning network, U-Net, on the images and discuss the results. We have observed that the optimal training time and convergence time for the Tversky loss function is better than the binary cross entropy and dice loss functions. This helps in getting better results faster, implying a quicker diagnosis.

Scene classification of high-resolution remote sensing images (HRRSI) is one of the most important means of land-cover classification. Deep learning techniques, especially the convolutional neural network (CNN) have been widely applied to the scene classification of HRRSI due to the advancement of graphic processing units (GPU). However, they tend to extract features from the whole images rather than discriminative regions. The visual attention mechanism can force the CNN to focus on discriminative regions, but it may suffer from the influence of intra-class diversity and repeated texture. Motivated by these problems, we propose an attention-based deep feature fusion (ADFF) framework that constitutes three parts, namely attention maps generated by Gradient-weighted Class Activation Mapping (Grad-CAM), a multiplicative fusion of deep features and the center-based cross-entropy loss function. First of all, we propose to make attention maps generated by Grad-CAM as an explicit input in order to force the network to concentrate on discriminative regions. Then, deep features derived from original images and attention maps are proposed to be fused by multiplicative fusion in order to consider both improved abilities to distinguish scenes of repeated texture and the salient regions. Finally, the center-based cross-entropy loss function that utilizes both the cross-entropy loss and center loss function is proposed to backpropagate fused features so as to reduce the effect of intra-class diversity on feature representations. The proposed ADFF architecture is tested on three benchmark datasets to show its performance in scene classification. The experiments confirm that the proposed method outperforms most competitive scene classification methods with an average overall accuracy of 94% under different training ratios.

Basketball foul action usually involves multiple features, and it is difficult to extract effective features from complex and diverse features. Therefore, a feature extraction algorithm for basketball players' foul action based on attention mechanism is proposed. On the basis of ResNet50 network model, the attention mechanism is integrated to build a basketball player foul action feature extraction model. The machine vision system is used to obtain basketball player action video as the input of the model. The space attention module and time attention module are, respectively, introduced into the video slow frame rate branch and video fast frame rate branch of ResNet50 network. By making the network give greater weight to the key areas of a single frame of video, and improving the network's attention to important video frames, the best spatio-temporal characteristics of foul actions are obtained. After fusion, the feature fusion results are input into the classifier with the cross-entropy loss function as the loss function, and the probability value of each possible foul action and the tag target probability are output, complete basketball player foul action feature extraction. The experimental results show that the algorithm can effectively identify the identification of basketball players, the cumulative matching feature value of foul action recognition can reach more than 95%, and the average accuracy is more than 70%; the F1 value and stability are high, which can reduce the error caused by data fluctuation and noise; the error rate of real-time detection is less than 4.5%, the omission rate is less than 4.7%, the detection time is lower than 14 ms, and the application effect is good.

Plant fungal diseases are one of the most important causes of crop yield losses. Therefore, plant disease identification algorithms have been seen as a useful tool to detect them at early stages to mitigate their effects. Although deep-learning based algorithms can achieve high detection accuracies, they require large and manually annotated image datasets that is not always accessible, specially for rare and new diseases. This study focuses on the development of a plant disease detection algorithm and strategy requiring few plant images (Few-shot learning algorithm). We extend previous work by using a novel challenging dataset containing more than 100,000 images. This dataset includes images of leaves, panicles and stems of five different crops (barley, corn, rape seed, rice, and wheat) for a total of 17 different diseases, where each disease is shown at different disease stages. In this study, we propose a deep metric learning based method to extract latent space representations from plant diseases with just few images by means of a Siamese network and triplet loss function. This enhances previous methods that require a support dataset containing a high number of annotated images to perform metric learning and few-shot classification. The proposed method was compared over a traditional network that was trained with the cross-entropy loss function. Exhaustive experiments have been performed for validating and measuring the benefits of metric learning techniques over classical methods. Results show that the features extracted by the metric learning based approach present better discriminative and clustering properties. Davis-Bouldin index and Silhouette score values have shown that triplet loss network improves the clustering properties with respect to the categorical-cross entropy loss. Overall, triplet loss approach improves the DB index value by 22.7% and Silhouette score value by 166.7% compared to the categorical cross-entropy loss model. Moreover, the F-score parameter obtained from the Siamese network with the triplet loss performs better than classical approaches when there are few images for training, obtaining a 6% improvement in the F-score mean value. Siamese networks with triplet loss have improved the ability to learn different plant diseases using few images of each class. These networks based on metric learning techniques improve clustering and classification results over traditional categorical cross-entropy loss networks for plant disease identification.