A Gender-Aware Deep Neural Network Structure for Speech Recognition

DBMiP: A pre-training method for information propagation over deep networks

Recently deep-learning (DL) techniques have been widely adopted in side-channel power analysis. A DL-assisted SCA generally consists of two phases: a deep neural network (DNN) training phase and a follow-on attack phase using the trained DNN. However, currently the two phases are not well aligned, as there is no conclusion on what metric used in the training can result in the most effective attack in the second phase. When traditional loss functions such as negative log-likelihood (NLL) are used in training a DNN, the trained model does not yield optimal follow-on attack. Recently some information theoretical SCA leakage metrics are proposed, either as the validation metric to stop the DNN training with traditional loss functions, or as both the validation metric and the training loss function. None of those proposed metrics, however, directly measures the SCA effectiveness. We propose to conduct DNN training directly with a common SCA effectiveness metric, Guessing Entropy (GE). We overcome the prior practical difficulty of using GE in DNN training by utilizing the GEEA estimation algorithm introduced in CHES 2020. We show that using GEEA as either the validation metric or the loss function produces DNN models that lead to much more effective follow-on attacks. Our work consolidates the DL-assisted SCA framework with a consistent metric, which shows great potential to be adopted as the universal SCA-oriented DNN training framework.

Deep Learning (DL) has become a prominent machine learning technique due to the availability of efficient computational resources in the form of Graphics Processing Units (GPUs), large-scale datasets and a variety of models. The newer generation of GPUs are being designed with special emphasis on optimizing performance for DL applications. Also, the availability of easy-to-use DL frameworks—like PyTorch and TensorFlow— has enhanced productivity of domain experts to work on their custom DL applications from diverse domains. However, existing Deep Neural Network (DNN) training approaches may not fully utilize the newly emerging powerful GPUs like the NVIDIA A100—this is the primary issue that we address in this paper. Our motivating analyses show that the GPU utilization on NVIDIA A100 can be as low as 43% using traditional DNN training approaches for small-to-medium DL models and input data size. This paper proposes AccDP—a data-parallel distributed DNN training approach—to accelerate GPU-based DL applications. AccDP exploits the Message Passing Interface (MPI) communication library coupled with the NVIDIA’s Multi-Process Service (MPS) to increase the amount of work assigned to parallel GPUs resulting in higher utilization of compute resources. We evaluate our proposed design on different small-to-medium DL models and input sizes on the state-of-the-art HPC clusters. By injecting more parallelism into DNN training using our approach, the evaluation shows up to 58% improvement in training performance on a single GPU and up to 62% on 16 GPUs compared to regular DNN training. Furthermore, we conduct an in-depth characterization to determine the impact of several DNN training factors and best practices—including the batch size and the number of data loading workers— to optimally utilize GPU devices. To the best of our knowledge, this is the first work that explores the use of MPS and MPI to maximize the utilization of GPUs in distributed DNN training.

Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural network due to its deep architecture. In this paper we investigate the nature of unsupervised learning in restricted Boltzmann machine. We have proved that maximization of the log-likelihood input data distribution of restricted Boltzmann machine is equivalent to minimizing the cross-entropy and to special case of minimizing the mean squared error. Thus the nature of unsupervised learning is invariant to different training criteria. As a result we propose a new technique called "REBA" for the unsupervised training of deep neural networks. In contrast to Hinton's conventional approach to the learning of restricted Boltzmann machine, which is based on linear nature of training rule, the proposed technique is founded on nonlinear training rule. We have shown that the classical equations for RBM learning are a special case of the proposed technique. As a result the proposed approach is more universal in contrast to the traditional energy-based model. We demonstrate the performance of the REBA technique using wellknown benchmark problem. The main contribution of this paper is a novel view and new understanding of an unsupervised learning in deep neural networks.

A point cloud deep neural network metamodel method for aerodynamic prediction

Survey on Deep Neural Networks in Speech and Vision Systems

This study presents a fault diagnosis system for vehicle heating, ventilation and air conditioning (HVAC) acoustic signal with various feature extractions in deep learning neural network. Traditionally, sound used for fault diagnosis or signal classification is observed the difference of energy in time or frequency domains. Unfortunately, the frequency smearing effect often arises in some critical conditions. In the present study, discrete wavelet transform (DWT) and wavelet packet transform (WPT) are proposed in fault diagnosis. Meanwhile, when using mechanical learning methods, the data are relatively large, in order to reduce the amount of data, DWT and WPT low-frequency decomposition could be used to improve the performance. Furthermore, the signal characteristics more comprehensive, this study attempts to use the feature extraction method of wavelet packet conversion to improve the signal characteristics. In the experiment process, the operation state of the blade blower in the vehicle air conditioner, four different faults were designed, test database was established through sound to classify, and identify the data using deep neural networks to achieve the purpose of blower fault diagnosis. In data analysis, the original signal is presented through wavelet packet decomposition and discrete packet conversion technology, compared with traditional time and frequency domain signals to explore the identification rate, identification speed and related issues. Experimental results show that using WPT combined with deep neural networks have good fault diagnosis and discrimination capabilities, training, and identification time is shorter than time-frequency domain signals.

In recent years, growing interest in deep learning neural networks has raised a question on how they can be used for effective processing of high-dimensional datasets produced by hyperspectral imaging (HSI). HSI, traditionally viewed as being within the scope of remote sensing, is used in non-invasive substance classification. One of the areas of potential application is forensic science, where substance classification on the scenes is important. An example problem from that area—blood stain classification—is a case study for the evaluation of methods that process hyperspectral data. To investigate the deep learning classification performance for this problem we have performed experiments on a dataset which has not been previously tested using this kind of model. This dataset consists of several images with blood and blood-like substances like ketchup, tomato concentrate, artificial blood, etc. To test both the classic approach to hyperspectral classification and a more realistic application-oriented scenario, we have prepared two different sets of experiments. In the first one, Hyperspectral Transductive Classification (HTC), both a training and a test set come from the same image. In the second one, Hyperspectral Inductive Classification (HIC), a test set is derived from a different image, which is more challenging for classifiers but more useful from the point of view of forensic investigators. We conducted the study using several architectures like 1D, 2D and 3D convolutional neural networks (CNN), a recurrent neural network (RNN) and a multilayer perceptron (MLP). The performance of the models was compared with baseline results of Support Vector Machine (SVM). We have also presented a model evaluation method based on t-SNE and confusion matrix analysis that allows us to detect and eliminate some cases of model undertraining. Our results show that in the transductive case, all models, including the MLP and the SVM, have comparative performance, with no clear advantage of deep learning models. The Overall Accuracy range across all models is 98–100% for the easier image set, and 74–94% for the more difficult one. However, in a more challenging inductive case, selected deep learning architectures offer a significant advantage; their best Overall Accuracy is in the range of 57–71%, improving the baseline set by the non-deep models by up to 9 percentage points. We have presented a detailed analysis of results and a discussion, including a summary of conclusions for each tested architecture. An analysis of per-class errors shows that the score for each class is highly model-dependent. Considering this and the fact that the best performing models come from two different architecture families (3D CNN and RNN), our results suggest that tailoring the deep neural network architecture to hyperspectral data is still an open problem.

At present, deep learning is becoming one of the most popular approach to creation of the artificial intelligences systems such as speech recognition, natural language processing, computer vision and so on. The paper presents a historical overview of deep learning in neural networks. The model of the artificial neural network is described as well as the learning algorithms for neural networks including the error backpropagation algorithm, which is used to train deep neural networks. The development of neural networks architectures is presented including neocognitron, autoencoders, convolutional neural networks, restricted Boltzmann machine, deep belief networks, long short-term memory, gated recurrent neural networks, and residual networks. Training deep neural networks with many hidden layers is impeded by the vanishing gradient problem. The paper describes the approaches to solve this problem that provide the ability to train neural networks with more than hundred layers. An overview of popular deep learning libraries is presented. Nowadays, for computer vision tasks convolutional neural networks are utilized, while for sequence processing, including natural language processing, recurrent networks are preferred solution, primarily long short-term memory networks and gated recurrent neural networks.

The sensitivity of wide parameter-space searches for continuous gravitational waves (CWs) is limited by their high computational cost. Deep learning is being studied as an alternative method to replace various aspects of a CW search. In previous work [], new design principles were presented for deep neural network (DNN) search of CWs and such DNNs were trained to perform a targeted search with matched-filtering sensitivity. In this paper, we adapt these design principles to build a DNN architecture for wide parameter-space searches in 10 days of data from two detectors (H1 and L1). We train a DNN for each of the benchmark cases: six all-sky searches and eight directed searches at different frequencies in the search band of 20–1000 Hz. We compare our results to the DNN sensitivity achieved from Dreissigacker and Prix [] and find that our trained DNNs are more sensitive in all the cases. The absolute improvement in detection probability ranges from 6.5% at 20 Hz to 38% at 1000 Hz in the all-sky cases and from 1.5% at 20 Hz to 59.4% at 500 Hz in the directed cases. An all-sky DNN trained on the entire search band of 20–1000 Hz shows a high sensitivity at all frequencies, providing a proof of concept for training a single DNN to perform the entire search. We also study the generalization of the DNN performance to signals with different signal amplitude, frequency, and the dependence of the DNN sensitivity on sky position. Published by the American Physical Society 2024

Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. Driven partly by rapidly-expanding literature on DNN approximation theory showing they can approximate a rich variety of functions, such tools are increasingly being considered for problems in scientific computing. Yet, unlike traditional algorithms in this field, little is known about DNNs from the principles of numerical analysis, e.g., stability, accuracy, computational efficiency and sample complexity. In this paper we introduce a computational framework for examining DNNs in practice, and use it to study empirical performance with regard to these issues. We study performance of DNNs of different widths & depths on test functions in various dimensions, including smooth and piecewise smooth functions. We also compare DL against best-in-class methods for smooth function approx. based on compressed sensing (CS). Our main conclusion from these experiments is that there is a crucial gap between the approximation theory of DNNs and their practical performance, with trained DNNs performing relatively poorly on functions for which there are strong approximation results (e.g. smooth functions), yet performing well in comparison to best-in-class methods for other functions. To analyze this gap further, we provide some theoretical insights. We establish a practical existence theorem, asserting existence of a DNN architecture and training procedure that offers the same performance as CS. This establishes a key theoretical benchmark, showing the gap can be closed, albeit via a strategy guaranteed to perform as well as, but no better than, current best-in-class schemes. Nevertheless, it demonstrates the promise of practical DNN approx., by highlighting potential for better schemes through careful design of DNN architectures and training strategies.

Gait recognition is a noncontact biometric procedure that determines the identity or health status of a person by analyzing his or her walking posture and habits, including skeletal and joint movements. The most remarkable feature of this method is the possibility of conducting recognition without demanding much cooperation from participants. Therefore, this recognition technique has attracted much attention from scholars. Additionally, because of the rapid development of graphics processing unit technology, related hardware and computation performance, the applications of deep-learning technology are considerably enhanced. The objective of this study was to apply a deep neural network (DNN), which employs deep-learning technology, to achieve gait-based automatic pedestrian detection and recognition. In contrast to using wearable devices to precisely capture skeletal and joint movements, pedestrian color-image sequences were used as input in this study. Subsequently, a pretraining convolutional neural network (CNN) was employed to capture pedestrian location and extract pedestrian dense optical flow to serve as concrete low-level feature inputs. Then, a finely-tuned DNN based on the wide residual network was employed to extract high-level abstract features. In addition, to overcome the difficulty of obtaining local temporal features by using a 2D CNN, part of the 3D convolutional structure was introduced into the CNN. This design enabled use of limited memory to acquire more effective features and enhance the DNN performance. The experimental results show that the proposed method has exceptional performance for pedestrian detection and recognition.

One of the key aspects of a successful drilling is effective optimization of ROP (Rate of Penetration). Because of the complexity and heterogeneity of formation permeability, the traditional ROP analysis method are limited by drilling prediction. With the accumulation of geological data and drilling records, new methods such as artificial neural network and particle swarm optimization have become powerful tools for obtaining optimization parameters. A ROP optimization method based on deep learning neural network and particle swarm optimization is proposed. Firstly, the prediction model of target wells is established from well logging data by using deep learning neural network. Secondly, the optimized wellbore operation parameters are obtained by using PSO algorithm. At last, the RNN learning algorithm is updated by introducing recovery factor. And also, for the sake of the realization of constraints, a penalty function is introduced. After analyzed logging data of a group of wells in Shunbei area, the experimental results showed that this method can effectively use engineering data to predict drilling rate and optimize drilling parameters.

A method that enables an industrial robot to accomplish the peg-in-hole task for holes in concrete is proposed. The proposed method involves slightly detaching the peg from the wall, when moving between search positions, to avoid the negative influence of the concrete’s high friction coefficient. It uses a deep neural network (DNN), trained via reinforcement learning, to effectively find holes with variable shape and surface finish (due to the brittle nature of concrete) without analytical modeling or control parameter tuning. The method uses displacement of the peg toward the wall surface, in addition to force and torque, as one of the inputs of the DNN. Since the displacement increases as the peg gets closer to the hole (due to the chamfered shape of holes in concrete), it is a useful parameter for inputting in the DNN. The proposed method was evaluated by training the DNN on a hole 500 times and attempting to find 12 unknown holes. The results of the evaluation show the DNN enabled a robot to find the unknown holes with average success rate of 96.1% and average execution time of 12.5 seconds. Additional evaluations with random initial positions and a different type of peg demonstrate the trained DNN can generalize well to different conditions. Analyses of the influence of the peg displacement input showed the success rate of the DNN is increased by utilizing this parameter. These results validate the proposed method in terms of its effectiveness and applicability to the construction industry.

Seed is a little embryonic plant that can be used to introduce plant infections to new areas while also allowing them to survive from one cropping season to the next. Seed health is a well-known component in modern agricultural science for achieving the required plant population and yield. Seed-borne fungus are a major biotic constraint in seed production around the world. The detection of seed-borne pathogens by seed health testing is a crucial step in the treatment of crop diseases. Speed and accuracy are critical requirements for longterm economic growth, competitiveness, and sustainability in agricultural output. Because human judgements in identifying objects and situations are variable, subjective, and delayed, seed prediction activities are costly and unreliable. Machine vision technology provides a nondestructive, cost-effective, quick, and accurate option for automated procedures. Seed variety, seed type (country seed or hybrid seed), seed health, and purity prediction were the four basic processes. We began the first procedure by aligning the seed bodies in the same direction using a seed orientation approach. Then, to detect atypical physical seed samples, a quality screening procedure was used. Their physical characteristics, such as shape, colour, and texture, were retrieved to serve as data representations for the prediction. This research introduces a new fuzzy cognitive map (FCM) model based on deep learning neural networks that predicts seed purity tests using data from biological investigations. The relevant data features from the seed test are extracted by FCM, which then effectively initialises the deep neural networks. The Levenberg–Marquardt (LM) technique for deep neural networks was discovered to improve seed purity test prediction. Four statistical machine learning algorithms (BP-ANN, Multivariate regression, and FCMLM deep learning). Furthermore, we demonstrated an improvement in the system''''s overall performance in terms of data quality, including seed orientation and quality screening. In independent numerical testing, the correlation coefficient between predicted values and true values acquired from experiments reached 0.9.