Modeling nonlinear dynamic objects using pre-trained time delay neural networks

The paper is devoted to resolving the contradiction between the accuracy of modeling nonlinear dynamics and the speed of model construction under conditions of limited computing resources. The aim of the work is to reduce the time for building models of nonlinear dynamic objects with continuous characteristics in the form of neural networks while ensuring a given modeling accuracy. This goal is achieved by developing a new method for synthesizing intelligent systems based on the superposition of pre-trained (support) models in the form of neural networks that reflect the basic properties of the object. The scientific novelty of the work lies in the development of a method for identifying nonlinear dynamic objects in the form of neural networks with time delays based on a set of pre-trained neural network models that reflect the basic properties of the subject area. In contrast to the traditional approach to pre-training, the developed method allows building models of lower complexity and with shorter training time while ensuring the required accuracy. To determine the initial parameters of the model, expressions based on the superposition of reference models in the form of neural networks are proposed. The practical utility of the work lies in the development of an algorithm for the method of reference models for training neural networks with time delays, which can significantly reduce the training time of neural networks without losing model accuracy

In this paper, an evaluation of various discriminant neural networks classifiers for text- independent speaker verification problem is presented. Each person to be verified has a personalized neural network model. A new classifier called neural tree network (NTN) is also examined for this application. The memoryless feedforward neural network architecture makes decisions based on static features. Time delay neural network (TDNNs) have proved to be an efficient way to handle the dynamic nature of speech. Furthermore, a model called recurrent time delay neural networks (RTDNNs), obtained through a local feedback connection at the first hidden layer level of TDNNs is investigated. The training is carried out by backpropagation for sequence algorithm. The database used is a subset of the TIMIT database consisting of 38 speakers from the same dialect region. The NTN is compared with the MLP, TDNN, and RTDNN. It is shown that NTN is found to perform better than the other neural networks classifiers. Also, a little bit performance improvement was achieved due to the addition of temporal information for text-independent speaker verification problem using TDNNs and RTDNNs. Finally, we described the experimental results obtained using different neural network models.© (1994) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Classification of acute myeloid leukemia by pre-trained deep neural networks: A comparison with different activation functions.

Neural style transfer has recently received significant attention and demonstrated amazing results. An efficient solution proposed by Johnson et al. trains feed-forward convolutional neural networks by defining and optimizing perceptual loss functions. Such methods are typically based on high-level features extracted from pre-trained neural networks, where the loss functions contain two components: style loss and content loss. However, such pre-trained networks are originally designed for object recognition, and hence the high-level features often focus on the primary target and neglect other details. As a result, when input images contain multiple objects potentially at different depths, the resulting images are often unsatisfactory because image layout is destroyed and the boundary between the foreground and background as well as different objects becomes obscured. We observe that the depth map effectively reflects the spatial distribution in an image and preserving the depth map of the content image after stylization helps produce an image that preserves its semantic content. In this paper, we introduce a novel approach for neural style transfer that integrates depth preservation as additional loss, preserving overall image layout while performing style transfer.

In this paper, we propose a deep-learning-based algorithm for screening neurological diseases. We proposed various examination protocols for screening neurological diseases and collected data by video-recording persons performing these protocols. We converted video data into human landmarks that capture action information with a much smaller data dimension. We also used voice data which are also effective indicators of neurological disorders. We designed a subnetwork for each protocol to extract features from landmarks or voice and a feature aggregator that combines all the information extracted from the protocols to make a final decision. Multitask learning was applied to screen two neurological diseases. To capture meaningful information about these human landmarks and voices, we applied various pre-trained models to extract preliminary features. The spatiotemporal characteristics of landmarks are extracted using a pre-trained graph neural network, and voice features are extracted using a pre-trained time-delay neural network. These extracted high-level features are then passed onto the subnetworks and an additional feature aggregator that are simultaneously trained. We also used various data augmentation techniques to overcome the shortage of data. Using a frame-length staticizer that considers the characteristics of the data, we can capture momentary tremors without wasting information. Finally, we examine the effectiveness of different protocols and different modalities (different body parts and voice) through extensive experiments. The proposed method achieves AUC scores of 0.802 for stroke and 0.780 for Parkinson’s disease, which is effective for a screening system.

Recently, x-vector [1] has been a successful and popular approach for speaker verification, which employs a time delay neural network (TDNN) and statistics pooling to extract speaker characterizing embedding from variable-length utterances. Improvement upon the x-vector has been an active research area, and enormous neural networks have been elaborately designed based on the x-vector, e.g., extended TDNN (E-TDNN) [2], factorized TDNN (F-TDNN) [3], and densely connected TDNN (D-TDNN) [4]. In this work, we try to identify the optimal architectures from a TDNN based search space employing neural architecture search (NAS), named SpeechNAS. Leveraging the recent advances in the speaker recognition, such as high-order statistics pooling, multi-branch mechanism, D-TDNN and angular additive margin softmax (AAM) loss with a minimum hyper-spherical energy (MHE), SpeechNAS automatically discovers five network architectures, from SpeechNAS-1 to SpeechNAS-5, of various numbers of parameters and GFLOPs on the large-scale text-independent speaker recognition dataset VoxCelebl. Our derived best neural network achieves an equal error rate (EER) of 1.02% on the standard test set of VoxCelebl, which surpasses previous TDNN based state-of-the-art approaches by a large margin.

Routine inspection of a power line insulator system for early problem detection and maintenance is required for the effective transmission of electrical power to customers. Unlike the traditional manual inspection methods such as foot patrol, helicopter assisted methods, flying and climbing robots are time consuming, dangerous, labor intensive, and expensive. This work primarily focused on the insulator image classification of the Chinese Power Line Insulator Dataset (CPLID). Convolutional Neural Network (CNN) and pre-trained neural networks with different optimizers like Adam, Adamax, Adagrad, Adadelta, Nadam, Ftrl, and RMSprop were used to find out the best combination of CNN and pre-trained neural network with the optimizer to provide robust performance and higher accuracy. Finally, CNN with Adamax optimizer provided overall accuracy of 95.27% and the pre-trained CNN Densenet 201 with Adam optimizer gave overall accuracy of 100%. In the automated inspection of transmission line insulators, the deployment of UAVs and deep learning algorithms can be a useful starting point for many researchers throughout the world.

Despite major advances in recent years, the field of Machine Learning continues to face research and technical challenges. Mostly, these stem from big data and streaming data, which require models to be frequently updated or re-trained, at the expense of significant computational resources. One solution is the use of distributed learning algorithms, which can learn in a distributed manner, from distributed datasets. In this paper, we describe CEDEs—a distributed learning system in which models are heterogeneous distributed Ensembles, i.e., complex models constituted by different base models, trained with different and distributed subsets of data. Specifically, we address the issue of predicting the training time of a given model, given its characteristics and the characteristics of the data. Given that the creation of an Ensemble may imply the training of hundreds of base models, information about the predicted duration of each of these individual tasks is paramount for an efficient management of the cluster’s computational resources and for minimizing makespan, i.e., the time it takes to train the whole Ensemble. Results show that the proposed approach is able to predict the training time of Decision Trees with an average error of 0.103 s, and the training time of Neural Networks with an average error of 21.263 s. We also show how results depend significantly on the hyperparameters of the model and on the characteristics of the input data.

A method for ankle torque prediction ahead of the current time is proposed in this paper. The mean average value of EMG signals from four muscles, alongside the joint angle and angular velocity of the right ankle, were used as input parameters to train a time-delayed artificial neural network. Data collected from five healthy subjects were used to generate the dataset to train and test the model. The model predicted ankle torque for five different future times from zero to 2 seconds. Model predictions were compared to torque calculated from inverse dynamics for each subject. The model predicted ankle torque up to 1 second ahead of time with normalized root mean squared error of less than 15 percent while the coefficient of determination was over 0.85.Clinical Relevance- the potential of the model for predicting joint torque ahead of time is helpful to establish an intuitive interaction between human and assistive robots. This model has application to assist patients with neurological disorders.

Pre-trained single-channel neural networks have become more prevalent for noise reduction in recent years. However, unlike their multichannel counterparts, these monoaural approaches do not exploit spatial information during the optimization process. Furthermore, while multichannel neural networks exploit spatial information, they are optimized for a specific microphone array configuration; extensive data collection and training are required if a new array configuration is deployed. We propose a transfer learning approach that leverages existing pre-trained single-channel neural networks for the optimization of multichannel neural networks. Simulation results on the CHiME-3 dataset show that the proposed method outperforms the state-of-the-art multichannel neural network and neural beamformer.

This paper is devoted to the efficiency analysis of detectors for the recognition of digits, which are mechanically stamped by a special machine on a steel cast billet. Such detectors are based on pre-trained deep neural networks. In the study, we analyze the performance of four different Faster R-CNN-based detectors. These neural networks have been trained and tested on our own training dataset obtained from the electro-metallurgical plant. According to the experiments, the best results are achieved by the Faster-RCNN Inception-Resnet v2 neural network detector. Its accuracy is about 98% on the test set.

The present work is devoted to a problem of Synthesis of the algorithms for adaptive control by nonlinear dynamic objects with the incomplete mathematical description. The method of synthesis adaptive neural networks is considered on the basis of application of some positions of a method of analytical designing. The law of adaptation is defined on a condition of maintenance of stability of the closed system with the help of the second method of Lyapunov. The resulting control systems can operate in uncertain conditions caused by external and internal disturbances. The designed parameter adaptation law of the controllers admits simple implementation, thereby facilitating the on-line adaptation process.

Neural networks with time delays offer the potential of providing massive parallelism and adaptation, and can provide excellent discrimination for temporal signal processing tasks. In a time delay neural network (TDNN), the shift-invariance capability required in speech recognition is achieved by explicitly making time-shifted copies of the inputs and internal responses and linking their corresponding weights. In this paper, we propose a limited feedback TDNN (LFB-TDNN), which feedbacks only those errors with significant magnitudes over a limited time span during the semi-batch back-propagation learning. Simulations show that the LFB-TDNN achieves favorable performance in isolated speech recognition tasks when compared with the standard TDNN, the discrete density and the continuous density hidden Markov models.

This article presents the application of a new approach to segmentation of images of the iris and pupil of the human eye using the pretrained neural network model YoLoV5. Despite the use of modern neural network technologies and computer vision methods for image processing and eye segmentation, not all of them guarantee obtaining a highly accurate result of the desired circles. Examples of incorrect segmentation of the iris and pupil obtained using the Mediapipe framework and the FloodFill algorithm are given. Smoothing filters were previously used to eliminate artifacts, noise and outliers in the data. At the same time, the use of smoothing filters contributes to the distortion of the data they process, especially when working in real-time systems with the property of observation duration. Since the diagnosis of the functional state of a person by the pupillometry method is sensitive to the quality of the original data, the use of smoothing filters in a number of diagnostic cases is not acceptable. To improve the accuracy of iris and eye image segmentation and avoid data distortion by smoothing filters, the authors developed a computer vision module based on the eye circumference segmentation mechanism using the pre-trained YoLoV5 neural network model. Examples of eye circumference segmentation by the YoLoV5 neural network are given. The accuracy of iris and pupil image segmentation was assessed by comparing the number of outliers and omissions in time series values obtained using the approaches discussed in the article. The approach to using the YoLoV5 neural network model proposed in the article is implemented in solving the problem of driver fatigue monitoring and is used in a prototype of a specialized device developed by the authors.

Reducing environmental noise in urban settings, i.e., unwanted or harmful outdoor sounds produced by human activity, has become an important issue in recent years. Most countries have established regulations that set maximum permitted noise levels. However, enforcing these regulations effectively remains challenging as it requires active monitoring networks and audio analysis performed by trained specialists. The manual evaluation of the audio recordings is laborious, time-consuming, and inefficient since many audios exceeding the noise level threshold do not correspond to a sound event considered by the regulation. To address this challenge, this work proposes a computational pipeline to assist specialists in detecting noise sources in the built environment that do not comply with the Chilean noise regulation. The system incorporates a deep neural model following a pre-trained audio neural network architecture transferred to a dataset compiled from public sources and recordings in Valdivia, Chile. The target dataset follows a customized taxonomy of urban sound events. The system also uses a public API so potential users can post audio files to obtain a prediction matrix reporting the presence of noise sources contributing to environmental noise pollution. Experiments using recordings from two continuous noise monitoring stations showed that the amount of data to be inspected by the specialist is decreased by 97% when the deep-learning tools are used. Therefore, this system efficiently assists trained experts in enforcing noise legislation through machine-assisted environmental noise monitoring.