A Study on Network Size Reduction Using Sparse Input Representation in Time Delay Neural Networks

In this paper, an evaluation of various discriminant neural networks classifiers for text- independent speaker verification problem is presented. Each person to be verified has a personalized neural network model. A new classifier called neural tree network (NTN) is also examined for this application. The memoryless feedforward neural network architecture makes decisions based on static features. Time delay neural network (TDNNs) have proved to be an efficient way to handle the dynamic nature of speech. Furthermore, a model called recurrent time delay neural networks (RTDNNs), obtained through a local feedback connection at the first hidden layer level of TDNNs is investigated. The training is carried out by backpropagation for sequence algorithm. The database used is a subset of the TIMIT database consisting of 38 speakers from the same dialect region. The NTN is compared with the MLP, TDNN, and RTDNN. It is shown that NTN is found to perform better than the other neural networks classifiers. Also, a little bit performance improvement was achieved due to the addition of temporal information for text-independent speaker verification problem using TDNNs and RTDNNs. Finally, we described the experimental results obtained using different neural network models.© (1994) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

The Sixth International Symposium on Neural Networks (ISNN 2009)

Proposed is an efficient face recognition algorithm using the discrete cosine transform DCT Technique for reducing dimensionality and image parameterization. These DCT coefficients are examined by a MLP (Multi-Layer Perceptron) and radial basis function RBF neural networks. Their purpose is to present a face recognition system that is a combination of discrete cosine transform (DCT) algorithm with a MLP and RBF neural networks. Neural networks have been widely applied in pattern recognition for the reason that neural-networks-based classifiers can incorporate both statistical and structural information and achieve better performance than the simple minimum distance classifiers. The authors demonstrate experimentally that when DCT coefficients are fed into a back propagation neural network for classification, a high recognition rate can be achieved by using a very small proportion of transform coefficients. Comparison with other statistical methods like Principal component Analysis (PCA) and Linear Discriminant Analysis (LDA) is presented. Their face recognition system is tested on the computer vision science research projects and the ORL database.

At the core of any inference procedure, deep neural networks are dot product operations, which are the component that requires the highest computational resources. For instance, deep neural networks, such as VGG-16, require up to 15-G operations in order to perform the dot products present in a single forward pass, which results in significant energy consumption and thus limits their use in resource-limited environments, e.g., on embedded devices or smartphones. One common approach to reduce the complexity of the inference is to prune and quantize the weight matrices of the neural network. Usually, this results in matrices whose entropy values are low, as measured relative to the empirical probability mass distribution of its elements. In order to efficiently exploit such matrices, one usually relies on, inter alia, sparse matrix representations. However, most of these common matrix storage formats make strong statistical assumptions about the distribution of the elements; therefore, cannot efficiently represent the entire set of matrices that exhibit low-entropy statistics (thus, the entire set of compressed neural network weight matrices). In this paper, we address this issue and present new efficient representations for matrices with low-entropy statistics. Alike sparse matrix data structures, these formats exploit the statistical properties of the data in order to reduce the size and execution complexity. Moreover, we show that the proposed data structures can not only be regarded as a generalization of sparse formats but are also more energy and time efficient under practically relevant assumptions. Finally, we test the storage requirements and execution performance of the proposed formats on compressed neural networks and compare them to dense and sparse representations. We experimentally show that we are able to attain up to ×42 compression ratios, ×5 speed ups, and ×90 energy savings when we lossless convert the state-of-the-art networks, such as AlexNet, VGG-16, ResNet152, and DenseNet, into the new data structures and benchmark their respective dot product.

Retrieving the layout of different cluttered indoor scenes from monocular images is a challenging task, using a smaller deep neural network than the existing proposals. Previous geometric solutions are prone to failure in the presence of cluttered scenes because they depend strongly on hand-engineering features and the expectation of the possibility of the vanishing points’ calculation. With the growth of neural networks, the geometric methods were either replaced or fused within the emerging area of deep learning. The more recent solutions rely on dense neural networks with additional adjustments, either by the calculation of the vanishing points, position based, or by layout ranking. All these methods presented valid solutions to this challenge with the flaw of being computationally demanding. Here, we present a more lightweight solution, running the segmentation on a smaller neural network and introducing a discriminative classifier for the posterior layout ranking and optimization. Our proposed method is evaluated by two standard dataset benchmarks, achieving near state of the art results even with a fraction of the required parameters than the available state of the art methods.

The discrete cosine transform (DCT) offers superior energy compaction properties for a large class of functions and has been employed as a standard tool in many signal and image processing applications. However, it suffers from spurious behavior in the vicinity of edge discontinuities in piecewise smooth signals. To leverage the sparse representation provided by the DCT, in this article, we derive a framework for the inverse polynomial reconstruction in the DCT expansion. It yields the expansion of a piecewise smooth signal in terms of polynomial coefficients, obtained from the DCT representation of the same signal. Taking advantage of this framework, we show that it is feasible to recover piecewise smooth signals from a relatively small number of DCT coefficients with high accuracy. Furthermore, automatic methods based on minimum description length principle and cross-validation are devised to select the polynomial orders, as a requirement of the inverse polynomial reconstruction method in practical applications. The developed framework can considerably enhance the performance of the DCT in sparse representation of piecewise smooth signals. Numerical results show that denoising and image approximation algorithms based on the proposed framework indicate significant improvements over wavelet counterparts for this class of signals.

Fault detection and diagnosis for non-linear processes empowered by dynamic neural networks

Various signal processing techniques have been used for the enhancement of defect detection and defect characterisation. Cross-correlation, filtering, autoregressive analysis, deconvolution, neural network, wavelet transform and sparse signal representations have all been applied in attempts to analyse ultrasonic signals. In ultrasonic nondestructive evaluation (NDE) applications, a large number of materials have multilayered structures. NDE of multilayered structures leads to some specific problems, such as penetration, echo overlap, high attenuation and low signal-to-noise ratio. The signals recorded from a multilayered structure are a class of very special signals comprised of limited echoes. Such signals can be assumed to have a sparse representation in a proper signal dictionary. Recently, a number of digital signal processing techniques have been developed by exploiting the sparse constraint. This paper presents a review of research to date, showing the up-to-date developments of signal processing techniques made in ultrasonic NDE. A few typical ultrasonic signal processing techniques used for NDE of multilayered structures are elaborated. The practical applications and limitations of different signal processing methods in ultrasonic NDE of multilayered structures are analysed.

A novel neural network is proposed for the dynamic classification of spatio-temporal signals. The network is designed to classify signals of different durations, taking into account correlations among different signal segments. Such a network is applicable to SONAR and speech signal classification problems, among others. Network parameters are adapted based on the biologically observed habituation mechanism. This allows the storage of contextual information, without a substantial increase in network complexity. Experiments on classification of high dimensional feature vectors obtained from Banzhaf sonograms, demonstrate that the proposed network performs better than time delay neural networks while using a substantially simpler structure. The mathematical power of the network is discussed, including its ability to realize any function realizable by a TDNN. Additionally, principal component analysis is used to introduce a further improvement to the network design by reducing the dimensionality of the encoded temporal information.

Recently, x-vector [1] has been a successful and popular approach for speaker verification, which employs a time delay neural network (TDNN) and statistics pooling to extract speaker characterizing embedding from variable-length utterances. Improvement upon the x-vector has been an active research area, and enormous neural networks have been elaborately designed based on the x-vector, e.g., extended TDNN (E-TDNN) [2], factorized TDNN (F-TDNN) [3], and densely connected TDNN (D-TDNN) [4]. In this work, we try to identify the optimal architectures from a TDNN based search space employing neural architecture search (NAS), named SpeechNAS. Leveraging the recent advances in the speaker recognition, such as high-order statistics pooling, multi-branch mechanism, D-TDNN and angular additive margin softmax (AAM) loss with a minimum hyper-spherical energy (MHE), SpeechNAS automatically discovers five network architectures, from SpeechNAS-1 to SpeechNAS-5, of various numbers of parameters and GFLOPs on the large-scale text-independent speaker recognition dataset VoxCelebl. Our derived best neural network achieves an equal error rate (EER) of 1.02% on the standard test set of VoxCelebl, which surpasses previous TDNN based state-of-the-art approaches by a large margin.

Machine learning methods are ubiquitous in communication systems and have proven powerful for applications including radio-frequency (RF) fingerprinting, automatic modulation classification, and signal recovery in communication systems. However, the high throughput requirement of a communication link makes AI models difficult to implement in real-time on edge devices. In this work, we address this issue by improving both the algorithm and hardware to target real-time AI processing in communication systems. For algorithm development, we propose the first compact deep network consisting of a silicon photonic recurrent neural network model in combination with a simplified convolutional neural network classifier to identify RF emitters by their random transmissions. Our model achieves 96.32% classification accuracy over a set of 30 identical ZigBee devices when using 50 times fewer training parameters than an existing state-of-the-art CNN classifier (Merchant et al., 2018). Thanks to the large reduction in network size, we emulate the system using a small-scale FPGA board, the PYNQ-Z1, and demonstrate real-time RF fingerprinting with 0.219 ms latency. In addition, for hardware implementation, we further demonstrate a fully-integrated silicon photonic neural network for fiber nonlinearity compensation (Huang et al., 2021), which improves the received signal by 0.60 dB.

<p>This Paper proposes a new facial feature extraction approach, Wash-Hadamard Transform (WHT). This approach is based on correlation between local pixels of the face image. Its primary advantage is the simplicity of its computation. The paper compares the proposed approach, WHT, which was traditionally used in data compression with two other known approaches: the Principal Component Analysis (PCA) and the Discrete Cosine Transform (DCT) using the face database of Olivetti Research Laboratory (ORL). In spite of its simple computation, the proposed algorithm (WHT) gave very close results to those obtained by the PCA and DCT. This paper initiates the research into WHT and the family of frequency transforms and examines their suitability for feature extraction in face recognition applications.</p>

The work is devoted to resolving the contradiction between the accuracy of modelling nonlinear dynamics and the speed of model construction under conditions of limited computing resources. The purposeof the work is to reduce the time for building time delay neural networks while ensuring a given accuracy in the tasks of identifying nonlinear dynamic objects with continuous characteristics. This goal is achieved by developing a method for pre-training neural networks that reflect the basic characteristics of the subject area. The scientific novelty of the work is the development of a method for identifying nonlinear dynamic objectsin the form of time delay neural networks based on the use of a set of basic pre-trained neural networks that reflect the typical properties of the subject area. In contrast to the traditional approach to pre-training, the developed method allows building models of lower complexity.A formal criterion is proposed for determining the moment of termination of the neural network pre-training, the use of which allows avoiding retraining of the base model and ensuring a significant reduction in the model training time on the target data set. The practical utility of the work lies in the development of an algorithm for the method of pre-training time delay neural networks in the tasks of identifying nonlinear dynamic objects with continuous characteristics, which allows to significantlyreduce the training time of neural networks without losing model accuracy. The value of this study is todetermine the area of effective use of the proposed method, namely, when the general and target datasets do not have significant differences and the target dataset is of sufficient size to reflect the properties of the research object.

Neural networks with time delays offer the potential of providing massive parallelism and adaptation, and can provide excellent discrimination for temporal signal processing tasks. In a time delay neural network (TDNN), the shift-invariance capability required in speech recognition is achieved by explicitly making time-shifted copies of the inputs and internal responses and linking their corresponding weights. In this paper, we propose a limited feedback TDNN (LFB-TDNN), which feedbacks only those errors with significant magnitudes over a limited time span during the semi-batch back-propagation learning. Simulations show that the LFB-TDNN achieves favorable performance in isolated speech recognition tasks when compared with the standard TDNN, the discrete density and the continuous density hidden Markov models.

Nowadays, sparse representation has been widely used in Magnetic Resonance Imaging (MRI). The commonly used sparse representation methods are based on symmetrical partition, which have not considered the complex structure of MRI image. In this paper, we proposed a sparse representation method for the brain MRI image, called GNAMlet transform, which is based on the gradient information and the non-symmetry and anti-packing model. The proposed sparse representation method can reduce the lost detail information, improving the reconstruction accuracy. The experiment results show the superiority of the proposed transform for the brain MRI image representation in comparison with some state-of-the-art sparse representation methods.