State-Clustering Based Multiple Deep Neural Networks Modeling Approach for Speech Recognition

Simple SummaryThe growing evidence suggested that competing endogenous RNAs (ceRNAs) have significant associations with tumor occurrence and progression, yet the regulatory mechanism of them in lung adenocarcinoma remains unclear. Identification of the regulatory modules for lung adenocarcinoma is a critical and fundamental step towards understanding the regulatory mechanisms during carcinogenesis. Deep neural network (DNN) models have become a powerful tool to intelligently recognize the sophisticated relationships of ceRNAs appropriately. In this paper, multiple deep neural network models were constructed using the expression data to identify regulatory modules for lung adenocarcinoma in biological networks. Three identified regulatory modules association with lung adenocarcinoma were validated from three aspects, i.e., literature review, functional enrichment analysis, and an independent dataset. The regulatory relationships between RNAs were validated in various datasets, including CPTAC, TCGA and an expression profile from the GEO database. Our study will contribute to improving the understanding of regulatory mechanisms in the carcinogenesis of lung adenocarcinoma and provide schemes for identifying novel regulatory modules of other cancers.Lung adenocarcinoma is the most common type of primary lung cancer, but the regulatory mechanisms during carcinogenesis remain unclear. The identification of regulatory modules for lung adenocarcinoma has become one of the hotspots of bioinformatics. In this paper, multiple deep neural network (DNN) models were constructed using the expression data to identify regulatory modules for lung adenocarcinoma in biological networks. First, the mRNAs, lncRNAs and miRNAs with significant differences in the expression levels between tumor and non-tumor tissues were obtained. MRNA DNN models were established and optimized to mine candidate mRNAs that significantly contributed to the DNN models and were in the center of an interaction network. Another DNN model was then constructed and potential ceRNAs were screened out based on the contribution of each RNA to the model. Finally, three modules comprised of miRNAs and their regulated mRNAs and lncRNAs with the same regulation direction were identified as regulatory modules that regulated the initiation of lung adenocarcinoma through ceRNAs relationships. They were validated by literature and functional enrichment analysis. The effectiveness of these regulatory modules was evaluated in an independent lung adenocarcinoma dataset. Regulatory modules for lung adenocarcinoma identified in this study provided a reference for regulatory mechanisms during carcinogenesis.

Reducing the inference time of Deep Neural Networks (DNNs) is critical when running time sensitive applications on mobile devices. Existing research has shown that partitioning a DNN and offloading a part of its computation to cloud servers can reduce the inference time. The single DNN partition problem has been extensively investigated recently. However, in real-world applications, a mobile device usually generates multiple DNN inference jobs simultaneously, and little attention has been paid to this case. We aim to minimize the makespan of multiple DNNs by jointly optimizing their partitioning and scheduling. Our observations show that the local computation time on a mobile device follows an increasing function, while the communication workload for offloading is usually decreasing as more DNN layers are computed. Based on this, we first relax our problem on continuous domain and show that partitioning all line-structure DNNs at the same layer is sufficient for makespan optimization. Then, for the discrete domain, two types of partitions are sufficient when the time difference between two adjacent partition layers is not drastic, subject to a given condition. An algorithm based on the binary search that efficiently finds optimal partition layers is illustrated. We also extend our approach to general-structure DNNs and offer a heuristic solution. Experiments have been conducted to evaluate the performance of different partition and scheduling methods on sample DNNs. Results validate the optimality of our theoretical results.

This paper presents a deep neural network (DNN) based large vocabulary continuous speech recognition (LVCSR) system for Serbian, developed using the open-source Kaldi speech recognition toolkit. The DNNs are initialized using stacked restricted Boltzmann machines (RBMs) and trained using cross-entropy as the objective function and the standard error backpropagation procedure in order to provide posterior probability estimates for the hidden Markov model (HMM) states. Emission densities of HMM states are represented as Gaussian mixture models (GMMs). The recipes were modified based on the particularities of the Serbian language in order to achieve the optimal results. A corpus of approximately 90 hours of speech (21000 utterances) is used for the training. The performances are compared for two different sets of utterances between the baseline GMM-HMM algorithm and various DNN settings.KeywordsKaldi speech recognition toolkitContinuous speech recognitionDeep neural networksSerbian

Distant speech recognition (DSR) remains to be an open challenge, even for the state-of-the-art deep neural network (DNN) models. Previous work has attempted to improve DNNs under constantly distant speech. However, in real applications, the speaker-microphone distance (SMD) can be quite dynamic, varying even within a single utterance. This paper investigates how to alleviate the impact of dynamic SMD on DNN models. Our solution is to incorporate the frame-level SMD information into DNN training. Generation of the SMD information relies on a universal extractor that is learned on a meeting corpus. We study the utility of different architectures in instantiating the SMD extractor. On our target acoustic modeling task, two approaches are proposed to build distance-aware DNN models using the SMD information: simple concatenation and distance adaptive training (DAT). Our experiments show that in the simplest case, incorporating the SMD descriptors improves word error rates of DNNs by 5.6% relative. Further optimizing SMD extraction and integration results in more gains.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

Dialect Identification is the process of identifies the dialects of particular standard language. The Telugu Language is one of the historical and important languages. Like any other language Telugu also contains mainly three dialects Telangana, Costa Andhra and Rayalaseema. The research work in dialect identification is very less compare to Language identification because of dearth of database. In any dialects identification system, the database and feature engineering play vital roles because of most the words are similar in pronunciation and also most of the researchers apply statistical approaches like Hidden Markov Model (HMM), Gaussian Mixture Model (GMM), etc. to work on speech processing applications. But in today's world, neural networks play a vital role in all application domains and produce good results. One of the types of the neural networks is Deep Neural Networks (DNN) and it is used to achieve the state of the art performance in several fields such as speech recognition, speaker identification. In this, the Deep Neural Network (DNN) based model Multilayer Perceptron is used to identify the regional dialects of the Telugu Language using enhanced Mel Frequency Cepstral Coefficients (MFCC) features. To do this, created a database of the Telugu dialects with the duration of 5h and 45m collected from different speakers in different environments. The results produced by DNN model compared with HMM and GMM model and it is observed that the DNN model provides good performance.

Optimizing makespan and resource utilization for multi-DNN training in GPU cluster

Real-time model predictive cooling control for an HVAC system in a factory building

Deep neural networks (DNNs) have become a critical component for inference in modern mobile applications, but the efficient provisioning of DNNs is non-trivial. Existing mobile- and server-based approaches compromise either the inference accuracy or latency. Instead, a hybrid approach can reap the benefits of the two by splitting the DNN at an appropriate layer and running the two parts separately on the mobile and the server respectively. Nevertheless, the DNN throughput in the hybrid approach has not been carefully examined, which is particularly important for edge servers where limited compute resources are shared among multiple DNNs. This article presents HiTDL, a runtime framework for managing multiple DNNs provisioned following the hybrid approach at the edge. HiTDL's mission is to improve edge resource efficiency by optimizing the combined throughput of all co-located DNNs, while still guaranteeing their SLAs. To this end, HiTDL first builds comprehensive performance models for DNN inference latency and throughout with respect to multiple factors including resource availability, DNN partition plan, and cross-DNN interference. HiTDL then uses these models to generate a set of candidate partition plans with SLA guarantees for each DNN. Finally, HiTDL makes global throughput-optimal resource allocation decisions by selecting partition plans from the candidate set for each DNN via solving a fairness-aware multiple-choice knapsack problem. Experimental results based on a prototype implementation show that HiTDL improves the overall throughput of the edge by <inline-formula><tex-math notation="LaTeX">$4.3\times$</tex-math></inline-formula> compared with the state-of-the-art.

Multi-model ensemble is an important fundamental technique of practical value for many artificial intelligence applications. However, the usage for multi-model ensemble has been limited when it is combined with deep neural networks to construct ensemble of deep neural networks. Due to the big time and computing resources required to train and to integrate multiple deep neural networks for the achievement of multi-model ensemble, the engineering application field where developing time and computing resources are usually restricted, has not yet widespreadly benefited from ensemble of deep neural networks. To alleviate this situation, we present a new multi-model ensemble approach entitled feature transferring based multi-model ensemble (FTBME), for ensemble of deep neural networks. Primarily, we propose a feature transferring based multi-model training strategy to more affordably find multiple extra models based on a given previously optimized deep neural network model. Sequentially, to develop better ensemble solutions, we design a more effective random greedy based ensemble selection strategy to filter out models non-positive to ensemble generalization. Finally, inspired by the idea of averaging parameter points, we propose to fuse the obtained models in weight space which eventually reduces the expense of ensemble at the testing stage to a single deep neural network model while retaining the generalization. These three advances constitute the resulting technique FTBME. We conducted extensive experiments using deep neural networks, from light weight to complex, on ImageNet, CIFAR-10 and CIFAR-100. Results show that, given a deep neural network model which has been well-optimized and reaching its limit, FTBME can obtain better generalization with minor extra training expense while maintaining the expense to a single model at ensemble testing. This promising property of FTBME make us believe that it could be leveraged to broaden the usage for ensemble of deep neural networks, alleviating the situation that the engineering application field has not yet widespreadly benefited from ensemble of deep neural networks.

Deep Neural Network (DNN) models have achieved excellent performance in many inference tasks and have been widely used in many intelligent applications. However, DNN models often require a lot of computational resources to complete the inference tasks, which hinders the deployment of such models to resource‐constrained edge devices. In order to extend the application scenarios of DNN models, the edge‐cloud collaborative inference methods, represented by model partition, have attracted much research attention in recent years. In scenarios that have multiple edge devices deployed, the edge‐cloud collaborative inference method requires partial migration of tasks, but traditional scheduling methods only migrate tasks at the task level. In this paper, we propose two task scheduling methods, which can solve the problem of partial migration of tasks in multiedge scenarios. The first scheduling method is based on the optimal cutting of a single DNN. The cutting positions of all the models are the same, regardless of the influence of external factors. This method is suitable for chain and directed acyclic graph‐ (DAG‐) type DNNs. The second scheduling method takes external factors such as congestion and queuing delay at the cloud side into consideration, which dynamically selects the cutting position of each DNN to optimize the overall delay and thus is applicable to chain DNN models. The experimental results show that, compared with the baseline method, our proposed scheduling method can reduce the delay by up to 6.48x.

Chaha is a very low-resource language, which is suffered from lack of language resources to develop human language technologies, namely, speech recognition. Moreover, Chaha writing system is syllabic with a consonant-vowel (CV) syllable structure. The Chaha orthography is a one-to-one correspondence with syllable sound units. By considering the above facts of Chaha, this study is the first endeavor that explores the use of CV syllables as acoustic modeling units for developing speech recognizers, using the Gaussian mixture model (GMM) and unilingual and transfer learning deep neural network (DNN) models. Our experimental results demonstrate that the syllablebased unilingual DNN and transfer learning DNN models outperform the corresponding GMM and unilingual DNN models with absolute performance improvements of 2.8 to 3.09% and 1.07 to 4.94%, respectively. The best performing syllable-based recognizer is achieved using a shared hidden layer (SHL) time delay deep neural network (TDNN) model with a word error rate (WER) of 23.11%. Hence, the CV syllables are suitable acoustic units to develop Chaha speech recognition systems under sufficient training corpus.

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .