Structure-Based Testing Criteria and Testing Case Generation for Deep Learning Systems

<p>Deep neural networks have achieved tremendous success in many domains (e.g., computer vision~\cite{Alexnet12,vggnet15,fastrcnn15}, speech recognition~\cite{hinton2012deep,dahl2012context}, natural language processing~\cite{dahl2012context,collobert2011natural}, games~\cite{silver2017mastering,silver2016mastering}), however, there are still many challenges in deep learning comunity such as how to speed up training large deep neural networks, how to compress large nerual networks for mobile/embed device without performance loss, how to automatically design the optimal network structures for a certain task, and how to further design the optimal networks with improved performance and certain model size with reduced computation cost.</p> <p>To speed up training large neural networks, we propose to use multinomial sampling for dropout, i.e., sampling features or neurons according to a multinomial distribution with different probabilities for different features/neurons. To exhibit the optimal dropout probabilities, we analyze the shallow learning with multinomial dropout and establish the risk bound for stochastic optimization. By minimizing a sampling dependent factor in the risk bound, we obtain a distribution-dependent dropout with sampling probabilities dependent on the second order statistics of the data distribution. To tackle the issue of evolving distribution of neurons in deep learning, we propose an efficient adaptive dropout (named evolutional dropout) that computes the sampling probabilities on-the-fly from a mini-batch of examples.</p> <p>To compress large neural network structures, we propose a simple yet powerful method for compressing the size of deep Convolutional Neural Networks (CNNs) based on parameter binarization. The striking difference from most previous work on parameter binarization/quantization lies at different treatments of $1\times 1$ convolutions and $k\times k$ convolutions ($k>1$), where we only binarize $k\times k$ convolutions into binary patterns. By doing this, we show that previous deep CNNs such as GoogLeNet and Inception-type Nets can be compressed dramatically with marginal drop in performance. Second, in light of the different functionalities of $1\times 1$ (data projection/transformation) and $k\times k$ convolutions (pattern extraction), we propose a new block structure codenamed the pattern residual block that adds transformed feature maps generated by $1\times 1$ convolutions to the pattern feature maps generated by $k\times k$ convolutions, based on which we design a small network with $\sim 1$ million parameters. Combining with our parameter binarization, we achieve better performance on ImageNet than using similar sized networks including recently released Google MobileNets.</p> <p>To automatically design neural networks, we study how to design a genetic programming approach for optimizing the structure of a CNN for a given task under limited computational resources yet without imposing strong restrictions on the search space. To reduce the computational costs, we propose two general strategies that are observed to be helpful: (i) aggressively selecting strongest individuals for survival and reproduction, and killing weaker individuals at a very early age; (ii) increasing mutation frequency to encourage diversity and faster evolution. The combined strategy with additional optimization techniques allows us to explore a large search space but with affordable computational costs.</p> <p>To further design the optimal networks with improved performance and certain model size under reduced computation cost, we propose an ecologically inspired genetic approach for neural network structure search , that includes two types of succession: primary and secondary succession as well as accelerated extinction. Specifically, we first use primary succession to rapidly evolve a community of poor initialized neural network structures into a more diverse community, followed by a secondary succession stage for fine-grained searching based on the networks from the primary succession. Accelerated extinction is applied in both stages to reduce computational cost. In addition, we also introduce the gene duplication to further utilize the novel block of layers that appeared in the discovered network structure.</p>

The article deals with the problem of path planning in a two-dimensional environment based on deep learning neural networks. Deep neural networks require large amounts of data and place high computational requirements on computing tools. The lack of sufficient data leads to a decrease in the accuracy of the neural network, and high computational requirements at the learning stage limit the use of this technology in engineering practice. In this paper, the forms of representation of the environment for the input of a neural network are studied. Vector form allows to reduce the amount of information supplied to the input of a neural network, but it leads to the need to use more complex neural networks. In this article, a combined form of representation is proposed, including a vector global and local map layout. The vector part of the map includes the position of the robot, the position of the target point and a description of obstacles. The local raster map describes the area closest to the robot. Using numerical research, the effectiveness of this form of data representation for a precise neural network is shown, compared with the raster representation of the map. In this article, two structures of neural networks are studied, one of which uses 8 possible directions of movement, and the other uses 3 possible directions of movement. It is shown that when using 3 possible directions, the cycling of trajectories planned by the neural network is eliminated, which leads to an increase in accuracy.

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

In this study, the deep multi-layered group method of data handling (GMDH)-type neural network algorithm using revised heuristic self-organization method is proposed and applied to medical image diagnosis of liver cancer. The deep GMDH-type neural network can automatically organize the deep neural network architecture which has many hidden layers. The structural parameters such as the number of hidden layers, the number of neurons in hidden layers and useful input variables are automatically selected to minimize prediction error criterion defined as Akaike’s information criterion (AIC) or prediction sum of squares (PSS). The architecture of the deep neural network is automatically organized using the revised heuristic self-organization method which is a type of the evolutionary computation. This new neural network algorithm is applied to the medical image diagnosis of the liver cancer and the recognition results are compared with the conventional 3-layered sigmoid function neural network.

A novel electrical net-load forecasting model based on deep neural networks and wavelet transform integration

A floating foundation vibration reduction system with an air spring as a vibration isolation element has been widely used in the foundation of large ultra-precision instruments. With the excellent vibration isolation performance of the air spring, its complex dynamic nonlinear behavior is always a research difficulty. In this paper, a dynamic model of a floating foundation vibration reduction system based on the restoring force of air spring is derived. The structure of the recurrent convolution neural network (RCNN) is proposed based on combining the working characteristics of a convolution neural network and a long short-term memory neural network, and the dynamic model of a floating foundation vibration reduction system is established with the restoring force of the air spring calculated by the RCNN as the input. Finally, a test experiment was designed to compare the dynamic characteristics of the traditional numerical model and three deep neural network models in the floating foundation. The results show that convolution neural network, long short-term memory, and RCNN models could predict the vibration response of floating foundation vibration reduction system, and the RCNN model had better performance for a floating foundation.

Federated Learning is a machine learning technique where independent devices (clients) cooperatively train a machine learning model by working on decentralized training data. A fundamental challenge in federated learning is the client model aggregation. The goal is to combine and preserve the knowledge that each client has acquired during its local training phase and generate an aggregated model with superior performance than any of the clients. Most current state-of-the-art algorithms for model aggregation still rely on simple weight parameter averaging. Though effective, these methods do not consider the internal structure of a deep neural network. In addition, the element-wise aggregation of the model parameters may create a conflict and loss of knowledge if a particular weight parameter serves different classification tasks over other learning clients. To decrease the probability of such conflicts, we propose FedGrid, a method that involves rotated (shifted) weight parameter grids during the clients' local training phase, which stimulates the models to build non-overlapping neural network structures. FedGrid can be applied to any existing algorithm that performs parameter averaging to improve the aggregation of deep NN models. To estimate the method's performance, we generate an ensemble of datasets with preset heterogeneity, measured via entropy balance metric, ranging from 0.1 to 1.0. Our results indicate that FedGrid outperforms the state-of-the-art methods when applied to deep neural network models. Moreover, the proposed method is simple and does not add computational complexity to the learning process. We believe that our effective method for grid application provides a new field for future research and improvement of the federation of deep neural network models.

Deep convolutional neural networks have achieved great success in recent years. They have been widely used in various applications such as optical and SAR image scene classification, object detection and recognition, semantic segmentation, and change detection. However, deep neural networks rely on large-scale high-quality training data, and can only guarantee good performance when the training and test data are independently sampled from the same distribution. Deep convolutional neural networks are found to be vulnerable to subtle adversarial perturbations. This adversarial vulnerability prevents the deployment of deep neural networks in security-sensitive applications such as medical, surveillance, autonomous driving and military scenarios. This paper first presents a holistic view of security issues for deep convolutional neural network-based image recognition systems. The entire information processing chain is analyzed regarding safety and security risks. In particular, poisoning attacks and evasion attacks on deep convolutional neural networks are analyzed in detail. The root causes of adversarial vulnerabilities of deep recognition models are also discussed. Then, we give a formal definition of adversarial robustness and present a comprehensive review of adversarial attacks, adversarial defense, and adversarial robustness evaluation. Rather than listing existing research, we focus on the threat models for the adversarial attack and defense arms race. We perform a detailed analysis of several representative adversarial attacks on SAR image recognition models and provide an example of adversarial robustness evaluation. Finally, several open questions are discussed regarding recent research progress from our workgroup. This paper can be further used as a reference to develop more robust deep neural network-based image recognition models in dynamic adversarial scenarios.

A methodology to construct deep neural network- (DNN) and recurrent neural network- (RNN) based proxy flow models is presented; these can reduce computational time of the flow simulation runs in the routine reservoir engineering workflows, such as history matching or optimization. A comparison of these two techniques shows that the DNN model generates predictions more quickly, but the RNN model provides better quality. In addition, RNN-based proxy flow models can make predictions for times after those included in the training data set. Both approaches can reduce computational time by a factor of up to 100 in comparison to the full-physics flow simulator. An example of the proxy flow model application is successfully demonstrated in an exhaustive search history matching exercise. All developments are shown on a synthesized Brugge petroleum reservoir.

Online dynamic stability analysis plays a key role in modern power systems and is becoming even more vital with the penetration of Distributed Energy Resources (DERs). Identification of an accurate lower-order model for the ‘external system” makes it possible to carry out the stability analysis of a study system faster than the analysis for the original detailed system. However, the identified lower-order model becomes invalid when the operating state changes in the power system. ‘Black box’ representation of the ‘external system’ with the application of Deep Neural Networks (DNN) is widely discussed in the literature. In such equivalent models, the DNN response changes based on two aspects i.e. the operating point of the power system and the location/type of the fault. In this paper, a clustering algorithm is proposed to identify the clusters of the ‘external system’ models for different states of the power system, and a DNN is trained for each cluster. The clustering algorithm is based on the pre-fault steady-state power flow from the ‘external system’ to the ‘study system’, which is a reflection of changes in the system state. Mean Absolute Error (MAE) between responses of the actual fault and the predicted fault is used to evaluate the accuracy of the identified system. The proposed technique is easy to implement and requires a lesser number of DNN models compared to identifying an individual DNN model for each operating point to represent the ‘external system’.

Security researchers have used Natural Language Processing (NLP) and Deep Learning techniques for programming code analysis tasks such as automated bug detection and vulnerability prediction or classification. These studies mainly generate the input vectors for the deep learning models based on the NLP embedding methods. Nevertheless, while there are many existing embedding methods, the structures of neural networks are diverse and usually heuristic. This makes it difficult to select effective combinations of neural models and the embedding techniques for training the code vulnerability detectors. To address this challenge, we extended a benchmark system to analyze the compatibility of four popular word embedding techniques with four different neural networks, including the standard Bidirectional Long Short-Term Memory (Bi-LSTM), the Bi-LSTM applied attention mechanism, the Convolutional Neural Network (CNN), and the classic Deep Neural Network (DNN). We trained and tested the models by using two types of vulnerable function datasets written in C code. Our results revealed that the Bi-LSTM model combined with the FastText embedding technique showed the most efficient detection rate on a real-world but not on an artificially constructed dataset. Further comparisons with the other combinations are also discussed in detail in our result.

We propose a hybrid neural network hidden Markov model (NN-HMM) approach for automatic story segmentation. A story is treated as an instance of an underlying topic (a hidden state) and words are generated from the distribution of the topic. The transition from one topic to another indicates a story boundary. Different from the traditional HMM approach, in which the emission probability of each state is calculated from a topic-dependent language model, we use deep neural network (DNN) to directly map the word distribution into topic posterior probabilities. DNN is known to be able to learn meaningful continuous features for words and hence has better discriminative and generalization capability than n-gram models. Specifically, we investigate three neural network structures: a feed-forward neural network, a recurrent neural network with long short-term memory cells (LSTM-RNN) and a modified LSTM-RNN with multi-task learning ability. Experimental results on the TDT2 corpus show that the proposed NN-HMM approach outperforms the traditional HMM approach significantly and achieves state-of-the-art performance in story segmentation.

Aiming at the problems of low prediction accuracy and efficiency and poor prediction effect in the current psychological pressure prediction methods, a psychological pressure prediction method for college students based on deep neural network is proposed. The structure and algorithm of depth neural network and gray theory model are analyzed. Using the deep neural network, this paper establishes the sample set data of college students’ psychological pressure prediction and constructs the college students’ psychological pressure prediction model combined with the deep neural network algorithm of gray theory. The physical network information model is formed through the relationship between neurons. According to the dynamic changes of college students’ psychological pressure in each neuron of the physical network, the prediction of college students’ psychological pressure is completed. The experimental results show that the proposed method is effective in predicting college students’ psychological pressure and can effectively improve the accuracy and efficiency of college students’ psychological pressure prediction.

Video hyperlinking represents a classical example of multimodal problems. Common approaches to such problems are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially deep autoencoders, have proven promising both for crossmodal translation and for early fusion via multimodal embedding. A particular architecture, bidirectional symmetrical deep neural networks, have been proven to yield improved multimodal embeddings over classical autoencoders, while also being able to perform crossmodal translation. In this work, we focus firstly at evaluating good single-modal continuous representations both for textual and for visual information. Word2Vec and paragraph vectors are evaluated for representing collections of words, such as parts of automatic transcripts and multiple visual concepts, while different deep convolutional neural networks are evaluated for directly embedding visual information, avoiding the creation of visual concepts. Secondly, we evaluate methods for multimodal fusion and crossmodal translation, with different single-modal pairs, in the task of video hyperlinking. Bidirectional (symmetrical) deep neural networks were shown to successfully tackle downsides of multimodal autoencoders and yield a superior multimodal representation. In this work, we extensively tests them in different settings, with different single-modal representations, within the context of video-hyperlinking. Our novel bidirectional symmetrical deep neural networks are compared to classical autoencoders and are shown to yield significantly improved multimodal embeddings that significantly (alpha=0.0001) outperform multimodal embeddings obtained by deep autoencoders with an absolute improvement in precision at 10 of 14.1% when embedding visual concepts and automatic transcripts and an absolute improvement of 4.3% when embedding automatic transcripts with features obtained with very deep convolutional neural networks, yielding 80% of precision at 10.

In order to explore the application of the image recognition model based on multi-stage convolutional neural network (MS-CNN) in the deep learning neural network in the intelligent recognition of commodity images and the recognition performance of the method, in the study, the features of color, shape, and texture of commodity images are first analyzed, and the basic structure of deep convolutional neural network (CNN) model is analyzed. Then, 50,000 pictures containing different commodities are constructed to verify the recognition effect of the model. Finally, the MS-CNN model is taken as the research object for improvement to explore the influence of label errors (p = 0.03, 0.05, 0.07, 0.09, 0.12) with different parameter settings and different probabilities (size of convolutional kernel, Dropout rate) on the recognition accuracy of MS-CNN model, at the same time, a CIR system platform based on MS-CNN model is built, and the recognition performance of salt and pepper noise images with different SNR (0, 0.03, 0.05, 0.07, 0.1) was compared, then the performance of the algorithm in the actual image recognition test was compared. The results show that the recognition accuracy is the highest (97.8%) when the convolution kernel size in the MS-CNN model is 2*2 and 3*3, and the average recognition accuracy is the highest (97.8%) when the dropout rate is 0.1; when the error probability of picture label is 12%, the recognition accuracy of the model constructed in this study is above 96%. Finally, the commodity image database constructed in this study is used to identify and verify the model. The recognition accuracy of the algorithm in this study is significantly higher than that of the Minitch stochastic gradient descent algorithm under different SNR conditions, and the recognition accuracy is the highest when SNR = 0 (99.3%). The test results show that the model proposed in this study has good recognition effect in the identification of commodity images in scenes of local occlusion, different perspectives, different backgrounds, and different light intensity, and the recognition accuracy is 97.1%. To sum up, the CIR platform based on MS-CNN model constructed in this study has high recognition accuracy and robustness, which can lay a foundation for the realization of subsequent intelligent commodity recognition technology.