On decomposing a deep neural network into modules

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

Deep neural network modeling of unknown partial differential equations in nodal space

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

Automated image classification is an essential task of the computer vision field. The tagging of images into a set of predefined groups is referred to as image classification. The implementation of computer vision to automate image classification would be beneficial because manual image evaluation and identification can be time-consuming, particularly when there are many images of different classes. Deep learning approaches are proven to overperform existing machine learning techniques in many fields in recent years, and computer vision is one of the most notable examples. The very deep neural network (VDCNN) is a powerful deep learning model for image classification, and this paper examines it briefly using MNIST handwritten digit dataset. This dataset is used to prove the efficacy of very deep neural networks over other deep learning models. The proposed study aims to comprehend the very deep neural network architecture used to accomplish a handwritten digit recognition task. The feasibility of the proposed model is evaluated using mean accuracy, validation accuracy, and standard deviation. The study results of the very deep neural network model are compared to a convolutional neural network and convolutional neural network with batch normalization. According to the results of the comparison study, very deep neural networks achieve high accuracy of 99.1% for a handwritten dataset. The outcome of the proposed work is used to interpret how well a very deep neural network performs when compared to the other two models of deep neural networks. This proposed architecture may be used to automate the classification of handwritten digits dataset.

Background and Objectives: This study investigated the usefulness of deep neural network (DNN) models based on 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) and blood inflammatory markers to assess the therapeutic response in pyogenic vertebral osteomyelitis (PVO). Materials and Methods: This was a retrospective study with prospectively collected data. Seventy-four patients diagnosed with PVO underwent clinical assessment for therapeutic responses based on clinical features during antibiotic therapy. The decisions of the clinical assessment were confirmed as ‘Cured’ or ‘Non-cured’. FDG-PETs were conducted concomitantly regardless of the decision at each clinical assessment. We developed DNN models depending on the use of attributes, including C-reactive protein (CRP), erythrocyte sedimentation ratio (ESR), and maximum standardized FDG uptake values of PVO lesions (SUVmax), and we compared their performances to predict PVO remission. Results: The 126 decisions (80 ‘Cured’ and 46 ‘Non-cured’ patients) were randomly assigned with training and test sets (7:3). We trained DNN models using a training set and evaluated their performances for a test set. DNN model 1 had an accuracy of 76.3% and an area under the receiver operating characteristic curve (AUC) of 0.768 [95% confidence interval, 0.625–0.910] using CRP and ESR, and these values were 79% and 0.804 [0.674–0.933] for DNN model 2 using ESR and SUVmax, 86.8% and 0.851 [0.726–0.976] for DNN model 3 using CRP and SUVmax, and 89.5% and 0.902 [0.804–0.999] for DNN model 4 using ESR, CRP, and SUVmax, respectively. Conclusions: The DNN models using SUVmax showed better performances when predicting the remission of PVO compared to CRP and ESR. The best performance was obtained in the DNN model using all attributes, including CRP, ESR, and SUVmax, which may be helpful for predicting the accurate remission of PVO.

Object detection based on deep learning can be widely used in all kinds of agricultural applications. In this paper, we present a deep neural network (DNN) model for quince and raspberry detection on RGB images. The trained DNN model is based on YOLOv5 architecture and it has 7 berry classes related to the berry development stage. YOLOv5 provides sufficiently good performance and precision trade-off. It is useful in the process of quince and raspberry phenotyping for the agriculture experts, where the yield and berry size parameters have to be estimated. Using our DNN model we have shown that it is possible to achieve a mean Average Precision close to 80.9 % and in some cases (Average Precision) close to 95 % for some classes. The DNN model is trained on labeled data gathered during the AKFEN project. The developed raspberry and quince detector is freely available at the GIT repository [1]. Further, the research on Sensor Networks, Wireless Systems, 3D point cloud processing and multi-spectral image processing has to be carried out leading to high-throughput phenotyping.

Dynamic properties, such as shear modulus and damping ratio, are critical for civil engineering applications and essential for accurate dynamic response analysis. This study introduces a novel Deep Neural Network (DNN) approach to predict the normalized shear modulus (G/Gmax ) and damping ratio (D) of granular soils over a wide strain range. Utilising a comprehensive dataset from cyclic triaxial (CT) and resonant column (RC) tests, we developed a Deep Feed-Forward Neural Network (DFFNN) model. The model incorporates grading characteristics, shear strain, void ratio, mean effective confining pressure, consolidation stress ratio, and specimen preparation method as inputs. The DFFNN demonstrated high accuracy with testing results of 0.9830 for G/Gmax and 0.9396 for D, outperforming traditional empirical models and other intelligent techniques such as Shallow Neural Network (SNN), Support Vector Regression (SVR), and Gradient Boosting Regression (GBR). This data-driven approach offers a robust and adaptable method for predicting the dynamic properties of granular soils across diverse conditions.

Deep Neural Network (DNN) models have achieved excellent performance in many inference tasks and have been widely used in many intelligent applications. However, DNN models often require a lot of computational resources to complete the inference tasks, which hinders the deployment of such models to resource‐constrained edge devices. In order to extend the application scenarios of DNN models, the edge‐cloud collaborative inference methods, represented by model partition, have attracted much research attention in recent years. In scenarios that have multiple edge devices deployed, the edge‐cloud collaborative inference method requires partial migration of tasks, but traditional scheduling methods only migrate tasks at the task level. In this paper, we propose two task scheduling methods, which can solve the problem of partial migration of tasks in multiedge scenarios. The first scheduling method is based on the optimal cutting of a single DNN. The cutting positions of all the models are the same, regardless of the influence of external factors. This method is suitable for chain and directed acyclic graph‐ (DAG‐) type DNNs. The second scheduling method takes external factors such as congestion and queuing delay at the cloud side into consideration, which dynamically selects the cutting position of each DNN to optimize the overall delay and thus is applicable to chain DNN models. The experimental results show that, compared with the baseline method, our proposed scheduling method can reduce the delay by up to 6.48x.

The superiority of various Deep Neural Networks (DNN) models, such as Convolutional Neural Networks (CNN), Generative Adversarial Networks (GAN), and Recurrent Neural Networks (RNN), has been proven in various real-world applications and has received much attention. However, different DNN models include various types of operations. For example, CNN models are usually composed of convolutional layers (Conv) and fully-connected layers (FC). Furthermore, some light CNN models such as MobileNet adopt depthwise separable convolution with depthwise convolution (DWC) and pointwise convolution (PWC) to compress the models. In addition to regular convolution, de-convolution (De-Conv) is also widely used in many GAN models. Moreover, many RNN models also employ long-short-term memory (LSTM) to control update of internal states and data. Such a high diversity of various DNN operations poses great design challenges in implementing reconfigurable Deep Learning (DL) accelerators, which can support various types of DNN operations. Most recent DL accelerators focus only on some DNN operations, which lacks computing flexibility. In this paper, by exploiting the sparsity in current DNN models, we design sparsity-aware DL hardware accelerators that can support efficient computation of various DNN operations, including Conv, DeConv, DWC, PWC, FC, and LSTM. Through reconfiguring dataflow and parallelizing different operations, the proposed designs not only improve system performance but also increase hardware utilization with a significant reduction of power consumption in memory accesses and arithmetic computations.

Decision of the model complexity is a significant challenge in contemporary data-driven modeling applications. Designing neural architecture involves the process of determining the optimal model complexity for deep neural networks (DNNs) models in order to uncover relationships in real-world data patterns. Consequently, optimizing DNN architectures is crucial for enhancing the practical approximation performance of DNN models to real-world data. This study implements a data-driven neuroevolution scheme for the optimal neural architecture search (NAS) and demonstrates a data-driven engineering application for cooling load estimation. The proposed neuroevolution scheme aims at evolving to the best generalizing DNN model that well suits the modeling complexity requirements of the dataset. To this end, the objective function for the neural architecture optimization process is simplified to the mean square error of the test dataset, which enables to reduce the risk of insufficient generalization during NAS. By employing this objective function for evolution field optimization (EFO), the proposed neuroevolution process can automatically achieve the optimal model complexity, preventing overfitting and underfitting cases, and thereby attaining almost the best generalization for the dataset. For this purpose, this approach combines parametric learning with the backpropagation algorithm and structural learning with EFO-based neural architecture search to address data-driven, optimal complexity DNN model generation problems. Effectiveness of the method is demonstrated in the cooling load estimation problem of residential buildings, and performances of the optimal DNN models with four objective functions are analyzed. The design of objective function for the best generalizing model is also elaborated.

In this article, for the first time, we propose channel characteristic-based deep neural network (DNN) models for accurate eye-height (EH) and eye-width (EW) estimation of high bandwidth memory (HBM) silicon interposer channels. The proposed models preprocess the design parameters that are highly relevant to the characteristics of the HBM channels. By taking account of the contribution of each design parameter to the eye diagram, the proposed models can accurately estimate the EH and EW of the channels even with a limited number of datasets. For verification, the proposed DNN models were applied to the microstrip and stripline channels of an HBM silicon interposer. Only redistributed layer (RDL) was used to clearly see the effect of reflecting the channel characteristics of the proposed method. We compared the proposed DNN models with various regression methods and a conventional fully connected multilayer DNN model. As a result, the proposed DNN models reduced the EH and EW error rates by 22.7 and 43.9% compared to the other regression methods. In addition, the proposed DNN models not only reduced the error rates by 22.0–28.4% but also reduced the computing cost by 8.0–9.4%, compared to the conventional DNN model. Moreover, we compared the proposed models with various DNN models having other preprocessing structures. By showing 26.7 and 28.8% lower EH and EW error rates than the other DNN models, we validated that the proposed models properly consider the most dominant design factors in preprocessing.

A deep learning algorithm-driven approach to predicting repair costs associated with natural disaster indicators: The case of accommodation facilities

Batteries are the most expensive component of battery electric vehicles (BEVs), but they degrade over time and battery operation. State of health (SOH) forecasting models learn how battery operation over long-time periods of weeks or months influences battery aging. Currently, existing methods for SOH forecasting of lithium-ion batteries based on deep neural network (DNN) models lack explainability of their forecasts due to their inherent black box character. However, the explainability of forecasts is essential to build user trust into the forecasting models. In this work, we address this problem from two perspectives: First, we compared four machine learning (ML) models like decision tree and random forest, which are inherently transparent, to two new DNN architectures with a more inherent black box character. Second, we proposed a new method using Gaussian-filtered saliency maps to visualize battery operational states that are relevant to DNN models. This method is applied to the best DNN models previously trained. We used an extensive data corpus consisting of five public data sets with different operational conditions, battery types, and aging trajectories. Furthermore, we show that the Gaussian-filtered saliency maps meaningfully visualize battery operational states that are consistent with findings from controlled laboratory aging experiments. Thus, this work was able to add transparency and interpretability to the SOH forecasting results of two state-of-the-art DNNs, while maintaining their superior performance compared to transparent ML models, while mitigating their inherent black box character.

Prediction of drilling induced delamination and circularity deviation in GFRP nanocomposites using deep neural network

Objective. Epidermal growth factor receptor (EGFR) mutation genotyping plays a pivotal role in targeted therapy for non-small cell lung cancer (NSCLC). We aimed to develop a computed tomography (CT) image-based hybrid deep radiomics model to predict EGFR mutation status in NSCLC and investigate the correlations between deep image and quantitative radiomics features. Approach. First, we retrospectively enrolled 818 patients from our centre and 131 patients from The Cancer Imaging Archive database to establish a training cohort (N = 654), an independent internal validation cohort (N = 164) and an external validation cohort (N = 131). Second, to predict EGFR mutation status, we developed three CT image-based models, namely, a multi-task deep neural network (DNN), a radiomics model and a feature fusion model. Third, we proposed a hybrid loss function to train the DNN model. Finally, to evaluate the model performance, we computed the areas under the receiver operating characteristic curves (AUCs) and decision curve analysis curves of the models. Main results. For the two validation cohorts, the feature fusion model achieved AUC values of 0.86 ± 0.03 and 0.80 ± 0.05, which were significantly higher than those of the single-task DNN and radiomics models (all P < 0.05). There was no significant difference between the feature fusion and the multi-task DNN models (P > 0.8). The binary prediction scores showed excellent prognostic value in predicting disease-free survival (P = 0.02) and overall survival (P < 0.005) for validation cohort 2. Significance. The results demonstrate that (1) the feature fusion and multi-task DNN models achieve significantly higher performance than that of the conventional radiomics and single-task DNN models, (2) the feature fusion model can decode the imaging phenotypes representing NSCLC heterogeneity related to both EGFR mutation and patient NSCLC prognosis, and (3) high correlations exist between some deep image and radiomics features.