Yielding Multi-Fold Training Strategy for Image Classification of Imbalanced Weeds

Deep neural network (DNN) models typically have many hyperparameters that can be configured to achieve optimal performance on a particular dataset. Practitioners usually tune the hyperparameters of their DNN models by training a number of trial models with different configurations of the hyperparameters, to find the optimal hyperparameter configuration that maximizes the training accuracy or minimizes the training loss. As such hyperparameter tuning usually focuses on the model accuracy or the loss function, it is not clear and remains under-explored how the process impacts other performance properties of DNN models, such as inference latency and model size. On the other hand, standard DNN models are often large in size and computing-intensive, prohibiting them from being directly deployed in resource-bounded environments such as mobile devices and Internet of Things (IoT) devices. To tackle this problem, various model optimization techniques (e.g., pruning or quantization) are proposed to make DNN models smaller and less computing-intensive so that they are better suited for resource-bounded environments. However, it is neither clear how the model optimization techniques impact other performance properties of DNN models such as inference latency and battery consumption, nor how the model optimization techniques impact the effect of hyperparameter tuning (i.e., the compounding effect). Therefore, in this paper, we perform a comprehensive study on four representative and widely-adopted DNN models, i.e., CNN image classification , Resnet-50 , CNN text classification , and LSTM sentiment classification , to investigate how different DNN model hyperparameters affect the standard DNN models, as well as how the hyperparameter tuning combined with model optimization affect the optimized DNN models, in terms of various performance properties (e.g., inference latency or battery consumption). Our empirical results indicate that tuning specific hyperparameters has heterogeneous impact on the performance of DNN models across different models and different performance properties. In particular, although the top tuned DNN models usually have very similar accuracy, they may have significantly different performance in terms of other aspects (e.g., inference latency). We also observe that model optimization has a confounding effect on the impact of hyperparameters on DNN model performance. For example, two sets of hyperparameters may result in standard models with similar performance but their performance may become significantly different after they are optimized and deployed on the mobile device. Our findings highlight that practitioners can benefit from paying attention to a variety of performance properties and the confounding effect of model optimization when tuning and optimizing their DNN models.

Deep learning is being incorporated in many modern software systems. Deep learning approaches train a deep neural network (DNN) model using training examples, and then use the DNN model for prediction. While the structure of a DNN model as layers is observable, the model is treated in its entirety as a monolithic component. To change the logic implemented by the model, e.g. to add/remove logic that recognizes inputs belonging to a certain class, or to replace the logic with an alternative, the training examples need to be changed and the DNN needs to be retrained using the new set of examples. We argue that decomposing a DNN into DNN modules— akin to decomposing a monolithic software code into modules—can bring the benefits of modularity to deep learning. In this work, we develop a methodology for decomposing DNNs for multi-class problems into DNN modules. For four canonical problems, namely MNIST, EMNIST, FMNIST, and KMNIST, we demonstrate that such decomposition enables reuse of DNN modules to create different DNNs, enables replacement of one DNN module in a DNN with another without needing to retrain. The DNN models formed by composing DNN modules are at least as good as traditional monolithic DNNs in terms of test accuracy for our problems.

Background and Objectives: This study investigated the usefulness of deep neural network (DNN) models based on 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) and blood inflammatory markers to assess the therapeutic response in pyogenic vertebral osteomyelitis (PVO). Materials and Methods: This was a retrospective study with prospectively collected data. Seventy-four patients diagnosed with PVO underwent clinical assessment for therapeutic responses based on clinical features during antibiotic therapy. The decisions of the clinical assessment were confirmed as ‘Cured’ or ‘Non-cured’. FDG-PETs were conducted concomitantly regardless of the decision at each clinical assessment. We developed DNN models depending on the use of attributes, including C-reactive protein (CRP), erythrocyte sedimentation ratio (ESR), and maximum standardized FDG uptake values of PVO lesions (SUVmax), and we compared their performances to predict PVO remission. Results: The 126 decisions (80 ‘Cured’ and 46 ‘Non-cured’ patients) were randomly assigned with training and test sets (7:3). We trained DNN models using a training set and evaluated their performances for a test set. DNN model 1 had an accuracy of 76.3% and an area under the receiver operating characteristic curve (AUC) of 0.768 [95% confidence interval, 0.625–0.910] using CRP and ESR, and these values were 79% and 0.804 [0.674–0.933] for DNN model 2 using ESR and SUVmax, 86.8% and 0.851 [0.726–0.976] for DNN model 3 using CRP and SUVmax, and 89.5% and 0.902 [0.804–0.999] for DNN model 4 using ESR, CRP, and SUVmax, respectively. Conclusions: The DNN models using SUVmax showed better performances when predicting the remission of PVO compared to CRP and ESR. The best performance was obtained in the DNN model using all attributes, including CRP, ESR, and SUVmax, which may be helpful for predicting the accurate remission of PVO.

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from intellectual property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this article, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labeled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This article is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

Decision of the model complexity is a significant challenge in contemporary data-driven modeling applications. Designing neural architecture involves the process of determining the optimal model complexity for deep neural networks (DNNs) models in order to uncover relationships in real-world data patterns. Consequently, optimizing DNN architectures is crucial for enhancing the practical approximation performance of DNN models to real-world data. This study implements a data-driven neuroevolution scheme for the optimal neural architecture search (NAS) and demonstrates a data-driven engineering application for cooling load estimation. The proposed neuroevolution scheme aims at evolving to the best generalizing DNN model that well suits the modeling complexity requirements of the dataset. To this end, the objective function for the neural architecture optimization process is simplified to the mean square error of the test dataset, which enables to reduce the risk of insufficient generalization during NAS. By employing this objective function for evolution field optimization (EFO), the proposed neuroevolution process can automatically achieve the optimal model complexity, preventing overfitting and underfitting cases, and thereby attaining almost the best generalization for the dataset. For this purpose, this approach combines parametric learning with the backpropagation algorithm and structural learning with EFO-based neural architecture search to address data-driven, optimal complexity DNN model generation problems. Effectiveness of the method is demonstrated in the cooling load estimation problem of residential buildings, and performances of the optimal DNN models with four objective functions are analyzed. The design of objective function for the best generalizing model is also elaborated.

Deep Neural Network (DNN) models have achieved excellent performance in many inference tasks and have been widely used in many intelligent applications. However, DNN models often require a lot of computational resources to complete the inference tasks, which hinders the deployment of such models to resource‐constrained edge devices. In order to extend the application scenarios of DNN models, the edge‐cloud collaborative inference methods, represented by model partition, have attracted much research attention in recent years. In scenarios that have multiple edge devices deployed, the edge‐cloud collaborative inference method requires partial migration of tasks, but traditional scheduling methods only migrate tasks at the task level. In this paper, we propose two task scheduling methods, which can solve the problem of partial migration of tasks in multiedge scenarios. The first scheduling method is based on the optimal cutting of a single DNN. The cutting positions of all the models are the same, regardless of the influence of external factors. This method is suitable for chain and directed acyclic graph‐ (DAG‐) type DNNs. The second scheduling method takes external factors such as congestion and queuing delay at the cloud side into consideration, which dynamically selects the cutting position of each DNN to optimize the overall delay and thus is applicable to chain DNN models. The experimental results show that, compared with the baseline method, our proposed scheduling method can reduce the delay by up to 6.48x.

In this article, for the first time, we propose channel characteristic-based deep neural network (DNN) models for accurate eye-height (EH) and eye-width (EW) estimation of high bandwidth memory (HBM) silicon interposer channels. The proposed models preprocess the design parameters that are highly relevant to the characteristics of the HBM channels. By taking account of the contribution of each design parameter to the eye diagram, the proposed models can accurately estimate the EH and EW of the channels even with a limited number of datasets. For verification, the proposed DNN models were applied to the microstrip and stripline channels of an HBM silicon interposer. Only redistributed layer (RDL) was used to clearly see the effect of reflecting the channel characteristics of the proposed method. We compared the proposed DNN models with various regression methods and a conventional fully connected multilayer DNN model. As a result, the proposed DNN models reduced the EH and EW error rates by 22.7 and 43.9% compared to the other regression methods. In addition, the proposed DNN models not only reduced the error rates by 22.0–28.4% but also reduced the computing cost by 8.0–9.4%, compared to the conventional DNN model. Moreover, we compared the proposed models with various DNN models having other preprocessing structures. By showing 26.7 and 28.8% lower EH and EW error rates than the other DNN models, we validated that the proposed models properly consider the most dominant design factors in preprocessing.

Artificial Intelligence (AI) in healthcare enables predicting diabetes using data-driven methods instead of the traditional ways of screening the disease, which include hemoglobin A1c (HbA1c), oral glucose tolerance test (OGTT), and fasting plasma glucose (FPG) screening techniques, which are invasive and limited in scale. Machine learning (ML) and deep neural network (DNN) models that use large datasets to learn the complex, nonlinear feature interactions, but the conventional ML algorithms are data sensitive and often show unstable predictive accuracy. Conversely, DNN models are more robust, though the ability to reach a high accuracy rate consistently on heterogeneous datasets is still an open challenge. For predicting diabetes, this work proposed a hybrid DNN approach by integrating a bidirectional long short-term memory (BiLSTM) network with a bidirectional gated recurrent unit (BiGRU). A robust DL model, developed by combining various datasets with weighted coefficients, dense operations in the connection of deep layers, and the output aggregation using batch normalization and dropout functions to avoid overfitting. The goal of this hybrid model is better generalization and consistency among various datasets, which facilitates the effective management and early intervention. The proposed DNN model exhibits an excellent predictive performance as compared to the state-of-the-art and baseline ML and DNN models for diabetes prediction tasks. The robust performance indicates the possible usefulness of DL-based models in the development of disease prediction in healthcare and other areas that demand high-quality analytics.

A small dataset commonly affects generalization, robustness, and overall performance of deep neural networks (DNNs) in medical imaging research. Since gathering large clinical databases is always difficult, we proposed an analytical method for producing a large realistic/diverse dataset. Clinical brain PET/CT/MR images including full-dose (FD), low-dose (LD) corresponding to only 5 % of events acquired in the FD scan, non-attenuated correction (NAC) and CT-based measured attenuation correction (MAC) PET images, CT images and T1 and T2 MR sequences of 35 patients were included. All images were registered to the Montreal Neurological Institute (MNI) template. Laplacian blending was used to make a natural presentation using information in the frequency domain of images from two separate patients, as well as the blending mask. This classical technique from the computer vision and image processing communities is still widely used and unlike modern DNNs, does not require the availability of training data. A modified ResNet DNN was implemented to evaluate four image-to-image translation tasks, including LD to FD, LD+MR to FD, NAC to MAC, and MRI to CT, with and without using the synthesized images. Quantitative analysis using established metrics, including the peak signal-to-noise ratio (PSNR), structural similarity index metric (SSIM), and joint histogram analysis was performed for quantitative evaluation. The quantitative comparison between the registered small dataset containing 35 patients and the large dataset containing 350 synthesized plus 35 real dataset demonstrated improvement of the RMSE and SSIM by 29% and 8% for LD to FD, 40% and 7% for LD+MRI to FD, 16% and 8% for NAC to MAC, and 24% and 11% for MRI to CT mapping task, respectively. The qualitative/quantitative analysis demonstrated that the proposed model improved the performance of all four DNN models through producing images of higher quality and lower quantitative bias and variance compared to reference images.

Deep neural network modeling of unknown partial differential equations in nodal space

Industrial software technology: R Mitchell (ed) Peter Peregrinus Ltd (on behalf of the Institution of Electrical Engineers), London, UK (1987) 289pp £37.00

Experimental assessment of the efficiency of deep learning method in predicting the mechanical properties of polymer concretes and composites

Accurate physical layer modeling for optical fiber communication systems is one of the key aspects of implementing an intelligent optical network. In this paper, we propose and study a data-driven model enabled by digital signal processing (DSP) methods for predicting the electrical and optical behavior of an intensity modulation direct detection (IM/DD) system based on an electro-absorption modulated laser (EML). This model captures both the linear and nonlinear characteristics of an IM/DD system with an adaptive finite impulse response (FIR) filter and a look-up table (LUT). Similar to conventional supervised machine learning problems, the proposed model is measurement-informed by iterating through a PAM-M signal training dataset. Additionally, we demonstrate that the proposed model parameters can be generalized by applying 2D piecewise bilinear interpolation across different driving voltages and received optical powers. To highlight the model advantages, we benchmark the proposed model against an existing equivalent circuit model and a deep neural network (DNN) model using multiple performance metrics, including signal root mean square error (RMSE), bit error rate (BER), training time, and model complexity. We demonstrate that the proposed FIR-LUT model outperforms the conventional equivalent circuit model using the PAM-2 dataset by achieving 0.021 sample RMSE and 0.018 symbol RMSE. Similarly, a 0.020 sample RMSE and 0.015 symbol RMSE are achieved with the PAM-4 dataset. Compared with the DNN alternative, the proposed FIR-LUT model delivers comparable accuracy but significantly reduced model complexity. The FIR-LUT model achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$34\times$</tex-math></inline-formula> PAM-2 and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$24\times$</tex-math></inline-formula> PAM-4 training phase speed-up and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$486\times$</tex-math></inline-formula> PAM-2 and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$283\times$</tex-math></inline-formula> PAM-4 inference phase speed-up when compared with the DNN model.

Long-term prediction for railway track geometry based on an optimised DNN method

It is a challenging task to resist illegal usage of Deep Neural Network (DNN) models in applications from edge computing. The existed protection method is developed based on encrypting all weights of DNN models and has achieved a promising result, which however suffers from a high computation cost. In this paper, we design a critical-weight based method to lock the DNN model to defend unauthorized usage, leading to a significant decrease in the time cost. To be specific, we analyse and figure out the critical weights of a DNN model, and then encrypt the critical weights to lock the DNN model. A set of preliminary experiments are conducted to testify the effectiveness of the proposed approach.