Prototype Surgery: Tailoring Neural Prototypes via Soft Labels for Efficient Machine Unlearning

Recently deep-learning (DL) techniques have been widely adopted in side-channel power analysis. A DL-assisted SCA generally consists of two phases: a deep neural network (DNN) training phase and a follow-on attack phase using the trained DNN. However, currently the two phases are not well aligned, as there is no conclusion on what metric used in the training can result in the most effective attack in the second phase. When traditional loss functions such as negative log-likelihood (NLL) are used in training a DNN, the trained model does not yield optimal follow-on attack. Recently some information theoretical SCA leakage metrics are proposed, either as the validation metric to stop the DNN training with traditional loss functions, or as both the validation metric and the training loss function. None of those proposed metrics, however, directly measures the SCA effectiveness. We propose to conduct DNN training directly with a common SCA effectiveness metric, Guessing Entropy (GE). We overcome the prior practical difficulty of using GE in DNN training by utilizing the GEEA estimation algorithm introduced in CHES 2020. We show that using GEEA as either the validation metric or the loss function produces DNN models that lead to much more effective follow-on attacks. Our work consolidates the DL-assisted SCA framework with a consistent metric, which shows great potential to be adopted as the universal SCA-oriented DNN training framework.

With the vigorous development of MOOC open online courses, low completion rate of courses has become an important side effect factor which is affecting the promotion and application of MOOC courses in worldwide. Through analysis of edX open data sets, it is found that loss of learners is one of the factors which are leading to low completion rate of MOOC courses, and also there is a close correlation between learning behavior and final scores with loss of learners. Based on this challenge, some research results show that low completion rate of MOOC courses can be greatly improved by predicting the relationship between learning behavior and scores. This article focuses on application of Deep Neural Network (DNN) in prediction and analysis of MOOC learners' learning behavior. First, data analysis of course performance and learning behavior are discussed, then prediction model based on DNN is given for learning behavior and final scores. Second, experiment is done for DNN model, the results show that DNN can be used for analysis and prediction of complex correlation between learning behavior and final scores, thus which can realize more accurate prediction, reduce loss of learner, and carry out individualized teaching intervention to potential learners.

Online dynamic stability analysis plays a key role in modern power systems and is becoming even more vital with the penetration of Distributed Energy Resources (DERs). Identification of an accurate lower-order model for the ‘external system” makes it possible to carry out the stability analysis of a study system faster than the analysis for the original detailed system. However, the identified lower-order model becomes invalid when the operating state changes in the power system. ‘Black box’ representation of the ‘external system’ with the application of Deep Neural Networks (DNN) is widely discussed in the literature. In such equivalent models, the DNN response changes based on two aspects i.e. the operating point of the power system and the location/type of the fault. In this paper, a clustering algorithm is proposed to identify the clusters of the ‘external system’ models for different states of the power system, and a DNN is trained for each cluster. The clustering algorithm is based on the pre-fault steady-state power flow from the ‘external system’ to the ‘study system’, which is a reflection of changes in the system state. Mean Absolute Error (MAE) between responses of the actual fault and the predicted fault is used to evaluate the accuracy of the identified system. The proposed technique is easy to implement and requires a lesser number of DNN models compared to identifying an individual DNN model for each operating point to represent the ‘external system’.

Deep Learning (DL) has become a prominent machine learning technique due to the availability of efficient computational resources in the form of Graphics Processing Units (GPUs), large-scale datasets and a variety of models. The newer generation of GPUs are being designed with special emphasis on optimizing performance for DL applications. Also, the availability of easy-to-use DL frameworks—like PyTorch and TensorFlow— has enhanced productivity of domain experts to work on their custom DL applications from diverse domains. However, existing Deep Neural Network (DNN) training approaches may not fully utilize the newly emerging powerful GPUs like the NVIDIA A100—this is the primary issue that we address in this paper. Our motivating analyses show that the GPU utilization on NVIDIA A100 can be as low as 43% using traditional DNN training approaches for small-to-medium DL models and input data size. This paper proposes AccDP—a data-parallel distributed DNN training approach—to accelerate GPU-based DL applications. AccDP exploits the Message Passing Interface (MPI) communication library coupled with the NVIDIA’s Multi-Process Service (MPS) to increase the amount of work assigned to parallel GPUs resulting in higher utilization of compute resources. We evaluate our proposed design on different small-to-medium DL models and input sizes on the state-of-the-art HPC clusters. By injecting more parallelism into DNN training using our approach, the evaluation shows up to 58% improvement in training performance on a single GPU and up to 62% on 16 GPUs compared to regular DNN training. Furthermore, we conduct an in-depth characterization to determine the impact of several DNN training factors and best practices—including the batch size and the number of data loading workers— to optimally utilize GPU devices. To the best of our knowledge, this is the first work that explores the use of MPS and MPI to maximize the utilization of GPUs in distributed DNN training.

Survey on Deep Neural Networks in Speech and Vision Systems

Sedimentary basins, integral to Earth's geological history and energy resource exploration, undergo complex changes driven by sedimentation, subsidence and geological processes. Gravity anomaly inversion is a crucial technique offering insights into subsurface structures and density variations. Our study addresses the challenge of complex subsurface structure assessment by leveraging deep neural networks to invert observed gravity anomalies. Optimization approaches traditionally incorporate known density distributions obtained from borehole data or geological logging for inverting basement depth in sedimentary basins using observed gravity anomalies. Our study explores the application of deep neural networks in accurate architectural assessment of sedimentary basins and demonstrates their significance in mineral and hydrocarbon exploration. Recent years have witnessed a surge in the use of machine learning in geophysics, with deep learning models playing a pivotal role. Integrating deep neural networks, such as the feedforward neural networks, has revolutionized subsurface density distribution and basement depth estimation. This study introduces a deep neural network specifically tailored for inverting observed gravity anomalies to estimate two‐dimensional basement relief topographies in sedimentary basins. To enhance computational efficiency, a one‐dimensional discrete cosine transform based discretization approach is employed. Synthetic data, generated using non‐Gaussian fractals, compensates for the scarcity of true datasets for training the deep neural network model. The algorithm's robustness is validated through noise introduction with comparisons against an efficient and traditional global optimization‐based approach. Gravity anomalies of real sedimentary basins further validate the algorithm's efficacy, establishing it as a promising methodology for accurate and efficient subsurface imaging in geological exploration.

This paper aims to overcome a fundamental problem in the theory and application of deep neural networks (DNNs). We propose a method to solve the local minimum problem in training DNNs directly. Our method is based on the cross-entropy loss criterion's convexification by transforming the cross-entropy loss into a risk averting error (RAE) criterion. To alleviate numerical difficulties, a normalized RAE (NRAE) is employed. The convexity region of the cross-entropy loss expands as its risk sensitivity index (RSI) increases. Making the best use of the convexity region, our method starts training with an extensive RSI, gradually reduces it, and switches to the RAE as soon as the RAE is numerically feasible. After training converges, the resultant deep learning machine is expected to be inside the attraction basin of a global minimum of the cross-entropy loss. Numerical results are provided to show the effectiveness of the proposed method.

Recently, data-driven approaches motivated by modern deep learning have been applied to optical communications in place of traditional model-based counterparts. The application of deep neural networks (DNN) allows flexible statistical analysis of complicated fiber-optic systems without relying on any specific physical models. Due to the inherent nonlinearity in DNN, various equalizers based on DNN have shown significant potential to mitigate fiber nonlinearity. In this paper, we propose turbo equalization (TEQ) based on DNN as a new alternative framework to deal with nonlinear fiber impairments. The proposed DNN-TEQ is constructed with nested deep residual networks (ResNet) to train extrinsic likelihood given soft-information feedback from channel decoding. Through extrinsic information transfer (EXIT) analysis, we verify that our DNN-TEQ can accelerate decoding convergence to achieve a significant gain in achievable throughput by 0.61 b/s/Hz. We also demonstrate that optimizing irregular low-density parity-check (LDPC) codes based on the EXIT chart of the DNN-TEQ can improve achievable rates by up to 0.12 b/s/Hz.

With the explosive growth in seismic data acquisition and the successful application of deep convolutional neural networks (DCNN) to various image processing tasks within multidisciplinary fields, many researchers have begun to research DCNN based automatic seismic interpretation techniques. Due to the vast number of parameters considered in deep neural networks, deep learning methods usually require a large amount of data for training. However, collecting a large number of expert interpretations is very time consuming, so related research usually uses synthetic datasets and ignores the practical problems of field datasets. In this paper, we open-source a multi-gigabyte expert-labelled field dataset in response to the challenge of accessing large-scale expert-labelled field datasets. We show that 2D fault recognition within this dataset is an image segmentation or edge detection problem in the computer vision field, that can be expressed as a pixel-level fault/non-fault binary classification. Both types of DCNNs are compared, and we propose a novel fault recognition workflow, which involves processing and screening of seismic images and labels, training DCNNs and automatic numerical evaluation. We have also demonstrated for three case study datasets that effective image augmentation methods can reduce the required labelled crosslines while maintaining satisfactory performance. Our experimental results show that our workflow not only outperforms two state-of-the-art DCNN solutions but also achieves performance comparable to humans on an expert labelled image dataset, even predicting subtle faults that an expert interpreter did not annotate. We suggest that the proposed workflow could reduce the fault interpretation life cycle from months to hours and improve the quality, and define the confidence, of fault interpretation results.

As applications using deep neural networks (DNNs) are increasingly deployed on mobile devices, researchers are exploring various methods to achieve low energy consumption and high performance. Recently, advances in STT-MRAM have shown promise in offering non-volatility, high performance, and low energy consumption when used to replace DRAM in main memory. Most of the memory space used in DNN applications is occupied by weight and activation data. Typically, the contents of weight data remain unchanged during inference, whereas activation data are modified to store intermediate results across the layers of the DNN. As large DNN applications consume significant energy and require high performance, STT-MRAM is an ideal candidate to fully or partially replace DRAM in main memory. However, the long write latency of STT-MRAM compared to DRAM presents a performance bottleneck. In this work, we propose a reduced retention-time MRAM-based main memory to address this issue. We divide the MRAM into multiple partitions, with each partition implemented with different retention times, tailored for DNN applications. In this approach, the DNN weights can be assigned to the long retention-time partition, while the activation data can be allocated to the short retention-time partition, optimizing for the varying characteristics of data reuse. To achieve high performance, we propose two mapping schemes: intra-segment and inter-segment circular buffers, which dynamically map DNN activation data (i.e., virtual pages of streaming data) to physical pages in a circular fashion to exploit reuse patterns of DNN data. These circular buffers are mapped to the short retention-time MRAM partition as much as possible. The intra-circular buffer mapping scheme achieves an average improvement of 14.4% in bandwidth and 12.6% in inference latency compared to DRAM for on-device DNN applications. Furthermore, the inter-circular buffer mapping scheme offers 11.1% bandwidth and 11.2% latency improvements on average using only 16 MBytes of the short retention-time partition.

The objective of this paper is 2-fold. First, it develops a prediction system to help the credit card issuer model the credit card delinquency risk. Second, it seeks to explore the potential of deep learning (also called deep neural network), an emerging artificial intelligence technology, in credit risk domain. With a real-life credit card data linked to 711,397 credit card holders from a large bank in Brazil, this study develops a deep neural network to evaluate the risk of credit card delinquency based on the client’s personal characteristics and the spending behaviors. Compared to machine learning algorithms of logistic regression, naive Bayes, traditional artificial neural network, and decision tree, deep neural network has a better overall predictive performance with the highest F scores and AUC. The successful application of deep learning implies that artificial intelligence has great potential to support and automate credit risk assessment for financial institutions and credit bureaus.

Small data sets make developing calibration models using deep neural networks difficult because it is easy to overfit the system. We developed two deep neural network architectures by revising two existing network architectures: the U-Net and the attention mechanism. The major changes were to use 1D convolutional layers to replace the fully connected layers. We also designed and combined average pooling and maximum pooling in our revised networks, respectively. We applied these revised network architectures to three publicly available data sets and the resulting calibration models can generate acceptable results for general quantitative analysis. It also generated rather good results for data sets that concern calibration transfer. It demonstrates that constructing network architectures by properly revising existing successful network architectures may provide additional choices in the exploration of the application of deep neural network in analytical chemistry.

The problem of application of deep neural networks (DNNs) for extremely small dataset is considered to find the effect of dataset size and quality on performance metrics. Classification task is considered for the set of problems where images of Lyme disease symptoms were collected and structured in the open-access dataset with the different subsets of the whole dataset: clean images only (C), dirty images only (D), and clean and dirty images in the equal proportions (C_D) (https://www.kaggle.com/yoctoman/lyme-clean-and-dirty) . Two different data preprocessing techniques were applied: normalization without data augmentation (NODA) and normalization with data augmentation (DA). The two versions of the modern NASNet DNN architectures of different sizes were used, namely NASNetLarge and NASNetMobile, with the weights obtained after training on ImageNet dataset. These DNNs were trained in two experiments, namely with frozen layers, aka transfer learning (TL), and with unfrozen layers when all weights were updated (NOTL). After fivefold cross-validated training the best results for mean validation metrics were demonstrated by the models trained in C_NOTL_NODA, C_NOTL_DA, and C_TL_NODA experiments for NASNetLarge, and in C_TL_NODA by NASNetMobile. But the testing on the new unknown subset of Lyme disease images with the equal proportion of clean and dirty images gave quite different results. The models trained in the before-mentioned experiments demonstrated poor performance in comparison with the models trained in C_NOTL_DA, D_NOTL_DA experiments, and, especially in D_C_NOTL_DA ones. The successful training strategy should include tuning ImageNet-trained weights without their freezing (NOTL) and intensive data augmentation (DA), but it would not be enough if some diverse relevant background will not be added. That is why the additional methods of “context-relevant data augmentation” (CRDA) mimicked by inclusion of D_C data configuration should be considered. These findings can be used for the more effective selection of training experiments of DNNs in the healthcare context in the view of the small number of available medical data in practice.KeywordsNeural networksDeep learningSmall dataset problemData augmentationTransfer learningHealth careLyme disease

Centralized machine learning algorithms in vehicular networks face privacy and resource constraints.Federated Learning (FL) addresses these by enabling collaborative model training without sharing raw data. To incentivize vehicle participation despite resource constraints, we propose a contract between Roadside Units (RSUs) and vehicles, specifying required computation resources and rewards based on local data quality. The contract is designed to maximize the utility of the RSU, subject to constraints that ensure the participation of the vehicles in the task and truthful reporting of their data quality parameter. However, designing an optimal contract with classical methods is limited to specific utility functions. To extend the design methodology of optimal contracts to a broader range of utility functions, our study proposes a deep learning-based solution. Specifically, we present a novel application of Deep Neural Networks (DNNs), marking the first instance of applying DNNs in contract theory-based incentive mechanisms to achieve near-optimal contracts. We design the DNN's structure to inherently satisfy specific constraints of the optimization problem, enhancing its efficiency and accuracy in determining the optimal contract. We utilize an augmented Lagrangian method to approximate the optimal contract in federated learning tasks. A case study on traffic sign recognition demonstrates the applicability and effectiveness of our method. Comparing our approach with the well-known generative diffusion model shows that in the FL scenarios, the DNN results are closer to the optimal contract compared to those of the generative diffusion model.

Recent years have witnessed growing interest in the application of deep neural networks (DNNs) for receiver design, which can potentially be applied in complex environments without relying on knowledge of the channel model. However, the dynamic nature of communication channels often leads to rapid distribution shifts, which may require periodically retraining. This paper formulates a data-efficient two-stage training method that facilitates rapid online adaptation. Our training mechanism uses a predictive meta-learning scheme to train rapidly from data corresponding to both current and past channel realizations. Our method is applicable to any deep neural network (DNN)-based receiver, and does not require transmission of new pilot data for training. To illustrate the proposed approach, we study DNN-aided receivers that utilize an interpretable model-based architecture, and introduce a modular training strategy based on predictive meta-learning. We demonstrate our techniques in simulations on a synthetic linear channel, a synthetic non-linear channel, and a COST 2100 channel. Our results demonstrate that the proposed online training scheme allows receivers to outperform previous techniques based on self-supervision and joint-learning by a margin of up to 2.5 dB in coded bit error rate in rapidly-varying scenarios.