Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization

As applications using deep neural networks (DNNs) are increasingly deployed on mobile devices, researchers are exploring various methods to achieve low energy consumption and high performance. Recently, advances in STT-MRAM have shown promise in offering non-volatility, high performance, and low energy consumption when used to replace DRAM in main memory. Most of the memory space used in DNN applications is occupied by weight and activation data. Typically, the contents of weight data remain unchanged during inference, whereas activation data are modified to store intermediate results across the layers of the DNN. As large DNN applications consume significant energy and require high performance, STT-MRAM is an ideal candidate to fully or partially replace DRAM in main memory. However, the long write latency of STT-MRAM compared to DRAM presents a performance bottleneck. In this work, we propose a reduced retention-time MRAM-based main memory to address this issue. We divide the MRAM into multiple partitions, with each partition implemented with different retention times, tailored for DNN applications. In this approach, the DNN weights can be assigned to the long retention-time partition, while the activation data can be allocated to the short retention-time partition, optimizing for the varying characteristics of data reuse. To achieve high performance, we propose two mapping schemes: intra-segment and inter-segment circular buffers, which dynamically map DNN activation data (i.e., virtual pages of streaming data) to physical pages in a circular fashion to exploit reuse patterns of DNN data. These circular buffers are mapped to the short retention-time MRAM partition as much as possible. The intra-circular buffer mapping scheme achieves an average improvement of 14.4% in bandwidth and 12.6% in inference latency compared to DRAM for on-device DNN applications. Furthermore, the inter-circular buffer mapping scheme offers 11.1% bandwidth and 11.2% latency improvements on average using only 16 MBytes of the short retention-time partition.

Online dynamic stability analysis plays a key role in modern power systems and is becoming even more vital with the penetration of Distributed Energy Resources (DERs). Identification of an accurate lower-order model for the ‘external system” makes it possible to carry out the stability analysis of a study system faster than the analysis for the original detailed system. However, the identified lower-order model becomes invalid when the operating state changes in the power system. ‘Black box’ representation of the ‘external system’ with the application of Deep Neural Networks (DNN) is widely discussed in the literature. In such equivalent models, the DNN response changes based on two aspects i.e. the operating point of the power system and the location/type of the fault. In this paper, a clustering algorithm is proposed to identify the clusters of the ‘external system’ models for different states of the power system, and a DNN is trained for each cluster. The clustering algorithm is based on the pre-fault steady-state power flow from the ‘external system’ to the ‘study system’, which is a reflection of changes in the system state. Mean Absolute Error (MAE) between responses of the actual fault and the predicted fault is used to evaluate the accuracy of the identified system. The proposed technique is easy to implement and requires a lesser number of DNN models compared to identifying an individual DNN model for each operating point to represent the ‘external system’.

With the vigorous development of MOOC open online courses, low completion rate of courses has become an important side effect factor which is affecting the promotion and application of MOOC courses in worldwide. Through analysis of edX open data sets, it is found that loss of learners is one of the factors which are leading to low completion rate of MOOC courses, and also there is a close correlation between learning behavior and final scores with loss of learners. Based on this challenge, some research results show that low completion rate of MOOC courses can be greatly improved by predicting the relationship between learning behavior and scores. This article focuses on application of Deep Neural Network (DNN) in prediction and analysis of MOOC learners' learning behavior. First, data analysis of course performance and learning behavior are discussed, then prediction model based on DNN is given for learning behavior and final scores. Second, experiment is done for DNN model, the results show that DNN can be used for analysis and prediction of complex correlation between learning behavior and final scores, thus which can realize more accurate prediction, reduce loss of learner, and carry out individualized teaching intervention to potential learners.

Deep Neural Networks meet computation offloading in mobile edge networks: Applications, taxonomy, and open issues

Tensor decomposition has been proved to be effective for solving many problems in signal processing and machine learning[1]. Recently, tensor decomposition finds its advantage for compressing deep neural networks. In many applications of deep neural networks, it is critical to reduce the number of parameters and computation workload to accelerate inference speed in deployment of the network. Modern deep neural network consists of multiple layers with multi-array weights where tensor decomposition is a natural way to perform compression. It is achieved by decomposing the weight tensors in convolutional layers or fully-connected layers with specified tensor ranks (e.g. canonical ranks, tensor train ranks). Conventional tensor decomposition in compressing deep neural networks selects the ranks manually that requires tedious human efforts to finetune the performance. To overcome this issue, we propose a novel rank selection scheme, which is inspired by reinforcement learning, to automatically select ranks in recently studied tensor ring decomposition in each convolutional layer. Experimental results validate that our learning based rank selection significantly outperforms hand-crafted rank selection heuristics on a number of benchmark datasets, for the purpose of effectively compressing deep neural networks while maintaining comparable accuracy.

Exploration seismology uses reflected and refracted seismic waves, emitted from a controlled (active) source into the ground, and recorded by an array of seismic sensors (receivers) to image the subsurface geologic structures. These seismic images are the main resources for energy and resource exploration and scientific investigation of the crust and upper mantle. We survey recent advances in applications of machine-learning methods, more specifically deep neural networks (DNNs), in exploration seismology. We provide a technically oriented review of DNN applications for seismic data acquisition; data preprocessing tasks such as interpolation/extrapolation, denoising, first-break picking, velocity picking, and seismic migration; data processing tasks such as geologic and structural interpretations; and data modeling tasks such as the inference of subsurface structures and lithologic and petrophysical properties. DNNs have entered almost every sector of exploration seismology. They have outperformed many traditional algorithms for the automation of seismic data acquisition, data preprocessing, data processing, interpretations, and data modeling tasks. However, despite the impressive performances of DNN-based approaches, the out-of-distribution generalization and interpretability of these models remain challenging. To overcome these challenges, incorporating domain knowledge into the DNNs is a promising path and a focus of current deep-learning research in seismology.

Sedimentary basins, integral to Earth's geological history and energy resource exploration, undergo complex changes driven by sedimentation, subsidence and geological processes. Gravity anomaly inversion is a crucial technique offering insights into subsurface structures and density variations. Our study addresses the challenge of complex subsurface structure assessment by leveraging deep neural networks to invert observed gravity anomalies. Optimization approaches traditionally incorporate known density distributions obtained from borehole data or geological logging for inverting basement depth in sedimentary basins using observed gravity anomalies. Our study explores the application of deep neural networks in accurate architectural assessment of sedimentary basins and demonstrates their significance in mineral and hydrocarbon exploration. Recent years have witnessed a surge in the use of machine learning in geophysics, with deep learning models playing a pivotal role. Integrating deep neural networks, such as the feedforward neural networks, has revolutionized subsurface density distribution and basement depth estimation. This study introduces a deep neural network specifically tailored for inverting observed gravity anomalies to estimate two‐dimensional basement relief topographies in sedimentary basins. To enhance computational efficiency, a one‐dimensional discrete cosine transform based discretization approach is employed. Synthetic data, generated using non‐Gaussian fractals, compensates for the scarcity of true datasets for training the deep neural network model. The algorithm's robustness is validated through noise introduction with comparisons against an efficient and traditional global optimization‐based approach. Gravity anomalies of real sedimentary basins further validate the algorithm's efficacy, establishing it as a promising methodology for accurate and efficient subsurface imaging in geological exploration.

With the rapid development of the Internet of Things (IoT) and communication technology, deep neural network (DNN) applications, such as medical imaging, speech transcription, handwritten text recognition, have been widely used in IoT devices. However, due to resource constraints on these devices, e.g., limited memory capacity, weak computing capacity, and low battery capacity, IoT devices cannot support complicated DNN operation effectively and, thus, fail to fulfill the requirements of Quality of Service of mobile users. One promising approach is to migrate the DNN model to a remote cloud server to reduce the computing burden on IoT devices. Unfortunately, it still suffers from high delay and low bandwidth when communicating with cloud servers. Although the transmission delay of the edge server is low, its computing capacity lacks scalability and elasticity.To make matters worse, in the real world, the wireless connection between IoT devices and the cloud is intermittent, which can cause offloading failures during large-scale DNN data transmission. In this article, we describe a DNN model migration framework to overcome the abovementioned challenges, which consists of three parts: DNN model preprocessing, partition-offloading plan, and partition-uploading plan. Accordingly, we introduce the operation of the DNN migration and the available methods for each part. In addition, we improve the DNN partition-uploading plan in a multiuser edge-cloud collaborative computing environment. Finally, we highlight the important challenges of achieving more efficient DNN migration and point out the unresolved issues of DNN migration, which may shed light on future research directions.

Recently, data-driven approaches motivated by modern deep learning have been applied to optical communications in place of traditional model-based counterparts. The application of deep neural networks (DNN) allows flexible statistical analysis of complicated fiber-optic systems without relying on any specific physical models. Due to the inherent nonlinearity in DNN, various equalizers based on DNN have shown significant potential to mitigate fiber nonlinearity. In this paper, we propose turbo equalization (TEQ) based on DNN as a new alternative framework to deal with nonlinear fiber impairments. The proposed DNN-TEQ is constructed with nested deep residual networks (ResNet) to train extrinsic likelihood given soft-information feedback from channel decoding. Through extrinsic information transfer (EXIT) analysis, we verify that our DNN-TEQ can accelerate decoding convergence to achieve a significant gain in achievable throughput by 0.61 b/s/Hz. We also demonstrate that optimizing irregular low-density parity-check (LDPC) codes based on the EXIT chart of the DNN-TEQ can improve achievable rates by up to 0.12 b/s/Hz.

The rapid advancements and widespread application of deep neural networks (DNNs), coupled with their reliance on sensitive and private data, have sparked growing concerns regarding data privacy and the ''right to be forgotten''. To address these concerns, machine unlearning has been proposed to efficiently eliminate the influence of specific training data from trained DNNs. However, existing machine unlearning methods struggle with the large number of parameters in trained DNNs, which lead to slow execution and high memory consumption, making them impractical for large-scale models. In this paper, we shift our focus to the small set of weights in the final classification layer of DNNs, which are defined as as ''prototypes'' for different classes. Our key observation is that the prototype associated with the unlearned training data undergoes a significant shift, whereas prototypes of unrelated classes exhibit only minor changes when comparing the prototypes of original and retrained models. Based on this observation, we propose a novel machine unlearning approach that efficiently achieves machine unlearning by directly adjusting the prototypes of DNNs. We first introduce Naive Prototype Surgery (Naive PS), a fast and simplified method that uses a closed-form solution to approximate unlearning effect by directly adjusting the prototype associated with the unlearned data. Next, we propose Prototype Surgery (PS), which incorporates soft label information to fine-tune the prototypes of all classes, to achieve a more effective unlearning. Both methods achieve data unlearning by only modifying the prototypes in the DNNs, thus avoiding the challenges posed by the large number of model parameters. Extensive experiments on four datasets demonstrate that our methods significantly accelerate the unlearning process while achieving comparable results to five existing methods in terms of both unlearning performance and privacy guarantee.

Recently, the applications of deep neural network (DNN) have been very prominent in many fields due to its superior feature extraction performance. However, the high-dimension parameter model and large-scale mathematical calculation restrict the execution efficiency, especially for the Internet of Things (IoT) devices. Different from the previous cloud/edge-only pattern that brings significant pressure for uplink communication and device-only fashion that undertakes unaffordable calculation strength, we highlight the collaborative computation between the device and edge for DNN models, which can achieve a good balance between the communication load and execution accuracy. Specifically, a systematic on-demand co-inference framework is proposed to exploit the multi-branch structure, in which the pre-trained Alexnet is right-sized through early-exit and partitioned at an intermediate DNN layer. The integer quantization is enforced to further compress transmission bits. As a result, we establish a new Deep Reinforcement Learning (DRL) optimizer-Soft Actor Critic for discrete (SAC-d), which generates the exit point, partition point, and compressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an optimizer can well adapt to the complex environment like dynamic wireless channel and arbitrary CPU processing, and is capable of supporting the 5G URLLC. Real-world experiment on Raspberry Pi 4 and PC shows the effective performance of the proposed solution.

With the widespread applications of Deep Neural Networks (DNNs), the safety of DNNs has become a significant issue. The vulnerability of the neural networks against adversarial examples deepens concerns about the safety of DNNs applications. This paper proposed a novel defence method to improve the adversarial robustness of DNN classifiers without using adversarial training. This method introduces two new loss functions. First, a zero-cross-entropy loss is used to punish overconfidence and find the appropriate confidence for different instances. Second, a logit balancing loss is proposed to protect DNNs from non-targeted attacks by regularising incorrect classes’ logits distribution. This method achieved competitive adversarial robustness compared to advanced adversarial training methods. Meanwhile, a novel robustness diagram is proposed to analyse, interpret and visualise the robustness of DNN classifiers against adversarial attacks. Furthermore, a Log-Softmax-pattern-based adversarial attack detection method is proposed. This detection method can distinguish clean inputs and multiple adversarial attacks via one multi-classification MLP. In particular, it is state-of-the-art in identifying white-box gradient-based attacks; it achieved at least 95.5% accuracy for classifying four white-box gradient-based attacks with maximum 0.1% false positive ratio.

Centralized machine learning algorithms in vehicular networks face privacy and resource constraints.Federated Learning (FL) addresses these by enabling collaborative model training without sharing raw data. To incentivize vehicle participation despite resource constraints, we propose a contract between Roadside Units (RSUs) and vehicles, specifying required computation resources and rewards based on local data quality. The contract is designed to maximize the utility of the RSU, subject to constraints that ensure the participation of the vehicles in the task and truthful reporting of their data quality parameter. However, designing an optimal contract with classical methods is limited to specific utility functions. To extend the design methodology of optimal contracts to a broader range of utility functions, our study proposes a deep learning-based solution. Specifically, we present a novel application of Deep Neural Networks (DNNs), marking the first instance of applying DNNs in contract theory-based incentive mechanisms to achieve near-optimal contracts. We design the DNN's structure to inherently satisfy specific constraints of the optimization problem, enhancing its efficiency and accuracy in determining the optimal contract. We utilize an augmented Lagrangian method to approximate the optimal contract in federated learning tasks. A case study on traffic sign recognition demonstrates the applicability and effectiveness of our method. Comparing our approach with the well-known generative diffusion model shows that in the FL scenarios, the DNN results are closer to the optimal contract compared to those of the generative diffusion model.

The cloud-only and edge-computing approaches have recently been proposed to satisfy the requirements of complex neural networks. However, the cloud-only approach generates a latency challenge because of the high data volumes that must be sent to a centralized location in the cloud. Less-powerful edge computing resources require a compression model for computation reduction, which degrades the model trading accuracy. To address this challenge, deep neural network (DNN) partitioning has become a recent trend, with DNN models being sliced into head and tail portions executed at the mobile edge devices and cloud server, respectively. We propose Edgepipe, a novel partitioning method based on pipeline inference with an image stream to automatically partition DNN computation between the edge device and cloud server, thereby reducing the global latency and enhancing the system-wide real-time performance. This method adapts to various DNN architectures, hardware platforms, and networks. Here, when evaluated on a suite of five DNN applications, Edgepipe achieves average latency speedups of 1.241× and 1.154× over the cloud-only approach and the state-of-the-art approach known as “Neurosurgeon”, respectively.

The growing complexity and diversity of deep neural network (DNN) applications have inspired intensive research on specialized DNN accelerators and also the design automation frameworks. Previous specialized NN acceleratos roughly fall into two categories of implementation, either the no-pipelined architecture that relies on a generic processing unit (PU) to sequentially execute the DNN layers in a layer-wise way, or the fully-pipelined architecture that dedicates interconnected customized PUs to the corresponding DNN layers in the model. Thus, such designs often suffer from either the resource under-utilization issue faced by no-pipelined accelerators or the resource scalability problem brought by the over-deep pipeline designs. In this work, we propose a novel class of design solution for DNN acceleration, segment-grained pipeline architecture (SPA). In the SPA accelerator, the targeted workload of DNN models will be divided into many segments and each segment will be sequentially executed on the shared interconnected PUs in a pipeline manner, so that they will benefit from both the efficiency of pipelined execution and also the flexibility of sharing PUs across different model layers. Particularly, we found that the efficiency of the implemented SPA accelerator significantly depends on the segmentation strategies of the models and the hardware resources assignment policy for PUs. Therefore, we introduce an automated design framework, AutoSeg, that includes a parameterized SPA accelerator template and a co-design engine that will generate the efficient model segmentation solution and hardware pipeline design parameters for the acceleration workload. Experimental results show that the SPA solutions generated by the AutoSeg framework achieve $1.2\times to 6.3\times$ speedup when compared to ASIC-based general DNN processors, and the FPGA designs implemented by AutoSeg also achieve as high as $3.4\times$ DSP efficiency and $3.6\times$ throughput improvement.