Online Meta-Learning for Hybrid Model-Based Deep Receivers

With the vigorous development of MOOC open online courses, low completion rate of courses has become an important side effect factor which is affecting the promotion and application of MOOC courses in worldwide. Through analysis of edX open data sets, it is found that loss of learners is one of the factors which are leading to low completion rate of MOOC courses, and also there is a close correlation between learning behavior and final scores with loss of learners. Based on this challenge, some research results show that low completion rate of MOOC courses can be greatly improved by predicting the relationship between learning behavior and scores. This article focuses on application of Deep Neural Network (DNN) in prediction and analysis of MOOC learners' learning behavior. First, data analysis of course performance and learning behavior are discussed, then prediction model based on DNN is given for learning behavior and final scores. Second, experiment is done for DNN model, the results show that DNN can be used for analysis and prediction of complex correlation between learning behavior and final scores, thus which can realize more accurate prediction, reduce loss of learner, and carry out individualized teaching intervention to potential learners.

Online dynamic stability analysis plays a key role in modern power systems and is becoming even more vital with the penetration of Distributed Energy Resources (DERs). Identification of an accurate lower-order model for the ‘external system” makes it possible to carry out the stability analysis of a study system faster than the analysis for the original detailed system. However, the identified lower-order model becomes invalid when the operating state changes in the power system. ‘Black box’ representation of the ‘external system’ with the application of Deep Neural Networks (DNN) is widely discussed in the literature. In such equivalent models, the DNN response changes based on two aspects i.e. the operating point of the power system and the location/type of the fault. In this paper, a clustering algorithm is proposed to identify the clusters of the ‘external system’ models for different states of the power system, and a DNN is trained for each cluster. The clustering algorithm is based on the pre-fault steady-state power flow from the ‘external system’ to the ‘study system’, which is a reflection of changes in the system state. Mean Absolute Error (MAE) between responses of the actual fault and the predicted fault is used to evaluate the accuracy of the identified system. The proposed technique is easy to implement and requires a lesser number of DNN models compared to identifying an individual DNN model for each operating point to represent the ‘external system’.

Sedimentary basins, integral to Earth's geological history and energy resource exploration, undergo complex changes driven by sedimentation, subsidence and geological processes. Gravity anomaly inversion is a crucial technique offering insights into subsurface structures and density variations. Our study addresses the challenge of complex subsurface structure assessment by leveraging deep neural networks to invert observed gravity anomalies. Optimization approaches traditionally incorporate known density distributions obtained from borehole data or geological logging for inverting basement depth in sedimentary basins using observed gravity anomalies. Our study explores the application of deep neural networks in accurate architectural assessment of sedimentary basins and demonstrates their significance in mineral and hydrocarbon exploration. Recent years have witnessed a surge in the use of machine learning in geophysics, with deep learning models playing a pivotal role. Integrating deep neural networks, such as the feedforward neural networks, has revolutionized subsurface density distribution and basement depth estimation. This study introduces a deep neural network specifically tailored for inverting observed gravity anomalies to estimate two‐dimensional basement relief topographies in sedimentary basins. To enhance computational efficiency, a one‐dimensional discrete cosine transform based discretization approach is employed. Synthetic data, generated using non‐Gaussian fractals, compensates for the scarcity of true datasets for training the deep neural network model. The algorithm's robustness is validated through noise introduction with comparisons against an efficient and traditional global optimization‐based approach. Gravity anomalies of real sedimentary basins further validate the algorithm's efficacy, establishing it as a promising methodology for accurate and efficient subsurface imaging in geological exploration.

Recently, data-driven approaches motivated by modern deep learning have been applied to optical communications in place of traditional model-based counterparts. The application of deep neural networks (DNN) allows flexible statistical analysis of complicated fiber-optic systems without relying on any specific physical models. Due to the inherent nonlinearity in DNN, various equalizers based on DNN have shown significant potential to mitigate fiber nonlinearity. In this paper, we propose turbo equalization (TEQ) based on DNN as a new alternative framework to deal with nonlinear fiber impairments. The proposed DNN-TEQ is constructed with nested deep residual networks (ResNet) to train extrinsic likelihood given soft-information feedback from channel decoding. Through extrinsic information transfer (EXIT) analysis, we verify that our DNN-TEQ can accelerate decoding convergence to achieve a significant gain in achievable throughput by 0.61 b/s/Hz. We also demonstrate that optimizing irregular low-density parity-check (LDPC) codes based on the EXIT chart of the DNN-TEQ can improve achievable rates by up to 0.12 b/s/Hz.

The rapid advancements and widespread application of deep neural networks (DNNs), coupled with their reliance on sensitive and private data, have sparked growing concerns regarding data privacy and the ''right to be forgotten''. To address these concerns, machine unlearning has been proposed to efficiently eliminate the influence of specific training data from trained DNNs. However, existing machine unlearning methods struggle with the large number of parameters in trained DNNs, which lead to slow execution and high memory consumption, making them impractical for large-scale models. In this paper, we shift our focus to the small set of weights in the final classification layer of DNNs, which are defined as as ''prototypes'' for different classes. Our key observation is that the prototype associated with the unlearned training data undergoes a significant shift, whereas prototypes of unrelated classes exhibit only minor changes when comparing the prototypes of original and retrained models. Based on this observation, we propose a novel machine unlearning approach that efficiently achieves machine unlearning by directly adjusting the prototypes of DNNs. We first introduce Naive Prototype Surgery (Naive PS), a fast and simplified method that uses a closed-form solution to approximate unlearning effect by directly adjusting the prototype associated with the unlearned data. Next, we propose Prototype Surgery (PS), which incorporates soft label information to fine-tune the prototypes of all classes, to achieve a more effective unlearning. Both methods achieve data unlearning by only modifying the prototypes in the DNNs, thus avoiding the challenges posed by the large number of model parameters. Extensive experiments on four datasets demonstrate that our methods significantly accelerate the unlearning process while achieving comparable results to five existing methods in terms of both unlearning performance and privacy guarantee.

The objective of this paper is 2-fold. First, it develops a prediction system to help the credit card issuer model the credit card delinquency risk. Second, it seeks to explore the potential of deep learning (also called deep neural network), an emerging artificial intelligence technology, in credit risk domain. With a real-life credit card data linked to 711,397 credit card holders from a large bank in Brazil, this study develops a deep neural network to evaluate the risk of credit card delinquency based on the client’s personal characteristics and the spending behaviors. Compared to machine learning algorithms of logistic regression, naive Bayes, traditional artificial neural network, and decision tree, deep neural network has a better overall predictive performance with the highest F scores and AUC. The successful application of deep learning implies that artificial intelligence has great potential to support and automate credit risk assessment for financial institutions and credit bureaus.

Small data sets make developing calibration models using deep neural networks difficult because it is easy to overfit the system. We developed two deep neural network architectures by revising two existing network architectures: the U-Net and the attention mechanism. The major changes were to use 1D convolutional layers to replace the fully connected layers. We also designed and combined average pooling and maximum pooling in our revised networks, respectively. We applied these revised network architectures to three publicly available data sets and the resulting calibration models can generate acceptable results for general quantitative analysis. It also generated rather good results for data sets that concern calibration transfer. It demonstrates that constructing network architectures by properly revising existing successful network architectures may provide additional choices in the exploration of the application of deep neural network in analytical chemistry.

Information security has emerged as a critical concern alongside the development of multimedia technology. Among the myriad security challenges, the secure transmission of sensitive information between parties has become a focal point of researchers. Encryption, involving mathematical techniques to ensure data security, is explored in this study. Specifically, the application of deep LSTM neural networks in concealing textual information within digital images is investigated. The approach involves embedding one image within another in a manner that prevents detection of the hidden image within the cover image, while textual content is covertly embedded within the image. The proposed method demonstrates superior performance based on three evaluation metrics—Peak Signal-to-Noise Ratio (PSNR) in decibels, Mean Squared Error (MSE), and accuracy rate in percentage—compared to three other benchmark images (lena.png, peppers.png, mandril.png, and monkey.png), achieving values of 93.665275 dB, 0.6945 MSE, and 97.23% accuracy, respectively.

The problem of application of deep neural networks (DNNs) for extremely small dataset is considered to find the effect of dataset size and quality on performance metrics. Classification task is considered for the set of problems where images of Lyme disease symptoms were collected and structured in the open-access dataset with the different subsets of the whole dataset: clean images only (C), dirty images only (D), and clean and dirty images in the equal proportions (C_D) (https://www.kaggle.com/yoctoman/lyme-clean-and-dirty) . Two different data preprocessing techniques were applied: normalization without data augmentation (NODA) and normalization with data augmentation (DA). The two versions of the modern NASNet DNN architectures of different sizes were used, namely NASNetLarge and NASNetMobile, with the weights obtained after training on ImageNet dataset. These DNNs were trained in two experiments, namely with frozen layers, aka transfer learning (TL), and with unfrozen layers when all weights were updated (NOTL). After fivefold cross-validated training the best results for mean validation metrics were demonstrated by the models trained in C_NOTL_NODA, C_NOTL_DA, and C_TL_NODA experiments for NASNetLarge, and in C_TL_NODA by NASNetMobile. But the testing on the new unknown subset of Lyme disease images with the equal proportion of clean and dirty images gave quite different results. The models trained in the before-mentioned experiments demonstrated poor performance in comparison with the models trained in C_NOTL_DA, D_NOTL_DA experiments, and, especially in D_C_NOTL_DA ones. The successful training strategy should include tuning ImageNet-trained weights without their freezing (NOTL) and intensive data augmentation (DA), but it would not be enough if some diverse relevant background will not be added. That is why the additional methods of “context-relevant data augmentation” (CRDA) mimicked by inclusion of D_C data configuration should be considered. These findings can be used for the more effective selection of training experiments of DNNs in the healthcare context in the view of the small number of available medical data in practice.KeywordsNeural networksDeep learningSmall dataset problemData augmentationTransfer learningHealth careLyme disease

Centralized machine learning algorithms in vehicular networks face privacy and resource constraints.Federated Learning (FL) addresses these by enabling collaborative model training without sharing raw data. To incentivize vehicle participation despite resource constraints, we propose a contract between Roadside Units (RSUs) and vehicles, specifying required computation resources and rewards based on local data quality. The contract is designed to maximize the utility of the RSU, subject to constraints that ensure the participation of the vehicles in the task and truthful reporting of their data quality parameter. However, designing an optimal contract with classical methods is limited to specific utility functions. To extend the design methodology of optimal contracts to a broader range of utility functions, our study proposes a deep learning-based solution. Specifically, we present a novel application of Deep Neural Networks (DNNs), marking the first instance of applying DNNs in contract theory-based incentive mechanisms to achieve near-optimal contracts. We design the DNN's structure to inherently satisfy specific constraints of the optimization problem, enhancing its efficiency and accuracy in determining the optimal contract. We utilize an augmented Lagrangian method to approximate the optimal contract in federated learning tasks. A case study on traffic sign recognition demonstrates the applicability and effectiveness of our method. Comparing our approach with the well-known generative diffusion model shows that in the FL scenarios, the DNN results are closer to the optimal contract compared to those of the generative diffusion model.

As applications using deep neural networks (DNNs) are increasingly deployed on mobile devices, researchers are exploring various methods to achieve low energy consumption and high performance. Recently, advances in STT-MRAM have shown promise in offering non-volatility, high performance, and low energy consumption when used to replace DRAM in main memory. Most of the memory space used in DNN applications is occupied by weight and activation data. Typically, the contents of weight data remain unchanged during inference, whereas activation data are modified to store intermediate results across the layers of the DNN. As large DNN applications consume significant energy and require high performance, STT-MRAM is an ideal candidate to fully or partially replace DRAM in main memory. However, the long write latency of STT-MRAM compared to DRAM presents a performance bottleneck. In this work, we propose a reduced retention-time MRAM-based main memory to address this issue. We divide the MRAM into multiple partitions, with each partition implemented with different retention times, tailored for DNN applications. In this approach, the DNN weights can be assigned to the long retention-time partition, while the activation data can be allocated to the short retention-time partition, optimizing for the varying characteristics of data reuse. To achieve high performance, we propose two mapping schemes: intra-segment and inter-segment circular buffers, which dynamically map DNN activation data (i.e., virtual pages of streaming data) to physical pages in a circular fashion to exploit reuse patterns of DNN data. These circular buffers are mapped to the short retention-time MRAM partition as much as possible. The intra-circular buffer mapping scheme achieves an average improvement of 14.4% in bandwidth and 12.6% in inference latency compared to DRAM for on-device DNN applications. Furthermore, the inter-circular buffer mapping scheme offers 11.1% bandwidth and 11.2% latency improvements on average using only 16 MBytes of the short retention-time partition.

In recent years, there is remarkable growing concern for marketing team to retain their customers. This can be achieved by predicting accurately ahead of time, whether a terminal for buying is valuable in the foreseeable future or not. This paper presents the application of Deep Neural Network in the issue of classifying the payment terminals in different branches of Parsian bank specifically. The paper uses real data for classifying various payment terminals in 6 classes of terminal by a 5 layer deep neural network and RFM model. The empirical results reveal that utilizing the deep network generate significantly better accuracy in comparison with other popular methods.

The International Classification of Diseases (ICD) is a standard for categorizing physical conditions, which has been widely used for analyzing clinical data and monitoring health issues. Manual ICD coding takes a long time and is vulnerable to errors, so people pay more and more attention to the application of deep neural networks in ICD automatic coding. However, there is still no comprehensive review of these studies and prospects for further research. This paper is not limited to the study of deep neural networks, but gives a formal definition of ICD coding problems, and then systematically reviews the existing literature on how to design deep neural networks to address the four major challenges of ICD coding tasks. This paper also summarizes the public data sets and future research directions, to provide a guidance for the research of ICD coding in medical field.

As data visibility in factories has increased with the deployment of sensors, data-driven maintenance has become popular in industries. Machine learning has been a promising tool for fault detection, but the problem is that the amount of fault data is much less than that of normal data which causes a data imbalance. In this study, we designed a deep neural network for fault detection and diagnosis, and compared the oversampling by a generative adversarial network to standard oversampling techniques. Simulation results indicate that oversampling by the generative adversarial network performs well under the given condition and the deep neural network designed is capable of classifying the faults of an induction motor with high accuracy.

Authorship Identification as a task has been long studied and explored. Historically, authorship claims were ratified for copyright issues after the death of the author for unpublished work through style matching. The immense growth in the reach of internet technologies has once again brought to the fore the importance of authorship identification. An application opening up in areas like Intellectual Property Right settlement, Copyrights, Plagiarism, Cyber Crime and Forensics, authorship identification is now an area of active research. The current work presents a Deep Neural Network based approach to authorship identification from a large corpus. The experiments carried out bring out the applicability of Deep Neural Networks for the task and also highlights the importance of hyperparameter tuning for the purpose. Results show that a proper choice and balance in the hyperparameter setting can improve already established outcomes.