Generalizing Projected Gradient Descent for Deep-Learning-Aided Massive MIMO Detection

The paper presents a deep learning-aided iterative detection algorithm for massive overloaded MIMO systems. Since the proposed algorithm is based on the projected gradient descent method with trainable parameters, it is named as trainable projected descent-detector (TPG-detector). The trainable internal parameters can be optimized with standard deep learning techniques such as back propagation and stochastic gradient descent algorithms. This approach referred to as data-driven tuning brings notable advantages of the proposed scheme such as fast convergence. The numerical experiments show that TPG-detector achieves comparable detection performance to those of the known algorithms for massive overloaded MIMO channels with lower computation cost.

This study discusses the negative impact of the derivative of the activation functions in the output layer of artificial neural networks, in particular in continual learning. We propose Hebbian descent as a theoretical framework to overcome this limitation, which is implemented through an alternative loss function for gradient descent we refer to as Hebbian descent loss. This loss is effectively the generalized log-likelihood loss and corresponds to an alternative weight update rule for the output layer wherein the derivative of the activation function is disregarded. We show how this update avoids vanishing error signals during backpropagation in saturated regions of the activation functions, which is particularly helpful in training shallow neural networks and deep neural networks where saturating activation functions are only used in the output layer. In combination with centering, Hebbian descent leads to better continual learning capabilities. It provides a unifying perspective on Hebbian learning, gradient descent, and generalized linear models, for all of which we discuss the advantages and disadvantages. Given activation functions with strictly positive derivative (as often the case in practice), Hebbian descent inherits the convergence properties of regular gradient descent. While established pairings of loss and output layer activation function (e.g., mean squared error with linear or cross-entropy with sigmoid/softmax) are subsumed by Hebbian descent, we provide general insights for designing arbitrary loss activation function combinations that benefit from Hebbian descent. For shallow networks, we show that Hebbian descent outperforms Hebbian learning, has a performance similar to regular gradient descent, and has a much better performance than all other tested update rules in continual learning. In combination with centering, Hebbian descent implements a forgetting mechanism that prevents catastrophic interference notably better than the other tested update rules. When training deep neural networks, our experimental results suggest that Hebbian descent has better or similar performance as gradient descent.

ZeroGrad: Costless conscious remedies for catastrophic overfitting in the FGSM adversarial training

The numerical solution of a ordinary differential equation (ODE) basically describes any physical phenomenon of interest. Traditionally, the numerical solution of ODE is solved by a discretization process using the classical finite difference approximation. Recently, many studies have used deep neural networks (DNN) to approximate ODE solutions without discretization. However, without the optimization process, DNN cannot run independently. In this work, we proposed an alternative method that combines DNN with a genetic algorithm (GA) as an optimization method. GA initiates a population of DNN trainable parameters, making the DNN training process more streamlined on a single-generation GA. We focused on DNN-GA hybrid solution for linear and nonlinear ordinary differential equations (ODE‘s). Our method is achieved by solving several test problems and demonstrating a comparison with stochastic gradient descent (SGD) optimizers. In most ODE problems, using the same number of iterations and architecture, DNN-GA produces a better loss function than using DNN-SGD.

The design of a security scheme for beamforming prediction is critical for next-generation wireless networks (5G, 6G, and beyond). However, there is no consensus about protecting beamforming prediction using deep learning algorithms in these networks. This paper presents the security vulnerabilities in deep learning for beamforming prediction using deep neural networks in 6G wireless networks, which treats the beamforming prediction as a multi-output regression problem. It is indicated that the initial DNN model is vulnerable to adversarial attacks, such as Fast Gradient Sign Method , Basic Iterative Method , Projected Gradient Descent , and Momentum Iterative Method , because the initial DNN model is sensitive to the perturbations of the adversarial samples of the training data. This study offers two mitigation methods, such as adversarial training and defensive distillation, for adversarial attacks against artificial intelligence-based models used in the millimeter-wave (mmWave) beamforming prediction. Furthermore, the proposed scheme can be used in situations where the data are corrupted due to the adversarial examples in the training data. Experimental results show that the proposed methods defend the DNN models against adversarial attacks in next-generation wireless networks.

Time series anomaly detection is extensively studied in statistics, economics, and computer science. Over the years, numerous methods have been proposed for time series anomaly detection using deep learning-based methods. Many of these methods demonstrate state-of-the-art performance on benchmark datasets, giving the false impression that these systems are robust and deployable in many practical and industrial real-world scenarios. In this paper, we demonstrate that the performance of state-of-the-art anomaly detection methods is degraded substantially by adding only small adversarial perturbations to the sensor data. We use different scoring metrics such as prediction errors, anomaly, and classification scores over several public and private datasets ranging from aerospace applications, server machines, to cyber-physical systems in power plants. Under well-known adversarial attacks from Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) methods, we demonstrate that state-of-the-art deep neural networks (DNNs) and graph neural networks (GNNs) methods, which claim to be robust against anomalies and have been possibly integrated in real-life systems, have their performance drop to as low as 0%. To the best of our understanding, we demonstrate, for the first time, the vulnerabilities of anomaly detection systems against adversarial attacks. The overarching goal of this research is to raise awareness towards the adversarial vulnerabilities of time series anomaly detectors.

Models built with deep neural network (DNN) can handle complicated real-world data extremely well, seemingly without suffering from the curse of dimensionality or the non-convex optimization. To contribute to the theoretical understanding of deep learning, this work studies the nonparametric perspective of DNNs by considering the following questions: (1) What is the underlying estimation problem and what are the most appropriate data assumptions? (2) What is the corresponding optimal convergence rate and does the curse of dimensionality occur? (3) Is the optimal rate achievable for DNN estimators and is there any optimization guarantee? These questions are investigated on two of the most fundamental problems --- regression and classification. Specifically, statistical optimality of DNN estimators is established under various settings with special focuses on the curse of dimensionality and optimization guarantee.In the classic binary classification problem, statistical optimal convergence rates that suffer less from the curse of dimensionality are established under two settings:(1) Under the smooth boundary assumption, I show that DNN classifiers with proper architectures can benefit from the compositional smoothness structure underlying the high dimensional data in the sense that the optimal convergence rates only depend on some effective dimension d*, potentially much smaller than the data dimension d. (2) Under a novel teacher-student framework that assumes the Bayes classifier to be expressed as ReLU neural networks, I obtain a dimension-free rate of convergence O(n^{-2/3}) for DNN classifiers, which is also proven optimal. The optimization of DNN is highly complicated with generally no algorithmic guarantee on finding the global minimizer. To this end, I turn to the recently proposed neural tangent kernel (NTK) literature where the similarity between overparametrized DNN trained by gradient descent (GD) and kernel methods are established. Specifically, through a comprehensive analysis of L2-regularized GD trajectories, I prove that for overparametrized one-hidden-layer ReLU neural networks with L2 regularization, the output from GD is close to that from the kernel ridge regression with the corresponding NTK and optimal rate of L2 estimation error can be achieved.

Nowadays, the influence of machine learning algorithms on our lives is booming gradually, and gradient descent (GD) method is a typical big data analysis method that can be applied to various machine learning algorithms. Traditional GD method has its advantages in dealing with the optimal parameter finding issue for partial loss functions, but the optimal parameter finding process may be trapped into local minimum sometimes if the loss function with certain convexity; moreover, traditional GD cannot meet the requirement of high accuracy. Fractional order GD, which is the core of fixed memory step GD, has better performance by comparing with traditional GD, but still have some problems in the aspect of convergence. On the other hand, t-distributed machine learning algorithm for dimensionality reduction, can reduce the dimensionality of data in high-dimensional space to two-dimensional or three-dimensional space, and can convert the original data into more comprehensible images. In order to solve the problem mentioned above reasonably, in this paper, a fixed-memory step GD method combined with t-distributed stochastic neighbor embedding (t-SNE) is employed to replace the traditional GD method, so as to keep the consistency of low-dimensional data being mapped from high-dimensional data, and to achieve efficient data visualization. The effectiveness of proposed t-SNE algorithm based on fixed memory step gradient descent method has been verified finally via numerical experiment.

This study explores the application of gradient descent optimization and loss functions for depression prediction through multidimensional analysis of behavioral, psychological, and social factors. A logistic regression model was trained on a set of 1,631 records. Experimental results indicate an overall accuracy of 91.17 %, a weighted precision of 0.92, a recall of 0.91, and an F1 score of 0.91. These performances demonstrate the robustness of the model and its potential for early detection of depression. This research highlights the role of artificial intelligence in decision support and prevention in mental health. This article explores in depth the gradient descent algorithm, a fundamental pillar of machine learning, focusing on its essential interaction with the loss function (or cost function). Far from a simple theoretical analysis, this talk applies these fundamental concepts to a crucial public health problem: the prediction of depression. Depression is a complex disorder, and its diagnosis and early prediction remain major challenges. To develop reliable predictive models of depression, it is imperative to identify and integrate a diverse set of relevant factors. We will rely on a set of 24 input variables, grouped into 8 symptomatic and contextual categories (such as concentration difficulties, feelings of guilt, sleep disturbances, intense fatigue, etc.), to build our predictive model. The challenge is to navigate the space of these multidimensional parameters in order to find the optimal combination minimizing the prediction error of depression. This paper details how gradient descent, guided by a carefully chosen loss function, addresses this prediction challenge: Modeling of input variables, Application and optimization of gradient descent and Convergence and performance analysis. By integrating these specific variables into a rigorous gradient descent optimization approach, this paper will seek to demonstrate the feasibility and effectiveness of a machine learning model for more accurate and meaningful prediction of depression.

Projected gradient descent method for cardinality-constrained portfolio optimization

Huber Loss function is a regression type loss function, mainly used for optimization problems.In order to make the predicted value close to the true value and minimize the distance between them, the traditional loss function generally uses the gradient descent method to find the minimum value. However, the gradient descent method takes too long and consumes a lot of memory. It may not necessarily find the global optimal value, and it is easy to fall into the problem of local optimization. The data of the unconstrained function is complicated and huge, and the Huber Loss function sets the hyper-parameter δ to optimize MSE and MAE, we must iterate continuously to train the hyper-parameters to obtain more accurate parameter values, which will cause a problem of large amount of calculations. The beetle antennae search algorithm is a simple biological intelligence optimization algorithm that can provide the best solution based on huge data, with fast convergence speed, small amount of calculation, simple operation, and no need to know the specific function. In order to improve the accuracy of finding the global optimal value, this paper changed the traditional fixed-step antennae search algorithm to variable-step beetle antennae search algorithm. This article will discuss the application of the gradient descent method in the regression loss function, and then use the variable step size beetle antennae search algorithm to optimize Huber Loss function.

We present a novel hybrid algorithm for training Deep Neural Networks that combines the state-of-the-art Gradient Descent (GD) method with a Mixed Integer Linear Programming (MILP) solver, outperforming GD and variants in terms of accuracy, as well as resource and data efficiency for both regression and classification tasks. Our GD+Solver hybrid algorithm, called GDSolver, works as follows: given a DNN D as input, GDSolver invokes GD to partially train D until it gets stuck in a local minima, at which point GDSolver invokes an MILP solver to exhaustively search a region of the loss landscape around the weight assignments of D’s final layer parameters with the goal of tunnelling through and escaping the local minima. The process is repeated until desired accuracy is achieved. In our experiments, we find that GDSolver not only scales well to additional data and very large model sizes, but also outperforms all other competing methods in terms of rates of convergence and data efficiency. For regression tasks, GDSolver produced models that, on average, had 31.5% lower MSE in 48% less time, and for classification tasks on MNIST and CIFAR10, GDSolver was able to achieve the highest accuracy over all competing methods, using only 50% of the training data that GD baselines required.

The successful emergence of deep learning (DL) in wireless system applications has raised concerns about new security-related challenges. One such security challenge is adversarial attacks. Although there has been much work demonstrating the susceptibility of DL-based classification tasks to adversarial attacks, regression-based problems in the context of a wireless system have not been studied so far from an attack perspective. The aim of this paper is twofold: (i) we consider a regression problem in a wireless setting and show that adversarial attacks can break the DL-based approach and (ii) we analyze the effectiveness of adversarial training as a defensive technique in adversarial settings and show that the robustness of DL-based wireless system against attacks improves significantly. Specifically, the wireless application considered in this paper is the DL-based power allocation in the downlink of a multicell massive multi-input-multi-output system, where the goal of the attack is to yield an infeasible solution by the DL model. We extend the gradient-based adversarial attacks: fast gradient sign method (FGSM), momentum iterative FGSM, and projected gradient descent method to analyze the susceptibility of the considered wireless application with and without adversarial training. We analyze the deep neural network (DNN) models performance against these attacks, where the adversarial perturbations are crafted using both the white-box and black-box attacks.

Where is my attention? An explainable AI exploration in water detection from SAR imagery

The zero-forcing (ZF) and minimum mean square error (MMSE) based detectors can approach optimal performance in the uplink of massive multiple-input multiple-output (MIMO) systems. However, they require inverting a matrix whose complexity is cubic in relation to the matrix dimension. This can lead to the high computational effort, especially in massive MIMO systems. To mitigate this, several iterative methods have been proposed in the literature. In this paper, we consider accelerated Chebyshev SOR (AC-SOR) and accelerated Chebyshev AOR (AC-AOR) algorithms, which improve the detection performance of conventional Successive Over-Relaxation (SOR) and Accelerated Over-Relaxation (AOR) methods, respectively. Additionally, we propose using a deep unfolding network (DUN) to optimize the parameters of the iterative AC-SOR and AC-AOR algorithms, leading to the AC-AORNet and AC-SORNet methods, respectively. The proposed DUN-based method leads to significant performance improvements compared to conventional iterative detectors for various massive MIMO channels. The results demonstrate that the AC-AORNet and AC-SORNet are effective, outperforming other state-of-the-art algorithms. Furthermore, they are highly effective, particularly for high-order modulations such as 256-QAM (Quadrature Amplitude Modulation). Moreover, the proposed AC-AORNet and AC-SORNet require almost the same number of computations as AC-AOR and AC-SOR methods, respectively, since the use of deep unfolding has a negligible impact on the system’s detection complexity. Furthermore, the proposed DUN features a fast and stable training scheme due to its smaller number of trainable parameters.