• Home
  • Search
  • Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank
  • Open Access IconOpen Access
  • Cite Icon10
  • https://doi.org/10.1016/j.acha.2023.101595Copy DOI Icon

Gradient descent for deep matrix factorization: Dynamics and implicit bias towards low rank

Show More
  • Abstract
  • Highlights & Summary
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

In deep learning, it is common to use more network parameters than training points. In such scenario of over-parameterization, there are usually multiple networks that achieve zero training error so that the training algorithm induces an implicit bias on the computed solution. In practice, (stochastic) gradient descent tends to prefer solutions which generalize well, which provides a possible explanation of the success of deep learning. In this paper we analyze the dynamics of gradient descent in the simplified setting of linear networks and of an estimation problem. Although we are not in an overparameterized scenario, our analysis nevertheless provides insights into the phenomenon of implicit bias. In fact, we derive a rigorous analysis of the dynamics of vanilla gradient descent, and characterize the dynamical convergence of the spectrum. We are able to accurately locate time intervals where the effective rank of the iterates is close to the effective rank of a low-rank projection of the ground-truth matrix. In practice, those intervals can be used as criteria for early stopping if a certain regularity is desired. We also provide empirical evidence for implicit bias in more general scenarios, such as matrix sensing and random initialization. This suggests that deep learning prefers trajectories whose complexity (measured in terms of effective rank) is monotonically increasing, which we believe is a fundamental concept for the theoretical understanding of deep learning.

Similar Papers
  • PDF
  • Research Article
  • Citations65

Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup* *This article is an updated version of: Goldt S, Advani M S, Saxe A M, Krzakala F and Zdeborova L 2019 Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup Advances in Neural Information Processing Systems pp 6981–91.

  • Dec 01, 2020
  • Journal of Statistical Mechanics (Online)
  • Sebastian Goldt +4
  • Research Article

Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks

  • Nov 18, 2025
  • Machine Learning: Science and Technology
  • Chanju Park +2
  • Research Article

The Module Gradient Descent Algorithm via L2 Regularization for Wavelet Neural Networks

  • Dec 04, 2025
  • Axioms
  • Khidir Shaib Mohamed +8
  • Research Article
  • Citations9

Information Bottleneck and its Applications in Deep Learning

  • Dec 29, 2018
  • حسن حافظ کلاهی +1
  • PDF
  • Research Article

Dual stochastic natural gradient descent

  • Apr 04, 2025
  • Information Geometry
  • Borja Sánchez-López +1
  • Research Article
  • Citations20

Deep convolutional neural network and IoT technology for healthcare.

  • Jan 01, 2024
  • DIGITAL HEALTH
  • Sobia Wassan +6
  • Research Article
  • Citations9

Laziness, barren plateau, and noises in machine learning

  • Mar 01, 2024
  • Machine Learning: Science and Technology
  • Junyu Liu +2
  • Research Article

Fractional-order Gradient Descent for Enhancing Deep Learning Optimization

  • Jul 29, 2025
  • Asian Research Journal of Mathematics
  • Okundalaye Oluwaseun Olumide
  • Preprint Article

Theory and implementation of least-squares-based deep learning

  • Mar 18, 2025
  • Alireza Amiri-Simkooei
  • Conference Article

Toeplitz Covariance Estimation via Overparametrized Gradient Descent

  • Dec 14, 2025
  • Daniel Busbib +1
  • Research Article
  • Citations2

Applying statistical learning theory to deep learning

  • Oct 30, 2024
  • Journal of Statistical Mechanics: Theory and Experiment
  • Cédric Gerbelot +5
  • Single Book
  • Citations2

Deep Learning and Beyond: AI's New Horizons

  • Dec 20, 2024
  • Dr Rajasekar P +4
  • Conference Article

The Evolutionary Deep Learning Model for Electrical Load Forecasting

  • Nov 01, 2020
  • Fei Peng +5
  • Research Article
  • Citations6

Steepest descent algorithm implementation for multichannel blind signal recovery

  • Dec 18, 2012
  • IET Communications
  • H.-D Han +1
  • Research Article
  • Citations13

Implicit Stereotypes in Pre-Trained Classifiers

  • Jan 01, 2021
  • IEEE Access
  • Nassim Dehouche
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.