Abstract
The optimization of deep learning models is traditionally interpreted as a discrete iterative algorithm such as SGD, but this view arguably clouds the continuous-time dynamics which govern training behavior. In the recent years, as a phenomenological framework of modelling and analysing stochastic optimization of neural network parameters, we have seen an increasing use of stochastic differential equations (SDEs) [65]. This paper studies the dynamics of deep learning training viewed as stochastic differential equations (SDEs), and provides new theoretical insights into convolution convergence, generalization, and robustness. We start with the correspondence between discrete optimization algorithms and continuous-time stochastic processes, showing that SGD can be approximated as an SDE featuring both drift and diffusion components. The drift term encodes the deterministic gradient flow while he diffusion term models the stochastic noise resulting from mini-batch sampling. This expression provides insight into the way in which randomness determines the evolutionary path of optimization across high-dimensional non-convex loss landscapes. In addition, finding that anisotropic noise interacts with the geometry of the loss surface, we analyse how structured stochasticity directs the model towards flat minima which optimally promote generalization. The work will also focus on more advanced SDE-based frameworks, such as Lange in dynamics and the Fokker–Planck equation, which can shed light on how probability distributions over model parameters evolve. Such approaches yield a probabilistic perspective on training, connecting deep learning optimization with Bayesian inference and statistical physics. In this way, SDE modelling provides a principled explanation for the implicit regularization and noise-induced robustness observed in contemporary neural networks.
Next, we study the contribution of adaptive optimization methods, continuous-depth architectures and multi-scale stochastic effects in our SDE framework. Through the results of both empirical simulations and theoretical analysis, we confirm that SDE-based modelling can effectively predict training behavior, enhance robustness, and inform more stable learning algorithm designs. In summary, this work provides a unifying framework for understanding deep learning dynamics through stochastic differential equations, which connects discrete optimization and continuous stochastic processes, taking us one step closer to building more principled and robust AI systems.
Keywords
Stochastic Differential Equations, Deep Learning Optimization, Stochastic Gradient Descent, Lange in Dynamics, Fokker–Planck Equation, Training Dynamics, Noise-Induced Regularization, Loss Landscape, Bayesian Deep Learning Continuous-Time Models
References
[1] Y. Lacuna, Y. Bagnio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
[2] I. Good fellow, Y. Bagnio, and A. Carville, Deep Learning. MIT Press, 2016.
[3] S. Ruder, “An overview of gradient descent optimization algorithms,” arrive preprint arXiv:1609.04747, 2016.
[4] L. Bottom, F. E. Curtis, and J. Necedah, “Optimization methods for large-scale machine learning,” SIAM Review, vol. 60, no. 2, pp. 223–311, 2018.
[5] H. Risen, The Fokker–Planck Equation: Methods of Solution and Applications. Springer, 1996.
[6] B. Kendal, Stochastic Differential Equations: An Introduction with Applications. Springer, 2003.
[7] D. J. Hingham, “An algorithmic introduction to numerical simulation of stochastic differential equations,” SIAM Review, vol. 43, no. 3, pp. 525–546, 2001.
[8] M. Welling and Y. W. The, “Bayesian learning via stochastic gradient Lange in dynamics,” in Proc. ICML, 2011.
[9] M. Mandy, M. D. Hoffman, and D. M. Bali, “Stochastic gradient descent as approximate Bayesian inference,” J. Mach. Learn. Res., vol. 18, no. 1, pp. 4873–4907, 2017.
[10] X. Li, T. Chen, H. Girolami, and J. A. Turner, “Stochastic gradient Hamiltonian Monte Carlo,” in Proc. ICML, 2014.
[11] S. Goal et al., “Accurate, large minibatch SGD: Training Image Net in 1 hour,” arXiv:1706.02677, 2017.
[12] S. Chaudhari and S. Sotto, “Stochastic gradient descent performs variation inference,” arXiv:1710.11029, 2017.
[13] J. Sirignano and K. Iliopoulos, “Mean field analysis of neural networks,” SIAM J. Appl. Math., vol. 80, no. 2, pp. 725–752, 2020.
[14] J. Weaning E, “A proposal on machine learning via dynamical systems,” Communications in Mathematics and Statistics, vol. 5, pp. 1–11, 2017.
[15] R. Ten and M. Raginsky, “Theoretical guarantees for sampling and inference in generative models with latent diffusions,” COLT, 2019.
[16] Y. Li, T. Ma, and H. Zhang, “Learning over parameterized neural networks via stochastic gradient descent,” arXiv:1807.00263, 2018.
[17] N. S. Cesar et al., “On large-batch training for deep learning: Generalization gap and sharp minima,” ICLR, 2017.
[18] S. Hoch Reiter and J. Schmidhuber, “Flat minima,” Neural Computation, vol. 9, no. 1, pp. 1–42, 1997.
[19] P. Chaudhari et al., “Entropy-SGD: Biasing gradient descent into wide valleys,” ICLR, 2017.
[20] J. Zhang et al., “Understanding deep learning requires rethinking generalization,” ICLR, 2017.
[21] D. P. Kingman and J. Ba, “Adam: A method for stochastic optimization,” ICLR, 2015.
[22] T. Tie leman and G. Hinton, “RMSProp: Divide the gradient by a running average,” Neural Networks for Machine Learning, 2012.
[23] A. Krizhevsky, I. Sutskever, and G. Hinton, “Image Net classification with deep convolutional neural networks,” NIPS, 2012.
[24] A. Aswan et al., “Attention is all you need,” NIPS, 2017.
[25] R. Neal, Bayesian Learning for Neural Networks. Springer, 1996.
[26] C. Gardiner, Handbook of Stochastic Methods. Springer, 2004.
[27] T. Chen et al., “Neural ordinary differential equations,” Neutrals, 2018.
[28] J. Song and S. Sermon, “Generative modelling by estimating gradients of the data distribution,” Neutrals, 2019.
[29] Y. Song et al., “Score-based generative modelling through stochastic differential equations,” ICLR, 2021.
[30] M. Raginsky, A. Sakhalin, and M. Telgarsky, “Non-convex learning via stochastic gradient Lange in dynamics,” COLT, 2017.
[31] L. Sagan, L. Bottom, and Y. Lacuna, “Eigenvalues of the Hessian in deep learning,” arXiv:1611.07476, 2016.
[32] A. Yastrzemski et al., “Three factors influencing minima in SGD,” ICLR Workshop, 2018.
[33] K. Kawaguchi, “Deep learning without poor local minima,” Neutrals, 2016.
[34] S. Malady et al., “On the diffusion approximation of SGD,” arXiv:2006.05987, 2020.
[35] A. S. Sznitman, “Topics in propagation of chaos,” Springer Lecture Notes in Mathematics, 1991.