01
Optimal Transport-Based Learning for Distribution Alignment in High-Dimensional Spaces
Mohamed Ashraf, Mohamed Haji Ali, Abdhul Haider · Pages 1–14
This paper studies distributional alignment in high-dimensional spaces through the use of Optimal Transport-based learning approaches, as discussed in [6]. In contrast to standard divergence measures, like Fullback–Libeler divergence or Jensen–Shannon divergence, Optimal Transport yields a distance metric that takes into consideration the underlying geometry of data and is particularly well suited for complex structured datasets. In this paper, we give a thorough overview of the theoretical background underlying Optimal Transport, covering Monger versus Kantorovich formulation, Wasserstein distances and entropy-regularized transport theory.In addition, this paper explores the combination between optimal transport and modern machine learning approaches like deep neural networks, domain adaptation models, generative models.
Keywords: Wasserstein Distance, Domain Adaptation, Machine Learning, Deep Learning, High-Dimensional Data, Stinkhorn Algorithm
02
Entropy-Regularized Optimization for Robust Deep Learning Models
Suman Bansal, Sameer Joshi, Ashutosh Agarwal · Pages 15–31
Entropy-regularized optimization is a promising framework for improving robustness, stability and generalization of deep learning models in high-dimensional uncertainty settings. Standard optimization techniques for deep NNs are often based on empirical risk minimization, thereby resulting in overfitting, miscalibrated probabilities and sensitivity to adversarial perturbations. On the other hand, entropy-based regularization develops a principled way of controlling the distributional behaviour of model predictions by promoting smoothness in probability landscapes and discouraging overconfidence.
Keywords: Entropy regularization, Adversarial robustness, Robust optimization, Information geometry, Distributional stability, Deep neural networks, Optimal transport, Uncertainty calibration (UQ), Stochastic Optimization(summary)
03
Stochastic Differential Equation Modelling of Deep Learning Training Dynamics
Humayun Kabir, Abdullah al Mamun, Ruhul Quddus · Pages 32-50
The optimization of deep learning models is traditionally interpreted as a discrete iterative algorithm such as SGD, but this view arguably clouds the continuous-time dynamics which govern training behavior. In the recent years, as a phenomenological framework of modelling and analysing stochastic optimization of neural network parameters, we have seen an increasing use of stochastic differential equations (SDEs) [65]. This paper studies the dynamics of deep learning training viewed as stochastic differential equations (SDEs), and provides new theoretical insights into convolution convergence, generalization, and robustness.
Keywords: Stochastic Differential Equations, Deep Learning Optimization, Stochastic Gradient Descent, Lange in Dynamics, Fokker–Planck Equation, Training Dynamics, Noise-Induced Regularization, Loss Landscape, Bayesian Deep Learning Continuous-Time Models
04
Hyperbolic Embedding Methods for Hierarchical Data Representation
Ali Mejbel Fartoos, Abdullah Hussein Jebara, Youssef Amyn, Farhad Pirbal · Pages 51-66
Hyperbolic embedding methods are defined as state-of-the-art approaches in machine learning and data science for effectively modelling hierarchical structures. Traditional placement methods, rooted in Euclidean geometry, fall short when it comes to expressing hierarchical relations where exponential growth is inevitable. Existing approaches for hierarchical data modelling, such as taxonomies, knowledge graphs, phylogenetic trees and social networks, are often better modelled in non-Euclidean spaces due to their structural properties. Hyperbolic Geometry is a mathematically sound embedding space that is especially well-suited for minimally distorting high dimensional data due to its inherent negative curvature.
Keywords: Hyperbolic Embedding, Hierarchical Data Representation, Poincare Ball Model, Hyperboloid Model, Riemannian Optimization, Graph Neural Networks, Non-Euclidean Geometry; A Semantic Sense Check On Background Reading
05
Invariant Risk Minimization for Out-of-Distribution Generalization
Bilal Emrah1, Baytr Rustem2, · Pages 67-83
Invariant Risk Minimization (IRM) is an emergent framework for tackling what is arguably the biggest, and most important, problem in machine learning today — how to generalize out of sample. State of the art learning paradigms, especially Empirical Risk Minimisation (ERM), rely on correlations available in Training Data which may not exist under distributional shift which can manifest poorly in real world deployment. IRM overcomes this limitation through learning invariant features that are stable across multiple environments and ultimately provide strong out-of-distribution (OOD) generalization.This article contains a detailed review of IRM, with coverage of theoretical underpinnings, optimization methods and applications. IRMs main idea is to search for representations of data where the best classifier is invariant to environments (or domains).
Keywords: Invariant Risk Minimization, Out-of-Distribution Generalization, Domain Generalization, Causal Inference, Empirical Risk Minimization, Distribution Shift, Machine Learning Robustness; IRMv1; Risk Extrapolation Deep learning
