Abstract
This paper studies distributional alignment in high-dimensional spaces through the use of Optimal Transport-based learning approaches, as discussed in [6]. In contrast to standard divergence measures, like Fullback–Libeler divergence or Jensen–Shannon divergence, Optimal Transport yields a distance metric that takes into consideration the underlying geometry of data and is particularly well suited for complex structured datasets. In this paper, we give a thorough overview of the theoretical background underlying Optimal Transport, covering Monger versus Kantorovich formulation, Wasserstein distances and entropy-regularized transport theory.In addition, this paper explores the combination between optimal transport and modern machine learning approaches like deep neural networks, domain adaptation models, generative models. It focuses extensively on issues of scalability in high-dimensional settings where computational complexity and the curse of dimensionality are major obstacles. The study investigates approaches like Stinkhorn iterations, stochastic optimization and dimensionality reduction averaging to improve efficiency of applicability. It also demonstrates various optimal transport-based distribution alignment applications from cross-domain image classification to healthcare data harmonization and multimodal learning systems, among others. OT-based methods consistently outperform conventional alignment methods in terms of preserving structural information and improving model robustness on comparative evaluations [10,11].Using Optimal Transport provides a principled, theoretically justified and practically efficient way to reach solutions for distribution alignment problems within high dimensional machine learning problems. Existing constraints are described and a roadmap for future research to increase scalability, interpretability and integration with new AI foundry technologies is proposed.
Keywords
Wasserstein Distance, Domain Adaptation, Machine Learning, Deep Learning, High-Dimensional Data, Stinkhorn Algorithm
References
[1] Ceric Villain, Optimal Transport: Old and New, Springer, 2009.
[2] Gabriel Pyre and Marco Couture, “Computational Optimal Transport,” Foundations and Trends in Machine Learning, 2019.
[3] Marco Couture, “Stinkhorn Distances: Light speed Computation of Optimal Transport,” Advances in Neural Information Processing Systems, 2013.
[4] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, “Deep Learning,” Nature, 2015.
[5] Arthur Gretton et al., “A Kernel Two-Sample Test,” Journal of Machine Learning Research, 2012.
[6] Solomon Fullback and Richard Libeler, “On Information and Sufficiency,” 1951.
[7] Jian-Yu Audibert, “Fast Learning Rates in Statistical Inference,” 2009.
[8] Justin Solomon, “Optimal Transport on Discrete Domains,” 2018.
[9] Filippo Santambrogio, Optimal Transport for Applied Mathematicians, 2015.
[10] Nicolas Courty et al., “Optimal Transport for Domain Adaptation,” IEEE TPAMI, 2017.
[11] Ievgen Redko et al., “Optimal Transport for Multi-Source Domain Adaptation,” 2019.
[12] Bharath K. Sriperumbudur, “On the Universality of RKHS,” 2010.
[13] Bernhard Schölkopf and Alexander Smola, Learning with Kernels, 2002.
[14] Ian Goodfellow et al., “Generative Adversarial Nets,” 2014.
[15] Diederik P. Kingma and Max Welling, “Auto-Encoding Variational Bayes,” 2013.
[16] Sebastian Nowozin et al., “f-GAN: Training Generative Neural Samplers,” 2016.
[17] Dougal J. Sutherland, “Generative Models and Divergence Measures,” 2017.
[18] Michael Arbel et al., “Maximum Mean Discrepancy Gradient Flow,” 2019.
[19] Alain Rakotomamonjy et al., “Generalized Wasserstein Distance,” 2015.
[20] Mathieu Blondel et al., “Fast Differentiable Sorting and Ranking,” 2020.
[21] Francis Bach, “Breaking the Curse of Dimensionality,” 2017.
[22] Elad Hazan, Introduction to Online Convex Optimization, 2016.
[23] Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning, 2014.
[24] Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, 2009.
[25] Kevin P. Murphy, Machine Learning: A Probabilistic Perspective, 2012.
[26] Christopher M. Bishop, Pattern Recognition and Machine Learning, 2006.
[27] Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2018.
[28] Peter Flach, Machine Learning: The Art and Science of Algorithms, 2012.
[29] David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, 2003.