Slides 2020 Outline The 2020 course consists of the following topics: Lecture 1 Overview of Mathematics of Data Empirical Risk Minimization Statistical Learning with Maximum Likelihood Estimators Decomposition of error Lecture 2 Principles of iterative descent methods Gradient descent for smooth convex problems Gradient descent for smooth non-convex problems Lecture 3 Gradient descent Acceleration Adaptive gradient methods Lecture 4 Deficiency of smooth models Sparsity and compressive sensing Atomic norms Non-smooth minimization via Subgradient descent Lecture 5 Composite minimization Proximal gradient methods Introduction to Frank-Wolfe method Lecture 6 Time-data trade-offs Rate iteration-cost trade-offs Variance reduction Lecture 7 Introduction to Deep Learning The Deep Learning Paradigm Challenges in Deep Learning Theory and Applications Introduction to Generalization error bounds Uniform Convergence and Rademacher Complexity Generalization in Deep Learning (Part 1) Lecture 8 The classical trade-off between model complexity and risk The generalization mystery in Deep Learning Implicit regularization of optimization algorithms Double Descent curves Generalization bounds based on Algorithmic Stability Lecture 9 Scalable non-convex optimization with emphasis on deep learning Lecture 10 Adversarial Machine Learning (minmax) Adversarial training Generative adversarial networks Difficulty of minmax Lecture 11 Fundamentals of min-max optimization Gradient descent-ascent (GDA) methods: Simultaneous and alternating Lecture 12 Extragradient and other operator splitting methods Lecture 13 Storage optimal algorithms for constrained optimization
