Slides 2025 Outline The 2025 course consists of the following topics Lecture 01 Introduction. Overview of Mathematics of Data Empirical Risk Minimization Statistical Learning with Maximum Likelihood Estimators Lecture 02 Parametric statistics Gaussian linear regression model Logistic regression model: Classification Poisson regression model: Graphical model selection M-estimator examples and unifying perspective for generalized linear models Role of computation Lecture 03 Linear algebra: Norms, matrix norms, dual norms Analysis: Continuity, Lipschitz continuity, differentiation Convexity: Convex sets, convex functions, subdifferentials, L-Lipschitz gradient functions, strong convexity Convergence rates and convergence plots Lecture 04 Principles of iterative descent methods Gradient descent for smooth convex problems Gradient descent for smooth non-convex problems Lecture 05 Optimality of convergence rates Lower bounds Accelerated gradient descent Newton and Adaptive methods Tensor methods Lecture 06 Stochastic optimization Deficiency of smooth models Sparsity and compressive sensing Non-smooth minimization via Subgradient descent Atomic norms Lecture 07 Composite minimization Proximal gradient methods Introduction to Frank-Wolfe method Lecture 08 Variance reduction Introduction to deep learning Challenges in deep learning theory and applications Lecture 09 The classical trade-off between model complexity and risk Generalization bounds via uniform convergence The generalization mystery in deep learning Implicit regularization of optimization algorithms Double descent curves: Generalization bounds via bias-variance decomposition Scaling Laws Lecture 10 Scalable non-convex optimization with emphasis on deep learning Lecture 11 Adversarial machine learning (minmax) Adversarial Training Generative Adversarial Networks (GANs) Lecture 12 Difficulty of minmax Diffusion models Lecture 13 Primal-dual optimization-I: Fundamentals of minimax problems Fenchel conjugates Duality Lecture 14 Primal-dual optimization-II: Augmented Lagrangian grandient methods Semi-definite programming HCGM and CGAL algorithms Lecture 15 Basics of language models. Self-attention and transformer architectures Fundamentals of pre-training & fine-tuning & reinforcement learning with human feedback (RLHF) Generative pre-trained transformer (GPT) family Basics of prompting: zero-shot & few-shot & chain-of-thought & automatic
