Slides 2023 Outline The 2023 course consists of the following topics Lecture 01 Introduction. The role of models and data Maximum-likelihood formulation Sample complexity bound for estimation and prediction Lecture 02 Generalized linear model Logistic regression Lecture 03 Linear algebra reminder Gradients Reading convergence plots Lecture 04 Optimization algorithms Optimality measures. Structures in optimization Gradient descent. Gradient descent for smooth functions Lecture 05 Optimality of convergence rates Lower bounds Accelerated gradient descent Concept of total complexity Adaptive methods Tensor methods Lecture 06 Stochastic gradient descent Concise signal models Compressive sensing Sample complexity bounds for estimation and prediction Challenges to optimization algorithms for non-smooth optimization Subgradient method Lecture 07 Introduction to proximal-operators Proximal gradient methods Linear minimization oracles Conditional gradient method for constrained optimization Lecture 08 Variance reduction Introduction to deep learning Challenges in deep learning theory and applications Lecture 09 Generalization through uniform convergence bounds Rademacher complexity Double descent curves and overparameterization Implicit regularization Generalization bounds using stability Lecture 10 Escaping saddle points Adaptive gradient methods Lecture 11 Adversarial machine learning and generative adversarial networks (GANs) Wasserstein GAN Difficulty of of minimax optimization. Lecture 12 Robustness in deep learning Diffusion models Lecture 13 Primal-dual optimization-I: Fundamentals of minimax problems Fenchel conjugates Duality Extra gradient method Chambolle-Pock algorithm Stochastic primal-dual methods Lecture 14 Primal-dual optimization-II: Augmented Lagrangian grandient methods Semi-definite programming HCGM and CGAL algorithms Lecture 15 Language models: Basis of language models. Self attention and Transformer GTP family
