CMPT 409/981 - Optimization for Machine Learning (Fall 2022)

Instructor: Sharan Vaswani
Instructor office hours: Monday 4 pm - 5 pm (TASC-1 8221)

Course Objective: This course introduces the foundational concepts of convex and non-convex optimization with applications to machine learning. It will give the students experience in 1. Proving theoretical guarantees for optimization algorithms, 2. Analyzing machine learning (ML) problems from an optimization perspective and 3. Developing and analyzing new optimization methods for ML applications.

Prerequisites: Linear Algebra, Multivariable Calculus, (Undergraduate) Machine Learning

Textbook: There is no required textbook. We will use the following resources:

• Lectures on Convex Optimization, Nesterov, 2018
• Convex Optimization, Boyd and Vandenberghe, 2004
• Convex Optimization: Algorithms and Complexity, Bubeck, 2014
• Numerical Optimization, Nocedal and Wright, 2006
• First-order Methods in Optimization, Beck, 2017
• A Modern Introduction to Online Learning, Orabona, 2019

Piazza for course-related questions.

List of topics

• Basics: Subdifferentials, Optimality conditions, Lipschitz continuity, Convexity
• (Non)-Convex minimization: (Projected) Gradient Descent, Nesterov/Polyak momentum, Newton method
• (Non)-Convex minimization: Stochastic gradient descent (SGD), Variance reduction techniques, Adaptivity for SGD
• Applications to training ML models (logistic regression, neural networks)
• Online optimization: Regret minimization, Follow the (regularized) leader, Adaptive gradient methods (AdaGrad, Adam, AMSGrad)
• Applications to Imitation learning
• Min-Max optimization: Gradient Descent-Ascent, Extragradient
• Applications to GANs, Robust optimization

Schedule

• UBC'22
• UBC'20
• MIT
• UT Austin
• UCLA
• Stanford