CMPT 409/981 - Optimization for Machine Learning (Fall 2024)

Instructor: Sharan Vaswani
Instructor office hours: Thursday, 2.30 pm - 3.30 pm (TASC-1 8221)

Teaching Assistant; Qiushi Lin
TA office hours: Monday, 9.30 am - 10.30 am (ASB 9814)

Prerequisites: Probability (e.g. CMPT 210, STAT 270, STAT 271), Linear Algebra (e.g. MATH 240), Basics of Multivariable Calculus (e.g. MATH 251). It is encouraged (though not necessary) for students to have taken (or be simultaneously taking) an introductory course on machine learning (e.g. CMPT 410/726).

Textbook: There is no required textbook. We will use the following resources:

• Lectures on Convex Optimization, Nesterov, 2018
• Convex Optimization, Boyd and Vandenberghe, 2004
• Convex Optimization: Algorithms and Complexity, Bubeck, 2014
• Numerical Optimization, Nocedal and Wright, 2006
• First-order Methods in Optimization, Beck, 2017
• A Modern Introduction to Online Learning, Orabona, 2019

Piazza for course-related questions.

List of topics

• Basics: Subdifferentials, Optimality conditions, Lipschitz continuity, Convexity
• (Non)-Convex minimization: (Projected) Gradient Descent, Nesterov/Polyak momentum, Mirror Descent, Newton method
• (Non)-Convex minimization: Stochastic gradient descent (SGD), Variance reduction techniques, Adaptivity for SGD, Coordinate Descent
• Applications to training ML models (logistic regression, neural networks)
• Online optimization: Regret minimization, Follow the (regularized) leader, Adaptive gradient methods (AdaGrad, Adam, AMSGrad)
• Applications to Imitation learning, Reinforcement learning
• Min-Max optimization: Gradient Descent-Ascent, Extragradient
• Applications to GANs, Robust optimization

Schedule

• UBC'22
• UBC'20
• MIT
• UT Austin
• UCLA
• Stanford