An in-depth graduate course on classical and modern optimization methods for statistics, machine learning, and data science.
Course Overview
The course begins with optimization models for learning from data and the theory of convex optimization. It then studies stochastic gradient methods, modern nonconvex optimization, and optimizers used in deep learning and reinforcement learning. Applications include statistical modeling, causal learning, robust optimization, and generative systems.
Emphasis is placed on modeling intuition, algorithm design, mathematical analysis, performance guarantees, and the limits of each method. Students learn to choose and assess optimization approaches for different applications.
Course Information
- Cross-listed as 16:960:690:01 and 16:954:694:02.
- Thursdays, 12:10–3:10 p.m., PH-111.
- Prerequisites: solid analysis and linear algebra, familiarity with algorithms and computational complexity, and comfort with mathematical proofs.
Assessment and Project
One quiz contributes 20% of the grade and the course project contributes 80%. Five optional take-home questions can each add up to 5% bonus credit.
Students may work alone or in a group of two or three. Projects may develop an optimization algorithm, analyze one theoretically, or implement methods for an application. Students may select an instructor-provided topic or propose their own after discussion with the course team.
Tentative Weekly Outline
| Week / Date | Topic |
|---|---|
| 1 · Jan 22 | Optimization models, examples, and constrained versus unconstrained problems |
| 2 · Jan 29 | Convexity: sets, functions, strong convexity, smoothness, and geometry |
| 3 · Feb 5 | Stochastic optimization, empirical risk minimization, stability, and generalization |
| 4 · Feb 12 | Stochastic gradient descent: convergence, sample complexity, stability, and implementation |
| 5 · Feb 19 | Coordinate descent and Frank–Wolfe methods |
| 6 · Feb 26 | Mirror descent, Bregman divergence, nonsmoothness, and hidden convexity |
| 7 · Mar 5 | Deep-learning optimizers and variance-reduced gradients: Adam-mini, SARAH, SPIDER, STORM, and PAGE |
| 8 · Mar 12 | Matrix optimizers, including Muon variants |
| 9 · Mar 19 | Spring break |
| 10 · Mar 26 | Nonconvex optimization landscapes and convergence to stationary points |
| 11 · Apr 2 | Stochastic minimax, bilevel, and contextual optimization; biased gradients |
| 12 · Apr 9 | Distributionally robust optimization, dual formulations, computation, and generalization |
| 13 · Apr 16 | Causal learning as optimization: invariance, instrumental variables, and DAG learning |
| 14 · Apr 23 | Policy optimization in reinforcement learning and function approximation |
| 15 · Apr 30 | Variational inference, reinforcement learning from human feedback, and fine-tuning |
Reading Material
- Optimization course lecture notes, by Zebang Shen, Niao He, Bernd Gärtner, and Martin Jaggi.
- Convex Optimization, by Stephen Boyd and Lieven Vandenberghe.
- Convex Optimization: Algorithms and Complexity, by Sébastien Bubeck.
- High-Dimensional Statistics, by Martin J. Wainwright.
- Optimization Methods for Large-Scale Machine Learning, by Léon Bottou, Frank E. Curtis, and Jorge Nocedal.