Algorithmic Foundations of Interactive Learning

Spring 2025: 17-740

Tuesday / Thursday 11:00-12:20

Announcements 📣

Week 1 Announcements

We can’t wait to meet you! 👋

Course Overview 📝

Interactive learning is a dynamic approach to machine learning where systems learn and adapt through continuous interaction with their environment or users, receiving feedback and adjusting their behavior in real time. These techniques are currently experiencing a resurgence across various domains of artificial intelligence and machine learning, from robotics to language modeling. In this advanced theory course, students will explore interactive learning from its foundational principles to recent applications, including fine-tuning Large Language Models (LLMs) and robot learning from demonstration. Key topics include:

  1. Online Learning: Learning under distribution shift.
  2. Game Solving: Using no-regret algorithms to compute equilibria.
  3. Reinforcement Learning: Sequential decision making. Model-free, model-based, and hybrid RL.
  4. Imitation Learning & Applications to Robotics: Learning from demonstrations. Behavioral cloning, DAgger, and inverse RL.
  5. RL from Human Feedback & Applications to Language Modeling: Learning from preferences. PPO, DPO, SPO.

Schedule (Tentative) 📅

Unit Date Lecturer Topic Reading
Online Learning 1/14/2025 Steven Course Overview  
  1/16/2025 Drew Intro to No-Regret Online Learning  
  1/21/2025 Steven Online Gradient Descent  
  1/23/2025 Steven Hedge / Maximum Entropy  
  1/28/2025 Steven Follow-the-Leader  
  1/30/2025 Steven Boosting  
Game Theory 2/4/2025 Steven Regret Minimization  
  2/6/2025 Steven Minimax Equilibria  
Reinforcement Learning 2/11/2025 Steven Foundations of MDPs  
  2/13/2025 Gokul API and CPI  
  2/18/2025 Gokul Policy Gradients  
  2/20/2025 Gokul Natural Policy Gradients  
  2/25/2025 Gokul PPO & TRPO  
  2/27/2025 Guest Lecture: Yuda Song Hybrid RL  
  Spring Break      
  3/11/2025 Gokul Three Lemmas for Model-Based RL  
  3/13/2025 Gokul Representation Learning for MBRL  
Imitation Learning 3/18/2025 Gokul DAgger  
  3/20/2025 Gokul Inverse RL  
  3/25/2025 Gokul Fast Inverse RL  
  3/27/2025 Guest Lecture: Sanjiban Choudhury Practical IL  
RLHF 4/1/2025 Gokul RLHF I  
  4/3/2025 Gokul RLHF II  
  4/8/2025 Guest Lecture: Wen Sun REBEL / REFUEL  
  4/10/2025 Gokul SPO  
Proj. Presentations 4/15/2025   Proj. Presentations  
  4/17/2025   Proj. Presentations  
  4/22/2025   Proj. Presentations  
  4/24/2025   Proj. Presentations  

Instructors 👨‍🏫

Avatar
Avatar
Avatar

Resources 📚

Textbooks