Course Syllabus

Differentiable programming is an emerging interdisciplinary field (or you can call it a rebranding of automatic differentiation) that attempts to compute the derivative of general numerical programs and use them as our machine learning model. This allows us to build more expressible, interpretable, and efficient machine learning models by exploiting domain knowledge. However, it also leads to challenges in algorithms, programming languages, and systems when we want to efficiently and correctly differentiate general programs. People who work on programming languages, computer vision and graphics, robotics, machine learning, numerical/high-performance/scientific computing, or applied math could all be interested in this topic.

 

We will be reading papers that address challenges in algorithms (how to differentiate sorting, optimization, physics simulation, or 3D rendering), languages (semantics, modularity, and correctness of differentiation), and systems (how do we efficiently map the derivative code to modern hardware? how do we employ differentiable programming in practice?). I put up a tentative reading list here: https://docs.google.com/document/d/1Xh3Em10_YhbBnbf8e4mHEnUkoDu5BF5KnqBWAZ1OLGM/edit, but it can be adjusted based on the collective interests of enrolled students (and me!). Please feel free to edit the list, or send me the articles you want to read. Please also feel free to send me questions if you have any. Finally, please come and join this fun reading group and let's learn from each other!

 

The plan for this semester is as following: the first week I will briefly introduce what is automatic differentiation and why do we want to differentiate programs. We will also spend some time to know each other. In the following weeks, we will have students present and discuss related work in the area. Since we have 9 weeks and around 20 people, we will take the following format (inspired by the "role-playing seminar" people did at UNC & U of Toronto https://colinraffel.com/blog/role-playing-seminar.html (Links to an external site.)): for each week, a group of 2-3 students will examine a paper/article in detail and present their opinions. The class will start with a brief summarization of the article from the group (but all of the class are expected to have at least skimmed through the article). One or two students will play the "reviewer" role, who will assess the importance of the paper and attempt to find flaws in it. The rest will play the "rebutter" role (imagine you are writing a rebuttal to the reviewers) to respond to the reviewer's criticism -- how we could potentially address the flaws either by improving the article or doing future research. Others in the class are expected to join the conversation and post a question on the Discord channel beforehand (see below for the link).

 

Regarding units: let's do 2 units if you ever present a paper in the class.

 

The class is being held at Center Hall 218

 

Before the end of the first day of the class (9/29), please do the following:

1. Please sign up for a time slot at https://docs.google.com/spreadsheets/d/1gu2WE37koU0j-bM-hXP7kRz1cHOPYxtjZgv1ZeL51N8/edit?usp=sharing (Links to an external site.) You don't have to decide for what to present/read yet, but feel free to put something there (it doesn't need to be in the reading material). We can talk with the class to decide what to read in the subsequent weeks.
2. Join the Discord server here https://discord.gg/PQCxeZpQrT (Links to an external site.) and briefly introduce yourself: tell us your background, research interests, and what do you want to learn from this reading group. This will make deciding what to read for each week much easier. You are also encouraged to post meme pictures.

Course Summary:

Date Details Due