What's Inside

Machine Learning Algorithms

**A broad overview of the field of ML.**

Introduction to algorithms from a broad range of areas across machine learning—-generative models, support vector machines, tree-based algorithms, neural networks, gradient boosting, and more.

Mathematical Foundations

**A rigorous definition of key concepts.**

Algorithms derived from first principles using mathematical notation. Rigorous introduction to key concepts in machine learning.

Algorithm Implementations

**Every algorithm is implemented in Python.**

Executable lecture notes—-Jupyter notebooks that display algorithm definitions and their implementations side-by-side. Over 20 algorithms are implemented from scratch in Python.

General Information

What you will learn

Key elements of this course include:

- Foundational probabilistic principles of deep generative models.
- DGM learning algorithms, and popular model families.
- Applications in domains such as computer vision, NLP, and biomedicine.

Prerequisites

This PhD-level course requires a background in mathematics and programming at the level of Master's level courses.

- Basic knowledge about machine learning from at least one of: CS4780, CS4701, CS5785.
- Basic knowledge of probabilities and calculus: students will work with computational and mathematical models.
- Basic knowledge of deep neural networks (CNNs, RNNs; CS5787). Extensive experience implementing deep neural networks is not required but will be helpful for the class project.
- Proficiency in a programming language (preferably Python) will be helpful for completing the class project if you want to perform an implementation.

## Lecture 1: Introduction.

Natural agents excel at discovering patterns, extracting knowledge, and performing complex reasoning based on the data they observe. How can we build artificial learning systems to do the same?

## Lecture 2: Introduction to Probabilistic Modeling.

In this lecture, we define probabilistic Models of the Data, compare discriminative models with generative models, and give our audience a first glimpse of deep generative models.

## Lecture 3: Autoregressive Models.

We will discuss basic and modern autoregressive models. We will also discuss how recurrent neural networks can work as autoregressive models.

## Lecture 4: Maximum Likelihood Learning.

This lecture is all about maximizing the likelihood from data. We will discuss topics such as KL-Divergence and Monte Carlo Estimation, and look at statistical issues and bias/Variance tradeoff in maximum likelihood learning.

## Lecture 5: Latent Variable Models.

Latent Variable Models is a very useful tool in our generative models toolbox. We will compare and give examples of shallow and deep latent variable models, and take a look at how to approximate marginal likelihood using variational inference.

## Lecture 6: Learning Latent Variable Models.

We continue our exploration in deep latent variable models. In this lecture, we focus on gradient-based optimization of the ELBO and Inference Amortization. We will also explore the variational autoencoder.

## Lecture 7: Normalizing Flows.

This lecture contains an in-depth discussion of normalizing flows. We will discuss how to compose simple transformations and volume-preserving transformations.

## Lecture 8: Advanced Flow Models.

Now that we have some knowledge in normalizing flows, we can begin discussing advanced flow models using Triangular Jacobians, autoregressive flows, probability distillation, and parallel wavenet.

## Lecture 9: Generative Adversarial Networks.

We will now start walking into the direction of likelihood-free learning, and discuss the hot topic of Generative Adversarial Networks (GANs).

## Lecture 10: Advanced Topics in Generative Adversarial Networks.

We will dive deeper into support vector machines by introducing Lagrange duality, and define the dual form of support vector machines.

## Lecture 11: Energy-Based Models.

In this lecture, we explain energy-based models, representation, and learning of energy-based models.

## Lecture 12: Score-Based Generative Models.

In this lecture, we will explore score-based generative models and their connection to diffusion models.

## Lecture 13: Probabilistic Reasoning.

We will explore probabilistic reasoning, which is an approach to machine learning in which we work with structured probabilistic models that encode our understanding of the problem.

## Lecture 14: Combining Generative Model Families.

We have covered several useful building blocks: autoregressive, latent variable models, flow models, GANs. Can we combine them in many ways to achieve different tradeoffs?

## Lecture 15: Evaluating Generative Models.

Quantitative evaluation of generative models is a challenging task, how should we compare performance of different models?