Deep Generative Models

Probabilistic foundations and learning algorithms.

About Us

Generative models are a class of machine learning algorithms that operate over complex, high-dimensional objects such as images, sequences, and graphs. Recent advances have greatly improved the capabilities of generative models and have enabled new applications in computer-generated art, natural language processing, computational drug design, and other areas.


This course explores the foundational probabilistic principles of generative models, their learning algorithms, and popular model families, which include variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, and diffusion models. The course also covers applications in domains such as computer vision, natural language processing, and biomedicine, and draws connections to the field of reinforcement learning.


Instructors

Volodymyr Kuleshov

Assistant Professor
Cornell Tech


Volodymyr Kuleshov focuses on machine learning and its applications in scientific discovery, health, and sustainability. He is also the co-founder of Afresh, a startup that uses AI to significantly driving down food waste.

 

Aditya Grover

Assistant Professor
UCLA


Aditya Grover's research is centered around machine learning with limited supervision, and am currently focussing on probabilistic generative modeling and sequential decision making under uncertainty.

 

Yang Song

Assistant Professor
CalTech


Yang Song (宋飏) focuses on developing scalable methods for modeling, analyzing and generating complex, high-dimensional data.

 

Stefano Ermon

Associate Professor
Stanford


Stefano Ermon enables innovative solutions to problems of broad societal relevance through advances in probabilistic modeling, learning and inference.

 

Hongjun Wu

Producer,
Cornell Tech


Hongjun Wu (吴泓骏) is interested in applying machine learning techniques to optimize 3D animation and video games. He is working on automated procedural mesh and shader generation with artificial intelligence.

 

What's Inside

Core Concepts & Techniques

Discover the Building Blocks.

Delve into the fundamentals of generative modeling, focusing on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). Learn the theory behind these powerful models and their application in generating realistic and complex data.

 

Practical Applications & Projects

Apply Knowledge to Real-world Challenges.

Engage with hands-on projects that apply deep generative models across various domains. From image and text generation to complex data synthesis, this section encourages creativity and innovation in solving practical problems with generative AI.

 

Integration & Enhancement

Explore Cutting-edge Model Combinations.

Study advanced techniques for combining different generative models to create sophisticated systems capable of producing highly detailed and diverse outputs, including PixelVAE and FlowGAN examples.

 

General Information

What you will learn

Key elements of this course include:

  • Understanding Generative Models: Learn the fundamentals of how generative models like VAEs and GANs create new, complex data from learned distributions.
  • Model Architecture Mastery: Dive deep into the architecture, advantages, and limitations of various generative models, including PixelVAE, FlowGAN, and more.
  • Integration Techniques: Explore how to combine different generative models to leverage their strengths and overcome their weaknesses.
  • Real-world Applications: Apply your knowledge to real-world scenarios, creating innovative solutions across domains such as art, gaming, and synthetic data generation.
  • Ethical AI Use: Understand the ethical considerations and societal impacts of deploying generative models, fostering responsible AI development and use.

Prerequisites

This PhD-level course requires a background in mathematics and programming at the level of Master's level courses.

  • Basic knowledge about machine learning from at least one of: CS4780, CS4701, CS5785.
  • Basic knowledge of probabilities and calculus: students will work with computational and mathematical models.
  • Basic knowledge of deep neural networks (CNNs, RNNs; CS5787). Extensive experience implementing deep neural networks is not required but will be helpful for the class project.
  • Proficiency in a programming language (preferably Python) will be helpful for completing the class project if you want to perform an implementation.

 

Lecture 1: Introduction.

Natural agents excel at discovering patterns, extracting knowledge, and performing complex reasoning based on the data they observe. How can we build artificial learning systems to do the same?

Lecture 2: Introduction to Probabilistic Modeling.

This lecture explores probabilistic modeling, focusing on learning generative models from data, comparing discriminative and generative models, and introducing deep generative models, with detailed discussions on Bayesian networks and the foundational principles of probabilistic models.

Lecture 3: Autoregressive Models.

This lecture covers the fundamentals of autoregressive models, including their principles, types like Sigmoid Belief Networks and NADE, advancements in RNNs, and modern examples such as PixelRNN, PixelCNN, and WaveNet, illustrating their applications in generative modeling, anomaly detection, and adversarial attacks.

Lecture 4: Maximum Likelihood Learning.

This lecture covers maximum likelihood learning for generative models, including the KL divergence as a learning objective, Monte Carlo estimation and gradient descent for maximizing the likelihood from data, and discusses statistical issues like the bias-variance tradeoff.

Lecture 5: Latent Variable Models.

Latent Variable Models is a very useful tool in our generative models toolbox. This lecture covers latent variable models, their motivation and definition, examples like Gaussian mixture models and deep latent Gaussian models, and the challenges of maximum marginal likelihood learning for these models.

Lecture 6: Learning Latent Variable Models.

We continue our exploration in deep latent variable models. This lecture discusses learning latent variable models, including gradient-based optimization of the evidence lower bound (ELBO) using techniques like the REINFORCE estimator and the reparameterization trick, inference amortization, and the variational autoencoder.

Lecture 7: Normalizing Flows.

This lecture explores the concept of Normalizing Flows in the context of Deep Generative Models, discussing their foundation in volume-preserving transformations, the importance of the determinant and change of variables formula, and practical applications in creating complex data distributions from simpler ones through invertible transformations.

Lecture 8: Advanced Flow Models.

Now that we have some knowledge in normalizing flows! This lecture covers normalizing flow models, a type of latent variable model with tractable likelihoods, discussing different flow architectures like NICE, Real NVP, Masked Autoregressive Flow, and Inverse Autoregressive Flow, as well as techniques like probability distillation to accelerate sampling from autoregressive models like WaveNet.

Lecture 9: Generative Adversarial Networks.

We will now start walking into the direction of likelihood-free learning. This lecture covers generative adversarial networks (GANs), a likelihood-free approach to generative modeling where a generator network produces samples that aim to fool a discriminator network in distinguishing them from real data samples, discussing the GAN objective function, optimization challenges like mode collapse, and example results.

Lecture 10: Advanced Topics in Generative Adversarial Networks.

We will dive deeper into support vector machines by introducing Lagrange duality, and define the dual form of support vector machines.

Lecture 11: Energy-Based Models.

In this lecture, we explain energy-based models, representation, and learning of energy-based models.

Lecture 12: Score-Based Generative Models.

In this lecture, we will explore score-based generative models and their connection to diffusion models.

Lecture 13: Diffusion Models.

This lecture covers diffusion models, focusing on their theoretical underpinnings in score matching and Fisher divergences, practical sampling via Langevin dynamics, and their strengths and limitations in generative modeling.

Lecture 14: Evaluating Generative Models.

Quantitative evaluation of generative models is a challenging task, how should we compare performance of different models?

Lecture 15: Combining Generative Model Families.

We have covered several useful building blocks: autoregressive, latent variable models, flow models, GANs. Can we combine them in many ways to achieve different tradeoffs?

Lecture 16: Discrete Deep Generative Models.

We will explore probabilistic reasoning, which is an approach to machine learning in which we work with structured probabilistic models that encode our understanding of the problem.

Lecture 17: Probabilistic Reasoning.

We will explore probabilistic reasoning, which is an approach to machine learning in which we work with structured probabilistic models that encode our understanding of the problem.