How to Build a Diffusion Language Model
An introduction to diffusion language models and the research advances that underlie today's diffusion LLMs. We describe the building blocks of recent open-source models, starting from simple masking diffusion, and including techniques for iterative refinement, post-training, and variable-length generation. Material is adapted from workshop talks and lectures at ICLR 2026 and MLSS 2026.
-
Simple Guidance Mechanisms for Discrete Diffusion Models
Classifier-free and classifier-based guidance mechanisms for discrete diffusion models, enabling controllable generation over text and biological sequences. Published at ICLR 2025.
-
Simple and Effective Masked Diffusion Language Models
MDLM: a simple, well-engineered masked (absorbing-state) discrete diffusion recipe with a Rao-Blackwellized objective that closes much of the gap to autoregressive language models. Published at NeurIPS 2024.
-
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
A DNA foundation model that brings bi-directional, reverse-complement–equivariant long-range sequence modeling to genomics, built on a structured state-space (Mamba) backbone. Published at ICML 2024.
-
Diffusion Models With Learned Adaptive Noise
MuLAN learns a multivariate, input-adaptive noise schedule that applies noise at different rates across an image, tightening the ELBO and setting new density-estimation results on CIFAR-10 and ImageNet. NeurIPS 2024 (spotlight).
-
ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
A memory-efficient finetuning method that integrates LoRA with modular low-bit quantizers, enabling finetuning of 2-3 bit LLMs on a single consumer GPU. Published in TMLR 2023 and presented at ICLR 2024.