Simple and Effective Masked Langauge Diffusion Models

Sahoo, S., Arriola, M., Schiff, Y., Gokalsan, A., Marroquin, E., Chiu, J. T., Rush, A., Kuleshov, V.

AccMLBio ICML Workshop 2024 [Spotlight], SPIGM ICML Workshop 2024

While diffusion models excel at generating high-quality images, they have traditionally lagged behind autoregressive (AR) methods in language modeling. We apply an effective training recipe and derive a simplified, Rao-Blackwellized objective that improves the performance of language diffusion models.