Learning to Undo Damage
Ruining an image is easy. Add a little Gaussian noise to every pixel, then a little more, and after enough rounds the picture of a cat is indistinguishable from television static. Nobody needs a neural network for that part. The interesting question is whether a network can learn to run the process backwards, one small step at a time, until static turns into a cat that never existed.
That is the whole idea behind diffusion models. It was first written down by Jascha Sohl-Dickstein and colleagues at ICML 2015, in a paper that borrowed its framing from nonequilibrium thermodynamics. The results were modest and the paper stayed fairly quiet for five years. Then in 2020 Jonathan Ho, Ajay Jain and Pieter Abbeel published «Denoising Diffusion Probabilistic Models» (DDPM) at NeurIPS, reported a state-of-the-art FID of 3.17 on unconditional CIFAR-10, and the field changed direction. Within two years the systems people used to turn text into pictures were almost all diffusion models.
The Forward Process
A diffusion model has two halves, and only one of them is learned. The forward process is fixed. Starting from a real image, it adds Gaussian noise over T steps according to a schedule. In the DDPM paper T was 1,000 and the amount of noise added per step grew linearly from 0.0001 to 0.02. By the last step almost nothing of the original image survives, and the result is very close to pure Gaussian noise.
One property makes training cheap. Because each step adds Gaussian noise, the noisy image at any step t can be written directly in terms of the clean image: a scaled copy of the original plus a scaled sample of noise. You never have to simulate the 999 intermediate steps. To make a training example you pick a random t, draw one noise sample, and mix.
Predicting the Noise
The reverse process is where the network lives. In principle it should model the distribution of a slightly cleaner image given a noisier one. Ho and his coauthors showed that, after working through the variational bound, this reduces to something almost embarrassingly simple: show the network a noisy image and the step number t, and ask it to predict the exact noise that was added. The loss is the mean squared error between the predicted noise and the real one.
They also found that dropping the weighting terms from the full bound gave better-looking samples, even though it gave up some likelihood. The network itself was a U-Net, the encoder and decoder architecture with skip connections that had become standard for segmentation, with the step number fed in as an embedding.
Generation then means starting from pure noise and applying the network 1,000 times. At each step it estimates the noise, removes a portion of it, and adds back a smaller amount of fresh randomness. The output of one step is the input of the next.
Score Matching
A separate line of research reached the same place from another direction. The score of a probability distribution is the gradient of its log density with respect to the data. It is a vector field that points, from any location, toward regions where data is more likely. If you know the score everywhere, you can sample by following it with a bit of added noise, a procedure called Langevin dynamics.
Aapo Hyvärinen introduced score matching in 2005 as a way to estimate this gradient without knowing the normalizing constant of the density. Pascal Vincent showed in 2011 that a denoising autoencoder trained on noisy data implicitly learns the score. Yang Song and Stefano Ermon built a generative model on this at NeurIPS 2019. Their key observation was that scores are badly estimated in empty regions far from the data, so they trained one network on many noise levels at once and sampled by annealing from large noise to small.
Predicting the added noise and estimating the score turn out to be the same quantity up to a scaling factor. Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole made this explicit at ICLR 2021 by describing both methods as discretizations of a stochastic differential equation that slowly turns data into noise. Reversing that equation requires only the score. The same framework gave a deterministic version, an ordinary differential equation with the same marginal distributions, which allowed exact likelihood computation. The paper reported an FID of 2.20 on CIFAR-10 and the first 1024 by 1024 samples from a score-based model.
A Thousand Steps Is Too Many
The obvious weakness was speed. A GAN produces an image in one forward pass. A DDPM needed a thousand. Much of the work since 2020 has gone into cutting that number without retraining from scratch.
DDIM (Jiaming Song, Chenlin Meng and Stefano Ermon, ICLR 2021) kept the DDPM training objective but defined a non-Markovian reverse process that can skip steps. The authors reported samples of similar quality 10 to 50 times faster in wall-clock time. Viewing sampling as solving an ODE also opened the door to better numerical solvers, and dedicated samplers now get usable images in a few dozen steps or fewer. Distillation goes further: a student network is trained to match two or more steps of a teacher in one. Consistency models (Yang Song, Prafulla Dhariwal, Mark Chen and Ilya Sutskever, ICML 2023) train a network to map any point on a sampling trajectory directly to its endpoint, which makes single-step generation possible.
Guidance
An unconditional model produces random samples from whatever it was trained on. To ask for a specific class or a text prompt, the model needs conditioning, and conditioning alone tends to give images that only loosely follow the request.
Prafulla Dhariwal and Alex Nichol introduced classifier guidance in «Diffusion Models Beat GANs on Image Synthesis» (NeurIPS 2021). A separate classifier, trained on noisy images, supplies a gradient that pushes each denoising step toward the desired class. With it they reached an FID of 2.97 on ImageNet at 128 by 128 and matched BigGAN-deep with as few as 25 forward passes per sample.
Training a classifier on noisy images is awkward, though. Jonathan Ho and Tim Salimans proposed classifier-free guidance at a NeurIPS 2021 workshop. One network is trained with the condition dropped a small fraction of the time, so it learns both conditional and unconditional predictions. At sampling time the two predictions are combined, and the difference between them is amplified by a guidance weight. Raising the weight makes images follow the prompt more closely at the cost of diversity. Most text-to-image systems use this trick, and the guidance scale is one of the few knobs ordinary users touch.
Diffusion in a Latent Space
Running a U-Net a few dozen times over a full-resolution image is expensive, and most of that computation goes into pixel detail the eye barely notices. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Björn Ommer addressed this in «High-Resolution Image Synthesis with Latent Diffusion Models» (CVPR 2022). They first train an autoencoder that compresses images into a smaller latent grid while keeping perceptual quality, then run the diffusion process in that latent space. A decoder turns the final latent back into pixels.
The paper also added cross-attention layers to the U-Net, so conditioning inputs such as text embeddings or bounding boxes can steer the denoiser at every resolution. This design became Stable Diffusion. Its first versions diffused in a 64 by 64 latent to produce 512 by 512 images and used a frozen CLIP text encoder for the prompt. Because it was released with open weights and ran on consumer graphics cards, it was the version of diffusion most researchers and hobbyists first worked with. Later systems replaced the U-Net with Transformers operating on latent patches, following the Diffusion Transformer of William Peebles and Saining Xie (ICCV 2023), but the latent space idea stayed.
Case: Training Images That Come Back
A model that learns the score of its training distribution should, in theory, be able to reproduce training images. Nicholas Carlini and eight coauthors tested this at USENIX Security 2023 in «Extracting Training Data from Diffusion Models.» They generated large numbers of images from Stable Diffusion and Google’s Imagen using captions from the training data and searched for outputs that were near copies of training images. They extracted over a thousand training examples across the models they studied, including photographs of identifiable people and trademarked logos.
The memorized images were mostly ones that appeared many times in the training set. Deduplicating data reduces the problem but does not remove it. The authors also reported that the diffusion models they tested leaked more training data than the GANs they compared against. For a technique so often described as generating something new, that is an uncomfortable result.
Closing Thought
The most unusual thing about diffusion models is how the learned part and the procedure are separated. The network is only ever trained to do one modest thing: look at a noisy input and estimate the noise. Everything else, the number of steps, the solver, the guidance weight, the choice between a random or a deterministic path, is decided after training. DDIM, better ODE solvers and classifier-free guidance all improved results by changing how a network is used rather than what it learned. Most of deep learning puts its effort into the training objective. With diffusion, a large share of the progress came from the sampler instead.
By: Max Johnson B.
Deja una respuesta