Guidance

Classifier Guidance

• Let us begin with score-based formulation of a diffusion model (because the score is an easier object to manipulate that the source noise)

• Our goal is to learn $\nabla logp(x_t|y)$, the score of the conditional model, at arbitrary noise levels t.

• By Bayes rules, we have ${\nabla }_{x_t}logp(x_t|y)={\nabla }_{x_t}log[\frac{p(y|x_t)p(x_t)}{p(y)}]={\nabla }_{x_t}logp(y|x_t)+{\nabla }_{x_t}logp(x_t)$

  • adversarial gradient + unconditional score
  • adversarial gradient because it’s the gradient of the classifier $p(y|x)$ on noisy input $x_t$

• Guidance

  • ${\nabla }_{x_t}logp(x_t|y)=\gamma {\nabla }_{x_t}logp(y|x_t)+{\nabla }_{x_t}logp(x_t)$

• Drawbacks. Reliance on a separately learned classifier. Because the classifier must handle arbitrarily noisy inputs, which most existing pretrained classification models are not optimized to do, it must be learned ad hoc alongside the diffusion model.

Classifier-free guidance

• Let’s look again at the adversarial gradient

• We can rewrite it as ${\nabla }_{x_t}logp(y|x_t)={\nabla }_{x_t}log\frac{p(x_t|y)p(y)}{p(x_t)}={\nabla }_{x_t}logp(x_t|y)-{\nabla }_{x_t}logp(x_t)$

• Plug into the guidance:

  • ${\nabla }_{x_t}logp(x_t|y)=\gamma {\nabla }_{x_t}logp(x_t|y)+(1-\gamma ){\nabla }_{x_t}logp(x_t)$

• The unconditional score can be obtained by plugging the “empty” conditioning into our condition diffusion model.