Score-based Generative models

Statistical physics or Energy based models

• Can define arbitary flexible distribution using the Boltzmann distribution
  • $p_{\theta }(x)=\frac{1}{Z_{\theta }}{e}^{-f_{\theta }(x)}$
  • Normalization constant is intractable
  • The score function removes it thanks to the log + gradient trick
  • ${\nabla }_{x}log{p}_{\theta }(x)=-{\nabla }_x{f}_{\theta }(x)\approx s_{\theta }(x)$

Langevin dynamics sampling or Langevin MCMC

• The sampling procedure is (Markov-Chain Monte Carlo): $x_{i+1}\leftarrow x_i+c\nabla logp(x_i)+\sqrt{2c}ϵ$
• where $x_0$ is randomly sampled from a prior distribution (such as uniform) and $ϵ$ is some gaussian noise to prevent mode collapse.

Interpretation

• What does the score function represent? For every $x$, taking the gradient of its log likelihood with respect to $x$ essentially describes what direction in data space to move in order to further increase its likelihood.
• Intuitively, then, the score function defines a vector field over the entire space that data $x$ inhabits, pointing towards the modes.
  • 

Score-based interpretation (Tweedie’s Formula)

• Tweedie’s Formula states that the true mean of an exponential family distribution, given samples drawn from it, can be estimated by the maximum likelihood estimate of the samples (aka empirical mean) plus some correction term involving the score of the estimate. The score serves a correction in case of sample bias.

• For a Gaussian variable $z\sim \mathcal{N}(z;{\mu }_z,{\Sigma }_z)$, the Tweedie’s Formula states that: $\mathbb{E}[{\mu }_z|z]=z+{\Sigma }_z{\nabla }_{z}logp(z)$

• We know that our noisy samples $x_t\sim \mathcal{N}(x_t;\sqrt{{\bar{\alpha }}_t}{x}_0,(1-{\bar{\alpha }}_t)I)$

• Thus, by Tweedie’s Formula, $\mathbb{E}[{\mu }_{x_t}|x_t]=x_t+(1-{\bar{\alpha }}_t){\nabla }_{x_t}logp(x_t)$

• Thus, according to the formula, our best estimate estimate for $x_0$ is : $x_0=\frac{x_t}{\sqrt{{\bar{\alpha }}_t}}+\frac{(1-{\bar{\alpha }}_t){\nabla }_{x_t}logp(x_t)}{\sqrt{{\bar{\alpha }}_t}}$

• Thus, we should have $s_{\theta }(x_t,t)$ is a neural network that learns to predict the score function ${\nabla }_{x_t}logp(x_t)$, which is the gradient of $x_t$ in data space, for any arbitrary noise level t. The score function measures how to move in data space to maximize the log probability.

• Source noise ${ϵ}_0$ and the score ${\nabla }_{x_t}logp(x_t)$ describe something very similar i.e. ${\nabla }_{x_t}logp(x_t)=-\frac{1}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_0$