Adam

Algortihm

• ${\beta }_1,{\beta }_2$ are the exponential decay rates for the moment estimates

• $ϵ$ is for stability reasons

• Given the gradient $g_t$ at timestep $t$, Adam

  • maintains a (biased) first moment of the gradient

    • $m_t\leftarrow {\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$

  • maintains a (biased) second raw moment of the gradient (elementwise-square)

    • $v_t\leftarrow {\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$

  • debiases the first and second moment before updating the params

    • ${\hat{m}}_t\leftarrow m_t/(1-{\beta }_1^t)$
    • ${\hat{v}}_t\leftarrow v_t/(1-{\beta }_2^t)$

  • Updates parameters by

    • ${\theta }_t\leftarrow {\theta }_{t-1}-\gamma \cdot {\hat{m}}_t/(\sqrt{{\hat{v}}_t}+ϵ)$

Intuition

• $m_t/\sqrt{{\hat{v}}_t}$ is the signal-to-noise ratio (SNR)
• Indeed, the effective stepsize taken in parameter space is bounded by $\gamma |m_t/\sqrt{{\hat{v}}_t}|$
• This also means that Adam is scale-invariant, rescaling the gradients by a factor $c$ will scale ${\hat{m}}_t$ by $c$ and ${\hat{v}}_t$ by $c^2$, which cancel out when computing the parameter update.

Takeaways

• If variance is low ⇒ SNR is high ⇒ we take larger effective step-sizes
• If SNR is low ⇒ mean gradient is getting low or variance is very high ⇒ in both cases, we want to take smaller steps
• Note that the SNR is maintained for each parameter, so we keep an estimate of which loss landscape has been smooth or not for each parameter