NTK

• At the infinite-width limit, every gradient update incurs such a small change on the parameters that the naive first-order taylor expansion holds i.e. $f(x,\theta )-f(x,{\theta }_0)=⟨{\nabla }_{\theta }f(x,{\theta }_0),\theta -{\theta }_0⟩$.
• So, in function space, $f_t-f_{t-1}=\eta \mathcal{K}\mathcal{L}(f_t,y)$ where $\mathcal{L}$ is loss, $y$ is label, and $\mathcal{K}$ is the NTK kernel $\mathcal{K}(x,z)=⟨{\nabla }_{\theta }f(x,{\theta }_0),{\nabla }_{\theta }f(z,{\theta }_0)⟩$
• The NTK limit does not learn features!