Time embeddings

Stable Diffusion

• Timesteps embeddings are like token embeddings (matrix $T\times t_{emb}$), you can

  • fix them e.g. sinus and cosines embeddings
  • learn them (stable diffusion does I think)

• Actually, the pipeline seems to be (https://github.com/huggingface/diffusers/blob/v0.27.2/src/diffusers/models/embeddings.py#L476):

  • project into sinus and cosinus embeddings (https://github.com/huggingface/diffusers/blob/v0.27.2/src/diffusers/models/embeddings.py#L26), embedding_dim=256, flip_sin_to_cos=True, downscale_freq_shift=1, scale: float = 1, max_period: int = 10000,

  • into FFN (https://github.com/huggingface/diffusers/blob/v0.27.2/src/diffusers/models/embeddings.py#L190)

  • Can be ultimately used in AdaLayerNormZero (DiT, https://github.com/huggingface/diffusers/blob/v0.27.2/src/diffusers/models/normalization.py#L63)

  • How to use in TransformerBlock (https://github.com/huggingface/diffusers/blob/v0.27.2/src/diffusers/models/attention.py#L184)

DiT Implementation

**AdaLayerNormZero https://github.com/facebookresearch/DiT/blob/main/models.py#L101 (WARNING: SHOULD INIT TO ZERO)