Breakdown of M-LLMs

• During training,
  • Modality Encoder, LLM Backbone, and Modality Generator are generally maintained in a frozen state.
  • The primary optimization emphasis is on Input and Output Projectors.
  • Given that Projectors are lightweight components, the proportion of trainable parameters in MM-LLMs is notably small compared to the total parameter count (typically around 2%).

General architecture components

Modality Encoder

• $F_x=M{E}_x(I_x)$, we want to extract features

• Vision: CLIP, SigLip

• Audio: Whisper, CLAP

• ImageBind: joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data.

  • allows for multimodal-conditioned generation
  • aligns each modality’s embeddings to the image embeddings (potentially extracted from CLIP)
  • no explicitly aligned pseudo-labeled dataset
  • 

Input Projector

• The Input Projector ${\theta }_{X\to T}$ is tasked with aligning the encoded features of other modalities $F_x$ with the text feature space $F_T$ . The aligned features as prompts $P_x$ are then fed into the LLM Backbone alongside the textual features $F_T$.
• Given $X$-text dataset $\{I_X,t\}$, the goal is to minimize the $X$-conditioned text generation loss $L_{txt-gen}(LLM(P_X,F_T),t)$
  • If we’re also generating the modality i.e. $t,S_X=LLM(P_x,F_T)$, the the loss becomes $L_{txt-gen}(LLM(P_X,F_T),(t,S_X))$

LLM backbone

• It produces (1) direct textual outputs t, and (2) signal tokens $S_X$ from other modalities (if any).
  • These signal tokens act as instructions to guide the generator on whether to produce MM contents
  • $t,S_X=LLM(P_x,F_T)$

Output Projector

• The Output Projector ${\theta }_{t\to x}$ maps the signal token representations $S_X$ from the LLM Backbone into features $H_X$ understandable to the following Modality Generator $M{G}_X$ .
  • $H_X={\theta }_{t\to x}(S_X)$
• Given $X$-text dataset $\{I_X,t\}$, $t$ is first fed into LLM to generate the corresponding $S_X$, then mapped into $H_X$
  • To facilitate alignment of the mapped features $H_X$ , the goal is to minimize the distance between $H_X$ and the conditional text representations of $M{G}_X$:
    • $argmin{L}_{mse}(H_X,{\tau }_X(t))$
    • ${\tau }_X$ is the textual condition encoder in $M{G}_X$

Modality Decoder/Generator

• The Modality Generator $M{G}_X$ is tasked with producing outputs in distinct modalities.
• Need to able to parse LLM response if the response is text only i.e. H_X is the identity
• Common:
  • Stable Diffusion (image)
  • Zeroscope, Videofusion (video)
  • AudioLDM-2 (audio)
• Can also compute text-conditioned noise-matching loss to tune the input and output projectors
  • ${\mathcal{L}}_{gen}=||ϵ-{ϵ}_X(z_t,t,H_x)|{|}_2^2$