MCQ normalization

https://blog.eleuther.ai/multiple-choice-normalization/

possible pitfalls: https://github.com/huggingface/blog/blob/main/evaluating-mmlu-leaderboard.md

Let $x_{0:m}$ be the prompt, and $x_{m:n_i}$ be the $i$th possible continuation with a token length of $n_i-m$. There are several ways to use a language model to rank multiple possible continuations to a prompt. Since the language model only gives (log) probabilities for the next token given the context (i.e $log(\mathbb{P}(x_i|x_{0:i}))$), there is ambiguity in handling scoring for arbitrary continuations. The following are several possible ways to resolve this problem:

Unormalized

• The score of continuation is determined is just the sum of the log-likelihoods
• ${\sum }_{j=m}^{n_i-1}log(\mathbb{P}(x_j)|x_{0:j})$
• Bias towards shorter responses as longer sequences tend to have lower log probabilities

Token-normalized

• ${\sum }_{j=m}^{n_i-1}log(\mathbb{P}(x_j)|x_{0:j})/(n_i-m)$
• average log probability per token
• not tokenization independent

Byte-length normalized

• ${\sum }_{j=m}^{n_i-1}log(\mathbb{P}(x_j)|x_{0:j})/{\sum }_{j^{\prime }=m}^{n_i-1}{L}_{x_j^{\prime }}$ where $L_{x_j}$ is the number of bytes represented by token $x_j$.

Unconditional likelihood normalized

• ${\sum }_{j=m}^{n_i-1}log(\mathbb{P}(x_j)|x_{0:j})-\mathbb{P}(x_j)$
• Intuitively, this approach measures the amount that the prompt increases the model’s probability of outputting each continuation from the probability of the model unconditionally producing that continuation. reduces the influence from very likely tokens e.g. “the”.