GPTQ-like methods

Sources:

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

• GPTQv2: Efficient Finetuning-Free Quantization for Asymmetric Calibration

• QQQ: Quality Quattuor-Bit Quantization for Large Language Models

• The goal of these methods is to optimally compress the weights such that the L2 distance between the original activations and the activations from the quantized model is minimal. The activations are obtained from a calibration dataset.

Layer-Wise Quantization

• For each layer with weights $W_l$ and layer input $X_l$ (corresponding to a set of activations obtained using a calibration dataset), we perform the following optimization

${\text{argmin}}_{\tilde{W}}||WX-\tilde{W}X|{|}_2^2$

• The quantization grid i.e. the set of value we map to (e.g. int8) is fixed prior to the optimization.
• A weight scalar value $w_{ij}$ can be mapped to an arbitrary point within the grid i.e. it doesn’t have to be “round-to-nearest”.

One way to solve it using whitening + projecting back

1. Decomposing each term by row

• Write $W=[w_1;w_2;\dots ;w_m]$ as its $m$ rows, and similarly $\tilde{W}=[{\tilde{w}}_1;{\tilde{w}}_2;\dots ;{\tilde{w}}_m]$. Then $WX-\tilde{W}X=\left[\begin{array}{c}(w_1-{\tilde{w}}_1)X\\ (w_2-{\tilde{w}}_2)X\\ ⋮\\ (w_m-{\tilde{w}}_m)X\end{array}\right].$ So the total squared error is just the sum of each row’s error: $\Vert WX-\tilde{W}X{\Vert }_F^2={\sum }_{i=1}^m\Vert (w_i-{\tilde{w}}_i)X{\Vert }_2^2.$

• Each row $(w_i,{\tilde{w}}_i)$ is independent of the others.

• If $X$ were the identity matrix, then $\Vert (w_i-{\tilde{w}}_i)X{\Vert }_2^2$ would just be the plain Euclidean distance $\Vert w_i-{\tilde{w}}_i{\Vert }_2^2$, and you’d clearly pick each component ${\tilde{w}}_{ij}$ to be the nearest allowed quantized value to $w_{ij}$.

• With a general $X$, each row error is $\Vert (w_i-{\tilde{w}}_i)X{\Vert }_2^2=(w_i-{\tilde{w}}_i)(X{X}^{\top })(w_i-{\tilde{w}}_i{)}^{\top }.$ You can think of $X{X}^{\top }$ as a “weighting matrix” that tells you which directions in $w_i$-space matter more: if along some direction $X{X}^{\top }$ is large, errors in that direction get penalized more heavily.

2. “Whitening”—turning the problem into plain Euclidean distance

We can remove that weighting by a simple change of variables:

• Compute the square-root of the matrix $X{X}^{\top }$. Call it $\sqrt{X{X}^{\top }}$. - Define a new row vector $w_i^{\prime }=w_i\sqrt{X{X}^{\top }},{\tilde{w}}_i^{\prime }={\tilde{w}}_i\sqrt{X{X}^{\top }}.$ - Then $\Vert (w_i-{\tilde{w}}_i)X{\Vert }_2^2=\Vert w_i^{\prime }-{\tilde{w}}_i^{\prime }{\Vert }_2^2,$ which is just ordinary Euclidean distance in the “primed” space.

3. Nearest-neighbor quantization in the primed space

Now in this transformed space, the best quantized ${\tilde{w}}_i^{\prime }$ simply picks each coordinate ${\tilde{w}}_{ij}^{\prime }=\arg {\min }_{q\in Q}|w_{ij}^{\prime }-q|.$ Here $Q$ is your codebook (e.g. the set of 256 8-bit values).

4. Transform back to the original space

Finally, you undo the change of variables: ${\tilde{w}}_i={\tilde{w}}_i^{\prime }(X{X}^{\top }{)}^{-1/2}.$ and quantize back again to nearest-neighbour.

Optimal Brain Quantization

• The Optimal Brain Quantization (OBQ) method solves the layer-wise quantization problem defined above.

• Then, OBQ handles each row $w$ independently, quantizing one weight at a time while always updating all not-yet-quantized weights, in order to compensate for the error incurred by quantizing a single weight.

• Since the corresponding objective is a quadratic, whose Hessian is $H_F=2X_F{X}_F^T$, where $F$ denotes the set of remaining full-precision weights, the greedy-optimal weight to quantize next, which we denote by $w_q$, and the corresponding optimal update of all weights in $F$, denoted by ${\delta }_F$, are given by the following formulas, where $\text{quant}(w)$ rounds $w$ to the nearest value on the quantization grid: $w_q=\arg {\min }_{w_q}\frac{(\text{quant}(w_q)-w_q{)}^2}{[H_F^{-1}{]}_{qq}}$ ${\delta }_F=-\frac{w_q-\text{quant}(w_q)}{[H_F^{-1}{]}_{qq}}\cdot (H_F^{-1}{)}_{:,q}$ OBQ quantizes weights iteratively using these two equations, until all the weights of $w$ are quantized.

• OBQ quantizes weights in greedy order, i.e. it always picks the weight which currently incurs the least additional quantization error.

GPTQ v1

GPTQ v1 is a bunch of (quite involved) tricks to make OBQ fast and stable numerically for large models.

GPTQ v2: Asymmetric Calibration and Parallelization

• GPTQ treats every layer independently.

• GPTQ v2 takes into account how quantized layers progressively transform the activation patterns. This becomes particularly acute as quantization proceeds through deeper layers.

• The layer-wise optimization objective can be written as

${\text{argmin}}_{\Delta w}||(w+\Delta w)X-wX|{|}_2^2\text{s.t.}\Delta w=\tilde{W}-W$

• GPTQv2 optimization objective takes the modified activation $\tilde{X}$ (potentially provoked both by activation quantization and weight quantization in previous layers.)

${\text{argmin}}_{\Delta w}||(w+\Delta w)X-w\tilde{X}|{|}_2^2\text{s.t.}\Delta w=\tilde{W}-W$

Practical details

• All methods are weight-only and asymmetric per-channel by default.

• Works out of the box with signed-INT4 kernels if you clamp min = -max (symmetric) or if your kernel supports per-channel zero-points.

  • Popular inference projects (ExLlama, Marlin) therefore tend to post-process GPTQ files into symmetric form.

How to post-process from asymmetric to symmetric

• For each channel (or group) with bit-width $b$

  • dequantize asymmetrically: $w_f=s(q-z)$
  • find the symmetric range: $\alpha =max|w|$
  • new symmetric scale: $s_{sym}=\frac{\alpha }{2^{b-1}-1}$
  • Re-quantize: $q_{sym}=clip(round(\frac{w_f}{s_{sym}}),-8,+7)$

• This a loss-less rewrite unless some values saturate to $\pm 7$

  • In practice, the hit on perplexity is $\le 0.5$ for Llama 2 7B