CODETIPS ACADEMY / FINE-TUNING METHODS

Part 6 — Quantization and QLoRA: Understand the Memory Trade-Off

Quantize a small matrix, measure reconstruction error, and train an update around a frozen quantized base.

Project: Workshop Assistant · Audience: basic Python developers · Run: CPU; optional external training only in Part 10.

Precision and adaptation are separate choices

A weight stored as a 32-bit float uses four bytes. A 16-bit value uses two. A value encoded in four bits needs half a byte when packed. Quantization represents numbers with a limited collection of codes plus information needed to reconstruct approximate values.

LoRA decides which new parameters are trained. Quantization decides how values are represented. QLoRA combines a frozen, quantized base model with trainable low-rank adapters. It does not mean that ordinary integer base weights are updated directly with gradients.

The original QLoRA method includes NF4 quantization, double quantization, and paged optimizers. Our CPU exercise below is deliberately simpler: symmetric uniform quantization with codes from -7 to 7. It explains rounding, error, and frozen-base training. It is not an NF4 implementation or a QLoRA performance benchmark.

Round three numbers by hand

Consider weights [-1, 0.2, 1]. With maximum absolute value 1, choose scale 1/7 ≈ 0.142857. Divide each value by the scale, round, and clamp to the permitted range. The codes become [-7,1,7]; reconstruct by multiplying by the scale. The middle value becomes about 0.142857, an error of about -0.057143.

More codes usually permit a closer approximation, but the distribution, grouping, scales, and quantization method all matter. Quantization error does not translate directly into an identical change in task accuracy.

import torch
from ft_lab import quantize4, QuantizedLoRA, seed
seed()
weight=torch.tensor([[-1.,.2,1.]])
codes,scale=quantize4(weight)
restored=codes.float()*scale
print('codes:',codes,'scale:',float(scale),'restored:',restored)
print('max absolute error:',float((weight-restored).abs().max()))
assert codes.min()>=-7 and codes.max()<=7

codes: tensor([[-7,  1,  7]], dtype=torch.int8) scale: 0.1428571492433548 restored: tensor([[-1.0000,  0.1429,  1.0000]])
max absolute error: 0.05714285373687744

Train only the correction

Now construct a small regression problem whose target is a frozen linear map plus a rank-one correction. Quantize the base; learn A and B around it. The target is synthetic and deliberately easy so we can inspect the mechanism. We do not use its loss as evidence about a real language model.

The stored codes are buffers. They participate in the forward calculation but are not trainable parameters. The update matrices are ordinary floating-point parameters. The backward pass flows through the computation to those matrices.

seed()
base=torch.randn(4,6)
x=torch.randn(64,6)
target=x@(base+.05*torch.ones_like(base)).T
model=QuantizedLoRA(base,rank=2)
before=model.codes.clone()
optimizer=torch.optim.Adam(model.parameters(),lr=.03)
losses=[]
for _ in range(160):
    loss=(model(x)-target).square().mean()
    optimizer.zero_grad(); loss.backward(); optimizer.step()
    losses.append(float(loss.detach()))
assert torch.equal(before,model.codes)
assert losses[-1]<losses[0]
print('MSE:',round(losses[0],5),'->',round(losses[-1],5))

MSE: 0.06253 -> 0.01504

The important memory accounting detail

Our code stores each quantized code in an int8 tensor and reconstructs a float matrix for calculation. It does not pack two four-bit codes into a byte, use a quantized kernel, or reduce measured model memory to half a byte per weight. The function name describes the code range, not a memory-saving storage implementation.

For an idealized billion-weight base, the weight payload alone is about 4 GB at float32, 2 GB at float16, or 0.5 GB at packed four-bit precision, using decimal GB. Real memory includes scales, metadata, adapters, gradients, optimizer tensors, activations, temporary buffers, and runtime overhead. A “0.5 GB weight payload” is not a promise that training fits in 0.5 GB of VRAM.

Longer sequences and larger batches increase activation memory. Gradient accumulation lets several microbatches contribute before an optimizer step, but it does not make a single overlong example fit. Gradient checkpointing exchanges recomputation for lower saved-activation memory. These controls are separate from choosing rank.

From this lab to an actual implementation

The optional Part 10 script has a --qlora branch using Transformers, PEFT, and bitsandbytes. It requests a four-bit base, prepares it for low-bit training, and adds LoRA. That branch needs a compatible installation and hardware; it was not run in the CPU verification for this release. Use its explicit setup instructions instead of assuming every Windows or GPU configuration behaves identically.

Do not start with quantization merely because it sounds advanced. First make an unquantized small-model experiment work. Then measure whether memory pressure justifies additional complexity, and compare quality before and after quantization on the same held-out examples.

Exercise: add an outlier value 100 to the matrix and observe what happens to smaller weights under a single shared scale. This shows why production quantizers use more sophisticated grouping and codebooks.

References: QLoRA paper, PEFT quantization guide, and Transformers bitsandbytes documentation.

← Part 5 · Part 7 →