Part 8 — Preference Tuning and DPO: Learn Which Answer Is Better

Calculate the DPO objective and train a tiny response-choice policy against a frozen reference.

Project: Workshop Assistant · Audience: basic Python developers · Run: CPU; optional external training only in Part 10.

A good answer can have several acceptable wordings

SFT supplies a desired answer. Preference data compares alternatives for the same prompt. For example, a reviewer may prefer “Which ESP32-CAM board and power supply are you using?” over an answer that confidently invents a wiring solution without enough context.

A preference record contains a prompt, a chosen response, and a rejected response. “Rejected” means less preferred under the review policy; it does not always mean every sentence is false. Reviewers need a consistent rubric, and both responses must address the same prompt for a meaningful comparison.

Direct Preference Optimization, or DPO, trains a policy using these pairs and a fixed reference model. A policy here simply means the model defining response probabilities. The reference anchors how much the chosen-versus-rejected preference has changed from the starting point. This lab illustrates the original sigmoid objective, not every DPO variant.

Work through one pair

Let the policy give chosen probability 0.6 and rejected probability 0.4. Let the reference give both 0.5. The policy's log-ratio is ln(0.6/0.4) ≈ 0.405; the reference log-ratio is zero. At beta 0.1, the scaled difference is 0.0405.

Apply sigmoid, 1/(1+exp(-z)), and take negative log: about 0.673. Equal policy and reference preferences give loss -ln(0.5) ≈ 0.693. A larger relative preference for the chosen response lowers this particular pair loss. It does not by itself measure factual correctness.

import torch
from ft_lab import dpo_loss, seed
policy=torch.tensor([[.6,.4]]).log()
reference=torch.tensor([[.5,.5]]).log()
loss=dpo_loss(policy,reference,beta=.1)
print('one-pair DPO loss:',round(float(loss),3))
assert abs(float(loss)-.673079) < .001

one-pair DPO loss: 0.673

A tiny, inspectable policy

For CPU clarity, our policy chooses between two complete candidate responses. Each prompt has its own pair of logits. Candidate zero is the chosen response; candidate one is rejected. This is a restricted response-choice policy, not a generative language model. We train actual parameters with the DPO objective while keeping the reference fixed.

In a generative model, compute the log probability of each complete response by summing its token log probabilities conditioned on the prompt and earlier response tokens. Exclude prompt and padding labels consistently. Do not substitute a single arbitrary token probability or silently length-normalize the sequence when claiming to implement the original objective.

seed()
pairs=[
 {'prompt':'Camera will not start','chosen':'Ask for board and power details','rejected':'Invent an exact wiring fix'},
 {'prompt':'Linux permission error','chosen':'Ask for command and file permissions','rejected':'Recommend making every file writable'},
]
logits=torch.nn.Parameter(torch.zeros(len(pairs),2))
reference=torch.zeros_like(logits).detach()
snapshot=reference.clone()
optimizer=torch.optim.Adam([logits],lr=.1)
losses=[]
for _ in range(100):
    loss=dpo_loss(logits,reference,beta=.2)
    optimizer.zero_grad();loss.backward();optimizer.step()
    losses.append(float(loss.detach()))
print('chosen probabilities:',logits.softmax(-1)[:,0].detach().tolist())
assert torch.equal(reference,snapshot)
assert losses[-1]<losses[0]

chosen probabilities: [0.9999978542327881, 0.9999978542327881]

The increasing probabilities only show that the optimizer learned two supplied preferences. It has no text encoder and cannot score an unseen prompt. This deliberately limited model makes the reference and loss easy to inspect. Do not interpret its high training probabilities as a general safety or quality score.

Where beta belongs

Beta scales the reference-adjusted log-ratio inside the loss. It affects optimization and the objective's relationship to staying near the reference. It is not a simple percentage of politeness or a universal learning rate. Learning rate still controls optimizer step size separately. Compare configurations using held-out preference pairs and direct task evaluations.

In practical adaptation, a supervised instruction model is often the starting policy and reference. Some implementations manage a frozen reference copy or equivalent reference calculation. Accidentally training the reference changes the intended comparison. Our detach() and equality assertion make that mistake visible.

Preference data can teach the wrong shortcut

If chosen responses are always longer, a model may learn verbosity rather than better reasoning. If rejected responses always contain a specific word, it may exploit that word. Vary wording, review hard cases, and record why a response is preferred. Disagreements between reviewers can reveal that the policy itself is ambiguous.

Before deployment, evaluate factual accuracy, format, abstention, and task completion separately. A model can prefer a courteous but wrong answer. For device advice, reviewed sources and deterministic constraints remain useful after preference tuning.

Exercise: reverse one preference and see how the policy changes. Then explain why more optimization cannot repair a wrong label. Next, we will put parameter count, accuracy, errors, and reproducibility into one comparison report.

Method reference: the original DPO paper.