Pretrain a tiny language model, mask prompt tokens, and teach a simple instruction-to-label format.
Project: Workshop Assistant · Audience: basic Python developers · Run: CPU; optional external training only in Part 10.
So far we have asked which parameters to update. Supervised fine-tuning, or SFT, says what training signal to use: examples of an input and the desired output. You can use SFT while updating every weight, only LoRA weights, or another supported parameter subset. SFT and LoRA are not competing entries in one flat list.
For an instruction-following language model, an example might be: user asks “Route this question: help with ESP32 camera,” assistant replies with a reviewed JSON object. The training objective increases the likelihood of the desired assistant tokens given the prompt and earlier answer tokens.
Our notebook uses a small recurrent language model, not a transformer. An embedding maps token IDs to vectors, a GRU tracks the token sequence, and an output layer predicts the next token. We first train it on three generic sentences. Then we fine-tune it on three routing instructions. This is real next-token training at a toy scale, with no claim of useful generalization.
For the sequence route esp32 answer iot <eos>, the model sees each prefix and predicts the next token. The input is all tokens except the last; the labels are all tokens except the first. This is the one-position shift. <eos> means the response has ended.
During training, earlier correct tokens are supplied, even if the model would have predicted a wrong token. This is called teacher forcing. During generation, the model must feed its own predictions back. The difference explains why low training loss is not a substitute for checking generated answers.
import torch
from ft_lab import language_setup, sft_example, sft_train, generate
model,vocab,pairs=language_setup()
x,labels=sft_example('route esp32','iot',vocab)
reverse={value:key for key,value in vocab.items()}
print('input:',[reverse[int(i)] for i in x[0]])
print('labels:',[reverse[int(i)] if i!=-100 else 'IGNORE' for i in labels[0]])
assert labels.tolist()[0][:2]==[-100,-100]
print('before:',generate(model,vocab,'route esp32'))
history=sft_train(model,vocab,pairs)
for prompt,answer in pairs:
print(prompt,'=>',generate(model,vocab,prompt),'expected:',answer)
assert history[-1] < history[0]Executed output (CPU teaching experiment):
input: ['route', 'esp32', 'answer', 'iot'] labels: ['IGNORE', 'IGNORE', 'iot', '<eos>'] before: device route python => python expected: python route linux => linux expected: linux route esp32 => iot expected: iot
We want the model to learn the assistant's response, not spend the loss budget reproducing the user's question. A label of -100 tells PyTorch cross-entropy to ignore that position. In this sequence, we ignore predictions of prompt tokens and train on the iot and <eos> targets.
Masking a label does not hide the prompt from the model. The prompt still appears in the input and helps condition the answer. Also, this recurrent example uses explicit word tokens. Production chat models require the model's own tokenizer and chat template; writing arbitrary role markers can change what the model sees.
Suppose the model assigns probabilities 0.8 to iot and 0.5 to <eos> at the two supervised positions. Their negative log losses are about 0.223 and 0.693. The mean is about 0.458. We average only over supervised positions, not ignored labels. If padding is added, its labels also need masking.
import math
example_loss=(-math.log(.8)-math.log(.5))/2
print('two-token mean loss:',round(example_loss,3))
import matplotlib.pyplot as plt
fig,ax=plt.subplots(figsize=(7,3))
ax.plot(history);ax.set(xlabel='Fine-tuning epoch',ylabel='Answer-token cross-entropy',title='Toy SFT training loss')
plt.show()Executed output (CPU teaching experiment):
two-token mean loss: 0.458
Write examples covering the behavior you want: answer structure, refusal when information is missing, concise explanations, and ambiguous requests. Do not fill a dataset with many copies of the same easy template. Review the answers, because SFT teaches mistakes as readily as good examples.
For an ESP32 assistant, separate reliable task behavior from device-specific facts. Training examples might teach asking for the board variant and camera module before suggesting pin settings. Current pin maps or product documents can still come from retrieval. Never train a model to invent a value merely to complete a JSON field.
Our three examples are a mechanism demonstration. A model that repeats them has memorized three training instructions; it has not become a routing service. A real experiment needs independent prompts, the unchanged-base comparison, and an explicit output validator. Part 10 provides a small external-model smoke-training path, also clearly separated from evidence of deployment quality.
If loss falls but responses repeat the question, inspect the label mask. If every token is ignored, there is no useful training signal. If output never stops, verify that the end token is supervised and generation has a maximum length. If results differ after reload, check the tokenizer, base revision, and saved adapter configuration.
Try masking nothing and compare behavior, but keep the evaluation prompts unchanged. Explain why the objective changed. Next, we will use pairs of preferred and rejected responses instead of only one desired response.
Reference: TRL's supervised fine-tuning documentation.