Create a labeled workshop dataset, keep paraphrases together, and learn what a baseline reveals.
Project: Workshop Assistant · Audience: basic Python developers · Run: CPU; optional external training only in Part 10.
Our first task is classification. Given a short question, return exactly one label: python, linux, or iot. For example, “help with dictionary” should go to Python, while “debug my gpio” should go to IoT. The output is a category, not a generated technical answer. That distinction makes our first experiments small enough to inspect.
A useful task definition also says what happens outside these classes. In a real application, a request about invoices should be routed to a person or an other class. Our small dataset does not support that behavior. We will not pretend that a three-class accuracy score solves out-of-domain routing.
Each record has text, a correct label, a group ID, and a split. A group identifies related examples that must stay together. If “help with gpio” is in training and “please explain gpio” is in testing, the score may mostly measure template recognition. We keep every paraphrase of the same keyword in the same split.
The synthetic dataset has 54 records: 3 topics × 6 keywords × 3 templates. Training has 36 examples; validation and test each have 9. These numbers are deliberately small so you can read every example. They are not a recommended production sample size. You will need representative, independently reviewed data for a real service.
Training data updates the weights. Validation data helps choose settings. Test data is reserved for the final comparison. If you repeatedly choose your best model using test accuracy, the test set has become another validation set.
from ft_lab import records, LABELS
rows = records()
sets = {split: [r for r in rows if r['split']==split]
for split in ['train','validation','test']}
groups = {split: {r['group'] for r in data} for split,data in sets.items()}
assert groups['train'].isdisjoint(groups['validation'])
assert groups['train'].isdisjoint(groups['test'])
assert groups['validation'].isdisjoint(groups['test'])
print({key: len(value) for key,value in sets.items()})
print(sets['train'][0])Executed output (CPU teaching experiment):
{'train': 36, 'validation': 9, 'test': 9}
{'text': 'help with python', 'label': 'python', 'group': 'python-python', 'split': 'train'}
A baseline is a simple reference system. Because the classes are balanced, always predicting Python gives accuracy 1/3, or about 33.3%. That tells us how easy it is to get a nonzero score without learning anything useful.
We can also build a keyword rule from training examples only. It will recognize the training keywords but will not know the unseen validation keywords. That is a valuable failure: the rule exposes the need for prior knowledge or more varied data. Do not add validation keywords to the rule after seeing errors and still call the result an untouched baseline.
from collections import Counter
from ft_lab import tokens
majority = Counter(r['label'] for r in sets['train']).most_common(1)[0][0]
score = sum(r['label']==majority for r in sets['validation'])/len(sets['validation'])
print('majority validation accuracy:', round(score,3))
train_keywords = {word: label for label, words in __import__('ft_lab').GROUPS
for word in words[:4]}
def rule(text):
return next((train_keywords[w] for w in tokens(text) if w in train_keywords), majority)
print('rule validation accuracy:', sum(rule(r['text'])==r['label'] for r in sets['validation'])/9)
assert abs(score-1/3) < 1e-8Executed output (CPU teaching experiment):
majority validation accuracy: 0.333 rule validation accuracy: 0.3333333333333333
Our router uses a bag-of-words vector. Make a vocabulary assigning each word a column. For a question, count how often each word occurs. Word order is discarded: “python help” and “help python” have the same vector. This is a limitation, not an unnoticed feature.
For vocabulary [help, python, linux], “help python” becomes [1,1,0]. A neural layer multiplies these numbers by weights. Unknown words contribute no known word feature. A modern tokenizer instead breaks text into reusable subword units, but the basic need to convert text into numbers remains.
Our source pretraining corpus deliberately includes every topic keyword in generic guides. Its vocabulary is available before target fine-tuning. This is disclosed source knowledge, not a vocabulary fitted to held-out target labels. In a real pretrained model, its tokenizer and earlier training also provide prior knowledge. Our test evaluates a tiny change of wording, not knowledge of completely new concepts.
For each wrong prediction, keep the text, expected label, predicted label, and model version. Ask whether the label is defensible. “Python script on Raspberry Pi” could reasonably belong to either queue. Write a routing policy and have reviewers apply it consistently; a larger model cannot repair an undefined task.
Before replacing synthetic data, remove personal information you do not need, verify permission to use examples, deduplicate related tickets, and keep conversations in a single split. Split by customer, device, time, or conversation when that matches deployment. Randomly shuffling individual sentences can hide leakage.
Add an other class on paper with five examples and define when it should win. Do not just change the label list in code: the head size, metrics, examples, and tests would all need updating. This shows why a precise task definition belongs before training.
Next, we will start from an explicitly pretrained small network and compare updating all of it with updating only its final classification layer. Our aim is to understand what changes, not to manufacture an impressive score.