GizStack
View .mdOpen in ChatGPTOpen in Claude

Chat model lab

In this lab you take a language model through every stage of training on one GPU with the giz ai command: pretraining on web text, supervised fine-tuning (SFT) on chats, preference pairs (DPO), rewards on answers (GRPO), facts written into the weights with no training, and a chat with the result. Each stage writes a Hugging Face model folder. The next stage starts from it, and giz ai chat runs it as it is.

The numbers come from one run on an RTX 4070 Ti SUPER (16 GB). The model has 112 M parameters. At the end it answers short questions in whole sentences, also some that it did not see in training. It is not a useful assistant: the lab shows the path and what each stage changes. Training language models explains every flag.

StageFolderDataTime here
Pretrainbase2.1 B tokens of web text21 h
Fine-tune on chatschat491 M tokens of chats23 min
Fine-tune on your own chatstalk60 short chats9 s
Preference pairsliked360 pairs11 s
Rewards on answerssolved198 problems37 s
Write factsedited171 facts28 s
Chat

Pretrain

Make token shards of your text, then train a model of the 100m shape from nothing:

sh
giz ai tokens text fineweb-edu/*.parquet ultra-fineweb/*.parquet
giz ai train text base 260000 --size 100m --rows 16 --context 512 --keep 5000
  • tokens first learns a tokenizer of 8,192 tokens from the files (each digit is a token of its own). Then it packs every document, and holds out one in fifty of each file.
  • 100m is dim 768 and 14 layers: 112 M parameters with this tokenizer. At 16 rows of 512 tokens it trains at about 26,000 tokens a second on this card. --rows 32 does not fit in 16 GB.
  • Every --keep steps the run measures the held-out loss and writes base. base/best keeps the best generation. A run whose held-out loss jumps goes back to it by itself. A stopped run goes on from the folder with the same command.
  • After 2.1 B tokens (21 hours) the held-out loss was 1.047 bits a byte. giz ai score measured 83.2% on 8 BLiMP files, 42.4% on ARC-Easy and 31.2% on HellaSwag.

The base model continues text: after "The capital of France is" it writes "Paris". It does not answer in the turns of a chat yet.

Fine-tune on chats

Pack chats in the base model's tokenizer, then train every weight of a copy of the base on them:

sh
mkdir sft && cp base/tokenizer.json sft/
giz ai tokens sft openorca-gpt4.parquet
giz ai train sft chat 6000 --base base --rows 16 --context 512 --muon 0.005 --adam 0.0005 --hold 0 --keep 1500
  • The Parquet rows of question, response and system_prompt (OpenOrca's) become chats in the model's template. Only the assistant's turns are targets of the loss. The prompt's tokens are read, not learned.
  • --base base starts the new folder chat as the base model, at step 0 with new optimizer state.
  • The rates are a quarter of pretraining's, and --hold 0 lets them fall from the start: the model keeps what it knows and learns the form of an answer.
  • Here: 491 M tokens of chats at 36,000 tokens a second. The held-out loss was 0.6201, 0.6152, 0.6110 and 0.6075 bits a byte at steps 1500 to 6000.

The model now answers in turns, but in the long and formal style of its data ("As an AI, I can provide you with ..."), and it often loses the point of a short question.

Fine-tune on your own chats

A few dozen chats in the style you want move the model to it quickly. examples/ai/posttrain.giz writes 60 short chats, 360 preference pairs and 198 problems for the stages that follow:

sh
giz examples/ai/posttrain.giz post
mkdir own && cp base/tokenizer.json own/
giz ai tokens own post/chats.jsonl
giz ai train own talk 40 --base chat --rows 16 --context 128 --muon 0.002 --adam 0.0002 --hold 0 --every 10 --keep 40
  • A chat is a JSON line of messages, each with a role and its content.
  • 60 chats are 1,758 tokens: a step of 16 rows reads about all of them. The loss fell from 1.52 to 0.72 in 40 steps.
  • The held-out part is one chat, shorter than a row: the run measures it in rows as long as it is.

Preference pairs

DPO teaches which of two answers people choose. A pair is a JSON line of a prompt (or the messages before the answer), the chosen answer and the rejected one:

sh
giz ai train sft liked 60 --base talk --pairs post/pairs.jsonl --rows 16 --context 128 --muon 0.001 --adam 0.0001 --hold 0 --every 20 --keep 60
  • DATA (here sft) gives only the tokenizer.
  • The model in --base is also the reference. The loss rewards the chosen answer's log-likelihood over the reference's, against the rejected one's. A new run starts at log 2 (0.693).
  • A step reads --rows sequences: 8 pairs of two answers each.
  • Here: the loss fell to 0.5476, 0.3138 and 0.2225 at steps 20, 40 and 60. The model preferred the chosen answer of 92.5%, then 100% of the pairs. The held-out pairs' loss was 0.2166, all of them chosen right.
  • Stop early. 300 steps at twice the rates took the loss to 0 and the model to one repeated token: on a few hundred pairs, a few passes are enough.

Rewards on answers

GRPO lets the model write answers, and raises the ones that earn a reward. A problem is a JSON line of a prompt and the answer that a right completion holds:

sh
giz ai train sft solved 40 --base liked --tasks post/tasks.jsonl --samples 8 --rows 64 --context 128 --recompute --beta 0.05 --muon 0.001 --adam 0.0001 --hold 0 --every 5 --keep 20
  • Each step the engine that serves models writes --samples completions of each of --rows / --samples problems (8 here), from the weights as they train: the trainer writes its weights over the engine's model in place, with no file. A completion that holds the answer earns 1, any other 0.
  • A completion's advantage is its reward against its problem's mean. Only problems where some completions are right and some are wrong teach anything.
  • --beta keeps the model near the reference (the --base model). --recompute keeps the activations of one layer at a time, so 64 completions of 128 tokens fit beside the engine.
  • Here: 40 steps in 37 s, with the GPU busy 46% of the run. The mean reward stayed between 3% and 15%, and none of the 3 held-out problems came out right: 180 of the problems are sums, and the model gets almost every sum wrong, so few problems have both right and wrong completions to learn from. The answers to other questions changed little. GRPO teaches what the model already does sometimes; a set of problems it solves a part of the time gives it more.

Write facts

Training teaches what the model does sometimes. A fact it never gets right can be written into its weights instead, with no training. A fact is a JSON line of a prompt and the answer the model should give; examples/ai/posttrain.giz wrote 171 of them (sums, colors, capitals and legs) and 91 checks that ask them another way:

sh
giz ai edit $PWD/solved $PWD/edited post/facts.jsonl --keep post/chats.jsonl --check post/checks.jsonl
  • edited grows one last layer. Each answer token the model gets wrong takes one cell of the layer's MLP, and the layer's output is solved on the GPU so that the token leads and the answers to --keep's chats stay.
  • Here: 28 s. Facts answered right went from 20 to 169 of 171; the checks from 29 to 39 of 91; 5 of the 60 kept chats changed their answer.
  • A written fact answers its own question, not the skill: of the 91 checks, which ask the facts another way, only 10 more came out right. Write the forms you need, or teach the skill by training.
Questionsolvededited
What color is a banana?A banana is a fruit.A banana is yellow.
What is 3 plus 4?4.3 plus 4 is 7.
How many legs does a spider have?A spider has two legs.A spider has eight legs.
What is the capital of Germany?The capital of Germany is Berlin.The capital of Germany is Berlin.

Chat

sh
giz ai chat $PWD/solved "What is the capital of Germany?"

The answers of each stage to the same questions (greedy, so each run gives the same answer). The first question is in the chats and pairs, and "What is 3 plus 4?" is one of the GRPO problems; the others are in no data of the post-training:

Questionchattalklikedsolved
What is the capital of France?France is located in the state of Gaetland. ...France isalso known as?The capital of France is Paris.The capital of France is Paris.
What is the capital of Germany?Germany.German is a country with a rich history and a rich culture.The capital of Germany is Berlin.The capital of Germany is Berlin.
How many legs does a cat have?To find the answer, I will need to search for information ...A cat has two legs.A cat has four legs.A cat has four legs.
What color is a banana?a banana?A banana is white.A banana is white.A banana is white.
What is 3 plus 4?101.10.101.10. The first word is "boy." ...
  • The answers took the form of the chats, then the facts the base model read in pretraining came through in that form: the capital of Germany and a cat's legs are in no chat or pair.
  • The model gets colors and sums wrong, and a longer answer drifts and repeats. 112 M parameters and a few hundred examples set those limits, not the stages.
  • giz ai serve $PWD/solved --port 8000 serves the same folder over the OpenAI API.

When a stage fails

What you seeCauseDo this
Out of GPU memory at the first stepThe activations of --rows x --context tokensHalve --rows, or add --recompute (a third more work)
DPO loss near 0, chats of one repeated wordToo many passes over few pairsFewer steps or lower rates; keep the step where the held-out pairs still improve
GRPO reward stays flatEvery completion of a problem scores the same, so no advantageProblems the model gets right sometimes, more --samples, higher rates
diverged 3 times from step NThe held-out loss rose more than 5% three timesLower rates; FOLDER/best holds the best model