# Chat model lab

In this lab you take a language model through every stage of training on one GPU with the `giz ai` command:
pretraining on web text, supervised fine-tuning (SFT) on chats, preference pairs (DPO), rewards on answers (GRPO),
facts written into the weights with no training, and a chat with the result. Each stage writes a Hugging Face model folder. The next stage starts from it, and
`giz ai chat` runs it as it is.

The numbers come from one run on an RTX 4070 Ti SUPER (16 GB). The model has 112 M parameters. At the end it answers
short questions in whole sentences, also some that it did not see in training. It is not a useful assistant: the
lab shows the path and what each stage changes. [Training language models](../guides/training.md) explains every
flag.

| Stage | Folder | Data | Time here |
| --- | --- | --- | --- |
| [Pretrain](#pretrain) | `base` | 2.1 B tokens of web text | 21 h |
| [Fine-tune on chats](#fine-tune-on-chats) | `chat` | 491 M tokens of chats | 23 min |
| [Fine-tune on your own chats](#fine-tune-on-your-own-chats) | `talk` | 60 short chats | 9 s |
| [Preference pairs](#preference-pairs) | `liked` | 360 pairs | 11 s |
| [Rewards on answers](#rewards-on-answers) | `solved` | 198 problems | 37 s |
| [Write facts](#write-facts) | `edited` | 171 facts | 28 s |
| [Chat](#chat) | | | |

## Pretrain

Make token shards of your text, then train a model of the `100m` shape from nothing:

```sh
giz ai tokens text fineweb-edu/*.parquet ultra-fineweb/*.parquet
giz ai train text base 260000 --size 100m --rows 16 --context 512 --keep 5000
```

- `tokens` first learns a tokenizer of 8,192 tokens from the files (each digit is a token of its own). Then it packs
  every document, and holds out one in fifty of each file.
- `100m` is dim 768 and 14 layers: 112 M parameters with this tokenizer. At 16 rows of 512 tokens it trains at
  about 26,000 tokens a second on this card. `--rows 32` does not fit in 16 GB.
- Every `--keep` steps the run measures the held-out loss and writes `base`. `base/best` keeps the best generation.
  A run whose held-out loss jumps goes back to it by itself. A stopped run goes on from the folder with the same
  command.
- After 2.1 B tokens (21 hours) the held-out loss was 1.047 bits a byte. `giz ai score` measured 83.2% on 8 BLiMP
  files, 42.4% on ARC-Easy and 31.2% on HellaSwag.

The base model continues text: after "The capital of France is" it writes "Paris". It does not answer in the turns
of a chat yet.

## Fine-tune on chats

Pack chats in the base model's tokenizer, then train every weight of a copy of the base on them:

```sh
mkdir sft && cp base/tokenizer.json sft/
giz ai tokens sft openorca-gpt4.parquet
giz ai train sft chat 6000 --base base --rows 16 --context 512 --muon 0.005 --adam 0.0005 --hold 0 --keep 1500
```

- The Parquet rows of `question`, `response` and `system_prompt` (OpenOrca's) become chats in the model's template.
  Only the assistant's turns are targets of the loss. The prompt's tokens are read, not learned.
- `--base base` starts the new folder `chat` as the base model, at step 0 with new optimizer state.
- The rates are a quarter of pretraining's, and `--hold 0` lets them fall from the start: the model keeps what it
  knows and learns the form of an answer.
- Here: 491 M tokens of chats at 36,000 tokens a second. The held-out loss was 0.6201, 0.6152, 0.6110 and 0.6075
  bits a byte at steps 1500 to 6000.

The model now answers in turns, but in the long and formal style of its data ("As an AI, I can provide you with
..."), and it often loses the point of a short question.

## Fine-tune on your own chats

A few dozen chats in the style you want move the model to it quickly.
[`examples/ai/posttrain.giz`](https://github.com/gizai/gizlang/blob/main/examples/ai/posttrain.giz) writes 60 short chats, 360 preference pairs and 198
problems for the stages that follow:

```sh
giz examples/ai/posttrain.giz post
mkdir own && cp base/tokenizer.json own/
giz ai tokens own post/chats.jsonl
giz ai train own talk 40 --base chat --rows 16 --context 128 --muon 0.002 --adam 0.0002 --hold 0 --every 10 --keep 40
```

- A chat is a JSON line of `messages`, each with a `role` and its `content`.
- 60 chats are 1,758 tokens: a step of 16 rows reads about all of them. The loss fell from 1.52 to 0.72 in 40
  steps.
- The held-out part is one chat, shorter than a row: the run measures it in rows as long as it is.

## Preference pairs

DPO teaches which of two answers people choose. A pair is a JSON line of a `prompt` (or the `messages` before the
answer), the `chosen` answer and the `rejected` one:

```sh
giz ai train sft liked 60 --base talk --pairs post/pairs.jsonl --rows 16 --context 128 --muon 0.001 --adam 0.0001 --hold 0 --every 20 --keep 60
```

- `DATA` (here `sft`) gives only the tokenizer.
- The model in `--base` is also the reference. The loss rewards the chosen answer's log-likelihood over the
  reference's, against the rejected one's. A new run starts at log 2 (0.693).
- A step reads `--rows` sequences: 8 pairs of two answers each.
- Here: the loss fell to 0.5476, 0.3138 and 0.2225 at steps 20, 40 and 60. The model preferred the chosen answer of
  92.5%, then 100% of the pairs. The held-out pairs' loss was 0.2166, all of them chosen right.
- Stop early. 300 steps at twice the rates took the loss to 0 and the model to one repeated token: on a few hundred
  pairs, a few passes are enough.

## Rewards on answers

GRPO lets the model write answers, and raises the ones that earn a reward. A problem is a JSON line of a `prompt`
and the `answer` that a right completion holds:

```sh
giz ai train sft solved 40 --base liked --tasks post/tasks.jsonl --samples 8 --rows 64 --context 128 --recompute --beta 0.05 --muon 0.001 --adam 0.0001 --hold 0 --every 5 --keep 20
```

- Each step the engine that serves models writes `--samples` completions of each of `--rows` / `--samples`
  problems (8 here), from the weights as they train: the trainer writes its weights over the engine's model in
  place, with no file. A completion that holds the answer earns 1, any other 0.
- A completion's advantage is its reward against its problem's mean. Only problems where some completions are right
  and some are wrong teach anything.
- `--beta` keeps the model near the reference (the `--base` model). `--recompute` keeps the activations of one layer
  at a time, so 64 completions of 128 tokens fit beside the engine.
- Here: 40 steps in 37 s, with the GPU busy 46% of the run. The mean reward stayed between 3% and 15%, and none of
  the 3 held-out problems came out right: 180 of the problems are sums, and the model gets almost every sum wrong,
  so few problems have both right and wrong completions to learn from. The answers to other questions changed little.
  GRPO teaches what the model already does sometimes; a set of problems it solves a part of the time gives it more.

## Write facts

Training teaches what the model does sometimes. A fact it never gets right can be written into its weights instead,
with no training. A fact is a JSON line of a `prompt` and the `answer` the model should give;
`examples/ai/posttrain.giz` wrote 171 of them (sums, colors, capitals and legs) and 91 checks that ask them another
way:

```sh
giz ai edit $PWD/solved $PWD/edited post/facts.jsonl --keep post/chats.jsonl --check post/checks.jsonl
```

- `edited` grows one last layer. Each answer token the model gets wrong takes one cell of the layer's MLP, and the
  layer's output is solved on the GPU so that the token leads and the answers to `--keep`'s chats stay.
- Here: 28 s. Facts answered right went from 20 to 169 of 171; the checks from 29 to 39 of 91; 5 of the 60 kept
  chats changed their answer.
- A written fact answers its own question, not the skill: of the 91 checks, which ask the facts another way, only 10
  more came out right. Write the forms you need, or teach the skill by training.

| Question | `solved` | `edited` |
| --- | --- | --- |
| What color is a banana? | A banana is a fruit. | A banana is yellow. |
| What is 3 plus 4? | 4. | 3 plus 4 is 7. |
| How many legs does a spider have? | A spider has two legs. | A spider has eight legs. |
| What is the capital of Germany? | The capital of Germany is Berlin. | The capital of Germany is Berlin. |

## Chat

```sh
giz ai chat $PWD/solved "What is the capital of Germany?"
```

The answers of each stage to the same questions (greedy, so each run gives the same answer). The first question is
in the chats and pairs, and "What is 3 plus 4?" is one of the GRPO problems; the others are in no data of the
post-training:

| Question | `chat` | `talk` | `liked` | `solved` |
| --- | --- | --- | --- | --- |
| What is the capital of France? | France is located in the state of Gaetland. ... | France isalso known as? | The capital of France is Paris. | The capital of France is Paris. |
| What is the capital of Germany? | Germany. | German is a country with a rich history and a rich culture. | The capital of Germany is Berlin. | The capital of Germany is Berlin. |
| How many legs does a cat have? | To find the answer, I will need to search for information ... | A cat has two legs. | A cat has four legs. | A cat has four legs. |
| What color is a banana? | a banana? | A banana is white. | A banana is white. | A banana is white. |
| What is 3 plus 4? | 101. | 10. | 101. | 10. The first word is "boy." ... |

- The answers took the form of the chats, then the facts the base model read in pretraining came through in that
  form: the capital of Germany and a cat's legs are in no chat or pair.
- The model gets colors and sums wrong, and a longer answer drifts and repeats. 112 M parameters and a few hundred
  examples set those limits, not the stages.
- `giz ai serve $PWD/solved --port 8000` serves the same folder over the OpenAI API.

## When a stage fails

| What you see | Cause | Do this |
| --- | --- | --- |
| Out of GPU memory at the first step | The activations of `--rows` x `--context` tokens | Halve `--rows`, or add `--recompute` (a third more work) |
| DPO loss near 0, chats of one repeated word | Too many passes over few pairs | Fewer steps or lower rates; keep the step where the held-out pairs still improve |
| GRPO reward stays flat | Every completion of a problem scores the same, so no advantage | Problems the model gets right sometimes, more `--samples`, higher rates |
| `diverged 3 times from step N` | The held-out loss rose more than 5% three times | Lower rates; `FOLDER/best` holds the best model |
