Training language models
giz ai tokens, giz ai train and giz ai score make a language model from nothing on your GPUs. A run writes a
Hugging Face model folder, so giz ai chat and giz ai serve run it with no conversion. The model is one that the
engine of Language models runs, and the trainer's logits equal the engine's on the same weights (a test of
each layer form holds that).
| Your task | Section |
|---|---|
| Make token shards from your text | Text to token shards |
| Train a model | Train |
| Go on training a model from elsewhere | Post-train a model |
| Adapt a large model in little memory | Train a LoRA adapter |
| Teach a model which answers people prefer | Preference pairs (DPO) |
| Take a model through every stage to a chat | Chat model lab |
| Choose the model's shape | Shapes |
| Set the steps, the batch and the rates | The plan |
| Train on several GPUs or hosts | Many GPUs and hosts |
| Measure a model | Score |
| Chat with or serve the result | Use the result |
| Call the trainer from a program | In a program |
Text to token shards
giz ai tokens data web.parquet books.jsonl notes.txt --vocab 8192- A file is Parquet (a column
textorcontent, or chat rows ofquestion,responseandsystem_prompt, read a row group at a time, so a file larger than memory packs), JSON lines with atexteach, or plain text whose documents an<|endoftext|>or an empty line parts. - When
datahas notokenizer.json,tokensfirst learns a byte-level BPE of--vocabtokens (8192 by default) from at most 32 MB of the files, an equal share from each. Each digit is a token of its own. The last three tokens are<|endoftext|>,<|im_start|>and<|im_end|>. - Then it packs every document as tokens of two bytes in
data/train-N.tok, and one document in fifty of each file indata/held-N.tok. No held-out document teaches the tokenizer or a step.train.jsonandheld.jsonsay how many bytes of text the tokens were, for loss in bits a byte. - A shard set may mark tokens that a model reads but does not learn (
quietin its JSON: a chat's prompt and tool output). Those tokens are no target of the loss.
Train
giz ai train data model 20000 --size 30m --rows 32train DATA FOLDER STEPStrains for STEPS steps. With no model in FOLDER it starts a new one of the flags' shape; with one, it goes on from the step, weights and optimizer state that FOLDER holds, and the shape flags are not read.- Every
--everysteps (100) a line says the mean loss of those steps, the tokens a second and the TFLOP a second. Every--keepsteps (2000) and at the end, the run measures the held-out loss and writes FOLDER. The folder is one generation:model.safetensorsholds the weights, the optimizers' state, the step, the draw of the batches and the held-out loss, and is written last. FOLDER/bestholds the generation of the least held-out loss, as hard links (no copy). A held-out loss more than 5% above the best is a divergence: the run does not write it, goes back to the best, and goes on with other batches (the next draw). A run that diverges three times from one best ends with an error, andbestkeeps the model.--cputrains on the CPU. Otherwise the run takes the GPU thatCUDA_VISIBLE_DEVICESshows first.
Post-train a model
A Qwen3, Qwen3.5 or LFM2 folder of one model.safetensors, in a form this trainer makes (the layer kinds of
Shapes), goes on training as your own run does:
mkdir data && cp base/tokenizer.json data/
giz ai tokens data chats.parquet
cp -r base model
giz ai train data model 2000 --rows 4 --context 1024 --muon 0 --adam 0.00002- The run starts at step 0 with new optimizer state, and the shape flags are not read: the folder's config says the
shape. A generation writes only
model.safetensors, so the folder keeps its own config, tokenizer and chat template. - The shards are in the model's tokenizer. A vocabulary of more than 65,536 tokens (Qwen's) takes four bytes a token.
A chat file (
messages, or Parquet chat rows) trains only on the assistant's turns. --muon 0moves every weight by AdamW. Muon orthogonalizes each matrix every step: for Qwen3-0.6B that is about 10 TFLOP a step whatever the batch, more than the model's own work on a few thousand tokens. Muon is for large batches.- Qwen3-0.6B (596 M parameters, a vocabulary of 151,936) post-trained at 1,060 tokens a second on a Radeon 8060S with
--rows 2 --context 512 --muon 0. Every weight, its gradient and the optimizers' state stay on the card in f32: 16 bytes a parameter and more. - A LoRA adapter of rank 16 of it trained at 1,260 tokens a second with
--rows 2in 15.7 GB of the card; with--recompute,--rows 2took 9.0 GB, and--rows 8took 14.9 GB at 1,122 tokens a second.
Train a LoRA adapter
giz ai train data adapter 2000 --lora 16 --base qwen --rows 4 --context 1024 --adam 0.0002
giz ai convert qwen served --lora adapter
giz ai serve /abs/path/served--lora Rtrains an adapter of rank R on each hidden matrix W of the model in--base: the model computes x W + (x A) B, and only A and B learn, by AdamW. The base stays as it is and is read, never written.- FOLDER is a PEFT adapter of the base:
adapter_config.json(alpha is R) andadapter_model.safetensors, which also holds the run's optimizer state and step undertrain., so a run goes on from it.giz ai convert --loramerges it into the base; PEFT reads it too. The query, key and value (and the gate and up) linears are one matrix here with one A: each of those modules gets that A and its own columns of B. - On a GPU the base's matrices stay in bf16, and no gradient or optimizer state of the base exists: about 2 bytes a parameter of the base, against 16 and more for full post-training. A step skips the base's weight gradients: 4 multiply-adds a parameter a token, not 6.
--basenames a Qwen3, Qwen3.5 or LFM2 folder of the forms of Shapes; the shards are in its tokenizer.- The adapter serves as it is, beside other adapters and none:
giz ai serve qwen --lora tuned=adapterruns a request whosemodelistunedwith it (LLM guide).
A LoRA of an image model
giz ai train pictures style 2000 --lora 16 --base z-image-turbo --side 768 --adam 0.0001
giz ai image "zxq style, a lighthouse at dusk" --model z-image-turbo --lora style- DATA is a folder of pictures (
.png,.jpg,.webp), each with its caption in NAME.txt beside it. Each picture is scaled to the area of a square of--sidepixels (768 by default), its shape kept, and cut to multiples of 16. - The model's encoders read every picture and caption once, then leave the card to the model. A step is flow
matching, as the model was trained: a picture's latents, noise at a sigma of a logit-normal draw moved by the
model's shift, and the error of the model's velocity.
--accumulate Ntakes the mean of N pictures a step. - The model's linears stay in their quantized form and train no gradient: a step makes each one's BF16 weights again for its product and its backward pass, and each block's values are made again in the backward pass. Only A and B and AdamW's state take memory beside the model: Z-Image Turbo trains on a 16 GB GPU.
- FOLDER is a PEFT adapter of the model's linears by their file names (ComfyUI's), with the run's state, so a run goes
on from it and gives what one run would.
giz ai image --loraand an image request'slorasread it. --basenames an image model of the catalog that trains a LoRA: z-image-turbo.
Preference pairs (DPO)
giz ai train data adapter 500 --lora 16 --base qwen --pairs pairs.jsonl --rows 4 --context 1024 --adam 0.00005--pairs FILEtrains by DPO (direct preference optimization) on JSON lines of aprompt(a user's text) ormessages(the chat before the answer), and the answerschosenandrejected, in the chat template's turns. DATA gives only the tokenizer.- The loss of a pair is -log sigmoid(beta ((c - c0) - (r - r0))): c and r are the log-likelihoods of the chosen and the
other answer, c0 and r0 the reference's: the model in
--base. With--lorathe reference is the base without the adapter, so the run holds one model. Without it the run trains every weight of a copy of the base and holds the reference beside it (in bf16 on a GPU).--beta(0.1) sets how far the model may move from it. - A step reads
--rowssequences:--rows/ 2 pairs, the memory of--rowsrows of pretraining. A line says the loss and the part of the pairs whose chosen answer the model prefers; every fiftieth pair is held out and measured every--keepsteps. A new run starts at a loss of log 2.
Rewards on answers (GRPO)
giz ai train data model 300 --base sft --tasks problems.jsonl --samples 8 --rows 64 --context 1024 --adam 0.00001--tasks FILEtrains by GRPO on JSON lines of apromptormessages(as pairs') and theanswera right completion holds. DATA gives only the tokenizer.- Each step the engine writes
--samples(8) completions of each of--rows/--samplesproblems, sampled at temperature 1, from the model as it trains: every--refreshsteps (1) the trainer writes its weights over the engine's model in place (no file, no new engine). A completion that holds the answer earns 1, any other 0. The reward loses--beta(0.1) times the completion's log-likelihood over the reference's (the model in--base;--beta 0needs none). - A completion's advantage is its reward over its problem's mean, in units of their spread (0 where all of them scored
the same), and the step raises the log-likelihood of the completions above the mean and lowers the others', over
their tokens. A line says the mean reward and the completions' length; every fiftieth problem is held out, and every
--keepsteps the run says the part of them whose greedy answer is right. - With
--lorathe adapter trains and the base is the reference, as in DPO.
Write facts into the weights
giz ai edit model edited facts.jsonl --keep chats.jsonl --check checks.jsonledit MODEL OUT FACTSwrites facts into a saved model with no training, to the new folder OUT. FACTS holds JSON lines of aprompt(ormessages) and theanswerthe model should give.- OUT grows one last layer that starts as the identity. Each answer token the model gets wrong takes one cell of its
MLP, which turns on where the stream is like that token's. The layer's down linear is solved on the GPU so that each
such token leads by 4 logits, and nothing changes at the tokens of
--keep(JSON lines of aprompt,messagesortext) and of the model's own answers to them. - A report gives the facts answered right (the greedy answer starts with the fact) before and after, the
--checklines (apromptand ananswertext the greedy answer holds) before and after, and the kept chats whose answer changed. - The facts must not outnumber the cells of the MLP: the answer tokens to change are at most
ffn.
Shapes
| Flag | Default | What it sets |
|---|---|---|
--size | 30m | A named shape: 10m (dim 384, 6 layers), 30m (512, 8), 50m (512, 14), 100m (768, 14). |
--dim, --layers, --ffn | of the size | Width, depth, MLP width. dim, ffn and the attention's width are multiples of 128. |
--heads, --kv | of the size | Attention heads, and key and value heads (GQA: fewer than heads). |
--head | dim / heads | The values of a head: 64, 128 or 256. |
--mix | a | The layers' mixers in turn: a attention, c LFM2's short causal convolution, d a gated delta rule (Qwen3.5's linear attention, heads of 128). cca repeats conv, conv, attention. |
--gated | off | A sigmoid gate on each attention head's output. |
--rotary | all | The head dimensions that turn by position (partial RoPE). |
--unique | the layers | Distinct layers that the layers repeat in turn. |
--tied | off | The head is the embedding. |
A model with c layers is saved as LFM2; with d layers or --gated, as Qwen3.5; else as Qwen3. Every layer has
RMS norms before its mixer and its gated SiLU MLP, and normed query and key heads.
The plan
| Flag | Default | What it sets |
|---|---|---|
--context | 512 | The tokens a row reads; a new model reads at most that many. |
--rows | 32 | Sequences of --context tokens a batch, for every objective: DPO's pairs take two each, GRPO's problems --samples. |
--accumulate | 1 | Batches whose mean gradient one step takes: a larger batch in the memory of one card. |
--recompute | off | Each layer keeps only its output, and the backward pass computes it again: the activations of one layer at a time, for a third more work. |
--muon, --adam | 0.02, 0.002 | The rates. Muon moves the hidden matrices, Adam the embedding, the head, the norms and the small tensors. --muon 0 moves all by Adam. |
--decay | 0 | Weight decay of the matrices, by their optimizer's rate a step. |
--hold, --fall | 0.4, line | The rates rise over 200 steps (a tenth of a shorter run), hold to that part of the run, then fall: line to a tenth, sqrt to nothing as 1 - sqrt of the fall gone. |
--bag, --bagged | 1, 0 | For the first --bagged part of the steps, a place reads the mean embedding of --bag tokens and learns the next bag's tokens: --bag times the text for the same work. Then each place is one token again. |
--future | none | Weights (0.5,0.25) of the loss of the tokens two and three places on, to two thirds of the run. |
--seed | 11 | The initial weights and the batches' draws. |
--lora, --base | none | An adapter of that rank of the model in that folder (LoRA). --base alone: a new folder starts as that model. |
--pairs, --beta | none, 0.1 | DPO on preference pairs (DPO). |
--tasks, --samples, --refresh | none, 8, 1 | GRPO on problems with answers (GRPO); --beta is its KL penalty. |
Many GPUs and hosts
A run on N GPUs is N processes of one group (gizai/dist): data parallel. Each rank draws its own rows, a step takes
the mean of the ranks' gradients, and every rank starts from the root's weights and step. Only the root
(rank 0) reads and writes FOLDER and says the lines, so the other ranks need neither the folder nor the root's files.
Each tensor's optimizer runs on one rank, its owner (ZeRO's first stage): the owner takes the group's sum of its gradients, steps, and sends its move to every rank. A rank holds the optimizer state of its own tensors only (Adam's two moments and Muon's momentum: a third to a half of a run's memory, divided by N) and runs only their steps (Muon's products, divided by N). The network carries as many bytes as a sum of the gradients. The root gathers the whole state when it writes FOLDER, so a run goes on from FOLDER on any number of ranks.
# One host, two GPUs: rank 0 is the root.
CUDA_VISIBLE_DEVICES=0 giz ai train data model 20000 --ranks 2 --rank 0 --root 127.0.0.1:29500 --accumulate 4 &
CUDA_VISIBLE_DEVICES=1 giz ai train data model 20000 --ranks 2 --rank 1 --root 127.0.0.1:29500 --accumulate 4- The root listens at
--rootHOST:PORT, and the other ranks connect to it. On several hosts each rank names the root's address and reads its own copy of DATA. - The gradients go through the host and TCP once a step, as bf16: half the bytes of f32, around a ring of the ranks
(each adds the part the one before sends it, in f32 on its GPU, then the sums go round). Each rank sends
2 (N - 1) / N of the gradients a step, so more ranks cost no rank more bytes.
--accumulatemakes a step of several batches, so they go once for all of them: on two GPUs of one host (RTX 4080 SUPER and 4070 Ti SUPER), a 17M model trained at 144k tokens a second with--accumulate 4, against 113k on the faster card alone. --ratebounds the bytes a second each rank sends, for a network that others use. Across two hosts on 1 Gbit/s Ethernet held to 300 Mbit/s, the same model trained at 64k tokens a second with--accumulate 8: the network, not the GPUs, sets that pace, so a larger--accumulategoes faster there.--sync Hsends the model once per H steps, not the gradients each step (DiLoCo). Each rank takes H steps alone, with its own optimizers. Then the group takes the mean of the ranks' moves since the last sync, and moves the model of that sync by it, with Nesterov momentum (rate 0.7, momentum 0.9). The network carries H times fewer bytes, so ranks on a slow network or on different hosts keep their GPUs busy; H of 50 to 500 is usual. The run syncs before it measures and writes the folder.- The ranks start in any order: a rank waits up to 10 minutes for the root to listen. A rank that does not answer
within 10 minutes, or closes, is lost: every other rank meets the group again at
--root, and the run goes on from FOLDER's last generation (the steps since it are taken again), at most--retries(3) times. Restart the lost rank's process (a service manager does) and it joins them; a run of one rank ends at its first failure. - Set
CUDA_DEVICE_ORDER=PCI_BUS_IDwhen the GPUs differ: CUDA numbers the fastest first,nvidia-smiby bus.
Score
giz ai score model --held data blimp/*.jsonl arc-easy.jsonl--held DATA: the loss on DATA's held-out shards, in nats a token and bits a byte, over 512 batches of 32 rows (8.4 M tokens at 512), the same ones each time. The 131 k tokens of a training line differ by 2% from draw to draw; compare models by this.- Each FILE is JSON lines, each a minimal pair (
sentence_good,sentence_bad: the likelier whole sentence wins, as BLiMP's files) or a question (context,endings,label: the ending with the highest log probability a byte wins, as lm-eval'sacc_norm). It reads them through the engine that serves the model.
Use the result
giz ai chat /abs/path/model "Hello"
giz ai serve /abs/path/model --port 8000A NAME that is a folder's absolute path runs it (Language models). The folder's tokenizer has a chat template whose tool calls the server reads back.
In a program
| Module | What it holds |
|---|---|
gizai/llm/corpus | Documents of files, tokenizer and pack. |
gizai/llm/learn | Shapes (Sizes), plans (Plan), setup of a run's flags and run: one loop for pretraining, DPO and GRPO. The Trainer of a language model around its Learner (learn, prefer, reinforce, heldLoss, keep, merged, and refresh: its weights over an engine's model in place), preference pairs (pairs), problems with answers (tasks), token shards (Shards), and the saved folder (save, kept). |
gizai/llm/score | heldOut and choices. |
gizai/nn | The step of any model: learner of a Run (parameters, optimizer state, steps) with its Rates, Learner.descend (accumulation, the group's mean, Muon and AdamW), Learner.state, and shared (the root's run on every rank). |
gizai/dist | Groups of ranks: join, Group.sum, Group.broadcast. |
import * as Learn from "gizai/llm/learn";
const run = Learn.run(Learn.setup(["data", "model", "1000", "--size", "10m"]), (line: string): void => { console.log(line); });