GizStack
View .mdOpen in ChatGPTOpen in Claude

Training language models

giz ai tokens, giz ai train and giz ai score make a language model from nothing on your GPUs. A run writes a Hugging Face model folder, so giz ai chat and giz ai serve run it with no conversion. The model is one that the engine of Language models runs, and the trainer's logits equal the engine's on the same weights (a test of each layer form holds that).

Your taskSection
Make token shards from your textText to token shards
Train a modelTrain
Go on training a model from elsewherePost-train a model
Adapt a large model in little memoryTrain a LoRA adapter
Teach a model which answers people preferPreference pairs (DPO)
Take a model through every stage to a chatChat model lab
Choose the model's shapeShapes
Set the steps, the batch and the ratesThe plan
Train on several GPUs or hostsMany GPUs and hosts
Measure a modelScore
Chat with or serve the resultUse the result
Call the trainer from a programIn a program

Text to token shards

sh
giz ai tokens data web.parquet books.jsonl notes.txt --vocab 8192
  • A file is Parquet (a column text or content, or chat rows of question, response and system_prompt, read a row group at a time, so a file larger than memory packs), JSON lines with a text each, or plain text whose documents an <|endoftext|> or an empty line parts.
  • When data has no tokenizer.json, tokens first learns a byte-level BPE of --vocab tokens (8192 by default) from at most 32 MB of the files, an equal share from each. Each digit is a token of its own. The last three tokens are <|endoftext|>, <|im_start|> and <|im_end|>.
  • Then it packs every document as tokens of two bytes in data/train-N.tok, and one document in fifty of each file in data/held-N.tok. No held-out document teaches the tokenizer or a step. train.json and held.json say how many bytes of text the tokens were, for loss in bits a byte.
  • A shard set may mark tokens that a model reads but does not learn (quiet in its JSON: a chat's prompt and tool output). Those tokens are no target of the loss.

Train

sh
giz ai train data model 20000 --size 30m --rows 32
  • train DATA FOLDER STEPS trains for STEPS steps. With no model in FOLDER it starts a new one of the flags' shape; with one, it goes on from the step, weights and optimizer state that FOLDER holds, and the shape flags are not read.
  • Every --every steps (100) a line says the mean loss of those steps, the tokens a second and the TFLOP a second. Every --keep steps (2000) and at the end, the run measures the held-out loss and writes FOLDER. The folder is one generation: model.safetensors holds the weights, the optimizers' state, the step, the draw of the batches and the held-out loss, and is written last.
  • FOLDER/best holds the generation of the least held-out loss, as hard links (no copy). A held-out loss more than 5% above the best is a divergence: the run does not write it, goes back to the best, and goes on with other batches (the next draw). A run that diverges three times from one best ends with an error, and best keeps the model.
  • --cpu trains on the CPU. Otherwise the run takes the GPU that CUDA_VISIBLE_DEVICES shows first.

Post-train a model

A Qwen3, Qwen3.5 or LFM2 folder of one model.safetensors, in a form this trainer makes (the layer kinds of Shapes), goes on training as your own run does:

sh
mkdir data && cp base/tokenizer.json data/
giz ai tokens data chats.parquet
cp -r base model
giz ai train data model 2000 --rows 4 --context 1024 --muon 0 --adam 0.00002
  • The run starts at step 0 with new optimizer state, and the shape flags are not read: the folder's config says the shape. A generation writes only model.safetensors, so the folder keeps its own config, tokenizer and chat template.
  • The shards are in the model's tokenizer. A vocabulary of more than 65,536 tokens (Qwen's) takes four bytes a token. A chat file (messages, or Parquet chat rows) trains only on the assistant's turns.
  • --muon 0 moves every weight by AdamW. Muon orthogonalizes each matrix every step: for Qwen3-0.6B that is about 10 TFLOP a step whatever the batch, more than the model's own work on a few thousand tokens. Muon is for large batches.
  • Qwen3-0.6B (596 M parameters, a vocabulary of 151,936) post-trained at 1,060 tokens a second on a Radeon 8060S with --rows 2 --context 512 --muon 0. Every weight, its gradient and the optimizers' state stay on the card in f32: 16 bytes a parameter and more.
  • A LoRA adapter of rank 16 of it trained at 1,260 tokens a second with --rows 2 in 15.7 GB of the card; with --recompute, --rows 2 took 9.0 GB, and --rows 8 took 14.9 GB at 1,122 tokens a second.

Train a LoRA adapter

sh
giz ai train data adapter 2000 --lora 16 --base qwen --rows 4 --context 1024 --adam 0.0002
giz ai convert qwen served --lora adapter
giz ai serve /abs/path/served
  • --lora R trains an adapter of rank R on each hidden matrix W of the model in --base: the model computes x W + (x A) B, and only A and B learn, by AdamW. The base stays as it is and is read, never written.
  • FOLDER is a PEFT adapter of the base: adapter_config.json (alpha is R) and adapter_model.safetensors, which also holds the run's optimizer state and step under train., so a run goes on from it. giz ai convert --lora merges it into the base; PEFT reads it too. The query, key and value (and the gate and up) linears are one matrix here with one A: each of those modules gets that A and its own columns of B.
  • On a GPU the base's matrices stay in bf16, and no gradient or optimizer state of the base exists: about 2 bytes a parameter of the base, against 16 and more for full post-training. A step skips the base's weight gradients: 4 multiply-adds a parameter a token, not 6.
  • --base names a Qwen3, Qwen3.5 or LFM2 folder of the forms of Shapes; the shards are in its tokenizer.
  • The adapter serves as it is, beside other adapters and none: giz ai serve qwen --lora tuned=adapter runs a request whose model is tuned with it (LLM guide).

A LoRA of an image model

sh
giz ai train pictures style 2000 --lora 16 --base z-image-turbo --side 768 --adam 0.0001
giz ai image "zxq style, a lighthouse at dusk" --model z-image-turbo --lora style
  • DATA is a folder of pictures (.png, .jpg, .webp), each with its caption in NAME.txt beside it. Each picture is scaled to the area of a square of --side pixels (768 by default), its shape kept, and cut to multiples of 16.
  • The model's encoders read every picture and caption once, then leave the card to the model. A step is flow matching, as the model was trained: a picture's latents, noise at a sigma of a logit-normal draw moved by the model's shift, and the error of the model's velocity. --accumulate N takes the mean of N pictures a step.
  • The model's linears stay in their quantized form and train no gradient: a step makes each one's BF16 weights again for its product and its backward pass, and each block's values are made again in the backward pass. Only A and B and AdamW's state take memory beside the model: Z-Image Turbo trains on a 16 GB GPU.
  • FOLDER is a PEFT adapter of the model's linears by their file names (ComfyUI's), with the run's state, so a run goes on from it and gives what one run would. giz ai image --lora and an image request's loras read it.
  • --base names an image model of the catalog that trains a LoRA: z-image-turbo.

Preference pairs (DPO)

sh
giz ai train data adapter 500 --lora 16 --base qwen --pairs pairs.jsonl --rows 4 --context 1024 --adam 0.00005
  • --pairs FILE trains by DPO (direct preference optimization) on JSON lines of a prompt (a user's text) or messages (the chat before the answer), and the answers chosen and rejected, in the chat template's turns. DATA gives only the tokenizer.
  • The loss of a pair is -log sigmoid(beta ((c - c0) - (r - r0))): c and r are the log-likelihoods of the chosen and the other answer, c0 and r0 the reference's: the model in --base. With --lora the reference is the base without the adapter, so the run holds one model. Without it the run trains every weight of a copy of the base and holds the reference beside it (in bf16 on a GPU). --beta (0.1) sets how far the model may move from it.
  • A step reads --rows sequences: --rows / 2 pairs, the memory of --rows rows of pretraining. A line says the loss and the part of the pairs whose chosen answer the model prefers; every fiftieth pair is held out and measured every --keep steps. A new run starts at a loss of log 2.

Rewards on answers (GRPO)

sh
giz ai train data model 300 --base sft --tasks problems.jsonl --samples 8 --rows 64 --context 1024 --adam 0.00001
  • --tasks FILE trains by GRPO on JSON lines of a prompt or messages (as pairs') and the answer a right completion holds. DATA gives only the tokenizer.
  • Each step the engine writes --samples (8) completions of each of --rows / --samples problems, sampled at temperature 1, from the model as it trains: every --refresh steps (1) the trainer writes its weights over the engine's model in place (no file, no new engine). A completion that holds the answer earns 1, any other 0. The reward loses --beta (0.1) times the completion's log-likelihood over the reference's (the model in --base; --beta 0 needs none).
  • A completion's advantage is its reward over its problem's mean, in units of their spread (0 where all of them scored the same), and the step raises the log-likelihood of the completions above the mean and lowers the others', over their tokens. A line says the mean reward and the completions' length; every fiftieth problem is held out, and every --keep steps the run says the part of them whose greedy answer is right.
  • With --lora the adapter trains and the base is the reference, as in DPO.

Write facts into the weights

sh
giz ai edit model edited facts.jsonl --keep chats.jsonl --check checks.jsonl
  • edit MODEL OUT FACTS writes facts into a saved model with no training, to the new folder OUT. FACTS holds JSON lines of a prompt (or messages) and the answer the model should give.
  • OUT grows one last layer that starts as the identity. Each answer token the model gets wrong takes one cell of its MLP, which turns on where the stream is like that token's. The layer's down linear is solved on the GPU so that each such token leads by 4 logits, and nothing changes at the tokens of --keep (JSON lines of a prompt, messages or text) and of the model's own answers to them.
  • A report gives the facts answered right (the greedy answer starts with the fact) before and after, the --check lines (a prompt and an answer text the greedy answer holds) before and after, and the kept chats whose answer changed.
  • The facts must not outnumber the cells of the MLP: the answer tokens to change are at most ffn.

Shapes

FlagDefaultWhat it sets
--size30mA named shape: 10m (dim 384, 6 layers), 30m (512, 8), 50m (512, 14), 100m (768, 14).
--dim, --layers, --ffnof the sizeWidth, depth, MLP width. dim, ffn and the attention's width are multiples of 128.
--heads, --kvof the sizeAttention heads, and key and value heads (GQA: fewer than heads).
--headdim / headsThe values of a head: 64, 128 or 256.
--mixaThe layers' mixers in turn: a attention, c LFM2's short causal convolution, d a gated delta rule (Qwen3.5's linear attention, heads of 128). cca repeats conv, conv, attention.
--gatedoffA sigmoid gate on each attention head's output.
--rotaryallThe head dimensions that turn by position (partial RoPE).
--uniquethe layersDistinct layers that the layers repeat in turn.
--tiedoffThe head is the embedding.

A model with c layers is saved as LFM2; with d layers or --gated, as Qwen3.5; else as Qwen3. Every layer has RMS norms before its mixer and its gated SiLU MLP, and normed query and key heads.

The plan

FlagDefaultWhat it sets
--context512The tokens a row reads; a new model reads at most that many.
--rows32Sequences of --context tokens a batch, for every objective: DPO's pairs take two each, GRPO's problems --samples.
--accumulate1Batches whose mean gradient one step takes: a larger batch in the memory of one card.
--recomputeoffEach layer keeps only its output, and the backward pass computes it again: the activations of one layer at a time, for a third more work.
--muon, --adam0.02, 0.002The rates. Muon moves the hidden matrices, Adam the embedding, the head, the norms and the small tensors. --muon 0 moves all by Adam.
--decay0Weight decay of the matrices, by their optimizer's rate a step.
--hold, --fall0.4, lineThe rates rise over 200 steps (a tenth of a shorter run), hold to that part of the run, then fall: line to a tenth, sqrt to nothing as 1 - sqrt of the fall gone.
--bag, --bagged1, 0For the first --bagged part of the steps, a place reads the mean embedding of --bag tokens and learns the next bag's tokens: --bag times the text for the same work. Then each place is one token again.
--futurenoneWeights (0.5,0.25) of the loss of the tokens two and three places on, to two thirds of the run.
--seed11The initial weights and the batches' draws.
--lora, --basenoneAn adapter of that rank of the model in that folder (LoRA). --base alone: a new folder starts as that model.
--pairs, --betanone, 0.1DPO on preference pairs (DPO).
--tasks, --samples, --refreshnone, 8, 1GRPO on problems with answers (GRPO); --beta is its KL penalty.

Many GPUs and hosts

A run on N GPUs is N processes of one group (gizai/dist): data parallel. Each rank draws its own rows, a step takes the mean of the ranks' gradients, and every rank starts from the root's weights and step. Only the root (rank 0) reads and writes FOLDER and says the lines, so the other ranks need neither the folder nor the root's files.

Each tensor's optimizer runs on one rank, its owner (ZeRO's first stage): the owner takes the group's sum of its gradients, steps, and sends its move to every rank. A rank holds the optimizer state of its own tensors only (Adam's two moments and Muon's momentum: a third to a half of a run's memory, divided by N) and runs only their steps (Muon's products, divided by N). The network carries as many bytes as a sum of the gradients. The root gathers the whole state when it writes FOLDER, so a run goes on from FOLDER on any number of ranks.

sh
# One host, two GPUs: rank 0 is the root.
CUDA_VISIBLE_DEVICES=0 giz ai train data model 20000 --ranks 2 --rank 0 --root 127.0.0.1:29500 --accumulate 4 &
CUDA_VISIBLE_DEVICES=1 giz ai train data model 20000 --ranks 2 --rank 1 --root 127.0.0.1:29500 --accumulate 4
  • The root listens at --root HOST:PORT, and the other ranks connect to it. On several hosts each rank names the root's address and reads its own copy of DATA.
  • The gradients go through the host and TCP once a step, as bf16: half the bytes of f32, around a ring of the ranks (each adds the part the one before sends it, in f32 on its GPU, then the sums go round). Each rank sends 2 (N - 1) / N of the gradients a step, so more ranks cost no rank more bytes. --accumulate makes a step of several batches, so they go once for all of them: on two GPUs of one host (RTX 4080 SUPER and 4070 Ti SUPER), a 17M model trained at 144k tokens a second with --accumulate 4, against 113k on the faster card alone.
  • --rate bounds the bytes a second each rank sends, for a network that others use. Across two hosts on 1 Gbit/s Ethernet held to 300 Mbit/s, the same model trained at 64k tokens a second with --accumulate 8: the network, not the GPUs, sets that pace, so a larger --accumulate goes faster there.
  • --sync H sends the model once per H steps, not the gradients each step (DiLoCo). Each rank takes H steps alone, with its own optimizers. Then the group takes the mean of the ranks' moves since the last sync, and moves the model of that sync by it, with Nesterov momentum (rate 0.7, momentum 0.9). The network carries H times fewer bytes, so ranks on a slow network or on different hosts keep their GPUs busy; H of 50 to 500 is usual. The run syncs before it measures and writes the folder.
  • The ranks start in any order: a rank waits up to 10 minutes for the root to listen. A rank that does not answer within 10 minutes, or closes, is lost: every other rank meets the group again at --root, and the run goes on from FOLDER's last generation (the steps since it are taken again), at most --retries (3) times. Restart the lost rank's process (a service manager does) and it joins them; a run of one rank ends at its first failure.
  • Set CUDA_DEVICE_ORDER=PCI_BUS_ID when the GPUs differ: CUDA numbers the fastest first, nvidia-smi by bus.

Score

sh
giz ai score model --held data blimp/*.jsonl arc-easy.jsonl
  • --held DATA: the loss on DATA's held-out shards, in nats a token and bits a byte, over 512 batches of 32 rows (8.4 M tokens at 512), the same ones each time. The 131 k tokens of a training line differ by 2% from draw to draw; compare models by this.
  • Each FILE is JSON lines, each a minimal pair (sentence_good, sentence_bad: the likelier whole sentence wins, as BLiMP's files) or a question (context, endings, label: the ending with the highest log probability a byte wins, as lm-eval's acc_norm). It reads them through the engine that serves the model.

Use the result

sh
giz ai chat /abs/path/model "Hello"
giz ai serve /abs/path/model --port 8000

A NAME that is a folder's absolute path runs it (Language models). The folder's tokenizer has a chat template whose tool calls the server reads back.

In a program

ModuleWhat it holds
gizai/llm/corpusDocuments of files, tokenizer and pack.
gizai/llm/learnShapes (Sizes), plans (Plan), setup of a run's flags and run: one loop for pretraining, DPO and GRPO. The Trainer of a language model around its Learner (learn, prefer, reinforce, heldLoss, keep, merged, and refresh: its weights over an engine's model in place), preference pairs (pairs), problems with answers (tasks), token shards (Shards), and the saved folder (save, kept).
gizai/llm/scoreheldOut and choices.
gizai/nnThe step of any model: learner of a Run (parameters, optimizer state, steps) with its Rates, Learner.descend (accumulation, the group's mean, Muon and AdamW), Learner.state, and shared (the root's run on every rank).
gizai/distGroups of ranks: join, Group.sum, Group.broadcast.
giz
import * as Learn from "gizai/llm/learn";

const run = Learn.run(Learn.setup(["data", "model", "1000", "--size", "10m"]), (line: string): void => { console.log(line); });