gizai/llm/learn
a language model trained from nothing, in the form the engine already runs. The model is Qwen3's (gizai/llm/recipes): RMS norms before the attention and the MLP, rotary positions, RMS norms of each query and key head, a gated SiLU MLP, no bias. train writes it as a Hugging Face folder (config.json, model.safetensors, tokenizer.json), so gizai/llm loads it, generates, chats and serves with no code of its own for this model, and so does any other program that reads Qwen3.
What is the trainer's own, and leaves the model's function as the engine computes it: - the hidden matrices move by Muon (gizai/nn), the embedding, the head and the norms by Adam; - the output projections of the attention and the MLP start at zero, so each layer starts as the identity; - unique layers may be fewer than layers: the layers then repeat them in turn (one set of weights computes several times), and the folder holds each layer's copy; - future weights also teach the logits of a place the tokens after the next one, a weight a distance.
import * as Learn from "gizai/llm/learn";Types
Sizes
type Sizes = { vocab: i64; context: i64; dim: i64; layers: i64; unique: i64; heads: i64; kv: i64; ffn: i64; tied: bool; gated: bool; mix: string; rotary: i64; head: i64; theta: f64; }A model's sizes. heads heads of head values (64, 128 or 256; dim / heads in the named sizes) over kv key
and value heads, turned by position with base theta; ffn the
MLP's width; unique distinct layers that layers repeat in turn; tied: the head is the embedding; gated: each
head's output is times a sigmoid gate the query linear also makes (Qwen3.5's attention, saved as that family).
mix names the distinct layers' mixers in turn: "a" attention, "c" a short causal convolution (LFM2's, saved as
that family, with no gate), "d" a gated delta rule of heads of DELTA_HEAD (Qwen3.5's linear attention, saved as
that family). rotary of a head's dimensions turn by position (0: all). dim, ffn and the attention's width are
multiples of 128 (the engine's blocks).
vocab:i64context:i64dim:i64layers:i64unique:i64heads:i64kv:i64ffn:i64tied:boolgated:boolmix:stringrotary:i64head:i64theta:f64
Weights
type Weights = { p: Vec<Var>; low: Vec<Vec<Var>>; scale: f64; }A model as values of a graph: p are init's parameters; low[i], when not empty, is LoRA's adapter [A, B] of
the hidden matrix p[i], A (inputs, rank) and B (rank, outputs), whose product times scale adds to it.
p:Vec<M19.Var>low:Vec<Vec<M19.Var>>scale:f64
Generation
type Generation = { step: i64; draw: i64; held: Option<f64>; }Where a generation stands in its run: its step, the draw of its batches (a rollback draws others: Trainer.steps) and its held-out loss (undefined when none was measured). Its weights file's metadata holds it.
step:i64draw:i64held:_gizlang_sdk_core.Option<f64>
Kept
type Kept = { z: Sizes; params: Vec<Tensor>; at: Generation; run: Vec<TensorCore.Named>; }What a folder that save wrote holds: the sizes, the parameters in init's order, its generation (Generation) and the
run's tensors (none and generation 0 for a model no run of this module wrote).
z:M66.Sizesparams:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>at:M66.Generationrun:Vec<M53.Named>
Shards
type Shards = { files: Vec<i64>; sizes: Vec<i64>; tokens: i64; bytes: i64; between: i64; quiet: i64; width: i64; }Tokens in files: each NAME-N.tok of a folder is tokens of width bytes (two, or four for a vocabulary of more
than 65,536 tokens; the low byte first), one after another, and NAME.json says the width, how many bytes of text
they were, which token ends a document, and quiet (0: none): a token
of quiet or more is token - quiet, read by the model but no target of the loss (a chat's prompt and tool output,
where the model learns only its own turns). The files stay on the disk: a batch reads its runs from them, so text of
any size trains in the memory of one batch.
files:Vec<i64>sizes:Vec<i64>tokens:i64bytes:i64between:i64quiet:i64width:i64
Plan
type Plan = { rows: i64; steps: i64; muon: f64; adam: f64; decay: f64; future: Vec<f64>; every: i64; keep: i64; seed: u64; hold: f64; root: bool; bag: i64; bagged: f64; accumulate: i64; context: i64; lora: i64; base: string; pairs: string; beta: f64; recompute: bool; sync: i64; tasks: string; samples: i64; refresh: i64; }A training run: steps steps of rows contexts of context tokens (0: the model's length). The rates are Muon's (0: every parameter
moves by Adam) and Adam's; decay shrinks the matrices, by their optimizer's rate times it a step; both rise over
the first 200 steps, hold to hold of the run and then fall to a tenth along a line, or with root to nothing as
1 - sqrt of the part of the fall gone. For the first bagged of the steps a place reads the mean embedding of
bag tokens and learns the bag tokens of the next place, each a share of the loss (bag training: bag times
the text for the same work); then each place is one token again. A step takes the mean gradient of accumulate
batches of rows (a larger batch in the memory of one). future[i] weighs the token i + 2 places on in the loss, and falls to
nothing by two thirds of the run. Every every steps a line is said, every keep steps the held-out loss is
measured and the folder is written. With lora above 0 the run trains LoRA adapters of that rank on the hidden
matrices of the model in base, which stays as it is, and the folder is the adapter (LoRA above). With pairs (a
JSON lines file, Pairs) the steps are DPO's on its preference pairs at strength beta, the base the reference;
with tasks (a JSON lines file, Problem) GRPO's on samples completions of each task, the engine taking the model
every refresh steps, a KL penalty of beta to the base (Trainer.reinforce). One loop takes every kind of run
(Trainer.steps).
recompute keeps one layer's activations at a time (logits). In a group each rank takes sync steps alone between
two syncs (Learner).
rows:i64steps:i64muon:f64adam:f64decay:f64future:Vec<f64>every:i64keep:i64seed:u64hold:f64root:boolbag:i64bagged:f64accumulate:i64context:i64lora:i64base:stringpairs:stringbeta:f64recompute:boolsync:i64tasks:stringsamples:i64refresh:i64
Trainer
type Trainer = { a: Arch; z: Sizes; learner: nn.Learner; base: Vec<Tensor>; reference: Vec<Tensor>; fixed: Vec<Tensor>; t: i64; rate: f64; recompute: bool; draw: i64; best: Option<f64>; }A model in training: its Learner (gizai/nn: the parameters, the optimizers, the steps and the group whose ranks
train it as one), the base model that a LoRA run adapts (none: the learner's parameters are the model), its tables
for contexts of t tokens where it trains, the tokens a second of train's last line (0 before one), the draw of
its batches (Generation) and the held-out loss of its best generation (the folder's best; none before one).
a:_gizlang_lib_ai__llm__shape.Archz:M66.Sizeslearner:M22.Learnerbase:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>reference:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>fixed:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>t:i64rate:f64recompute:booldraw:i64best:_gizlang_sdk_core.Option<f64>
Setup
type Setup = { data: string; tokenizer: string; folder: string; z: Sizes; plan: Plan; cpu: bool; ranks: i64; rank: i64; host: string; port: i64; rate: i64; retries: i64; }A run: its token shards (data, with their tokenizer.json), its folder, its steps, the sizes of a new model, the
plan, where it trains, and its place in a group.
data:stringtokenizer:stringfolder:stringz:M66.Sizesplan:M66.Plancpu:boolranks:i64rank:i64host:stringport:i64rate:i64retries:i64
Pair
type Pair = { prompt: Vec<i64>; chosen: Vec<i64>; rejected: Vec<i64>; }A preference pair as tokens: a prompt the model reads and two answers to it, the one people chose and the other.
prompt:Vec<i64>chosen:Vec<i64>rejected:Vec<i64>
Problem
type Problem = { prompt: Vec<i64>; answer: string; }A problem as tokens: a prompt the model reads (the chat template's turns, as a Pair's) and the text a right answer holds.
prompt:Vec<i64>answer:string
Grown
type Grown = { z: Sizes; params: Vec<Tensor>; }A model with one more distinct layer.
z:M66.Sizesparams:Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>
Functions
sizes
function sizes(name: string, vocab: i64, context: i64): Sizes throws stringThe sizes by name: "10m", "30m", "50m" or "100m" parameters near a vocabulary of 8,192.
Effects: fail, alloc.
convolved
function convolved(z: Sizes): boolWhether the model is LFM2's: some layer mixes by a short convolution.
Effects: fail.
config
function config(z: Sizes, ends: i64, pad: i64): stringThe config.json of a model of these sizes: Qwen3's; with gated or delta rules Qwen3.5's; with short convolutions
LFM2's. ends is the token that ends a turn, pad the one between documents.
Effects: fail, alloc.
arch
function arch(z: Sizes): Arch throws stringWhat the engine computes for these sizes: the Arch of their config.json.
Effects: fail, read, write, console, io, alloc.
init
function init(k: Key, z: Sizes): Vec<Tensor>The parameters, in order: the embedding (vocab, dim); each distinct layer's tensors; the last norm; the head (dim, vocab) when it is not tied. A linear is (inputs, outputs), as a row times it. Linears are normal(0, 0.02), the mixers' output and the MLP's down linears and the head zero, norms one, a convolution's taps normal(0, 0.02); a delta rule's A_log is log of 1 to 16 over the heads and dt_bias the softplus inverse of 0.001 to 0.1, as Mamba's.
Effects: fail, read, write, console, io, alloc.
count
function count(params: Vec<Tensor>): i64How many numbers the parameters hold.
Effects: fail.
tables
function tables(a: Arch, t: i64): Vec<Tensor>What the graph reads beside the parameters for contexts of t tokens: the cosines and the sines (t, head) of the
rotary turns, pairs i and i + rotary / 2 of the first rotary dimensions as the engine's (the others stay).
Effects: fail, alloc.
logits
function logits(g: mut Graph, a: Arch, w: Weights, f: Vec<Var>, ids: Vec<i64>, rows: i64, t: i64, bag: i64 = 1, recompute: bool = false): VarLogits (rows * t, vocab) of the token ids (rows of t), as the engine computes them for the model's folder.
w are the model's values and f tables' tensors, as values of g. With bag above 1, ids are rows of
t bags of bag tokens, and a place reads the mean of its bag's embeddings (training only: the engine reads one
token a place). With recompute each layer keeps only its output, and the backward pass computes it again
(Graph.checkpoint): the activations of one layer at a time, for a third more work.
Effects: fail, read, write, console, io, alloc.
read
function read(a: Arch, params: Vec<Tensor>, fixed: Vec<Tensor>, ids: Vec<i64>, rows: i64, t: i64): TensorThe logits on a new graph, in bf16 products on a GPU: for a reader that takes no gradient.
Effects: fail, read, write, console, io, alloc.
generationOf
function generationOf(path: string): GenerationThe generation of the weights file at path: step 0 and draw 0 for a file no run wrote.
Effects: fail, read, io, alloc.
save
function save(folder: string, z: Sizes, params: Vec<Tensor>, tokenizer: string, run: Vec<TensorCore.Named> = [], at: Generation = { step: 0, draw: 0, held: undefined }): void throws stringWrites the model as a Hugging Face folder of Qwen3: tokenizer is its tokenizer.json's text, with SPECIALS as its
last tokens. The folder is made when it is not there; the config and the tokenizer are written only then, so a
model from elsewhere (post-training) keeps its own. run are a training run's own tensors (the optimizers')
at generation at: they go in model.safetensors beside the weights, under train., so the weights and the run
that made them are one file, replaced whole (the last file written). A reader of the model does not read them.
Effects: fail, read, write, console, io, alloc.
kept
function kept(folder: string): Option<Kept> throws stringThe model of folder, or none when the folder holds no weights.
Effects: fail, read, write, console, io, alloc.
shards
function shards(folder: string, name: string): Shards throws stringThe shards name of folder, open while the program runs.
Effects: fail, read, io, alloc.
shard
function shard(folder: string, name: string, part: i64, tokens: Vec<i64>, width: i64 = 2): void throws stringWrites tokens as part part of the shards name of folder, each in width bytes.
Effects: fail, read, write, console, io, alloc.
shardBytes
function shardBytes(folder: string, name: string, size: i64, between: i64, quiet: i64 = 0, width: i64 = 2): void throws stringSays of the shards name of folder how many bytes of text they were, the token that ends a document, where
the tokens that are no target start (quiet, Shards) and the bytes of a token.
Effects: fail, read, write, console, io, alloc.
plan
function plan(steps: i64): PlanThe plan that is this module's first choice for steps steps.
trainer
function trainer(folder: string, z: Option<Sizes>, p: Plan, gpu: Option<Gpu>, group: Group = Dist.alone(), draws: i64 = 0): Trainer throws stringThe trainer of the model in folder, from where its run stopped (a model no run of this module wrote starts at
step 0 with new optimizers: post-training), or of a new model of z when the folder holds none. p gives the
optimizers' rates and the seed. In a group only the root reads the folder; every rank starts from the root's weights
and optimizer state (nn.shared), so the other ranks need neither the folder nor the root's files. A LoRA run (Plan
lora) reads its model from base and its adapter from folder. Its batches are the folder's draw, at least
draws (a rollback's: Learn.run).
Effects: fail, read, write, console, io, alloc.
bitsOf
function bitsOf(nats: f64, held: Shards): f64A loss of nats a token on held as bits a byte of its text: what compares models of other tokenizers.
Effects: fail.
setup
function setup(given: Vec<string>): Setup throws stringThe run that DATA FOLDER STEPS and the flags of given name.
Effects: fail, read, write, io, alloc.
run
function run(s: Setup, said: (line: string) => void): Trainer throws stringTrains the run of s (train): it joins its group first, and said takes the root's lines. When the group loses
a rank, every rank meets again (Dist.join) and the run goes on from the folder's last generation, at most
retries times.
Effects: fail, read, write, console, io, alloc.
pairs
function pairs(tok: Tok.Tokenizer, path: string): Vec<Pair> throws stringThe pairs of a JSON lines file in tok: each line a prompt (a user's text) or messages (a chat before the
answer, role and content each), and the assistant's answers chosen and rejected, in the chat template's turns.
Every fiftieth pair is held out (Pairs.held).
Effects: fail, read, write, io, alloc.
tasks
function tasks(tok: Tok.Tokenizer, path: string): Vec<Problem> throws stringThe tasks of a JSON lines file in tok: each line a prompt or messages (as pairs'), and its answer.
Effects: fail, read, write, io, alloc.
chats
function chats(tok: Tok.Tokenizer, path: string): Vec<Vec<i64>> throws stringThe chats of a JSON lines file in tok: each line a prompt (as pairs': a user's turn and the assistant's
opening), messages (every turn, role and content each) or text.
Effects: fail, read, write, io, alloc.
lastNorm
function lastNorm(z: Sizes): i64The place in init's order of the last norm; the head follows it when it is not tied.
Effects: fail, alloc.
grown
function grown(z: Sizes, params: Vec<Tensor>, gateUp: Tensor, down: Tensor): Grown throws stringThe model z, params with one more attention layer at the end that adds only what its MLP adds: gateUp (dim,
2 ffn) and down (ffn, dim). Its other tensors are init's, so its attention adds nothing (its output is 0).
Effects: fail, read, write, console, io, alloc.
Methods
Shards.batch
function Shards.batch(self: Shards, seed: u64, rows: i64, t: i64, ahead: i64): Vec<Vec<i64>> throws stringrows runs of t tokens drawn by seed: a run's tokens, then for each of ahead distances the tokens that far
on (1: the next token). A run starts at a place drawn evenly over the places a whole run fits at, each shard by how
many it has, and then at the first document that starts within t tokens of it, when one does: a context reads a
document from its start, as a reader of the model does. A quiet token (Shards) is its token in the run and -1 as a
token after (no target: gizai/grad crossEntropy). The same seed gives the same runs.
Effects: fail, read, io, alloc.
Trainer.merged
function Trainer.merged(self: Trainer): Vec<Tensor>The model's parameters in init's order where the model trains: a LoRA run's base with each adapter merged,
W + A B, in F32.
Effects: fail, read, write, console, io, alloc.
Trainer.learn
function Trainer.learn(self: mut Trainer, drawn: Vec<Vec<i64>>, weights: Vec<f64>, rows: i64, bag: i64 = 1): TensorOne step on rows of tokens: drawn are the tokens, the tokens after them, and for each of weights the tokens
one more place on (Shards.batch). The loss of the next token as a tensor where the model trains (read it only to say
it: reading waits for the GPU). With bag above 1 the rows are bags of tokens (Plan), drawn holds the tokens up
to bag places on, and the loss is the mean over the next bag's tokens.
Effects: fail, read, write, console, io, alloc.
Trainer.heldLoss
function Trainer.heldLoss(self: Trainer, held: Shards, rows: i64, batches: i64): f64 throws stringThe mean loss in nats a token of batches held-out batches of rows contexts (the same ones each time). A
held-out part too short for a context (a few chats: one document in fifty) is read in rows as long as its longest
shard holds (Shards.batch draws two rows' span).
Effects: fail, read, write, console, io, alloc.
Trainer.keep
function Trainer.keep(self: Trainer, folder: string, tokenizer: string, held: Option<f64> = undefined): void throws stringWrites the model in folder as save does, or a LoRA run's adapter, with the run's state in it: one generation,
whole or not there, of held-out loss held.
Effects: fail, read, write, console, io, alloc.
Trainer.refresh
function Trainer.refresh(self: Trainer, e: mut Engine): void throws stringWrites the model as it trains over the model of engine e (one of a folder this run wrote: the same config) in
place, and empties its slots (Model.refresh, Engine.forget). On the card each linear goes from the trainer's F32
buffer to the engine's by a kernel; the embedding and the small tensors go through the host.
Effects: fail, read, write, console, io, alloc.
Trainer.prefer
function Trainer.prefer(self: mut Trainer, ps: Vec<Pair>, beta: f64, step: bool = true): Vec<f64> throws stringDPO on pairs ps (Rafailov et al. 2023): the loss is -log sigmoid(beta ((c - c0) - (r - r0))) for the log-likelihoods
c and r of the chosen and the other answer, and c0 and r0 the base's (the reference: the model without its
adapters). One step when step, else only the measure. Says the mean loss and the part of the pairs whose chosen
answer the model prefers.
Effects: fail, read, write, console, io, alloc.
Trainer.reinforce
function Trainer.reinforce(self: mut Trainer, ts: Vec<Problem>, made: Vec<Vec<i64>>, rewards: Vec<f64>, k: i64, beta: f64, step: bool = true): Vec<f64> throws stringReinforcement on verifiable rewards (GRPO: Shao et al. 2024): made holds k completions of each of the tasks
ts in turn, and rewards theirs. With beta above 0 each reward loses beta times the completion's log-likelihood
over the reference's (the model in --base): the KL penalty in the reward. A completion's advantage is its reward
over its task's mean, in units of their spread (0 where its task's completions all scored the same), and the loss is
minus the advantages times the completions' log-likelihoods, over the completions' tokens. One step when step.
Says the mean reward and the completions' mean length.
Effects: fail, read, write, console, io, alloc.
Constants
SPECIALS
const SPECIALS: Vec<string> = ["<|endoftext|>", "<|im_start|>", "<|im_end|>"]The tokens a tokenizer of this module ends with: between documents, and around a chat's turn. Every text of this module that marks a turn is made of them.
TRAIN_FLAGS
const TRAIN_FLAGS: Vec<string> = ["--size", "--dim", "--layers", "--ffn", "--heads", "--head", "--kv", "--mix", "--rotary", "--unique", "--context", "--rows", "--accumulate", "--muon", "--adam", "--decay", "--hold", "--fall", "--bag", "--bagged", "--future", "--seed", "--every", "--keep", "--lora", "--base", "--pairs", "--beta", "--tasks", "--samples", "--refresh", "--ranks", "--rank", "--root", "--rate", "--sync", "--retries", "--side"]The flags of a training run, each with a value (giz ai train, as --rows 16): the sizes (--size 10m, 30m, 50m
or 100m, then --dim, --layers, --ffn, --heads, --head, --kv, --mix, --rotary, --unique), the plan
(--context, --rows, --accumulate, --muon, --adam, --decay, --hold, --fall line or sqrt, --bag, --bagged,
--future weights as 0.5,0.25, --seed, --every, --keep, --lora rank, --base folder, --pairs file,
--beta, --tasks file, --samples, --refresh; an image model's LoRA run: --side pixels) and the group (--ranks, --rank, --root
HOST:PORT, --rate bytes a second, --sync steps, --retries after a lost rank).
TRAIN_SWITCHES
const TRAIN_SWITCHES: Vec<string> = ["--gated", "--tied", "--cpu", "--recompute"]The flags of a training run without a value: --gated attention, a --tied head, --cpu, and --recompute.