# gizai/llm/learn

a language model trained from nothing, in the form the engine already runs. The model is Qwen3's (gizai/llm/recipes): RMS norms before the attention and the MLP, rotary positions, RMS norms of each query and key head, a gated SiLU MLP, no bias. `train` writes it as a Hugging Face folder (config.json, model.safetensors, tokenizer.json), so gizai/llm loads it, generates, chats and serves with no code of its own for this model, and so does any other program that reads Qwen3.

What is the trainer's own, and leaves the model's function as the engine computes it: - the hidden matrices move by Muon (gizai/nn), the embedding, the head and the norms by Adam; - the output projections of the attention and the MLP start at zero, so each layer starts as the identity; - `unique` layers may be fewer than `layers`: the layers then repeat them in turn (one set of weights computes several times), and the folder holds each layer's copy; - `future` weights also teach the logits of a place the tokens after the next one, a weight a distance.

```gizlang
import * as Learn from "gizai/llm/learn";
```

## Types

### `Sizes`

```gizlang
type Sizes = { vocab: i64; context: i64; dim: i64; layers: i64; unique: i64; heads: i64; kv: i64; ffn: i64; tied: bool; gated: bool; mix: string; rotary: i64; head: i64; theta: f64; }
```

A model's sizes. `heads` heads of `head` values (64, 128 or 256; dim / heads in the named sizes) over `kv` key
and value heads, turned by position with base `theta`; `ffn` the
MLP's width; `unique` distinct layers that `layers` repeat in turn; `tied`: the head is the embedding; `gated`: each
head's output is times a sigmoid gate the query linear also makes (Qwen3.5's attention, saved as that family).
`mix` names the distinct layers' mixers in turn: "a" attention, "c" a short causal convolution (LFM2's, saved as
that family, with no gate), "d" a gated delta rule of heads of DELTA_HEAD (Qwen3.5's linear attention, saved as
that family). `rotary` of a head's dimensions turn by position (0: all). `dim`, `ffn` and the attention's width are
multiples of 128 (the engine's blocks).

- `vocab`: `i64`
- `context`: `i64`
- `dim`: `i64`
- `layers`: `i64`
- `unique`: `i64`
- `heads`: `i64`
- `kv`: `i64`
- `ffn`: `i64`
- `tied`: `bool`
- `gated`: `bool`
- `mix`: `string`
- `rotary`: `i64`
- `head`: `i64`
- `theta`: `f64`

### `Weights`

```gizlang
type Weights = { p: Vec<Var>; low: Vec<Vec<Var>>; scale: f64; }
```

A model as values of a graph: `p` are `init`'s parameters; `low[i]`, when not empty, is LoRA's adapter [A, B] of
the hidden matrix p[i], A (inputs, rank) and B (rank, outputs), whose product times `scale` adds to it.

- `p`: `Vec<M19.Var>`
- `low`: `Vec<Vec<M19.Var>>`
- `scale`: `f64`

### `Generation`

```gizlang
type Generation = { step: i64; draw: i64; held: Option<f64>; }
```

Where a generation stands in its run: its step, the draw of its batches (a rollback draws others: Trainer.steps)
and its held-out loss (undefined when none was measured). Its weights file's metadata holds it.

- `step`: `i64`
- `draw`: `i64`
- `held`: `_gizlang_sdk_core.Option<f64>`

### `Kept`

```gizlang
type Kept = { z: Sizes; params: Vec<Tensor>; at: Generation; run: Vec<TensorCore.Named>; }
```

What a folder that `save` wrote holds: the sizes, the parameters in `init`'s order, its generation (Generation) and the
run's tensors (none and generation 0 for a model no run of this module wrote).

- `z`: `M66.Sizes`
- `params`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `at`: `M66.Generation`
- `run`: `Vec<M53.Named>`

### `Shards`

```gizlang
type Shards = { files: Vec<i64>; sizes: Vec<i64>; tokens: i64; bytes: i64; between: i64; quiet: i64; width: i64; }
```

Tokens in files: each `NAME-N.tok` of a folder is tokens of `width` bytes (two, or four for a vocabulary of more
than 65,536 tokens; the low byte first), one after another, and `NAME.json` says the width, how many bytes of text
they were, which token ends a document, and `quiet` (0: none): a token
of `quiet` or more is token - quiet, read by the model but no target of the loss (a chat's prompt and tool output,
where the model learns only its own turns). The files stay on the disk: a batch reads its runs from them, so text of
any size trains in the memory of one batch.

- `files`: `Vec<i64>`
- `sizes`: `Vec<i64>`
- `tokens`: `i64`
- `bytes`: `i64`
- `between`: `i64`
- `quiet`: `i64`
- `width`: `i64`

### `Plan`

```gizlang
type Plan = { rows: i64; steps: i64; muon: f64; adam: f64; decay: f64; future: Vec<f64>; every: i64; keep: i64; seed: u64; hold: f64; root: bool; bag: i64; bagged: f64; accumulate: i64; context: i64; lora: i64; base: string; pairs: string; beta: f64; recompute: bool; sync: i64; tasks: string; samples: i64; refresh: i64; }
```

A training run: `steps` steps of `rows` contexts of `context` tokens (0: the model's length). The rates are Muon's (0: every parameter
moves by Adam) and Adam's; `decay` shrinks the matrices, by their optimizer's rate times it a step; both rise over
the first 200 steps, hold to `hold` of the run and then fall to a tenth along a line, or with `root` to nothing as
1 - sqrt of the part of the fall gone. For the first `bagged` of the steps a place reads the mean embedding of
`bag` tokens and learns the `bag` tokens of the next place, each a share of the loss (bag training: `bag` times
the text for the same work); then each place is one token again. A step takes the mean gradient of `accumulate`
batches of `rows` (a larger batch in the memory of one). `future[i]` weighs the token i + 2 places on in the loss, and falls to
nothing by two thirds of the run. Every `every` steps a line is said, every `keep` steps the held-out loss is
measured and the folder is written. With `lora` above 0 the run trains LoRA adapters of that rank on the hidden
matrices of the model in `base`, which stays as it is, and the folder is the adapter (LoRA above). With `pairs` (a
JSON lines file, Pairs) the steps are DPO's on its preference pairs at strength `beta`, the base the reference;
with `tasks` (a JSON lines file, Problem) GRPO's on `samples` completions of each task, the engine taking the model
every `refresh` steps, a KL penalty of `beta` to the base (Trainer.reinforce). One loop takes every kind of run
(Trainer.steps).
`recompute` keeps one layer's activations at a time (logits). In a group each rank takes `sync` steps alone between
two syncs (Learner).

- `rows`: `i64`
- `steps`: `i64`
- `muon`: `f64`
- `adam`: `f64`
- `decay`: `f64`
- `future`: `Vec<f64>`
- `every`: `i64`
- `keep`: `i64`
- `seed`: `u64`
- `hold`: `f64`
- `root`: `bool`
- `bag`: `i64`
- `bagged`: `f64`
- `accumulate`: `i64`
- `context`: `i64`
- `lora`: `i64`
- `base`: `string`
- `pairs`: `string`
- `beta`: `f64`
- `recompute`: `bool`
- `sync`: `i64`
- `tasks`: `string`
- `samples`: `i64`
- `refresh`: `i64`

### `Trainer`

```gizlang
type Trainer = { a: Arch; z: Sizes; learner: nn.Learner; base: Vec<Tensor>; reference: Vec<Tensor>; fixed: Vec<Tensor>; t: i64; rate: f64; recompute: bool; draw: i64; best: Option<f64>; }
```

A model in training: its Learner (gizai/nn: the parameters, the optimizers, the steps and the group whose ranks
train it as one), the base model that a LoRA run adapts (none: the learner's parameters are the model), its tables
for contexts of `t` tokens where it trains, the tokens a second of `train`'s last line (0 before one), the draw of
its batches (Generation) and the held-out loss of its best generation (the folder's `best`; none before one).

- `a`: `_gizlang_lib_ai__llm__shape.Arch`
- `z`: `M66.Sizes`
- `learner`: `M22.Learner`
- `base`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `reference`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `fixed`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`
- `t`: `i64`
- `rate`: `f64`
- `recompute`: `bool`
- `draw`: `i64`
- `best`: `_gizlang_sdk_core.Option<f64>`

### `Setup`

```gizlang
type Setup = { data: string; tokenizer: string; folder: string; z: Sizes; plan: Plan; cpu: bool; ranks: i64; rank: i64; host: string; port: i64; rate: i64; retries: i64; }
```

A run: its token shards (`data`, with their tokenizer.json), its folder, its steps, the sizes of a new model, the
plan, where it trains, and its place in a group.

- `data`: `string`
- `tokenizer`: `string`
- `folder`: `string`
- `z`: `M66.Sizes`
- `plan`: `M66.Plan`
- `cpu`: `bool`
- `ranks`: `i64`
- `rank`: `i64`
- `host`: `string`
- `port`: `i64`
- `rate`: `i64`
- `retries`: `i64`

### `Pair`

```gizlang
type Pair = { prompt: Vec<i64>; chosen: Vec<i64>; rejected: Vec<i64>; }
```

A preference pair as tokens: a prompt the model reads and two answers to it, the one people chose and the other.

- `prompt`: `Vec<i64>`
- `chosen`: `Vec<i64>`
- `rejected`: `Vec<i64>`

### `Problem`

```gizlang
type Problem = { prompt: Vec<i64>; answer: string; }
```

A problem as tokens: a prompt the model reads (the chat template's turns, as a Pair's) and the text a right answer
holds.

- `prompt`: `Vec<i64>`
- `answer`: `string`

### `Grown`

```gizlang
type Grown = { z: Sizes; params: Vec<Tensor>; }
```

A model with one more distinct layer.

- `z`: `M66.Sizes`
- `params`: `Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>`

## Functions

### `sizes`

```gizlang
function sizes(name: string, vocab: i64, context: i64): Sizes throws string
```

The sizes by name: "10m", "30m", "50m" or "100m" parameters near a vocabulary of 8,192.

Effects: fail, alloc.

### `convolved`

```gizlang
function convolved(z: Sizes): bool
```

Whether the model is LFM2's: some layer mixes by a short convolution.

Effects: fail.

### `config`

```gizlang
function config(z: Sizes, ends: i64, pad: i64): string
```

The config.json of a model of these sizes: Qwen3's; with `gated` or delta rules Qwen3.5's; with short convolutions
LFM2's. `ends` is the token that ends a turn, `pad` the one between documents.

Effects: fail, alloc.

### `arch`

```gizlang
function arch(z: Sizes): Arch throws string
```

What the engine computes for these sizes: the Arch of their config.json.

Effects: fail, read, write, console, io, alloc.

### `init`

```gizlang
function init(k: Key, z: Sizes): Vec<Tensor>
```

The parameters, in order: the embedding (vocab, dim); each distinct layer's tensors; the last norm; the head
(dim, vocab) when it is not tied. A linear is (inputs, outputs), as a row times it. Linears are normal(0, 0.02),
the mixers' output and the MLP's down linears and the head zero, norms one, a convolution's taps normal(0, 0.02);
a delta rule's A_log is log of 1 to 16 over the heads and dt_bias the softplus inverse of 0.001 to 0.1, as Mamba's.

Effects: fail, read, write, console, io, alloc.

### `count`

```gizlang
function count(params: Vec<Tensor>): i64
```

How many numbers the parameters hold.

Effects: fail.

### `tables`

```gizlang
function tables(a: Arch, t: i64): Vec<Tensor>
```

What the graph reads beside the parameters for contexts of `t` tokens: the cosines and the sines (t, head) of the
rotary turns, pairs i and i + rotary / 2 of the first `rotary` dimensions as the engine's (the others stay).

Effects: fail, alloc.

### `logits`

```gizlang
function logits(g: mut Graph, a: Arch, w: Weights, f: Vec<Var>, ids: Vec<i64>, rows: i64, t: i64, bag: i64 = 1, recompute: bool = false): Var
```

Logits (rows * t, vocab) of the token `ids` (rows of t), as the engine computes them for the model's folder.
`w` are the model's values and `f` `tables`' tensors, as values of `g`. With `bag` above 1, `ids` are rows of
t bags of `bag` tokens, and a place reads the mean of its bag's embeddings (training only: the engine reads one
token a place). With `recompute` each layer keeps only its output, and the backward pass computes it again
(Graph.checkpoint): the activations of one layer at a time, for a third more work.

Effects: fail, read, write, console, io, alloc.

### `read`

```gizlang
function read(a: Arch, params: Vec<Tensor>, fixed: Vec<Tensor>, ids: Vec<i64>, rows: i64, t: i64): Tensor
```

The logits on a new graph, in bf16 products on a GPU: for a reader that takes no gradient.

Effects: fail, read, write, console, io, alloc.

### `generationOf`

```gizlang
function generationOf(path: string): Generation
```

The generation of the weights file at `path`: step 0 and draw 0 for a file no run wrote.

Effects: fail, read, io, alloc.

### `save`

```gizlang
function save(folder: string, z: Sizes, params: Vec<Tensor>, tokenizer: string, run: Vec<TensorCore.Named> = [], at: Generation = { step: 0, draw: 0, held: undefined }): void throws string
```

Writes the model as a Hugging Face folder of Qwen3: `tokenizer` is its tokenizer.json's text, with SPECIALS as its
last tokens. The folder is made when it is not there; the config and the tokenizer are written only then, so a
model from elsewhere (post-training) keeps its own. `run` are a training run's own tensors (the optimizers')
at generation `at`: they go in model.safetensors beside the weights, under `train.`, so the weights and the run
that made them are one file, replaced whole (the last file written). A reader of the model does not read them.

Effects: fail, read, write, console, io, alloc.

### `kept`

```gizlang
function kept(folder: string): Option<Kept> throws string
```

The model of `folder`, or none when the folder holds no weights.

Effects: fail, read, write, console, io, alloc.

### `shards`

```gizlang
function shards(folder: string, name: string): Shards throws string
```

The shards `name` of `folder`, open while the program runs.

Effects: fail, read, io, alloc.

### `shard`

```gizlang
function shard(folder: string, name: string, part: i64, tokens: Vec<i64>, width: i64 = 2): void throws string
```

Writes `tokens` as part `part` of the shards `name` of `folder`, each in `width` bytes.

Effects: fail, read, write, console, io, alloc.

### `shardBytes`

```gizlang
function shardBytes(folder: string, name: string, size: i64, between: i64, quiet: i64 = 0, width: i64 = 2): void throws string
```

Says of the shards `name` of `folder` how many bytes of text they were, the token that ends a document, where
the tokens that are no target start (`quiet`, Shards) and the bytes of a token.

Effects: fail, read, write, console, io, alloc.

### `plan`

```gizlang
function plan(steps: i64): Plan
```

The plan that is this module's first choice for `steps` steps.

### `trainer`

```gizlang
function trainer(folder: string, z: Option<Sizes>, p: Plan, gpu: Option<Gpu>, group: Group = Dist.alone(), draws: i64 = 0): Trainer throws string
```

The trainer of the model in `folder`, from where its run stopped (a model no run of this module wrote starts at
step 0 with new optimizers: post-training), or of a new model of `z` when the folder holds none. `p` gives the
optimizers' rates and the seed. In a group only the root reads the folder; every rank starts from the root's weights
and optimizer state (nn.shared), so the other ranks need neither the folder nor the root's files. A LoRA run (Plan
`lora`) reads its model from `base` and its adapter from `folder`. Its batches are the folder's draw, at least
`draws` (a rollback's: Learn.run).

Effects: fail, read, write, console, io, alloc.

### `bitsOf`

```gizlang
function bitsOf(nats: f64, held: Shards): f64
```

A loss of `nats` a token on `held` as bits a byte of its text: what compares models of other tokenizers.

Effects: fail.

### `setup`

```gizlang
function setup(given: Vec<string>): Setup throws string
```

The run that `DATA FOLDER STEPS` and the flags of `given` name.

Effects: fail, read, write, io, alloc.

### `run`

```gizlang
function run(s: Setup, said: (line: string) => void): Trainer throws string
```

Trains the run of `s` (`train`): it joins its group first, and `said` takes the root's lines. When the group loses
a rank, every rank meets again (Dist.join) and the run goes on from the folder's last generation, at most
`retries` times.

Effects: fail, read, write, console, io, alloc.

### `pairs`

```gizlang
function pairs(tok: Tok.Tokenizer, path: string): Vec<Pair> throws string
```

The pairs of a JSON lines file in `tok`: each line a `prompt` (a user's text) or `messages` (a chat before the
answer, role and content each), and the assistant's answers `chosen` and `rejected`, in the chat template's turns.
Every fiftieth pair is held out (Pairs.held).

Effects: fail, read, write, io, alloc.

### `tasks`

```gizlang
function tasks(tok: Tok.Tokenizer, path: string): Vec<Problem> throws string
```

The tasks of a JSON lines file in `tok`: each line a `prompt` or `messages` (as pairs'), and its `answer`.

Effects: fail, read, write, io, alloc.

### `chats`

```gizlang
function chats(tok: Tok.Tokenizer, path: string): Vec<Vec<i64>> throws string
```

The chats of a JSON lines file in `tok`: each line a `prompt` (as pairs': a user's turn and the assistant's
opening), `messages` (every turn, role and content each) or `text`.

Effects: fail, read, write, io, alloc.

### `lastNorm`

```gizlang
function lastNorm(z: Sizes): i64
```

The place in `init`'s order of the last norm; the head follows it when it is not tied.

Effects: fail, alloc.

### `grown`

```gizlang
function grown(z: Sizes, params: Vec<Tensor>, gateUp: Tensor, down: Tensor): Grown throws string
```

The model `z`, `params` with one more attention layer at the end that adds only what its MLP adds: `gateUp` (dim,
2 ffn) and `down` (ffn, dim). Its other tensors are `init`'s, so its attention adds nothing (its output is 0).

Effects: fail, read, write, console, io, alloc.

## Methods

### `Shards.batch`

```gizlang
function Shards.batch(self: Shards, seed: u64, rows: i64, t: i64, ahead: i64): Vec<Vec<i64>> throws string
```

`rows` runs of `t` tokens drawn by `seed`: a run's tokens, then for each of `ahead` distances the tokens that far
on (1: the next token). A run starts at a place drawn evenly over the places a whole run fits at, each shard by how
many it has, and then at the first document that starts within `t` tokens of it, when one does: a context reads a
document from its start, as a reader of the model does. A quiet token (Shards) is its token in the run and -1 as a
token after (no target: gizai/grad crossEntropy). The same seed gives the same runs.

Effects: fail, read, io, alloc.

### `Trainer.merged`

```gizlang
function Trainer.merged(self: Trainer): Vec<Tensor>
```

The model's parameters in `init`'s order where the model trains: a LoRA run's base with each adapter merged,
W + A B, in F32.

Effects: fail, read, write, console, io, alloc.

### `Trainer.learn`

```gizlang
function Trainer.learn(self: mut Trainer, drawn: Vec<Vec<i64>>, weights: Vec<f64>, rows: i64, bag: i64 = 1): Tensor
```

One step on rows of tokens: `drawn` are the tokens, the tokens after them, and for each of `weights` the tokens
one more place on (Shards.batch). The loss of the next token as a tensor where the model trains (read it only to say
it: reading waits for the GPU). With `bag` above 1 the rows are bags of tokens (Plan), `drawn` holds the tokens up
to `bag` places on, and the loss is the mean over the next bag's tokens.

Effects: fail, read, write, console, io, alloc.

### `Trainer.heldLoss`

```gizlang
function Trainer.heldLoss(self: Trainer, held: Shards, rows: i64, batches: i64): f64 throws string
```

The mean loss in nats a token of `batches` held-out batches of `rows` contexts (the same ones each time). A
held-out part too short for a context (a few chats: one document in fifty) is read in rows as long as its longest
shard holds (Shards.batch draws two rows' span).

Effects: fail, read, write, console, io, alloc.

### `Trainer.keep`

```gizlang
function Trainer.keep(self: Trainer, folder: string, tokenizer: string, held: Option<f64> = undefined): void throws string
```

Writes the model in `folder` as `save` does, or a LoRA run's adapter, with the run's state in it: one generation,
whole or not there, of held-out loss `held`.

Effects: fail, read, write, console, io, alloc.

### `Trainer.refresh`

```gizlang
function Trainer.refresh(self: Trainer, e: mut Engine): void throws string
```

Writes the model as it trains over the model of engine `e` (one of a folder this run wrote: the same config) in
place, and empties its slots (Model.refresh, Engine.forget). On the card each linear goes from the trainer's F32
buffer to the engine's by a kernel; the embedding and the small tensors go through the host.

Effects: fail, read, write, console, io, alloc.

### `Trainer.prefer`

```gizlang
function Trainer.prefer(self: mut Trainer, ps: Vec<Pair>, beta: f64, step: bool = true): Vec<f64> throws string
```

DPO on pairs `ps` (Rafailov et al. 2023): the loss is -log sigmoid(beta ((c - c0) - (r - r0))) for the log-likelihoods
c and r of the chosen and the other answer, and c0 and r0 the base's (the reference: the model without its
adapters). One step when `step`, else only the measure. Says the mean loss and the part of the pairs whose chosen
answer the model prefers.

Effects: fail, read, write, console, io, alloc.

### `Trainer.reinforce`

```gizlang
function Trainer.reinforce(self: mut Trainer, ts: Vec<Problem>, made: Vec<Vec<i64>>, rewards: Vec<f64>, k: i64, beta: f64, step: bool = true): Vec<f64> throws string
```

Reinforcement on verifiable rewards (GRPO: Shao et al. 2024): `made` holds `k` completions of each of the tasks
`ts` in turn, and `rewards` theirs. With `beta` above 0 each reward loses beta times the completion's log-likelihood
over the reference's (the model in --base): the KL penalty in the reward. A completion's advantage is its reward
over its task's mean, in units of their spread (0 where its task's completions all scored the same), and the loss is
minus the advantages times the completions' log-likelihoods, over the completions' tokens. One step when `step`.
Says the mean reward and the completions' mean length.

Effects: fail, read, write, console, io, alloc.

## Constants

### `SPECIALS`

```gizlang
const SPECIALS: Vec<string> = ["<|endoftext|>", "<|im_start|>", "<|im_end|>"]
```

The tokens a tokenizer of this module ends with: between documents, and around a chat's turn. Every text of this
module that marks a turn is made of them.

### `TRAIN_FLAGS`

```gizlang
const TRAIN_FLAGS: Vec<string> = ["--size", "--dim", "--layers", "--ffn", "--heads", "--head", "--kv", "--mix", "--rotary", "--unique", "--context", "--rows", "--accumulate", "--muon", "--adam", "--decay", "--hold", "--fall", "--bag", "--bagged", "--future", "--seed", "--every", "--keep", "--lora", "--base", "--pairs", "--beta", "--tasks", "--samples", "--refresh", "--ranks", "--rank", "--root", "--rate", "--sync", "--retries", "--side"]
```

The flags of a training run, each with a value (`giz ai train`, as `--rows 16`): the sizes (`--size` 10m, 30m, 50m
or 100m, then `--dim`, `--layers`, `--ffn`, `--heads`, `--head`, `--kv`, `--mix`, `--rotary`, `--unique`), the plan
(`--context`, `--rows`, `--accumulate`, `--muon`, `--adam`, `--decay`, `--hold`, `--fall` line or sqrt, `--bag`, `--bagged`,
`--future` weights as 0.5,0.25, `--seed`, `--every`, `--keep`, `--lora` rank, `--base` folder, `--pairs` file,
`--beta`, `--tasks` file, `--samples`, `--refresh`; an image model's LoRA run: `--side` pixels) and the group (`--ranks`, `--rank`, `--root`
HOST:PORT, `--rate` bytes a second, `--sync` steps, `--retries` after a lost rank).

### `TRAIN_SWITCHES`

```gizlang
const TRAIN_SWITCHES: Vec<string> = ["--gated", "--tied", "--cpu", "--recompute"]
```

The flags of a training run without a value: `--gated` attention, a `--tied` head, `--cpu`, and `--recompute`.

