GizStack
View .mdOpen in ChatGPTOpen in Claude

gizai/llm/learn

a language model trained from nothing, in the form the engine already runs. The model is Qwen3's (gizai/llm/recipes): RMS norms before the attention and the MLP, rotary positions, RMS norms of each query and key head, a gated SiLU MLP, no bias. train writes it as a Hugging Face folder (config.json, model.safetensors, tokenizer.json), so gizai/llm loads it, generates, chats and serves with no code of its own for this model, and so does any other program that reads Qwen3.

What is the trainer's own, and leaves the model's function as the engine computes it: - the hidden matrices move by Muon (gizai/nn), the embedding, the head and the norms by Adam; - the output projections of the attention and the MLP start at zero, so each layer starts as the identity; - unique layers may be fewer than layers: the layers then repeat them in turn (one set of weights computes several times), and the folder holds each layer's copy; - future weights also teach the logits of a place the tokens after the next one, a weight a distance.

gizlang
import * as Learn from "gizai/llm/learn";

Types

Sizes

gizlang
type Sizes = { vocab: i64; context: i64; dim: i64; layers: i64; unique: i64; heads: i64; kv: i64; ffn: i64; tied: bool; gated: bool; mix: string; rotary: i64; head: i64; theta: f64; }

A model's sizes. heads heads of head values (64, 128 or 256; dim / heads in the named sizes) over kv key and value heads, turned by position with base theta; ffn the MLP's width; unique distinct layers that layers repeat in turn; tied: the head is the embedding; gated: each head's output is times a sigmoid gate the query linear also makes (Qwen3.5's attention, saved as that family). mix names the distinct layers' mixers in turn: "a" attention, "c" a short causal convolution (LFM2's, saved as that family, with no gate), "d" a gated delta rule of heads of DELTA_HEAD (Qwen3.5's linear attention, saved as that family). rotary of a head's dimensions turn by position (0: all). dim, ffn and the attention's width are multiples of 128 (the engine's blocks).

  • vocab: i64
  • context: i64
  • dim: i64
  • layers: i64
  • unique: i64
  • heads: i64
  • kv: i64
  • ffn: i64
  • tied: bool
  • gated: bool
  • mix: string
  • rotary: i64
  • head: i64
  • theta: f64

Weights

gizlang
type Weights = { p: Vec<Var>; low: Vec<Vec<Var>>; scale: f64; }

A model as values of a graph: p are init's parameters; low[i], when not empty, is LoRA's adapter [A, B] of the hidden matrix p[i], A (inputs, rank) and B (rank, outputs), whose product times scale adds to it.

  • p: Vec<M19.Var>
  • low: Vec<Vec<M19.Var>>
  • scale: f64

Generation

gizlang
type Generation = { step: i64; draw: i64; held: Option<f64>; }

Where a generation stands in its run: its step, the draw of its batches (a rollback draws others: Trainer.steps) and its held-out loss (undefined when none was measured). Its weights file's metadata holds it.

  • step: i64
  • draw: i64
  • held: _gizlang_sdk_core.Option<f64>

Kept

gizlang
type Kept = { z: Sizes; params: Vec<Tensor>; at: Generation; run: Vec<TensorCore.Named>; }

What a folder that save wrote holds: the sizes, the parameters in init's order, its generation (Generation) and the run's tensors (none and generation 0 for a model no run of this module wrote).

  • z: M66.Sizes
  • params: Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>
  • at: M66.Generation
  • run: Vec<M53.Named>

Shards

gizlang
type Shards = { files: Vec<i64>; sizes: Vec<i64>; tokens: i64; bytes: i64; between: i64; quiet: i64; width: i64; }

Tokens in files: each NAME-N.tok of a folder is tokens of width bytes (two, or four for a vocabulary of more than 65,536 tokens; the low byte first), one after another, and NAME.json says the width, how many bytes of text they were, which token ends a document, and quiet (0: none): a token of quiet or more is token - quiet, read by the model but no target of the loss (a chat's prompt and tool output, where the model learns only its own turns). The files stay on the disk: a batch reads its runs from them, so text of any size trains in the memory of one batch.

  • files: Vec<i64>
  • sizes: Vec<i64>
  • tokens: i64
  • bytes: i64
  • between: i64
  • quiet: i64
  • width: i64

Plan

gizlang
type Plan = { rows: i64; steps: i64; muon: f64; adam: f64; decay: f64; future: Vec<f64>; every: i64; keep: i64; seed: u64; hold: f64; root: bool; bag: i64; bagged: f64; accumulate: i64; context: i64; lora: i64; base: string; pairs: string; beta: f64; recompute: bool; sync: i64; tasks: string; samples: i64; refresh: i64; }

A training run: steps steps of rows contexts of context tokens (0: the model's length). The rates are Muon's (0: every parameter moves by Adam) and Adam's; decay shrinks the matrices, by their optimizer's rate times it a step; both rise over the first 200 steps, hold to hold of the run and then fall to a tenth along a line, or with root to nothing as 1 - sqrt of the part of the fall gone. For the first bagged of the steps a place reads the mean embedding of bag tokens and learns the bag tokens of the next place, each a share of the loss (bag training: bag times the text for the same work); then each place is one token again. A step takes the mean gradient of accumulate batches of rows (a larger batch in the memory of one). future[i] weighs the token i + 2 places on in the loss, and falls to nothing by two thirds of the run. Every every steps a line is said, every keep steps the held-out loss is measured and the folder is written. With lora above 0 the run trains LoRA adapters of that rank on the hidden matrices of the model in base, which stays as it is, and the folder is the adapter (LoRA above). With pairs (a JSON lines file, Pairs) the steps are DPO's on its preference pairs at strength beta, the base the reference; with tasks (a JSON lines file, Problem) GRPO's on samples completions of each task, the engine taking the model every refresh steps, a KL penalty of beta to the base (Trainer.reinforce). One loop takes every kind of run (Trainer.steps). recompute keeps one layer's activations at a time (logits). In a group each rank takes sync steps alone between two syncs (Learner).

  • rows: i64
  • steps: i64
  • muon: f64
  • adam: f64
  • decay: f64
  • future: Vec<f64>
  • every: i64
  • keep: i64
  • seed: u64
  • hold: f64
  • root: bool
  • bag: i64
  • bagged: f64
  • accumulate: i64
  • context: i64
  • lora: i64
  • base: string
  • pairs: string
  • beta: f64
  • recompute: bool
  • sync: i64
  • tasks: string
  • samples: i64
  • refresh: i64

Trainer

gizlang
type Trainer = { a: Arch; z: Sizes; learner: nn.Learner; base: Vec<Tensor>; reference: Vec<Tensor>; fixed: Vec<Tensor>; t: i64; rate: f64; recompute: bool; draw: i64; best: Option<f64>; }

A model in training: its Learner (gizai/nn: the parameters, the optimizers, the steps and the group whose ranks train it as one), the base model that a LoRA run adapts (none: the learner's parameters are the model), its tables for contexts of t tokens where it trains, the tokens a second of train's last line (0 before one), the draw of its batches (Generation) and the held-out loss of its best generation (the folder's best; none before one).

  • a: _gizlang_lib_ai__llm__shape.Arch
  • z: M66.Sizes
  • learner: M22.Learner
  • base: Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>
  • reference: Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>
  • fixed: Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>
  • t: i64
  • rate: f64
  • recompute: bool
  • draw: i64
  • best: _gizlang_sdk_core.Option<f64>

Setup

gizlang
type Setup = { data: string; tokenizer: string; folder: string; z: Sizes; plan: Plan; cpu: bool; ranks: i64; rank: i64; host: string; port: i64; rate: i64; retries: i64; }

A run: its token shards (data, with their tokenizer.json), its folder, its steps, the sizes of a new model, the plan, where it trains, and its place in a group.

  • data: string
  • tokenizer: string
  • folder: string
  • z: M66.Sizes
  • plan: M66.Plan
  • cpu: bool
  • ranks: i64
  • rank: i64
  • host: string
  • port: i64
  • rate: i64
  • retries: i64

Pair

gizlang
type Pair = { prompt: Vec<i64>; chosen: Vec<i64>; rejected: Vec<i64>; }

A preference pair as tokens: a prompt the model reads and two answers to it, the one people chose and the other.

  • prompt: Vec<i64>
  • chosen: Vec<i64>
  • rejected: Vec<i64>

Problem

gizlang
type Problem = { prompt: Vec<i64>; answer: string; }

A problem as tokens: a prompt the model reads (the chat template's turns, as a Pair's) and the text a right answer holds.

  • prompt: Vec<i64>
  • answer: string

Grown

gizlang
type Grown = { z: Sizes; params: Vec<Tensor>; }

A model with one more distinct layer.

  • z: M66.Sizes
  • params: Vec<_gizlang_lib_ai__tensor__storage.Tensor<f32>>

Functions

sizes

gizlang
function sizes(name: string, vocab: i64, context: i64): Sizes throws string

The sizes by name: "10m", "30m", "50m" or "100m" parameters near a vocabulary of 8,192.

Effects: fail, alloc.

convolved

gizlang
function convolved(z: Sizes): bool

Whether the model is LFM2's: some layer mixes by a short convolution.

Effects: fail.

config

gizlang
function config(z: Sizes, ends: i64, pad: i64): string

The config.json of a model of these sizes: Qwen3's; with gated or delta rules Qwen3.5's; with short convolutions LFM2's. ends is the token that ends a turn, pad the one between documents.

Effects: fail, alloc.

arch

gizlang
function arch(z: Sizes): Arch throws string

What the engine computes for these sizes: the Arch of their config.json.

Effects: fail, read, write, console, io, alloc.

init

gizlang
function init(k: Key, z: Sizes): Vec<Tensor>

The parameters, in order: the embedding (vocab, dim); each distinct layer's tensors; the last norm; the head (dim, vocab) when it is not tied. A linear is (inputs, outputs), as a row times it. Linears are normal(0, 0.02), the mixers' output and the MLP's down linears and the head zero, norms one, a convolution's taps normal(0, 0.02); a delta rule's A_log is log of 1 to 16 over the heads and dt_bias the softplus inverse of 0.001 to 0.1, as Mamba's.

Effects: fail, read, write, console, io, alloc.

count

gizlang
function count(params: Vec<Tensor>): i64

How many numbers the parameters hold.

Effects: fail.

tables

gizlang
function tables(a: Arch, t: i64): Vec<Tensor>

What the graph reads beside the parameters for contexts of t tokens: the cosines and the sines (t, head) of the rotary turns, pairs i and i + rotary / 2 of the first rotary dimensions as the engine's (the others stay).

Effects: fail, alloc.

logits

gizlang
function logits(g: mut Graph, a: Arch, w: Weights, f: Vec<Var>, ids: Vec<i64>, rows: i64, t: i64, bag: i64 = 1, recompute: bool = false): Var

Logits (rows * t, vocab) of the token ids (rows of t), as the engine computes them for the model's folder. w are the model's values and f tables' tensors, as values of g. With bag above 1, ids are rows of t bags of bag tokens, and a place reads the mean of its bag's embeddings (training only: the engine reads one token a place). With recompute each layer keeps only its output, and the backward pass computes it again (Graph.checkpoint): the activations of one layer at a time, for a third more work.

Effects: fail, read, write, console, io, alloc.

read

gizlang
function read(a: Arch, params: Vec<Tensor>, fixed: Vec<Tensor>, ids: Vec<i64>, rows: i64, t: i64): Tensor

The logits on a new graph, in bf16 products on a GPU: for a reader that takes no gradient.

Effects: fail, read, write, console, io, alloc.

generationOf

gizlang
function generationOf(path: string): Generation

The generation of the weights file at path: step 0 and draw 0 for a file no run wrote.

Effects: fail, read, io, alloc.

save

gizlang
function save(folder: string, z: Sizes, params: Vec<Tensor>, tokenizer: string, run: Vec<TensorCore.Named> = [], at: Generation = { step: 0, draw: 0, held: undefined }): void throws string

Writes the model as a Hugging Face folder of Qwen3: tokenizer is its tokenizer.json's text, with SPECIALS as its last tokens. The folder is made when it is not there; the config and the tokenizer are written only then, so a model from elsewhere (post-training) keeps its own. run are a training run's own tensors (the optimizers') at generation at: they go in model.safetensors beside the weights, under train., so the weights and the run that made them are one file, replaced whole (the last file written). A reader of the model does not read them.

Effects: fail, read, write, console, io, alloc.

kept

gizlang
function kept(folder: string): Option<Kept> throws string

The model of folder, or none when the folder holds no weights.

Effects: fail, read, write, console, io, alloc.

shards

gizlang
function shards(folder: string, name: string): Shards throws string

The shards name of folder, open while the program runs.

Effects: fail, read, io, alloc.

shard

gizlang
function shard(folder: string, name: string, part: i64, tokens: Vec<i64>, width: i64 = 2): void throws string

Writes tokens as part part of the shards name of folder, each in width bytes.

Effects: fail, read, write, console, io, alloc.

shardBytes

gizlang
function shardBytes(folder: string, name: string, size: i64, between: i64, quiet: i64 = 0, width: i64 = 2): void throws string

Says of the shards name of folder how many bytes of text they were, the token that ends a document, where the tokens that are no target start (quiet, Shards) and the bytes of a token.

Effects: fail, read, write, console, io, alloc.

plan

gizlang
function plan(steps: i64): Plan

The plan that is this module's first choice for steps steps.

trainer

gizlang
function trainer(folder: string, z: Option<Sizes>, p: Plan, gpu: Option<Gpu>, group: Group = Dist.alone(), draws: i64 = 0): Trainer throws string

The trainer of the model in folder, from where its run stopped (a model no run of this module wrote starts at step 0 with new optimizers: post-training), or of a new model of z when the folder holds none. p gives the optimizers' rates and the seed. In a group only the root reads the folder; every rank starts from the root's weights and optimizer state (nn.shared), so the other ranks need neither the folder nor the root's files. A LoRA run (Plan lora) reads its model from base and its adapter from folder. Its batches are the folder's draw, at least draws (a rollback's: Learn.run).

Effects: fail, read, write, console, io, alloc.

bitsOf

gizlang
function bitsOf(nats: f64, held: Shards): f64

A loss of nats a token on held as bits a byte of its text: what compares models of other tokenizers.

Effects: fail.

setup

gizlang
function setup(given: Vec<string>): Setup throws string

The run that DATA FOLDER STEPS and the flags of given name.

Effects: fail, read, write, io, alloc.

run

gizlang
function run(s: Setup, said: (line: string) => void): Trainer throws string

Trains the run of s (train): it joins its group first, and said takes the root's lines. When the group loses a rank, every rank meets again (Dist.join) and the run goes on from the folder's last generation, at most retries times.

Effects: fail, read, write, console, io, alloc.

pairs

gizlang
function pairs(tok: Tok.Tokenizer, path: string): Vec<Pair> throws string

The pairs of a JSON lines file in tok: each line a prompt (a user's text) or messages (a chat before the answer, role and content each), and the assistant's answers chosen and rejected, in the chat template's turns. Every fiftieth pair is held out (Pairs.held).

Effects: fail, read, write, io, alloc.

tasks

gizlang
function tasks(tok: Tok.Tokenizer, path: string): Vec<Problem> throws string

The tasks of a JSON lines file in tok: each line a prompt or messages (as pairs'), and its answer.

Effects: fail, read, write, io, alloc.

chats

gizlang
function chats(tok: Tok.Tokenizer, path: string): Vec<Vec<i64>> throws string

The chats of a JSON lines file in tok: each line a prompt (as pairs': a user's turn and the assistant's opening), messages (every turn, role and content each) or text.

Effects: fail, read, write, io, alloc.

lastNorm

gizlang
function lastNorm(z: Sizes): i64

The place in init's order of the last norm; the head follows it when it is not tied.

Effects: fail, alloc.

grown

gizlang
function grown(z: Sizes, params: Vec<Tensor>, gateUp: Tensor, down: Tensor): Grown throws string

The model z, params with one more attention layer at the end that adds only what its MLP adds: gateUp (dim, 2 ffn) and down (ffn, dim). Its other tensors are init's, so its attention adds nothing (its output is 0).

Effects: fail, read, write, console, io, alloc.

Methods

Shards.batch

gizlang
function Shards.batch(self: Shards, seed: u64, rows: i64, t: i64, ahead: i64): Vec<Vec<i64>> throws string

rows runs of t tokens drawn by seed: a run's tokens, then for each of ahead distances the tokens that far on (1: the next token). A run starts at a place drawn evenly over the places a whole run fits at, each shard by how many it has, and then at the first document that starts within t tokens of it, when one does: a context reads a document from its start, as a reader of the model does. A quiet token (Shards) is its token in the run and -1 as a token after (no target: gizai/grad crossEntropy). The same seed gives the same runs.

Effects: fail, read, io, alloc.

Trainer.merged

gizlang
function Trainer.merged(self: Trainer): Vec<Tensor>

The model's parameters in init's order where the model trains: a LoRA run's base with each adapter merged, W + A B, in F32.

Effects: fail, read, write, console, io, alloc.

Trainer.learn

gizlang
function Trainer.learn(self: mut Trainer, drawn: Vec<Vec<i64>>, weights: Vec<f64>, rows: i64, bag: i64 = 1): Tensor

One step on rows of tokens: drawn are the tokens, the tokens after them, and for each of weights the tokens one more place on (Shards.batch). The loss of the next token as a tensor where the model trains (read it only to say it: reading waits for the GPU). With bag above 1 the rows are bags of tokens (Plan), drawn holds the tokens up to bag places on, and the loss is the mean over the next bag's tokens.

Effects: fail, read, write, console, io, alloc.

Trainer.heldLoss

gizlang
function Trainer.heldLoss(self: Trainer, held: Shards, rows: i64, batches: i64): f64 throws string

The mean loss in nats a token of batches held-out batches of rows contexts (the same ones each time). A held-out part too short for a context (a few chats: one document in fifty) is read in rows as long as its longest shard holds (Shards.batch draws two rows' span).

Effects: fail, read, write, console, io, alloc.

Trainer.keep

gizlang
function Trainer.keep(self: Trainer, folder: string, tokenizer: string, held: Option<f64> = undefined): void throws string

Writes the model in folder as save does, or a LoRA run's adapter, with the run's state in it: one generation, whole or not there, of held-out loss held.

Effects: fail, read, write, console, io, alloc.

Trainer.refresh

gizlang
function Trainer.refresh(self: Trainer, e: mut Engine): void throws string

Writes the model as it trains over the model of engine e (one of a folder this run wrote: the same config) in place, and empties its slots (Model.refresh, Engine.forget). On the card each linear goes from the trainer's F32 buffer to the engine's by a kernel; the embedding and the small tensors go through the host.

Effects: fail, read, write, console, io, alloc.

Trainer.prefer

gizlang
function Trainer.prefer(self: mut Trainer, ps: Vec<Pair>, beta: f64, step: bool = true): Vec<f64> throws string

DPO on pairs ps (Rafailov et al. 2023): the loss is -log sigmoid(beta ((c - c0) - (r - r0))) for the log-likelihoods c and r of the chosen and the other answer, and c0 and r0 the base's (the reference: the model without its adapters). One step when step, else only the measure. Says the mean loss and the part of the pairs whose chosen answer the model prefers.

Effects: fail, read, write, console, io, alloc.

Trainer.reinforce

gizlang
function Trainer.reinforce(self: mut Trainer, ts: Vec<Problem>, made: Vec<Vec<i64>>, rewards: Vec<f64>, k: i64, beta: f64, step: bool = true): Vec<f64> throws string

Reinforcement on verifiable rewards (GRPO: Shao et al. 2024): made holds k completions of each of the tasks ts in turn, and rewards theirs. With beta above 0 each reward loses beta times the completion's log-likelihood over the reference's (the model in --base): the KL penalty in the reward. A completion's advantage is its reward over its task's mean, in units of their spread (0 where its task's completions all scored the same), and the loss is minus the advantages times the completions' log-likelihoods, over the completions' tokens. One step when step. Says the mean reward and the completions' mean length.

Effects: fail, read, write, console, io, alloc.

Constants

SPECIALS

gizlang
const SPECIALS: Vec<string> = ["<|endoftext|>", "<|im_start|>", "<|im_end|>"]

The tokens a tokenizer of this module ends with: between documents, and around a chat's turn. Every text of this module that marks a turn is made of them.

TRAIN_FLAGS

gizlang
const TRAIN_FLAGS: Vec<string> = ["--size", "--dim", "--layers", "--ffn", "--heads", "--head", "--kv", "--mix", "--rotary", "--unique", "--context", "--rows", "--accumulate", "--muon", "--adam", "--decay", "--hold", "--fall", "--bag", "--bagged", "--future", "--seed", "--every", "--keep", "--lora", "--base", "--pairs", "--beta", "--tasks", "--samples", "--refresh", "--ranks", "--rank", "--root", "--rate", "--sync", "--retries", "--side"]

The flags of a training run, each with a value (giz ai train, as --rows 16): the sizes (--size 10m, 30m, 50m or 100m, then --dim, --layers, --ffn, --heads, --head, --kv, --mix, --rotary, --unique), the plan (--context, --rows, --accumulate, --muon, --adam, --decay, --hold, --fall line or sqrt, --bag, --bagged, --future weights as 0.5,0.25, --seed, --every, --keep, --lora rank, --base folder, --pairs file, --beta, --tasks file, --samples, --refresh; an image model's LoRA run: --side pixels) and the group (--ranks, --rank, --root HOST:PORT, --rate bytes a second, --sync steps, --retries after a lost rank).

TRAIN_SWITCHES

gizlang
const TRAIN_SWITCHES: Vec<string> = ["--gated", "--tied", "--cpu", "--recompute"]

The flags of a training run without a value: --gated attention, a --tied head, --cpu, and --recompute.