# gizml:nn what training a network needs on top of gizml:grad. Layers are functions of a Graph and their parameters (plain Tensors the caller keeps), losses are Graph values, and optimizers update the parameters in fused passes. Files of parameters are gizml:tensor's (save, load). ```gizlang import * as Nn from "gizml:nn"; ``` ## Types ### `Sgd` ```gizlang type Sgd ``` Stochastic gradient descent with momentum, as torch.optim.SGD: v = momentum v + g, p -= rate v. Without momentum there is no v, and p - rate g is one pass (or the epilogue of the product that makes g). - `rate`: `f64` - `momentum`: `f64` ### `Adam` ```gizlang type Adam ``` Adam (Kingma and Ba) with decoupled weight decay (AdamW when decay > 0), as torch.optim.AdamW. - `rate`: `f64` - `beta1`: `f64` - `beta2`: `f64` - `epsilon`: `f64` - `decay`: `f64` ### `GptConfig` ```gizlang type GptConfig = { vocab: i64; context: i64; dim: i64; heads: i64; layers: i64; dropout: f64; } ``` A decoder-only transformer as GPT-2 builds one: pre-norm blocks of causal multi-head attention and a 4x GELU MLP, learned positions, a final norm and an unbiased head. - `vocab`: `i64` - `context`: `i64` - `dim`: `i64` - `heads`: `i64` - `layers`: `i64` - `dropout`: `f64` ### `KvCache` ```gizlang type KvCache = { keys: Vec; values: Vec; length: i64; } ``` Inference state: per layer the keys and values of the tokens so far, (heads, tokens, hd) each, so a new token attends without recomputing them. - `keys`: `Vec` - `values`: `Vec` - `length`: `i64` ## Functions ### `linearInit` ```gizlang function linearInit(k: Key, inputs: i64, outputs: i64): Vec ``` A dense layer's parameters, [weight (inputs, outputs), bias (outputs)], uniform in +-1/sqrt(inputs) as PyTorch's nn.Linear starts. Effects: fail. ### `linear` ```gizlang function linear(g: mut Graph, x: Var, weight: Var, bias: Var): Var ``` x (..., inputs) times weight plus bias. Effects: fail, read, write, console, io. ### `inputs` ```gizlang function inputs(g: mut Graph, params: Vec): Vec ``` Each tensor of PARAMS as an input of G, in order. Effects: fail. ### `gradients` ```gizlang function gradients(grads: Grads, vars: Vec): Vec ``` The gradient of each of VARS, in order (for an optimizer's step), not yet computed: the step's update runs as each is stored. Effects: fail, read, write, console, io. ### `mse` ```gizlang function mse(g: mut Graph, y: Var, target: Var): Var ``` The mean of (y - target)^2. Effects: fail, read, write, console, io. ### `oneHot` ```gizlang function oneHot(labels: Vec, classes: i64): Tensor ``` Rows of 0s with a 1 at each label: (labels, classes). Effects: fail. ### `accuracy` ```gizlang function accuracy(logits: Tensor, labels: Vec): f64 ``` The share of rows whose largest logit is at the label. Effects: fail, read, write, console, io. ### `sgd` ```gizlang function sgd(rate: f64, momentum: f64): Sgd ``` A stochastic gradient descent optimizer with this learning rate and momentum. ### `adam` ```gizlang function adam(rate: f64): Adam ``` An Adam optimizer with this learning rate. ### `adamW` ```gizlang function adamW(rate: f64, decay: f64): Adam ``` An AdamW optimizer with this learning rate and weight decay. ### `gptInit` ```gizlang function gptInit(k: Key, c: GptConfig): Vec ``` Parameters in order: token and position embeddings; per layer ln1 (w, b), qkv (w, b), proj (w, b), ln2 (w, b), fc (w, b), fc2 (w, b); final norm (w, b); head. Weights normal(0, 0.02), biases 0, norms 1 and 0 (GPT-2's start). Effects: fail, read, write, console, io. ### `gpt` ```gizlang function gpt(g: mut Graph, c: GptConfig, p: Vec, ids: Vec, batch: i64, t: i64, k: Option): Var ``` Logits (batch * t, vocab) of the token ids (batch rows of t). `key` draws the dropout masks when training (c.dropout > 0). Effects: fail, read, write, console, io. ### `kvCache` ```gizlang function kvCache(): KvCache ``` An empty cache. ### `gptStep` ```gizlang function gptStep(c: GptConfig, p: Vec, cache: mut KvCache, id: i64): Tensor ``` The logits (vocab) after token `id` at the cache's next position, extending the cache. Effects: fail, read, write, console, io. ### `generate` ```gizlang function generate(c: GptConfig, p: Vec, prompt: Vec, count: i64): Vec ``` `count` tokens after `prompt`, each the most likely (greedy), with a KV cache. Effects: fail, read, write, console, io. ### `conv2dInit` ```gizlang function conv2dInit(k: Key, inputs: i64, outputs: i64, size: i64): Vec ``` A convolution's parameters, [weight (outputs, inputs, k, k), bias (outputs)], uniform in +-1/sqrt(inputs k k) as torch's nn.Conv2d starts. Effects: fail. ### `onGpu` ```gizlang function onGpu(params: Vec, gpu: Gpu): Vec ``` The parameters on `gpu`. Effects: fail, read, write, console, io. ## Methods ### `Sgd.step` ```gizlang function Sgd.step(self: mut Sgd, params: mut Vec, grads: Vec): void ``` One step: moves PARAMS against GRADS. Effects: fail, read, write, console, io. ### `Adam.step` ```gizlang function Adam.step(self: mut Adam, params: mut Vec, grads: Vec): void ``` One step: moves PARAMS against GRADS. Effects: fail, read, write, console, io.