# gizml:grad derivatives of tensor programs. A Graph records each operation on its values as it runs (a tape); `backward` walks it once in reverse for the gradient of a scalar against every value (reverse mode), and `tangent` walks it forward for the derivative along a direction (forward mode). Each rule is the textbook one over gizml:tensor's own operations, so gradients run fused on the CPU's lanes and cores like the forward pass. ```gizlang import * as Grad from "gizml:grad"; ``` ## Types ### `Var` ```gizlang type Var = { id: i64; } ``` A value in a Graph. - `id`: `i64` ### `Backward` ```gizlang type Backward = (gradient: Tensor) => Vec ``` The vector-Jacobian product of a custom operation: the output's gradient to one gradient per input, in the inputs' shapes. ### `Graph` ```gizlang type Graph = { nodes: Vec; customs: Vec; compute: i64; } ``` A tape of tensor operations. Operations take and return Vars; `value(v)` reads one. A Graph is cheap: make one per step. `compute` is the autocast type (Graph.autocast). - `nodes`: `Vec` - `customs`: `Vec` - `compute`: `i64` ### `Grads` ```gizlang type Grads ``` Gradients by Var; a value the result does not depend on has a zero gradient. Opaque: create and use it through this module's functions. ## Functions ### `graph` ```gizlang function graph(): Graph ``` An empty graph that computes in f32. ## Methods ### `Graph.autocast` ```gizlang function Graph.autocast(self: mut Graph, dtype: i64): void ``` Autocast, as torch.autocast: from now on the matrix products (matmul, batchMatmul, conv2d, forward and backward) of values on a GPU multiply in `dtype` (T.F16, T.BF16 or T.FP8) on its matrix cores, add in f32 and give `dtype` (T.productType; f32 for FP8, whose operands are scaled to its range, T.fp8Scaled). Elementwise operations on a product run in its store (T.lazyMatmul) and keep `dtype` while each other input has that type or one element. Every other operation reads any type and gives f32; T.F32 turns it off. On the CPU nothing changes. Effects: fail. ### `Graph.input` ```gizlang function Graph.input(self: mut Graph, value: Tensor): Var ``` A leaf: a value derivatives are taken against. Effects: fail. ### `Graph.constant` ```gizlang function Graph.constant(self: mut Graph, value: Tensor): Var ``` A leaf no derivative is taken against (data, labels): backward computes no gradient for it or for what depends on nothing else, as PyTorch skips a tensor without requires_grad; its gradient reads as zeros. Effects: fail. ### `Graph.value` ```gizlang function Graph.value(self: mut Graph, v: Var): Tensor ``` The value of V, computed now when it is still an expression. Effects: fail, read, write, console, io. ### `Graph.add` ```gizlang function Graph.add(self: mut Graph, x: Var, y: Var): Var ``` X + Y, element by element. Effects: fail, read, write, console, io. ### `Graph.sub` ```gizlang function Graph.sub(self: mut Graph, x: Var, y: Var): Var ``` X - Y, element by element. Effects: fail, read, write, console, io. ### `Graph.mul` ```gizlang function Graph.mul(self: mut Graph, x: Var, y: Var): Var ``` X * Y, element by element. Effects: fail, read, write, console, io. ### `Graph.div` ```gizlang function Graph.div(self: mut Graph, x: Var, y: Var): Var ``` X / Y, element by element. Effects: fail, read, write, console, io. ### `Graph.maximum` ```gizlang function Graph.maximum(self: mut Graph, x: Var, y: Var): Var ``` Elementwise larger; a tie sends the gradient to `x`. Effects: fail, read, write, console, io. ### `Graph.minimum` ```gizlang function Graph.minimum(self: mut Graph, x: Var, y: Var): Var ``` Elementwise smaller; a tie sends the gradient to `x`. Effects: fail, read, write, console, io. ### `Graph.neg` ```gizlang function Graph.neg(self: mut Graph, x: Var): Var ``` -X. Effects: fail, read, write, console, io. ### `Graph.exp` ```gizlang function Graph.exp(self: mut Graph, x: Var): Var ``` e to the power X, element by element. Effects: fail, read, write, console, io. ### `Graph.log` ```gizlang function Graph.log(self: mut Graph, x: Var): Var ``` The natural logarithm of X, element by element. Effects: fail, read, write, console, io. ### `Graph.tanh` ```gizlang function Graph.tanh(self: mut Graph, x: Var): Var ``` The hyperbolic tangent of X, element by element. Effects: fail, read, write, console, io. ### `Graph.sigmoid` ```gizlang function Graph.sigmoid(self: mut Graph, x: Var): Var ``` The sigmoid 1 / (1 + e^-x) of X, element by element. Effects: fail, read, write, console, io. ### `Graph.relu` ```gizlang function Graph.relu(self: mut Graph, x: Var): Var ``` max(x, 0); the gradient at 0 is 0. Effects: fail, read, write, console, io. ### `Graph.square` ```gizlang function Graph.square(self: mut Graph, x: Var): Var ``` X squared, element by element. Effects: fail, read, write, console, io. ### `Graph.sqrt` ```gizlang function Graph.sqrt(self: mut Graph, x: Var): Var ``` The square root of X, element by element. Effects: fail, read, write, console, io. ### `Graph.abs` ```gizlang function Graph.abs(self: mut Graph, x: Var): Var ``` |x|; the gradient at 0 is 0. Effects: fail, read, write, console, io. ### `Graph.matmul` ```gizlang function Graph.matmul(self: mut Graph, x: Var, y: Var): Var ``` (..., m, k) times (k, n), as Tensor.matmul. Effects: fail, read, write, console, io. ### `Graph.transpose` ```gizlang function Graph.transpose(self: mut Graph, x: Var): Var ``` X with its last two axes swapped. Effects: fail, read, write, console, io. ### `Graph.reshape` ```gizlang function Graph.reshape(self: mut Graph, x: Var, shape: Vec): Var ``` X with SHAPE (the same number of values). Effects: fail, read, write, console, io. ### `Graph.sum` ```gizlang function Graph.sum(self: mut Graph, x: Var): Var ``` The sum of every element (a scalar), in f64 lanes (Reduction.strict). Effects: fail, read, write, console, io. ### `Graph.mean` ```gizlang function Graph.mean(self: mut Graph, x: Var): Var ``` The mean of all values of X. Effects: fail, read, write, console, io. ### `Graph.sumTo` ```gizlang function Graph.sumTo(self: mut Graph, x: Var, shape: Vec): Var ``` Summed to `shape` over the axes it keeps as 1 (Tensor.sumTo): (b, n) to (b, 1) sums each row. Effects: fail, read, write, console, io. ### `Graph.maxTo` ```gizlang function Graph.maxTo(self: mut Graph, x: Var, shape: Vec): Var ``` Maxima to `shape` over the axes it keeps as 1 (Tensor.maxTo). Ties share the gradient evenly, as torch.amax. Effects: fail, read, write, console, io. ### `Graph.logSoftmax` ```gizlang function Graph.logSoftmax(self: mut Graph, x: Var): Var ``` Log softmax over the last axis: x - max - log(sum(exp(x - max))), from the operations above (both modes). Effects: fail, read, write, console, io. ### `Graph.softmaxCrossEntropy` ```gizlang function Graph.softmaxCrossEntropy(self: mut Graph, logits: Var, target: Var): Var ``` The mean over rows of -sum(target * log softmax(logits)) (logits and target (rows, classes); target is usually a constant, such as one-hot rows or soft labels). Graph.crossEntropy is the fused form for labels. Effects: fail, read, write, console, io. ### `Graph.custom` ```gizlang function Graph.custom(self: mut Graph, inputs: Vec, result: Tensor, backward: Backward): Var ``` An operation with its own gradient: `value` computed from `inputs`, and `backward` mapping the output's gradient to the inputs'. Reverse mode only. Effects: fail. ### `Graph.checkpoint` ```gizlang function Graph.checkpoint(self: mut Graph, inputs: Vec, f: (g: Graph, xs: Vec) => Graph): Var ``` `f` applied to `inputs` without keeping its intermediate values: only the result is stored, and the backward pass runs `f` again for its gradient (memory for time). `f` gets a fresh Graph and the inputs as its leaves and returns that Graph with the result as its last value. Effects: fail, read, write, console, io. ### `Graph.gelu` ```gizlang function Graph.gelu(self: mut Graph, x: Var): Var ``` GELU (tanh form, as torch's approximate="tanh"). Effects: fail, read, write, console, io. ### `Graph.geluErf` ```gizlang function Graph.geluErf(self: mut Graph, x: Var): Var ``` GELU (erf form, torch's default). Effects: fail, read, write, console, io. ### `Graph.attention` ```gizlang function Graph.attention(self: mut Graph, q: Var, k: Var, v: Var, scale: f64, causal: bool, rate: f64, key: T.Key): Var ``` Scaled dot-product attention as N.attention: q (b, tq, d), k (b, tk, d), v (b, tk, e). The backward pass makes the weights P again from q and k (no (tq, tk) matrix is kept), then dv = Pdᵀ dy, dPd = dy vᵀ, dS = P (dP - rowsum(dP P)) and dq = dS k scale, dk = dSᵀ q scale, where Pd is P with the dropout mask and dP is dPd with it. Effects: fail, read, write, console, io. ### `Graph.embedding` ```gizlang function Graph.embedding(self: mut Graph, table: Var, ids: Vec): Var ``` Rows `ids` of `table` (rows, width): (ids, width). Effects: fail, read, write, console, io. ### `Graph.layerNorm` ```gizlang function Graph.layerNorm(self: mut Graph, x: Var, weight: Var, bias: Var, eps: f64): Var ``` Layer normalization of the last axis with `weight` and `bias` (its length). Effects: fail, read, write, console, io. ### `Graph.softmax` ```gizlang function Graph.softmax(self: mut Graph, x: Var): Var ``` Softmax over the last axis. Effects: fail, read, write, console, io. ### `Graph.crossEntropy` ```gizlang function Graph.crossEntropy(self: mut Graph, logits: Var, labels: Vec): Var ``` The mean over rows of the softmax cross-entropy of logits (rows, classes) at integer labels. Effects: fail, read, write, console, io. ### `Graph.dropout` ```gizlang function Graph.dropout(self: mut Graph, x: Var, k: T.Key, p: f64): Var ``` x with each element zeroed at rate p (the mask from key k) and the rest scaled by 1/(1-p). Effects: fail, read, write, console, io. ### `Graph.slice` ```gizlang function Graph.slice(self: mut Graph, x: Var, axis: i64, start: i64, end: i64): Var ``` Elements start..end of `axis`. Effects: fail, read, write, console, io. ### `Graph.concat` ```gizlang function Graph.concat(self: mut Graph, xs: Vec, axis: i64): Var ``` The values joined along `axis`. Effects: fail, read, write, console, io. ### `Graph.permute` ```gizlang function Graph.permute(self: mut Graph, x: Var, axes: Vec): Var ``` The axes reordered: output axis i is input axis axes[i]. Effects: fail, read, write, console, io. ### `Graph.batchMatmul` ```gizlang function Graph.batchMatmul(self: mut Graph, x: Var, ta: bool, y: Var, tb: bool): Var ``` Batched matmul as T.batchMatmul: x (b, m, k) or, transposed, (b, k, m); y (b, k, n), (b, n, k), or one matrix for every batch. Effects: fail, read, write, console, io. ### `Graph.conv2d` ```gizlang function Graph.conv2d(self: mut Graph, x: Var, weight: Var, bias: Var, stride: i64, pad: i64): Var ``` A 2-D convolution of x (n, c, h, w) by weight (o, c, kh, kw) plus bias (o), at `stride` with `pad` zeros: (n, o, oh, ow), as one matmul over im2col's patches. Effects: fail, read, write, console, io. ### `Graph.pool2d` ```gizlang function Graph.pool2d(self: mut Graph, x: Var, size: i64, stride: i64, average: bool): Var ``` Max (or average) pooling over size x size windows at `stride`. Effects: fail, read, write, console, io. ### `Grads.of` ```gizlang function Grads.of (self: Grads, v: Var): Tensor ``` The gradient of v (each call computes it). Effects: fail, read, write, console, io. ### `Grads.pending` ```gizlang function Grads.pending(self: Grads, v: Var): Lazy ``` The gradient of v not yet computed: an optimizer extends it, so where it is a product, the update runs as the product is stored. Effects: fail, read, write, console, io. ### `Graph.backward` ```gizlang function Graph.backward(self: mut Graph, loss: Var): Grads ``` The gradient of `loss`, a one-element value, against every value before it. Effects: fail, read, write, console, io. ### `Graph.backwardFrom` ```gizlang function Graph.backwardFrom(self: mut Graph, output: Var, seed: Tensor): Grads ``` The vector-Jacobian product: `seed` (the output's shape) through the tape. Effects: fail, read, write, console, io. ### `Graph.tangent` ```gizlang function Graph.tangent(self: mut Graph, inputs: Vec, directions: Vec, output: Var): Tensor ``` The derivative of `output` along `directions` given for `inputs` (each the input's shape; other inputs held fixed): one forward walk of the tape (a Jacobian-vector product). Every built-in operation; not custom ones. Effects: fail, read, write, console, io.