# gizml:grad

derivatives of tensor programs. A Graph records each operation on its values as it runs (a tape); `backward` walks it once in reverse for the gradient of a scalar against every value (reverse mode), and `tangent` walks it forward for the derivative along a direction (forward mode). Each rule is the textbook one over gizml:tensor's own operations, so gradients run fused on the CPU's lanes and cores like the forward pass.

```gizlang
import * as Grad from "gizml:grad";
```

## Types

### `Var`

```gizlang
type Var = { id: i64; }
```

A value in a Graph.

- `id`: `i64`

### `Backward`

```gizlang
type Backward = (gradient: Tensor) => Vec<Tensor>
```

The vector-Jacobian product of a custom operation: the output's gradient
to one gradient per input, in the inputs' shapes.

### `Graph`

```gizlang
type Graph = { nodes: Vec<Node>; customs: Vec<Custom>; compute: i64; }
```

A tape of tensor operations. Operations take and return Vars; `value(v)`
reads one. A Graph is cheap: make one per step. `compute` is the autocast type (Graph.autocast).

- `nodes`: `Vec<M21.Node>`
- `customs`: `Vec<M21.Custom>`
- `compute`: `i64`

### `Grads`

```gizlang
type Grads
```

Gradients by Var; a value the result does not depend on has a zero gradient.

Opaque: create and use it through this module's functions.

## Functions

### `graph`

```gizlang
function graph(): Graph
```

An empty graph that computes in f32.

## Methods

### `Graph.autocast`

```gizlang
function Graph.autocast(self: mut Graph, dtype: i64): void
```

Autocast, as torch.autocast: from now on the matrix products (matmul, batchMatmul, conv2d, forward and backward) of
values on a GPU multiply in `dtype` (T.F16, T.BF16 or T.FP8) on its matrix cores, add in f32 and give `dtype`
(T.productType; f32 for FP8, whose operands are scaled to its range, T.fp8Scaled).
Elementwise operations on a product run in its store (T.lazyMatmul) and keep `dtype` while each other input has
that type or one element. Every other operation reads any type and gives f32; T.F32 turns it off. On the CPU
nothing changes.

Effects: fail.

### `Graph.input`

```gizlang
function Graph.input(self: mut Graph, value: Tensor): Var
```

A leaf: a value derivatives are taken against.

Effects: fail.

### `Graph.constant`

```gizlang
function Graph.constant(self: mut Graph, value: Tensor): Var
```

A leaf no derivative is taken against (data, labels): backward computes no
gradient for it or for what depends on nothing else, as PyTorch skips a
tensor without requires_grad; its gradient reads as zeros.

Effects: fail.

### `Graph.value`

```gizlang
function Graph.value(self: mut Graph, v: Var): Tensor
```

The value of V, computed now when it is still an expression.

Effects: fail, read, write, console, io.

### `Graph.add`

```gizlang
function Graph.add(self: mut Graph, x: Var, y: Var): Var
```

X + Y, element by element.

Effects: fail, read, write, console, io.

### `Graph.sub`

```gizlang
function Graph.sub(self: mut Graph, x: Var, y: Var): Var
```

X - Y, element by element.

Effects: fail, read, write, console, io.

### `Graph.mul`

```gizlang
function Graph.mul(self: mut Graph, x: Var, y: Var): Var
```

X * Y, element by element.

Effects: fail, read, write, console, io.

### `Graph.div`

```gizlang
function Graph.div(self: mut Graph, x: Var, y: Var): Var
```

X / Y, element by element.

Effects: fail, read, write, console, io.

### `Graph.maximum`

```gizlang
function Graph.maximum(self: mut Graph, x: Var, y: Var): Var
```

Elementwise larger; a tie sends the gradient to `x`.

Effects: fail, read, write, console, io.

### `Graph.minimum`

```gizlang
function Graph.minimum(self: mut Graph, x: Var, y: Var): Var
```

Elementwise smaller; a tie sends the gradient to `x`.

Effects: fail, read, write, console, io.

### `Graph.neg`

```gizlang
function Graph.neg(self: mut Graph, x: Var): Var
```

-X.

Effects: fail, read, write, console, io.

### `Graph.exp`

```gizlang
function Graph.exp(self: mut Graph, x: Var): Var
```

e to the power X, element by element.

Effects: fail, read, write, console, io.

### `Graph.log`

```gizlang
function Graph.log(self: mut Graph, x: Var): Var
```

The natural logarithm of X, element by element.

Effects: fail, read, write, console, io.

### `Graph.tanh`

```gizlang
function Graph.tanh(self: mut Graph, x: Var): Var
```

The hyperbolic tangent of X, element by element.

Effects: fail, read, write, console, io.

### `Graph.sigmoid`

```gizlang
function Graph.sigmoid(self: mut Graph, x: Var): Var
```

The sigmoid 1 / (1 + e^-x) of X, element by element.

Effects: fail, read, write, console, io.

### `Graph.relu`

```gizlang
function Graph.relu(self: mut Graph, x: Var): Var
```

max(x, 0); the gradient at 0 is 0.

Effects: fail, read, write, console, io.

### `Graph.square`

```gizlang
function Graph.square(self: mut Graph, x: Var): Var
```

X squared, element by element.

Effects: fail, read, write, console, io.

### `Graph.sqrt`

```gizlang
function Graph.sqrt(self: mut Graph, x: Var): Var
```

The square root of X, element by element.

Effects: fail, read, write, console, io.

### `Graph.abs`

```gizlang
function Graph.abs(self: mut Graph, x: Var): Var
```

|x|; the gradient at 0 is 0.

Effects: fail, read, write, console, io.

### `Graph.matmul`

```gizlang
function Graph.matmul(self: mut Graph, x: Var, y: Var): Var
```

(..., m, k) times (k, n), as Tensor.matmul.

Effects: fail, read, write, console, io.

### `Graph.transpose`

```gizlang
function Graph.transpose(self: mut Graph, x: Var): Var
```

X with its last two axes swapped.

Effects: fail, read, write, console, io.

### `Graph.reshape`

```gizlang
function Graph.reshape(self: mut Graph, x: Var, shape: Vec<i64>): Var
```

X with SHAPE (the same number of values).

Effects: fail, read, write, console, io.

### `Graph.sum`

```gizlang
function Graph.sum(self: mut Graph, x: Var): Var
```

The sum of every element (a scalar), in f64 lanes (Reduction.strict).

Effects: fail, read, write, console, io.

### `Graph.mean`

```gizlang
function Graph.mean(self: mut Graph, x: Var): Var
```

The mean of all values of X.

Effects: fail, read, write, console, io.

### `Graph.sumTo`

```gizlang
function Graph.sumTo(self: mut Graph, x: Var, shape: Vec<i64>): Var
```

Summed to `shape` over the axes it keeps as 1 (Tensor.sumTo): (b, n) to (b, 1) sums each row.

Effects: fail, read, write, console, io.

### `Graph.maxTo`

```gizlang
function Graph.maxTo(self: mut Graph, x: Var, shape: Vec<i64>): Var
```

Maxima to `shape` over the axes it keeps as 1 (Tensor.maxTo). Ties share the gradient evenly, as torch.amax.

Effects: fail, read, write, console, io.

### `Graph.logSoftmax`

```gizlang
function Graph.logSoftmax(self: mut Graph, x: Var): Var
```

Log softmax over the last axis: x - max - log(sum(exp(x - max))), from the operations above (both modes).

Effects: fail, read, write, console, io.

### `Graph.softmaxCrossEntropy`

```gizlang
function Graph.softmaxCrossEntropy(self: mut Graph, logits: Var, target: Var): Var
```

The mean over rows of -sum(target * log softmax(logits)) (logits and target (rows, classes); target is
usually a constant, such as one-hot rows or soft labels). Graph.crossEntropy is the fused form for labels.

Effects: fail, read, write, console, io.

### `Graph.custom`

```gizlang
function Graph.custom(self: mut Graph, inputs: Vec<Var>, result: Tensor, backward: Backward): Var
```

An operation with its own gradient: `value` computed from `inputs`, and
`backward` mapping the output's gradient to the inputs'. Reverse mode only.

Effects: fail.

### `Graph.checkpoint`

```gizlang
function Graph.checkpoint(self: mut Graph, inputs: Vec<Var>, f: (g: Graph, xs: Vec<Var>) => Graph): Var
```

`f` applied to `inputs` without keeping its intermediate values: only the
result is stored, and the backward pass runs `f` again for its gradient
(memory for time). `f` gets a fresh Graph and the inputs as its leaves and
returns that Graph with the result as its last value.

Effects: fail, read, write, console, io.

### `Graph.gelu`

```gizlang
function Graph.gelu(self: mut Graph, x: Var): Var
```

GELU (tanh form, as torch's approximate="tanh").

Effects: fail, read, write, console, io.

### `Graph.geluErf`

```gizlang
function Graph.geluErf(self: mut Graph, x: Var): Var
```

GELU (erf form, torch's default).

Effects: fail, read, write, console, io.

### `Graph.attention`

```gizlang
function Graph.attention(self: mut Graph, q: Var, k: Var, v: Var, scale: f64, causal: bool, rate: f64, key: T.Key): Var
```

Scaled dot-product attention as N.attention: q (b, tq, d), k (b, tk, d), v (b, tk, e). The backward pass makes the
weights P again from q and k (no (tq, tk) matrix is kept), then dv = Pdᵀ dy, dPd = dy vᵀ, dS = P (dP - rowsum(dP P))
and dq = dS k scale, dk = dSᵀ q scale, where Pd is P with the dropout mask and dP is dPd with it.

Effects: fail, read, write, console, io.

### `Graph.embedding`

```gizlang
function Graph.embedding(self: mut Graph, table: Var, ids: Vec<i64>): Var
```

Rows `ids` of `table` (rows, width): (ids, width).

Effects: fail, read, write, console, io.

### `Graph.layerNorm`

```gizlang
function Graph.layerNorm(self: mut Graph, x: Var, weight: Var, bias: Var, eps: f64): Var
```

Layer normalization of the last axis with `weight` and `bias` (its length).

Effects: fail, read, write, console, io.

### `Graph.softmax`

```gizlang
function Graph.softmax(self: mut Graph, x: Var): Var
```

Softmax over the last axis.

Effects: fail, read, write, console, io.

### `Graph.crossEntropy`

```gizlang
function Graph.crossEntropy(self: mut Graph, logits: Var, labels: Vec<i64>): Var
```

The mean over rows of the softmax cross-entropy of logits (rows, classes) at integer labels.

Effects: fail, read, write, console, io.

### `Graph.dropout`

```gizlang
function Graph.dropout(self: mut Graph, x: Var, k: T.Key, p: f64): Var
```

x with each element zeroed at rate p (the mask from key k) and the rest scaled by 1/(1-p).

Effects: fail, read, write, console, io.

### `Graph.slice`

```gizlang
function Graph.slice(self: mut Graph, x: Var, axis: i64, start: i64, end: i64): Var
```

Elements start..end of `axis`.

Effects: fail, read, write, console, io.

### `Graph.concat`

```gizlang
function Graph.concat(self: mut Graph, xs: Vec<Var>, axis: i64): Var
```

The values joined along `axis`.

Effects: fail, read, write, console, io.

### `Graph.permute`

```gizlang
function Graph.permute(self: mut Graph, x: Var, axes: Vec<i64>): Var
```

The axes reordered: output axis i is input axis axes[i].

Effects: fail, read, write, console, io.

### `Graph.batchMatmul`

```gizlang
function Graph.batchMatmul(self: mut Graph, x: Var, ta: bool, y: Var, tb: bool): Var
```

Batched matmul as T.batchMatmul: x (b, m, k) or, transposed, (b, k, m); y (b, k, n),
(b, n, k), or one matrix for every batch.

Effects: fail, read, write, console, io.

### `Graph.conv2d`

```gizlang
function Graph.conv2d(self: mut Graph, x: Var, weight: Var, bias: Var, stride: i64, pad: i64): Var
```

A 2-D convolution of x (n, c, h, w) by weight (o, c, kh, kw) plus bias (o), at
`stride` with `pad` zeros: (n, o, oh, ow), as one matmul over im2col's patches.

Effects: fail, read, write, console, io.

### `Graph.pool2d`

```gizlang
function Graph.pool2d(self: mut Graph, x: Var, size: i64, stride: i64, average: bool): Var
```

Max (or average) pooling over size x size windows at `stride`.

Effects: fail, read, write, console, io.

### `Grads.of`

```gizlang
function Grads.of (self: Grads, v: Var): Tensor
```

The gradient of v (each call computes it).

Effects: fail, read, write, console, io.

### `Grads.pending`

```gizlang
function Grads.pending(self: Grads, v: Var): Lazy
```

The gradient of v not yet computed: an optimizer extends it, so where it is a product, the update runs as the
product is stored.

Effects: fail, read, write, console, io.

### `Graph.backward`

```gizlang
function Graph.backward(self: mut Graph, loss: Var): Grads
```

The gradient of `loss`, a one-element value, against every value before it.

Effects: fail, read, write, console, io.

### `Graph.backwardFrom`

```gizlang
function Graph.backwardFrom(self: mut Graph, output: Var, seed: Tensor): Grads
```

The vector-Jacobian product: `seed` (the output's shape) through the tape.

Effects: fail, read, write, console, io.

### `Graph.tangent`

```gizlang
function Graph.tangent(self: mut Graph, inputs: Vec<Var>, directions: Vec<Tensor>, output: Var): Tensor
```

The derivative of `output` along `directions` given for `inputs` (each the
input's shape; other inputs held fixed): one forward walk of the tape (a
Jacobian-vector product). Every built-in operation; not custom ones.

Effects: fail, read, write, console, io.

gizml:grad

derivatives of tensor programs. A Graph records each operation on its values as it runs (a tape); backward walks it once in reverse for the gradient of a scalar against every value (reverse mode), and tangent walks it forward for the derivative along a direction (forward mode). Each rule is the textbook one over gizml:tensor's own operations, so gradients run fused on the CPU's lanes and cores like the forward pass.

gizlang

import * as Grad from "gizml:grad";

Types

`Var`

gizlang

type Var = { id: i64; }

A value in a Graph.

id: i64

`Backward`

gizlang

type Backward = (gradient: Tensor) => Vec<Tensor>

The vector-Jacobian product of a custom operation: the output's gradient to one gradient per input, in the inputs' shapes.

`Graph`

gizlang

type Graph = { nodes: Vec<Node>; customs: Vec<Custom>; compute: i64; }

A tape of tensor operations. Operations take and return Vars; value(v) reads one. A Graph is cheap: make one per step. compute is the autocast type (Graph.autocast).

nodes: Vec<M21.Node>
customs: Vec<M21.Custom>
compute: i64

`Grads`

gizlang

type Grads

Gradients by Var; a value the result does not depend on has a zero gradient.

Opaque: create and use it through this module's functions.

Functions

`graph`

gizlang

function graph(): Graph

An empty graph that computes in f32.

Methods

`Graph.autocast`

gizlang

function Graph.autocast(self: mut Graph, dtype: i64): void

Autocast, as torch.autocast: from now on the matrix products (matmul, batchMatmul, conv2d, forward and backward) of values on a GPU multiply in dtype (T.F16, T.BF16 or T.FP8) on its matrix cores, add in f32 and give dtype (T.productType; f32 for FP8, whose operands are scaled to its range, T.fp8Scaled). Elementwise operations on a product run in its store (T.lazyMatmul) and keep dtype while each other input has that type or one element. Every other operation reads any type and gives f32; T.F32 turns it off. On the CPU nothing changes.

Effects: fail.

`Graph.input`

gizlang

function Graph.input(self: mut Graph, value: Tensor): Var

A leaf: a value derivatives are taken against.

Effects: fail.

`Graph.constant`

gizlang

function Graph.constant(self: mut Graph, value: Tensor): Var

A leaf no derivative is taken against (data, labels): backward computes no gradient for it or for what depends on nothing else, as PyTorch skips a tensor without requires_grad; its gradient reads as zeros.

Effects: fail.

`Graph.value`

gizlang

function Graph.value(self: mut Graph, v: Var): Tensor

The value of V, computed now when it is still an expression.

Effects: fail, read, write, console, io.

`Graph.add`

gizlang

function Graph.add(self: mut Graph, x: Var, y: Var): Var

X + Y, element by element.

Effects: fail, read, write, console, io.

`Graph.sub`

gizlang

function Graph.sub(self: mut Graph, x: Var, y: Var): Var

X - Y, element by element.

Effects: fail, read, write, console, io.

`Graph.mul`

gizlang

function Graph.mul(self: mut Graph, x: Var, y: Var): Var

X * Y, element by element.

Effects: fail, read, write, console, io.

`Graph.div`

gizlang

function Graph.div(self: mut Graph, x: Var, y: Var): Var

X / Y, element by element.

Effects: fail, read, write, console, io.

`Graph.maximum`

gizlang

function Graph.maximum(self: mut Graph, x: Var, y: Var): Var

Elementwise larger; a tie sends the gradient to x.

Effects: fail, read, write, console, io.

`Graph.minimum`

gizlang

function Graph.minimum(self: mut Graph, x: Var, y: Var): Var

Elementwise smaller; a tie sends the gradient to x.

Effects: fail, read, write, console, io.

`Graph.neg`

gizlang

function Graph.neg(self: mut Graph, x: Var): Var

-X.

Effects: fail, read, write, console, io.

`Graph.exp`

gizlang

function Graph.exp(self: mut Graph, x: Var): Var

e to the power X, element by element.

Effects: fail, read, write, console, io.

`Graph.log`

gizlang

function Graph.log(self: mut Graph, x: Var): Var

The natural logarithm of X, element by element.

Effects: fail, read, write, console, io.

`Graph.tanh`

gizlang

function Graph.tanh(self: mut Graph, x: Var): Var

The hyperbolic tangent of X, element by element.

Effects: fail, read, write, console, io.

`Graph.sigmoid`

gizlang

function Graph.sigmoid(self: mut Graph, x: Var): Var

The sigmoid 1 / (1 + e^-x) of X, element by element.

Effects: fail, read, write, console, io.

`Graph.relu`

gizlang

function Graph.relu(self: mut Graph, x: Var): Var

max(x, 0); the gradient at 0 is 0.

Effects: fail, read, write, console, io.

`Graph.square`

gizlang

function Graph.square(self: mut Graph, x: Var): Var

X squared, element by element.

Effects: fail, read, write, console, io.

`Graph.sqrt`

gizlang

function Graph.sqrt(self: mut Graph, x: Var): Var

The square root of X, element by element.

Effects: fail, read, write, console, io.

`Graph.abs`

gizlang

function Graph.abs(self: mut Graph, x: Var): Var

|x|; the gradient at 0 is 0.

Effects: fail, read, write, console, io.

`Graph.matmul`

gizlang

function Graph.matmul(self: mut Graph, x: Var, y: Var): Var

(..., m, k) times (k, n), as Tensor.matmul.

Effects: fail, read, write, console, io.

`Graph.transpose`

gizlang

function Graph.transpose(self: mut Graph, x: Var): Var

X with its last two axes swapped.

Effects: fail, read, write, console, io.

`Graph.reshape`

gizlang

function Graph.reshape(self: mut Graph, x: Var, shape: Vec<i64>): Var

X with SHAPE (the same number of values).

Effects: fail, read, write, console, io.

`Graph.sum`

gizlang

function Graph.sum(self: mut Graph, x: Var): Var

The sum of every element (a scalar), in f64 lanes (Reduction.strict).

Effects: fail, read, write, console, io.

`Graph.mean`

gizlang

function Graph.mean(self: mut Graph, x: Var): Var

The mean of all values of X.

Effects: fail, read, write, console, io.

`Graph.sumTo`

gizlang

function Graph.sumTo(self: mut Graph, x: Var, shape: Vec<i64>): Var

Summed to shape over the axes it keeps as 1 (Tensor.sumTo): (b, n) to (b, 1) sums each row.

Effects: fail, read, write, console, io.

`Graph.maxTo`

gizlang

function Graph.maxTo(self: mut Graph, x: Var, shape: Vec<i64>): Var

Maxima to shape over the axes it keeps as 1 (Tensor.maxTo). Ties share the gradient evenly, as torch.amax.

Effects: fail, read, write, console, io.

`Graph.logSoftmax`

gizlang

function Graph.logSoftmax(self: mut Graph, x: Var): Var

Log softmax over the last axis: x - max - log(sum(exp(x - max))), from the operations above (both modes).

Effects: fail, read, write, console, io.

`Graph.softmaxCrossEntropy`

gizlang

function Graph.softmaxCrossEntropy(self: mut Graph, logits: Var, target: Var): Var

The mean over rows of -sum(target * log softmax(logits)) (logits and target (rows, classes); target is usually a constant, such as one-hot rows or soft labels). Graph.crossEntropy is the fused form for labels.

Effects: fail, read, write, console, io.

`Graph.custom`

gizlang

function Graph.custom(self: mut Graph, inputs: Vec<Var>, result: Tensor, backward: Backward): Var

An operation with its own gradient: value computed from inputs, and backward mapping the output's gradient to the inputs'. Reverse mode only.

Effects: fail.

`Graph.checkpoint`

gizlang

function Graph.checkpoint(self: mut Graph, inputs: Vec<Var>, f: (g: Graph, xs: Vec<Var>) => Graph): Var

f applied to inputs without keeping its intermediate values: only the result is stored, and the backward pass runs f again for its gradient (memory for time). f gets a fresh Graph and the inputs as its leaves and returns that Graph with the result as its last value.

Effects: fail, read, write, console, io.

`Graph.gelu`

gizlang

function Graph.gelu(self: mut Graph, x: Var): Var

GELU (tanh form, as torch's approximate="tanh").

Effects: fail, read, write, console, io.

`Graph.geluErf`

gizlang

function Graph.geluErf(self: mut Graph, x: Var): Var

GELU (erf form, torch's default).

Effects: fail, read, write, console, io.

`Graph.attention`

gizlang

function Graph.attention(self: mut Graph, q: Var, k: Var, v: Var, scale: f64, causal: bool, rate: f64, key: T.Key): Var

Scaled dot-product attention as N.attention: q (b, tq, d), k (b, tk, d), v (b, tk, e). The backward pass makes the weights P again from q and k (no (tq, tk) matrix is kept), then dv = Pdᵀ dy, dPd = dy vᵀ, dS = P (dP - rowsum(dP P)) and dq = dS k scale, dk = dSᵀ q scale, where Pd is P with the dropout mask and dP is dPd with it.

Effects: fail, read, write, console, io.

`Graph.embedding`

gizlang

function Graph.embedding(self: mut Graph, table: Var, ids: Vec<i64>): Var

Rows ids of table (rows, width): (ids, width).

Effects: fail, read, write, console, io.

`Graph.layerNorm`

gizlang

function Graph.layerNorm(self: mut Graph, x: Var, weight: Var, bias: Var, eps: f64): Var

Layer normalization of the last axis with weight and bias (its length).

Effects: fail, read, write, console, io.

`Graph.softmax`

gizlang

function Graph.softmax(self: mut Graph, x: Var): Var

Softmax over the last axis.

Effects: fail, read, write, console, io.

`Graph.crossEntropy`

gizlang

function Graph.crossEntropy(self: mut Graph, logits: Var, labels: Vec<i64>): Var

The mean over rows of the softmax cross-entropy of logits (rows, classes) at integer labels.

Effects: fail, read, write, console, io.

`Graph.dropout`

gizlang

function Graph.dropout(self: mut Graph, x: Var, k: T.Key, p: f64): Var

x with each element zeroed at rate p (the mask from key k) and the rest scaled by 1/(1-p).

Effects: fail, read, write, console, io.

`Graph.slice`

gizlang

function Graph.slice(self: mut Graph, x: Var, axis: i64, start: i64, end: i64): Var

Elements start..end of axis.

Effects: fail, read, write, console, io.

`Graph.concat`

gizlang

function Graph.concat(self: mut Graph, xs: Vec<Var>, axis: i64): Var

The values joined along axis.

Effects: fail, read, write, console, io.

`Graph.permute`

gizlang

function Graph.permute(self: mut Graph, x: Var, axes: Vec<i64>): Var

The axes reordered: output axis i is input axis axes[i].

Effects: fail, read, write, console, io.

`Graph.batchMatmul`

gizlang

function Graph.batchMatmul(self: mut Graph, x: Var, ta: bool, y: Var, tb: bool): Var

Batched matmul as T.batchMatmul: x (b, m, k) or, transposed, (b, k, m); y (b, k, n), (b, n, k), or one matrix for every batch.

Effects: fail, read, write, console, io.

`Graph.conv2d`

gizlang

function Graph.conv2d(self: mut Graph, x: Var, weight: Var, bias: Var, stride: i64, pad: i64): Var

A 2-D convolution of x (n, c, h, w) by weight (o, c, kh, kw) plus bias (o), at stride with pad zeros: (n, o, oh, ow), as one matmul over im2col's patches.

Effects: fail, read, write, console, io.

`Graph.pool2d`

gizlang

function Graph.pool2d(self: mut Graph, x: Var, size: i64, stride: i64, average: bool): Var

Max (or average) pooling over size x size windows at stride.

Effects: fail, read, write, console, io.

`Grads.of`

gizlang

function Grads.of (self: Grads, v: Var): Tensor

The gradient of v (each call computes it).

Effects: fail, read, write, console, io.

`Grads.pending`

gizlang

function Grads.pending(self: Grads, v: Var): Lazy

The gradient of v not yet computed: an optimizer extends it, so where it is a product, the update runs as the product is stored.

Effects: fail, read, write, console, io.

`Graph.backward`

gizlang

function Graph.backward(self: mut Graph, loss: Var): Grads

The gradient of loss, a one-element value, against every value before it.

Effects: fail, read, write, console, io.

`Graph.backwardFrom`

gizlang

function Graph.backwardFrom(self: mut Graph, output: Var, seed: Tensor): Grads

The vector-Jacobian product: seed (the output's shape) through the tape.

Effects: fail, read, write, console, io.

`Graph.tangent`

gizlang

function Graph.tangent(self: mut Graph, inputs: Vec<Var>, directions: Vec<Tensor>, output: Var): Tensor

The derivative of output along directions given for inputs (each the input's shape; other inputs held fixed): one forward walk of the tape (a Jacobian-vector product). Every built-in operation; not custom ones.

Effects: fail, read, write, console, io.