# gizml:netops

the operations of networks on gizml:tensor Tensors. Each one runs where its operand is, on the CPU or a GPU, and adds in one order on both: row terms and norms, losses, argmax, attention (plain, packed rows, 8-bit matrix cores), dropout, convolution and pooling, the modulated norm with INT8 rows, and the rows of a language model's step (argmax, sampling candidates, split sums, rotated norm and SwiGLU).

```gizlang
import * as Netops from "gizml:netops";
```

## Types

### `HeadPrep`

```gizlang
type HeadPrep = { qw: Tensor; kw: Tensor; cos: Tensor; sin: Tensor; eps: f64; }
```

Each head of q and k over its RMS (eps) times its norm weight (qw, kw: `[head]` F32), then its values e < 2 half
turned with e ± half by its row's cos and sin ([rows, half] F32, half = 0: none): a transformer's QK norm and rotary
(H3's split halves; FLUX.3's pairs, loaded even values first). The quantized path applies it as its passes read q
and k; the BF16 path first in place (prepKernel), so qkv's q and k may change.

- `qw`: `M32.Tensor`
- `kw`: `M32.Tensor`
- `cos`: `M32.Tensor`
- `sin`: `M32.Tensor`
- `eps`: `f64`

## Functions

### `affine`

```gizlang
function affine(x: Tensor, w: Tensor, b: Tensor, compute: i64): Tensor
```

x Wᵀ + b ([rows, out] f32), the products in `compute` on a GPU (as lazyMatmulIn).

Effects: fail, read, write, console, io.

### `layerNorm`

```gizlang
function layerNorm(x: Tensor, weight: Tensor, bias: Tensor, eps: f64): Vec<Tensor>
```

Layer normalization over the last axis: (x - mean) / sqrt(var + eps) * weight
+ bias, with mean and var over the row (1/n, as torch). Returns the output and
each row's mean and 1/sqrt(var + eps), which layerNormGrad takes.

Effects: fail, read, write, console, io.

### `layerNormGrad`

```gizlang
function layerNormGrad(dy: Tensor, x: Tensor, weight: Tensor, mean: Tensor, rstd: Tensor): Vec<Tensor>
```

The gradients of layerNorm against x, weight and bias from the output's
gradient dy and layerNorm's row means and rstds: dx = rstd (dy w - mean(dy w)
- xhat mean(dy w xhat)); dweight and dbias add over rows in order.

Effects: fail, read, write, console, io.

### `crossEntropyLoss`

```gizlang
function crossEntropyLoss(logits: Tensor, labels: Vec<i64>): f64
```

Softmax cross-entropy of logits (rows, classes) against integer labels: each
row's loss m + log(sum exp(x - m)) - x[label] (the sum in 16 lanes), and their
mean in f64 in row order.

Effects: fail, read, write, console, io.

### `crossEntropyRows`

```gizlang
function crossEntropyRows(logits: Tensor, labels: Vec<i64>): Tensor
```

Each row's loss of crossEntropyLoss, as a tensor of shape `[rows]` where the logits are (on a GPU: no wait).

Effects: fail, read, write, console, io.

### `crossEntropyGrad`

```gizlang
function crossEntropyGrad(logits: Tensor, labels: Vec<i64>, scale: f64): Tensor
```

Its gradient times `scale` (the loss's gradient over rows): (softmax - onehot) scale,
the softmax exp(x - m) times the f32 of 1/sum.

Effects: fail, read, write, console, io.

### `argmaxInto`

```gizlang
function argmaxInto(g: Gpu, rows: i64, vocab: i64, stride: i64, indirect: bool, top: u64, logits: u64, blocks: u64): void
```

argmaxKernel over `rows` rows of the logits at `logits` (rows of `stride`, of the head blocks in `blocks` when
`indirect`), each row's token as f32 into `top`.

Effects: fail, read, write, console, io.

### `candidatesInto`

```gizlang
function candidatesInto(g: Gpu, rows: i64, vocab: i64, stride: i64, indirect: bool, cand: u64, logits: u64, blocks: u64): void
```

candidatesKernel over `rows` rows of the logits at `logits` (as argmaxInto reads them) into `cand`.

Effects: fail, read, write, console, io.

### `splitSumInto`

```gizlang
function splitSumInto(g: Gpu, y: u64, p: u64, total: i64, splits: i64, add: bool): void
```

splitSumKernel over the `total` f32 values at y and the parts at p.

Effects: fail, read, write, console, io.

### `normRotateInto`

```gizlang
function normRotateInto(g: Gpu, rows: i64, width: i64, eps: f64, rotate: bool, stride: i64, offset: i64, y: u64, x: u64, w: u64, d: u64, map: u64): void
```

normRotateKernel over `rows` rows of `width` (the row map at `map`, as f32 indexes).

Effects: fail, read, write, console, io.

### `siluMulRotatedInto`

```gizlang
function siluMulRotatedInto(g: Gpu, rows: i64, ffn: i64, nUp: i64, y: u64, p: u64, d: u64): void
```

siluMulKernel over `rows` rows.

Effects: fail, read, write, console, io.

### `attention`

```gizlang
function attention(q: Tensor, k: Tensor, v: Tensor, scale: f64, causal: bool, rate: f64, key: Key): Tensor
```

Scaled dot-product attention, softmax(q kᵀ scale) v, without the (tq, tk) matrix of scores: q (b, tq, d), k
(b, tk, d) and v (b, tk, e) give (b, tq, e). With `causal`, query i sees the keys up to i + tk - tq (a decoder's
mask; with a KV cache tq < tk). With `rate` > 0 each weight is dropped at that rate (the mask of `key`'s word
(b tq + i) tk + j, as Tensor.dropout draws it for the weights) and the rest scaled by 1 / (1 - rate).

The CPU and a GPU take the same steps in f32, so they give the same bits: the keys go in blocks of 16; each score
adds q[p] k[j][p] by fma for p from 0, then is scaled; the block's maximum m' gives c = e^(m - m') and the weights
e^(s - m'); the sum l becomes l c plus the weights in order, and each output element a becomes a c plus the
weights times the values by fma in order; the result is a / l. attentionRows is the fast path on matrix cores for
a transformer's packed BF16 rows.

Effects: fail, read, write, console, io.

### `causalMask`

```gizlang
function causalMask(tq: i64, tk: i64): Tensor
```

The additive mask of causal attention, (tq, tk): 0 where query i sees key j (j <= i + tk - tq), else -inf.

Effects: fail.

### `im2col`

```gizlang
function im2col(x: Tensor, kh: i64, kw: i64, stride: i64, pad: i64): Tensor
```

The patches of x (n, c, h, w) under a kh x kw window at `stride` with `pad`
zeros around: (n * oh * ow, c * kh * kw), so a convolution is one matmul.

Effects: fail, read, write, console, io.

### `patchRows`

```gizlang
function patchRows(x: Tensor, kh: i64, kw: i64, pad: i64, r0: i64, r1: i64, dtype: i64): Tensor
```

The patches of x ([c, h, w]) under a kh x kw window (stride 1, `pad` zeros around) for output rows r0 to r1, laid
by patch entry: [c kh kw, (r1 - r0) ow] of `dtype` (ow = w + 2 pad - kw + 1), entry (ch kh kw + i kw + j,
(oy - r0) ow + ox) = x[ch, oy + i - pad, ox + j - pad]. W [o, c kh kw] times it is the convolution's rows r0 to r1,
channel first, with no padded copy of x; on a GPU its reads and its writes both run along pixels.

Effects: fail, read, write, console, io.

### `lazyConv`

```gizlang
function lazyConv(w: Tensor, x: Tensor, kh: i64, kw: i64, pad: i64, compute: i64): Lazy
```

w ([o, c kh kw]) times the patches of x ([c, h, w]) under a kh x kw window at stride 1 with `pad` zeros around
(patchRows): the convolution, [o, oh ow] channel first, not yet computed, so its bias and what follows run in the
product's store. On NVIDIA's matrix cores (BF16, 3 x 3, pad 1, w a multiple of 8) the product reads the patches
from x as it copies them (Product.patch, patchFetch), so they are never stored; elsewhere they are built first.

Effects: fail, read, write, console, io.

### `lazyConv3`

```gizlang
function lazyConv3(w: Tensor, x: Tensor): Lazy
```

w ([o, 27 c], its inputs in (time, channel, height, width) order) over frames 1 .. n of x ([n + 2, c, h, w] BF16,
the edge frames 0 and n + 1 given): the 3 x 3 x 3 convolution with zeros around in space, [n, o, h w], not yet
computed. On NVIDIA's matrix cores (w a multiple of 8) it is one product of n batches whose B is the patches of
frames t .. t + 2 as an image of 3 c channels (Product.patch); elsewhere a lazyConv a frame, each computed in F32.

Effects: fail, read, write, console, io.

### `pixelNormInto`

```gizlang
function pixelNormInto(x: Tensor, eps: f64, act: bool, out: Tensor): void
```

PixelNorm of x ([frames, c, pixels] BF16 or F32 on a GPU, channel first in each frame): each pixel's c values over
their RMS (eps), then SiLU when `act`, into out (x's shape and dtype; x itself is allowed): StyleGAN's and LTX's VAE
norm.

Effects: fail, read, write, console, io.

### `groupNorm`

```gizlang
function groupNorm(x: Tensor, groups: i64, w: Tensor, b: Tensor, eps: f64, act: bool, dtype: i64): Tensor
```

GroupNorm of x ([c, ...] f32, channel first): each of `groups` groups of c / groups channels less its mean, over
the square root of its variance plus eps, times w[ch] plus b[ch], then SiLU when `act`: `dtype` elements of x's
shape. On a GPU a group's sums run on GROUP_SLICES workgroups (f32 within one, f64 across), so a few long groups
fill the GPU; the host adds in f64.

Effects: fail, read, write, console, io.

### `col2im`

```gizlang
function col2im(cols: Tensor, n: i64, c: i64, h: i64, w: i64, kh: i64, kw: i64, stride: i64, pad: i64): Tensor
```

The adjoint of im2col: element (b, ch, y, x) adds every patch entry that
copied it, window offsets (i, j) in increasing order.

Effects: fail, read, write, console, io.

### `im2col1d`

```gizlang
function im2col1d(x: Tensor, k: i64, stride: i64, pad: i64, dilation: i64, first: i64 = 0, count: i64 = -1): Tensor
```

The patches of x (t, c), time first, under a window of k taps `dilation` apart at `stride`, with `pad` zeros
before and after: (ot, c * k), entry (o, ch * k + j) = x[o * stride + j * dilation - pad, ch], so a 1-D
convolution is one matmul (conv1d). With `count` >= 0, only the `count` rows from row `first`.

Effects: fail, read, write, console, io.

### `col2im1d`

```gizlang
function col2im1d(cols: Tensor, c: i64, length: i64, k: i64, stride: i64, pad: i64): Tensor
```

The adjoint of im2col1d at dilation 1 on cols (n, c * k): the (length, c) whose entry (o, ch) adds every
cols[i, ch * k + j] with i * stride + j - pad = o, j in increasing order. A transposed convolution is the matmul
x W then this (convTranspose1d).

Effects: fail, read, write, console, io.

### `conv1d`

```gizlang
function conv1d(x: Tensor, w: Tensor, bias: Vec<Tensor>, stride: i64, pad: i64, dilation: i64): Tensor
```

The 1-D convolution of x (t, c), time first, by w (o, c, k) with `bias` (o) unless empty: (ot, o), as PyTorch's
conv1d of xᵀ transposed.

Effects: fail, read, write, console, io.

### `convTranspose1d`

```gizlang
function convTranspose1d(x: Tensor, w: Tensor, bias: Vec<Tensor>, stride: i64, pad: i64): Tensor
```

The 1-D transposed convolution of x (n, c), time first, by w (c, o, k) at `stride` with `pad` with `bias` (o)
unless empty: ((n - 1) stride - 2 pad + k, o), as PyTorch's conv_transpose1d of xᵀ transposed.

Effects: fail, read, write, console, io.

### `pool2d`

```gizlang
function pool2d(x: Tensor, size: i64, stride: i64, average: bool): Tensor
```

Max (or, `average`, mean) pooling of x (n, c, h, w) over k x k windows at
`stride`: the max is the first largest in window order; the mean adds the
window in order and multiplies by the f32 of 1/(k k).

Effects: fail, read, write, console, io.

### `pool2dGrad`

```gizlang
function pool2dGrad(dy: Tensor, x: Tensor, size: i64, stride: i64, average: bool): Tensor
```

The gradient of pool2d against x from dy: each element adds, over the windows
holding it in order, dy/(k k) (average) or dy where it is the window's max.

Effects: fail, read, write, console, io.

### `attentionRows`

```gizlang
function attentionRows(qkv: Tensor, heads: i64, head: i64, quantized: bool, group: i64, out: Tensor, prep: Vec<HeadPrep> = [], queries: i64 = 0): void
```

Full attention of every head of qkv ([n, 3 heads head] BF16 on a GPU: each row's q, k and v of `heads` heads of
`head`, rotated as the model needs) into out ([n, heads head] BF16): softmax(q kᵀ / sqrt(head)) v a head, the
keys of each group of `group` rows alone (0: all rows). With `quantized`, where the GPU has 8-bit matrix cores
(K.hasFp8Mma) and heads are 64 or 128, Q Kᵀ runs on INT8 and P V on E4M3 (attention8, SageAttention's split);
else on BF16. `prep` (a HeadPrep or none) normalizes and turns each head of q and k as attention reads them: the
8-bit path in its quantizing pass, the BF16 path in place in qkv first.

Effects: fail, read, write, console, io.

### `attendedRows`

```gizlang
function attendedRows(qkv: Tensor, heads: i64, head: i64, quantized: bool, group: i64, prep: Vec<HeadPrep> = [], queries: i64 = 0): Tensor
```

attentionRows' output as a new tensor where qkv is: [rows (queries when given), heads × head], BF16 on a GPU. On
the host in F32 and exact (`quantized` has no effect): prep, then attention among the rows of each group.

Effects: fail, read, write, console, io.

### `headPrepInPlace`

```gizlang
function headPrepInPlace(qkv: Tensor, n: i64, heads: i64, head: i64, p: HeadPrep, row0: i64): void
```

p of q and k of the first n rows of qkv ([rows, 3 heads head] BF16 on a GPU, each row its q, k and v) in place
(prepKernel), `row0` the sequence row of the first.

Effects: fail, read, write, console, io.

### `crossAttentionRows`

```gizlang
function crossAttentionRows(q: Tensor, kv: Tensor, heads: i64, head: i64, out: Tensor): void
```

Attention of the rows of q ([nq, heads head] BF16 on a GPU) over the keys and values of kv ([nk, 2 heads head]:
each row's k, then its v), as attentionRows: a cross attention (a text's, another modality's). Into out ([nq, heads
head] BF16), on the BF16 matrix cores.

Effects: fail, read, write, console, io.

### `neighborhood3d`

```gizlang
function neighborhood3d(qkv: Tensor, grid: Vec<i64>, kernel: Vec<i64>, heads: i64, scale: f64): Tensor
```

3D neighborhood attention (NATTEN's na3d: dilation 1, not causal) of qkv ([T H W, 3 heads head] BF16 on a GPU, F32
on the host; rows in (t, h, w) order, each row its q, k and v, each its heads, as a fused QKV linear gives them; as
`attention` reads it): each query attends
the keys of a window of `kernel` [kt, kh, kw] positions (each odd) around it, shifted inward at the grid's edges and
as long as the axis at most, with scores q · k times `scale`. The output has q's layout, BF16 on a GPU (flash attention
over the box of keys each tile of 64 queries can see), F32 on the host (exact).

Effects: fail, read, write, console, io.

### `modulatedNorm`

```gizlang
function modulatedNorm(x: Tensor, w: Tensor, shift: Tensor, scale: Tensor, map: Tensor, eps: f64, centered: bool = false): Tensor
```

Each row of x ([rows, width] F32) over its RMS (with `centered`, LayerNorm's: less its mean, over its standard
deviation), times w, times 1 + scale and plus shift of its modulation row (map[r], an F32 row index into shift and
scale): a diffusion transformer's adaptive norm (AdaLN). F32 on the host, BF16 on a GPU.

Effects: fail, read, write, console, io.

### `modulatedNormInto`

```gizlang
function modulatedNormInto(x: Tensor, w: Tensor, shift: Tensor, scale: Tensor, map: Tensor, eps: f64, y: Tensor, centered: bool = false): void
```

modulatedNorm on a GPU into y (BF16).

Effects: fail, read, write, console, io.

### `gatedNormAddInto`

```gizlang
function gatedNormAddInto(x: Tensor, y: Tensor, w: Tensor, gate: Tensor, map: Tensor, eps: f64): void
```

x ([rows, width] F32, on a GPU) plus, each row, y's row (BF16) over its RMS (eps) times w, times the row map[r] of
gate ([modulation rows, width] F32): a block's output norm under its gate (Z-Image's sandwich norm). In place.

Effects: fail, read, write, console, io.

### `rowNormRopeInto`

```gizlang
function rowNormRopeInto(gpu: Gpu, at: u64, stride: i64, rows: i64, width: i64, head: i64, w: Tensor, eps: f64, cos: Vec<Tensor> = [], sin: Vec<Tensor> = []): void
```

Each of `rows` rows of `width` BF16 values (from address `at`, rows `stride` values apart, on `gpu`) over its RMS
across the whole row (eps), times w (`width` F32 values), then, with cos and sin ([rows, width / 2] F32), each head of
`head` values turned by split halves: value d < head / 2 with value d + head / 2 by its pair's angle (pair h head / 2
+ d of the row): LTX's QK norm, which spans every head, and its rotary. In place.

Effects: fail, read, write, console, io.

### `int8Rows`

```gizlang
function int8Rows(x: Tensor, bits: i64 = 8): Vec<Tensor>
```

Rows of x (BF16 on a GPU, a multiple of 256 wide) as INT8 over a scale a row, as ConvRot quantizes activations
(WanGP's asym_w4a8_int8, comfy-kitchen's int8_linear): each 256 values times the regular Hadamard matrix over 16
(Kernel.hadamard regular: its outputs in that order, so the weights take K.regularSource and K.regularSign), in F32
(y); s = max |y| / 127 a row ([rows, 1] F32, at least 1e-30); q = y / s to nearest even within [-128, 127]. Those
references round y and y / s to BF16 first, which moves a code by one where the BF16 step passes a half: LTX-2.5's
video velocity was 26.7% off its BF16-activation result with that rounding and 22.6% without it (ComfyUI's 20.8%).
[q, s]. With 4 `bits`, q is I4: s = max |y| / 7, q within [-8, 7]. On the host x holds BF16 values (as F32) and q
the integers (as F32): the same steps, the transform gizlang:linalg's.

Effects: fail, read, write, console, io.

### `int8RowsInto`

```gizlang
function int8RowsInto(x: Tensor, q: Tensor, s: Tensor): void
```

int8Rows into q ([rows, width] I8, or I4 for 4 bits) and s ([rows, 1] F32).

Effects: fail, read, write, console, io.

### `modulatedInt8RowsInto`

```gizlang
function modulatedInt8RowsInto(x: Tensor, w: Tensor, shift: Tensor, scale: Tensor, map: Tensor, eps: f64, y: Option<Tensor>, q: Tensor, s: Tensor, centered: bool = false): void
```

int8Rows of modulatedNorm's rows of x (F32) in one pass, into q and s; with `y` modulatedNorm's values there too.

Effects: fail, read, write, console, io.

## Methods

### `Tensor.argmax`

```gizlang
function Tensor.argmax(self: Tensor): Vec<i64>
```

The index of each row's largest element (the first of equals; NaN never wins).

Effects: fail, read, write, console, io.

### `Tensor.dropout`

```gizlang
function Tensor.dropout(self: Tensor, k: Key, p: f64): Tensor
```

x where word i of `key`'s draw (as `uniform` makes it) is at least p, scaled
by 1 / (1 - p), else 0: the same mask on the CPU and a GPU, so the backward
pass applies it again from the key.

Effects: fail, read, write, console, io.

## Constants

### `TOP_EACH`

```gizlang
const TOP_EACH = 4
```

The largest logits each of a candidates workgroup's 256 threads keeps of the values it reads, and the floats of a
row's candidates (a value and a token each: 256 TOP_EACH 2).

### `CANDIDATE_ROW`

```gizlang
const CANDIDATE_ROW = 2048
```

The floats of a row's candidates in a candidates workgroup.

gizml:netops

the operations of networks on gizml:tensor Tensors. Each one runs where its operand is, on the CPU or a GPU, and adds in one order on both: row terms and norms, losses, argmax, attention (plain, packed rows, 8-bit matrix cores), dropout, convolution and pooling, the modulated norm with INT8 rows, and the rows of a language model's step (argmax, sampling candidates, split sums, rotated norm and SwiGLU).

gizlang

import * as Netops from "gizml:netops";

Types

`HeadPrep`

gizlang

type HeadPrep = { qw: Tensor; kw: Tensor; cos: Tensor; sin: Tensor; eps: f64; }

Each head of q and k over its RMS (eps) times its norm weight (qw, kw: [head] F32), then its values e < 2 half turned with e ± half by its row's cos and sin ([rows, half] F32, half = 0: none): a transformer's QK norm and rotary (H3's split halves; FLUX.3's pairs, loaded even values first). The quantized path applies it as its passes read q and k; the BF16 path first in place (prepKernel), so qkv's q and k may change.

qw: M32.Tensor
kw: M32.Tensor
cos: M32.Tensor
sin: M32.Tensor
eps: f64

Functions

`affine`

gizlang

function affine(x: Tensor, w: Tensor, b: Tensor, compute: i64): Tensor

x Wᵀ + b ([rows, out] f32), the products in compute on a GPU (as lazyMatmulIn).

Effects: fail, read, write, console, io.

`layerNorm`

gizlang

function layerNorm(x: Tensor, weight: Tensor, bias: Tensor, eps: f64): Vec<Tensor>

Layer normalization over the last axis: (x - mean) / sqrt(var + eps) * weight

bias, with mean and var over the row (1/n, as torch). Returns the output and each row's mean and 1/sqrt(var + eps), which layerNormGrad takes.

Effects: fail, read, write, console, io.

`layerNormGrad`

gizlang

function layerNormGrad(dy: Tensor, x: Tensor, weight: Tensor, mean: Tensor, rstd: Tensor): Vec<Tensor>

The gradients of layerNorm against x, weight and bias from the output's gradient dy and layerNorm's row means and rstds: dx = rstd (dy w - mean(dy w)

xhat mean(dy w xhat)); dweight and dbias add over rows in order.

Effects: fail, read, write, console, io.

`crossEntropyLoss`

gizlang

function crossEntropyLoss(logits: Tensor, labels: Vec<i64>): f64

Softmax cross-entropy of logits (rows, classes) against integer labels: each row's loss m + log(sum exp(x - m)) - x[label] (the sum in 16 lanes), and their mean in f64 in row order.

Effects: fail, read, write, console, io.

`crossEntropyRows`

gizlang

function crossEntropyRows(logits: Tensor, labels: Vec<i64>): Tensor

Each row's loss of crossEntropyLoss, as a tensor of shape [rows] where the logits are (on a GPU: no wait).

Effects: fail, read, write, console, io.

`crossEntropyGrad`

gizlang

function crossEntropyGrad(logits: Tensor, labels: Vec<i64>, scale: f64): Tensor

Its gradient times scale (the loss's gradient over rows): (softmax - onehot) scale, the softmax exp(x - m) times the f32 of 1/sum.

Effects: fail, read, write, console, io.

`argmaxInto`

gizlang

function argmaxInto(g: Gpu, rows: i64, vocab: i64, stride: i64, indirect: bool, top: u64, logits: u64, blocks: u64): void

argmaxKernel over rows rows of the logits at logits (rows of stride, of the head blocks in blocks when indirect), each row's token as f32 into top.

Effects: fail, read, write, console, io.

`candidatesInto`

gizlang

function candidatesInto(g: Gpu, rows: i64, vocab: i64, stride: i64, indirect: bool, cand: u64, logits: u64, blocks: u64): void

candidatesKernel over rows rows of the logits at logits (as argmaxInto reads them) into cand.

Effects: fail, read, write, console, io.

`splitSumInto`

gizlang

function splitSumInto(g: Gpu, y: u64, p: u64, total: i64, splits: i64, add: bool): void

splitSumKernel over the total f32 values at y and the parts at p.

Effects: fail, read, write, console, io.

`normRotateInto`

gizlang

function normRotateInto(g: Gpu, rows: i64, width: i64, eps: f64, rotate: bool, stride: i64, offset: i64, y: u64, x: u64, w: u64, d: u64, map: u64): void

normRotateKernel over rows rows of width (the row map at map, as f32 indexes).

Effects: fail, read, write, console, io.

`siluMulRotatedInto`

gizlang

function siluMulRotatedInto(g: Gpu, rows: i64, ffn: i64, nUp: i64, y: u64, p: u64, d: u64): void

siluMulKernel over rows rows.

Effects: fail, read, write, console, io.

`attention`

gizlang

function attention(q: Tensor, k: Tensor, v: Tensor, scale: f64, causal: bool, rate: f64, key: Key): Tensor

Scaled dot-product attention, softmax(q kᵀ scale) v, without the (tq, tk) matrix of scores: q (b, tq, d), k (b, tk, d) and v (b, tk, e) give (b, tq, e). With causal, query i sees the keys up to i + tk - tq (a decoder's mask; with a KV cache tq < tk). With rate > 0 each weight is dropped at that rate (the mask of key's word (b tq + i) tk + j, as Tensor.dropout draws it for the weights) and the rest scaled by 1 / (1 - rate).

The CPU and a GPU take the same steps in f32, so they give the same bits: the keys go in blocks of 16; each score adds q[p] k[j][p] by fma for p from 0, then is scaled; the block's maximum m' gives c = e^(m - m') and the weights e^(s - m'); the sum l becomes l c plus the weights in order, and each output element a becomes a c plus the weights times the values by fma in order; the result is a / l. attentionRows is the fast path on matrix cores for a transformer's packed BF16 rows.

Effects: fail, read, write, console, io.

`causalMask`

gizlang

function causalMask(tq: i64, tk: i64): Tensor

The additive mask of causal attention, (tq, tk): 0 where query i sees key j (j <= i + tk - tq), else -inf.

Effects: fail.

`im2col`

gizlang

function im2col(x: Tensor, kh: i64, kw: i64, stride: i64, pad: i64): Tensor

The patches of x (n, c, h, w) under a kh x kw window at stride with pad zeros around: (n * oh * ow, c * kh * kw), so a convolution is one matmul.

Effects: fail, read, write, console, io.

`patchRows`

gizlang

function patchRows(x: Tensor, kh: i64, kw: i64, pad: i64, r0: i64, r1: i64, dtype: i64): Tensor

The patches of x ([c, h, w]) under a kh x kw window (stride 1, pad zeros around) for output rows r0 to r1, laid by patch entry: [c kh kw, (r1 - r0) ow] of dtype (ow = w + 2 pad - kw + 1), entry (ch kh kw + i kw + j, (oy - r0) ow + ox) = x[ch, oy + i - pad, ox + j - pad]. W [o, c kh kw] times it is the convolution's rows r0 to r1, channel first, with no padded copy of x; on a GPU its reads and its writes both run along pixels.

Effects: fail, read, write, console, io.

`lazyConv`

gizlang

function lazyConv(w: Tensor, x: Tensor, kh: i64, kw: i64, pad: i64, compute: i64): Lazy

w ([o, c kh kw]) times the patches of x ([c, h, w]) under a kh x kw window at stride 1 with pad zeros around (patchRows): the convolution, [o, oh ow] channel first, not yet computed, so its bias and what follows run in the product's store. On NVIDIA's matrix cores (BF16, 3 x 3, pad 1, w a multiple of 8) the product reads the patches from x as it copies them (Product.patch, patchFetch), so they are never stored; elsewhere they are built first.

Effects: fail, read, write, console, io.

`lazyConv3`

gizlang

function lazyConv3(w: Tensor, x: Tensor): Lazy

w ([o, 27 c], its inputs in (time, channel, height, width) order) over frames 1 .. n of x ([n + 2, c, h, w] BF16, the edge frames 0 and n + 1 given): the 3 x 3 x 3 convolution with zeros around in space, [n, o, h w], not yet computed. On NVIDIA's matrix cores (w a multiple of 8) it is one product of n batches whose B is the patches of frames t .. t + 2 as an image of 3 c channels (Product.patch); elsewhere a lazyConv a frame, each computed in F32.

Effects: fail, read, write, console, io.

`pixelNormInto`

gizlang

function pixelNormInto(x: Tensor, eps: f64, act: bool, out: Tensor): void

PixelNorm of x ([frames, c, pixels] BF16 or F32 on a GPU, channel first in each frame): each pixel's c values over their RMS (eps), then SiLU when act, into out (x's shape and dtype; x itself is allowed): StyleGAN's and LTX's VAE norm.

Effects: fail, read, write, console, io.

`groupNorm`

gizlang

function groupNorm(x: Tensor, groups: i64, w: Tensor, b: Tensor, eps: f64, act: bool, dtype: i64): Tensor

GroupNorm of x ([c, ...] f32, channel first): each of groups groups of c / groups channels less its mean, over the square root of its variance plus eps, times w[ch] plus b[ch], then SiLU when act: dtype elements of x's shape. On a GPU a group's sums run on GROUP_SLICES workgroups (f32 within one, f64 across), so a few long groups fill the GPU; the host adds in f64.

Effects: fail, read, write, console, io.

`col2im`

gizlang

function col2im(cols: Tensor, n: i64, c: i64, h: i64, w: i64, kh: i64, kw: i64, stride: i64, pad: i64): Tensor

The adjoint of im2col: element (b, ch, y, x) adds every patch entry that copied it, window offsets (i, j) in increasing order.

Effects: fail, read, write, console, io.

`im2col1d`

gizlang

function im2col1d(x: Tensor, k: i64, stride: i64, pad: i64, dilation: i64, first: i64 = 0, count: i64 = -1): Tensor

The patches of x (t, c), time first, under a window of k taps dilation apart at stride, with pad zeros before and after: (ot, c * k), entry (o, ch * k + j) = x[o * stride + j * dilation - pad, ch], so a 1-D convolution is one matmul (conv1d). With count >= 0, only the count rows from row first.

Effects: fail, read, write, console, io.

`col2im1d`

gizlang

function col2im1d(cols: Tensor, c: i64, length: i64, k: i64, stride: i64, pad: i64): Tensor

The adjoint of im2col1d at dilation 1 on cols (n, c * k): the (length, c) whose entry (o, ch) adds every cols[i, ch * k + j] with i * stride + j - pad = o, j in increasing order. A transposed convolution is the matmul x W then this (convTranspose1d).

Effects: fail, read, write, console, io.

`conv1d`

gizlang

function conv1d(x: Tensor, w: Tensor, bias: Vec<Tensor>, stride: i64, pad: i64, dilation: i64): Tensor

The 1-D convolution of x (t, c), time first, by w (o, c, k) with bias (o) unless empty: (ot, o), as PyTorch's conv1d of xᵀ transposed.

Effects: fail, read, write, console, io.

`convTranspose1d`

gizlang

function convTranspose1d(x: Tensor, w: Tensor, bias: Vec<Tensor>, stride: i64, pad: i64): Tensor

The 1-D transposed convolution of x (n, c), time first, by w (c, o, k) at stride with pad with bias (o) unless empty: ((n - 1) stride - 2 pad + k, o), as PyTorch's conv_transpose1d of xᵀ transposed.

Effects: fail, read, write, console, io.

`pool2d`

gizlang

function pool2d(x: Tensor, size: i64, stride: i64, average: bool): Tensor

Max (or, average, mean) pooling of x (n, c, h, w) over k x k windows at stride: the max is the first largest in window order; the mean adds the window in order and multiplies by the f32 of 1/(k k).

Effects: fail, read, write, console, io.

`pool2dGrad`

gizlang

function pool2dGrad(dy: Tensor, x: Tensor, size: i64, stride: i64, average: bool): Tensor

The gradient of pool2d against x from dy: each element adds, over the windows holding it in order, dy/(k k) (average) or dy where it is the window's max.

Effects: fail, read, write, console, io.

`attentionRows`

gizlang

function attentionRows(qkv: Tensor, heads: i64, head: i64, quantized: bool, group: i64, out: Tensor, prep: Vec<HeadPrep> = [], queries: i64 = 0): void

Full attention of every head of qkv ([n, 3 heads head] BF16 on a GPU: each row's q, k and v of heads heads of head, rotated as the model needs) into out ([n, heads head] BF16): softmax(q kᵀ / sqrt(head)) v a head, the keys of each group of group rows alone (0: all rows). With quantized, where the GPU has 8-bit matrix cores (K.hasFp8Mma) and heads are 64 or 128, Q Kᵀ runs on INT8 and P V on E4M3 (attention8, SageAttention's split); else on BF16. prep (a HeadPrep or none) normalizes and turns each head of q and k as attention reads them: the 8-bit path in its quantizing pass, the BF16 path in place in qkv first.

Effects: fail, read, write, console, io.

`attendedRows`

gizlang

function attendedRows(qkv: Tensor, heads: i64, head: i64, quantized: bool, group: i64, prep: Vec<HeadPrep> = [], queries: i64 = 0): Tensor

attentionRows' output as a new tensor where qkv is: [rows (queries when given), heads × head], BF16 on a GPU. On the host in F32 and exact (quantized has no effect): prep, then attention among the rows of each group.

Effects: fail, read, write, console, io.

`headPrepInPlace`

gizlang

function headPrepInPlace(qkv: Tensor, n: i64, heads: i64, head: i64, p: HeadPrep, row0: i64): void

p of q and k of the first n rows of qkv ([rows, 3 heads head] BF16 on a GPU, each row its q, k and v) in place (prepKernel), row0 the sequence row of the first.

Effects: fail, read, write, console, io.

`crossAttentionRows`

gizlang

function crossAttentionRows(q: Tensor, kv: Tensor, heads: i64, head: i64, out: Tensor): void

Attention of the rows of q ([nq, heads head] BF16 on a GPU) over the keys and values of kv ([nk, 2 heads head]: each row's k, then its v), as attentionRows: a cross attention (a text's, another modality's). Into out ([nq, heads head] BF16), on the BF16 matrix cores.

Effects: fail, read, write, console, io.

`neighborhood3d`

gizlang

function neighborhood3d(qkv: Tensor, grid: Vec<i64>, kernel: Vec<i64>, heads: i64, scale: f64): Tensor

3D neighborhood attention (NATTEN's na3d: dilation 1, not causal) of qkv ([T H W, 3 heads head] BF16 on a GPU, F32 on the host; rows in (t, h, w) order, each row its q, k and v, each its heads, as a fused QKV linear gives them; as attention reads it): each query attends the keys of a window of kernel [kt, kh, kw] positions (each odd) around it, shifted inward at the grid's edges and as long as the axis at most, with scores q · k times scale. The output has q's layout, BF16 on a GPU (flash attention over the box of keys each tile of 64 queries can see), F32 on the host (exact).

Effects: fail, read, write, console, io.

`modulatedNorm`

gizlang

function modulatedNorm(x: Tensor, w: Tensor, shift: Tensor, scale: Tensor, map: Tensor, eps: f64, centered: bool = false): Tensor

Each row of x ([rows, width] F32) over its RMS (with centered, LayerNorm's: less its mean, over its standard deviation), times w, times 1 + scale and plus shift of its modulation row (map[r], an F32 row index into shift and scale): a diffusion transformer's adaptive norm (AdaLN). F32 on the host, BF16 on a GPU.

Effects: fail, read, write, console, io.

`modulatedNormInto`

gizlang

function modulatedNormInto(x: Tensor, w: Tensor, shift: Tensor, scale: Tensor, map: Tensor, eps: f64, y: Tensor, centered: bool = false): void

modulatedNorm on a GPU into y (BF16).

Effects: fail, read, write, console, io.

`gatedNormAddInto`

gizlang

function gatedNormAddInto(x: Tensor, y: Tensor, w: Tensor, gate: Tensor, map: Tensor, eps: f64): void

x ([rows, width] F32, on a GPU) plus, each row, y's row (BF16) over its RMS (eps) times w, times the row map[r] of gate ([modulation rows, width] F32): a block's output norm under its gate (Z-Image's sandwich norm). In place.

Effects: fail, read, write, console, io.

`rowNormRopeInto`

gizlang

function rowNormRopeInto(gpu: Gpu, at: u64, stride: i64, rows: i64, width: i64, head: i64, w: Tensor, eps: f64, cos: Vec<Tensor> = [], sin: Vec<Tensor> = []): void

Each of rows rows of width BF16 values (from address at, rows stride values apart, on gpu) over its RMS across the whole row (eps), times w (width F32 values), then, with cos and sin ([rows, width / 2] F32), each head of head values turned by split halves: value d < head / 2 with value d + head / 2 by its pair's angle (pair h head / 2

d of the row): LTX's QK norm, which spans every head, and its rotary. In place.

Effects: fail, read, write, console, io.

`int8Rows`

gizlang

function int8Rows(x: Tensor, bits: i64 = 8): Vec<Tensor>

Rows of x (BF16 on a GPU, a multiple of 256 wide) as INT8 over a scale a row, as ConvRot quantizes activations (WanGP's asym_w4a8_int8, comfy-kitchen's int8_linear): each 256 values times the regular Hadamard matrix over 16 (Kernel.hadamard regular: its outputs in that order, so the weights take K.regularSource and K.regularSign), in F32 (y); s = max |y| / 127 a row ([rows, 1] F32, at least 1e-30); q = y / s to nearest even within [-128, 127]. Those references round y and y / s to BF16 first, which moves a code by one where the BF16 step passes a half: LTX-2.5's video velocity was 26.7% off its BF16-activation result with that rounding and 22.6% without it (ComfyUI's 20.8%). [q, s]. With 4 bits, q is I4: s = max |y| / 7, q within [-8, 7]. On the host x holds BF16 values (as F32) and q the integers (as F32): the same steps, the transform gizlang:linalg's.

Effects: fail, read, write, console, io.

`int8RowsInto`

gizlang

function int8RowsInto(x: Tensor, q: Tensor, s: Tensor): void

int8Rows into q ([rows, width] I8, or I4 for 4 bits) and s ([rows, 1] F32).

Effects: fail, read, write, console, io.

`modulatedInt8RowsInto`

gizlang

function modulatedInt8RowsInto(x: Tensor, w: Tensor, shift: Tensor, scale: Tensor, map: Tensor, eps: f64, y: Option<Tensor>, q: Tensor, s: Tensor, centered: bool = false): void

int8Rows of modulatedNorm's rows of x (F32) in one pass, into q and s; with y modulatedNorm's values there too.

Effects: fail, read, write, console, io.

Methods

`Tensor.argmax`

gizlang

function Tensor.argmax(self: Tensor): Vec<i64>

The index of each row's largest element (the first of equals; NaN never wins).

Effects: fail, read, write, console, io.

`Tensor.dropout`

gizlang

function Tensor.dropout(self: Tensor, k: Key, p: f64): Tensor

x where word i of key's draw (as uniform makes it) is at least p, scaled by 1 / (1 - p), else 0: the same mask on the CPU and a GPU, so the backward pass applies it again from the key.

Effects: fail, read, write, console, io.

Constants

`TOP_EACH`

gizlang

const TOP_EACH = 4

The largest logits each of a candidates workgroup's 256 threads keeps of the values it reads, and the floats of a row's candidates (a value and a token each: 256 TOP_EACH 2).

`CANDIDATE_ROW`

gizlang

const CANDIDATE_ROW = 2048

The floats of a row's candidates in a candidates workgroup.