gizml:tensor

n-dimensional f32 arrays on the CPU's vector lanes and cores. One Tensor type over one storage (row-major Vec<f32>), numpy broadcasting, elementwise chains fused into one pass (Lazy: a small tensor IR run 1024 elements at a time in L1), reductions with a policy for their order, and matmul on packed 6x32 FMA tiles across cores.

import * as Tensor from "gizml:tensor";

Types

Tensor

type Tensor

A tensor: in host memory (data, f32) or on a GPU (_gpu, with data empty). Operations run where their operands are; on(gpu) and host() move one. Operands of one operation are in one place.

• shape: Vec<i64>
• data: Vec<f32>

OnGpu

type OnGpu = { gpu: Gpu; buffer: Buffer; dtype: i64; }

A GPU tensor's memory: dtype elements (F32, F16, BF16 or FP8).

• gpu: M10.Gpu
• buffer: M10.Buffer
• dtype: i64

Reduction

type Reduction = | { kind: "strict"; } | { kind: "deterministic"; } | { kind: "fast"; }

The order a reduction adds in: strict is one fixed order and f64 partial sums on every machine (bitwise reproducible), deterministic the same result each run on one machine, fast any order across cores.

• strict
• deterministic
• fast

Key

type Key = { k0: u64; k1: u64; }

A random key: two 32-bit words. key(seed) makes one; split derives new ones.

• k0: u64
• k1: u64

Lazy

type Lazy

An elementwise expression over tensors, evaluated by eval. With a _product, input 0 is that product, not yet computed: on a GPU the expression runs as the product is stored (its epilogue), so C is written once.

• shape: Vec<i64>
• inputs: Vec<M32.Tensor>
• steps: Vec<M32.Step>

Product

type Product = { a: Tensor; ta: bool; b: Tensor; tb: bool; m: i64; k: i64; n: i64; compute: i64; low: Vec<Tensor>; lowAt: i64; patch: Vec<i64>; }

A times B as matmulT takes them: m x k times k x n, on a GPU with the operands rounded to compute (-1: as they are; lazyMatmulIn). low is [D, U] of a low-rank product D Uᵀ (D m x r, U n x r, both F16 or BF16 on a GPU) the chain adds as its input lowAt: on NVIDIA's matrix cores the product's epilogue computes it from the stage memory the main loop is done with, so D Uᵀ is never stored (a LoRA's up projection); elsewhere lowApart.

• a: M32.Tensor
• ta: bool
• b: M32.Tensor
• tb: bool
• m: i64
• k: i64
• n: i64
• compute: i64
• low: Vec<M32.Tensor>
• lowAt: i64
• patch: Vec<i64>

Named

type Named = { name: string; tensor: Tensor; }

A named tensor, as a safetensors file holds them.

• name: string
• tensor: M32.Tensor

Functions

vexp

function vexp(x: f32x16): f32x16

e^x per lane, within 2 ulp (Cephes expf): 2^n by the exponent bits, the rest by a degree-6 polynomial on |r| <= ln2/2. Overflow gives inf, underflow 0.

vlog

function vlog(x: f32x16): f32x16

ln x per lane, within 2 ulp (Cephes logf); negative gives NaN, 0 gives -inf.

bytesOf

function bytesOf(dtype: i64, count: i64): i64

Bytes of count elements of dtype: I4 packs two a byte.

Effects: fail.

fromData

function fromData(shape: Vec<i64>, data: Vec<f32>): Tensor

A host tensor of shape over data (as many as the shape has).

Effects: fail.

placed

function placed(shape: Vec<i64>, gpu: Gpu, buffer: Buffer, dtype: i64): Tensor

A tensor of SHAPE whose values are in BUFFER on GPU, of the element type DTYPE.

elements

function elements(shape: Vec<i64>): i64

The number of elements of a tensor of SHAPE.

Effects: fail.

tensor

function tensor(shape: Vec<i64>, values: Vec<f64>): Tensor

A tensor of shape holding values (as many as the shape has, rounded to f32).

Effects: fail.

full

function full(shape: Vec<i64>, value: f64): Tensor

A host tensor of SHAPE with every value VALUE.

Effects: fail.

zeros

function zeros(shape: Vec<i64>): Tensor

A host tensor of SHAPE with every value 0.

Effects: fail.

ones

function ones(shape: Vec<i64>): Tensor

A host tensor of SHAPE with every value 1.

Effects: fail.

scalar

function scalar(value: f64): Tensor

A 0-d tensor: a scalar that broadcasts to any shape.

Effects: fail.

arange

function arange(n: i64): Tensor

0, 1, ..., n-1.

Effects: fail.

key

function key(seed: u64): Key

The random key of SEED.

Effects: fail.

philox

function philox(k: Key, c0: u64, c1: u64, out: mut Vec<u64>, at: i64): void

Four 32-bit words for counter (c0, c1), written to out[at..at+4) as far as it reaches.

Effects: fail.

uniform

function uniform(k: Key, shape: Vec<i64>, low: f64, high: f64): Tensor

Uniform in [low, high): the top 24 bits of each word, so every value is exact in f32.

Effects: fail.

normal

function normal(k: Key, shape: Vec<i64>): Tensor

Standard normal (mean 0, deviation 1) by Box-Muller on pairs of words, in f64.

Effects: fail.

onDevice

function onDevice(shape: Vec<i64>, gpu: Gpu, buffer: Buffer, dtype: i64): Tensor

The tensor of shape whose dtype elements (row-major) are device memory buffer of gpu: a kernel's result, or weights uploaded as their bytes.

broadcastShape

function broadcastShape(a: Vec<i64>, b: Vec<i64>): Vec<i64>

The shape two shapes broadcast to (numpy: aligned at the right, 1 stretches).

Effects: fail.

lazy

function lazy(t: Tensor): Lazy

T as a lazy expression with no steps yet.

Effects: fail.

lazyMatmul

function lazyMatmul(x: Tensor, transposeA: bool, y: Tensor, transposeB: bool): Lazy

x times y as matmulT, not yet computed: the steps on it run as the product is stored.

Effects: fail, read, write, console, io.

lazyMatmulIn

function lazyMatmulIn(x: Tensor, transposeA: bool, y: Tensor, transposeB: bool, compute: i64): Lazy

As lazyMatmul, with operands on a GPU rounded to compute (F16, BF16 or FP8) as Tensor.cast rounds them; in host memory they stay f32. The matrix cores' kernel rounds an F32 operand of an F16 or BF16 product as it copies it, so the cast takes no pass of its own (the bits are those of the cast and then the product). FP8 operands are scaled to its range first (fp8Scaled), and the product's store multiplies by both scales.

Effects: fail, read, write, console, io.

lazyProduct

function lazyProduct(shape: Vec<i64>, product: Product): Lazy

The expression of PRODUCT alone, of shape: its input 0 stands for the product and has no values.

Effects: fail.

fp8Scaled

function fp8Scaled(t: Tensor): Vec<Tensor>

t scaled to the range of FP8 as tensorwise scaling (Transformer Engine) does it: [t / s as FP8, s], with s = max |t| / 448 a tensor of shape [] where t is.

Effects: fail, read, write, console, io.

fp4Blocks

function fp4Blocks(t: Tensor, across: bool): Vec<Tensor>

t (on a GPU) in NVFP4 blocks with its rows along k, as NVIDIA's Transformer Engine makes them: [q, s]. q is of type FP4 and shape [rows, k]: rows x k / 2 bytes of E2M1 pairs (row-major, the first in the low half), then k / 16 E4M3 block scales for each of lines (rows rounded up to 4; zeros past rows), in words of 4 (64 of k) by columns: word w of row r at w lines + r, so the words of a warp's mma lie together and copy in 16 bytes. s (F32, shape []) is max |t| / (6 x 448). The value at row r, p is its E2M1 times the scale of its block times s. across: t is [k, rows], else its last axis is k.

Effects: fail, read, write, console, io.

productType

function productType(a: i64, b: i64): i64

The type of a product of operands of types a and b, as in torch: theirs when both are F16 or both BF16 (the sums are f32, rounded once at the end), else F32.

matmulT

function matmulT(x: Tensor, transposeA: bool, y: Tensor, transposeB: bool): Tensor

A times B with either read transposed in place, no copy made: A is (..., m, k) or, with transposeA, a 2-d (k, m); B is (k, n) or, with transposeB, (n, k). Each sum adds a[i][p] b[p][j] by fma for p in order from 0, so the result is the same bits however the work is split. B is packed once in panels of 32 columns; rows go in blocks across the cores once there are 2^21 or more multiply-adds, as many blocks as cores (in multiples of 6 rows, at most 96), and within a block 6x32 FMA tiles stay in registers. The result is of productType.

Effects: fail, read, write, console, io.

gpuOf

function gpuOf(xs: Vec<Tensor>): Gpu

The GPU of the first operand there, every other operand moved to it.

Effects: fail.

keyOf

function keyOf(shape: Vec<i64>): string

The text of SHAPE as a key ("[2,3,]").

Effects: fail.

blocks256

function blocks256(threads: i64): i64

The blocks of 256 threads that cover THREADS threads.

Effects: fail.

longs

function longs(xs: Vec<i64>): bytes

XS as 64-bit little-endian words, for kernel arguments.

Effects: fail.

concat

function concat(xs: Vec<Tensor>, axis: i64): Tensor

The tensors joined along axis (equal elsewhere), in order.

Effects: fail, read, write, console, io.

batchMatmul

function batchMatmul(x: Tensor, transposeA: bool, y: Tensor, transposeB: bool): Tensor

Batched matmul: x (b, m, k) (or (b, k, m) with transposeA) times y (b, k, n) (or (b, n, k) with transposeB), or one y (k, n) for every batch.

Effects: fail, read, write, console, io.

within

function within(buffer: Buffer, gpu: Gpu, shape: Vec<i64>, dtype: i64): Tensor

A tensor of shape and dtype on the first bytes of buffer (work memory a caller reuses).

Effects: fail.

toSafetensors

function toSafetensors(tensors: Vec<Named>): bytes

TENSORS as a safetensors file of F32 data, in their order.

Effects: fail, read, write, console, io.

fromSafetensors

function fromSafetensors(data: bytes): Vec<Named> throws string

The tensors of a safetensors file, in the header's order, as f32 (gizlang:safetensors decoded).

Effects: fail.

placedOn

function placedOn(t: Tensor, gpu: Option<Gpu>): Tensor

Tensor t on gpu when there is one, else as it is: where a model runs.

Effects: fail, read, write, console, io.

storedOn

function storedOn(sys: System, s: Stored, gpu: Option<Gpu>, linear: bool): Tensor throws string

A stored tensor where it runs: a linear as BF16 on a GPU, anything else as f32 (BF16, F16, F32 and E4M3 bytes go to the GPU as they are stored and are cast there).

Effects: fail, read, write, console, io.

Methods

Key.words

function Key.words(self: Key, n: i64): Vec<u64>

N random 32-bit words: word i from counter (i / 4, 0).

Effects: fail.

Key.split

function Key.split(self: Key, n: i64): Vec<Key>

N keys independent of this one and of each other (counters (i, 1)).

Effects: fail.

Tensor.text

function Tensor.text(self: Tensor): string

The values nested by shape, as numpy prints them: [[1, 2], [3, 4]]. A GPU tensor is copied to the host first.

Effects: fail, read, write, console, io.

Tensor.size

function Tensor.size(self: Tensor): i64

The number of values: the product of the shape.

Effects: fail.

Tensor.gpu

function Tensor.gpu(self: Tensor): Option<Gpu>

Where the tensor is: none for host memory.

Tensor.isHost

function Tensor.isHost(self: Tensor): bool

Whether the tensor is in host memory.

Tensor.dtype

function Tensor.dtype(self: Tensor): i64

The element type: F32 for a host tensor.

Tensor.on

function Tensor.on(self: Tensor, gpu: Gpu): Tensor

A copy on gpu (the same tensor when it is there).

Effects: fail, read, write, console, io.

Tensor.host

function Tensor.host(self: Tensor): Tensor

A copy in host memory (the same tensor when it is there), as f32.

Effects: fail, read, write, console, io.

Tensor.cast

function Tensor.cast(self: Tensor, dtype: i64): Tensor

The same values stored as dtype (F32, F16, BF16 or FP8; rounded to nearest even, FP8 saturated at ±448): on a GPU only, as host tensors are f32.

Effects: fail, read, write, console, io.

Tensor.assign

function Tensor.assign(self: Tensor, x: Tensor): void

Writes x's values into this GPU tensor's memory (x of the same shape, on the same GPU or the host), so that the kernels of a recording (Gpu.record) that read this tensor read them.

Effects: fail, read, write, console, io.

Tensor.buffer

function Tensor.buffer(self: Tensor): Buffer

The buffer of a GPU tensor.

Effects: fail.

Tensor.onGpu

function Tensor.onGpu(self: Tensor): OnGpu

Where a GPU tensor is: its device, buffer and element type.

Effects: fail.

Tensor.placement

function Tensor.placement(self: Tensor): Option<OnGpu>

Where the tensor is: on a GPU (its device, buffer and element type), or none for host memory.

Tensor.deviceBuffer

function Tensor.deviceBuffer(self: Tensor): Buffer

A GPU tensor's device memory, for kernels made elsewhere (gizml:tile): its dtype() elements in row-major order.

Effects: fail.

Tensor.rank

function Tensor.rank(self: Tensor): i64

The number of axes.

Tensor.get

function Tensor.get(self: Tensor, index: Vec<i64>): f64

The element at a full index (R001 outside).

Effects: fail, read, write, console, io.

Tensor.values

function Tensor.values(self: Tensor): Vec<f64>

The values in row-major order, as f64, copied to the host when the tensor is on a GPU.

Effects: fail, read, write, console, io.

Tensor.reshape

function Tensor.reshape(self: Tensor, shape: Vec<i64>): Tensor

The same elements in another shape of the same size; one -1 is inferred.

Effects: fail.

Lazy.pending

function Lazy.pending(self: Lazy): bool

Whether input 0 is a product not yet evaluated (_product): a step over it evaluates that product first.

Lazy.expand

function Lazy.expand(self: Lazy, shape: Vec<i64>): Lazy

The expression broadcast to SHAPE.

Effects: fail, read, write, console, io.

Lazy.add

function Lazy.add(self: Lazy, y: Lazy): Lazy

SELF + Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Lazy.sub

function Lazy.sub(self: Lazy, y: Lazy): Lazy

SELF - Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Lazy.mul

function Lazy.mul(self: Lazy, y: Lazy): Lazy

SELF * Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Lazy.div

function Lazy.div(self: Lazy, y: Lazy): Lazy

SELF / Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Lazy.maximum

function Lazy.maximum(self: Lazy, y: Lazy): Lazy

The larger of SELF and Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Lazy.minimum

function Lazy.minimum(self: Lazy, y: Lazy): Lazy

The smaller of SELF and Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Lazy.greater

function Lazy.greater(self: Lazy, y: Lazy): Lazy

1 where self > y, else 0.

Effects: fail, read, write, console, io.

Lazy.neg

function Lazy.neg(self: Lazy): Lazy

-SELF, element by element.

Effects: fail.

Lazy.abs

function Lazy.abs(self: Lazy): Lazy

The absolute value of SELF, element by element.

Effects: fail.

Lazy.sqrt

function Lazy.sqrt(self: Lazy): Lazy

The square root of SELF, element by element.

Effects: fail.

Lazy.exp

function Lazy.exp(self: Lazy): Lazy

e to the power SELF, element by element.

Effects: fail.

Lazy.log

function Lazy.log(self: Lazy): Lazy

The natural logarithm of SELF, element by element.

Effects: fail.

Lazy.tanh

function Lazy.tanh(self: Lazy): Lazy

The hyperbolic tangent of SELF, element by element.

Effects: fail.

Lazy.sigmoid

function Lazy.sigmoid(self: Lazy): Lazy

The sigmoid 1 / (1 + e^-x) of SELF, element by element.

Effects: fail.

Lazy.relu

function Lazy.relu(self: Lazy): Lazy

max(SELF, 0), element by element.

Effects: fail.

Lazy.square

function Lazy.square(self: Lazy): Lazy

SELF squared, element by element.

Effects: fail.

Lazy.roundedBf16

function Lazy.roundedBf16(self: Lazy): Lazy

Each value rounded to BF16 (nearest even) and kept as F32: where a pipeline stores BF16 between two steps that a product's epilogue now runs together.

Effects: fail.

Lazy.gelu

function Lazy.gelu(self: Lazy): Lazy

GELU, tanh form (as GPT-2 and torch's approximate="tanh").

Effects: fail.

Lazy.geluGrad

function Lazy.geluGrad(self: Lazy, dy: Lazy): Lazy

The gradient through gelu: self is x, dy the output's gradient.

Effects: fail, read, write, console, io.

Lazy.geluErf

function Lazy.geluErf(self: Lazy): Lazy

GELU, erf form (torch's default and BERT's "gelu").

Effects: fail.

Lazy.geluErfGrad

function Lazy.geluErfGrad(self: Lazy, dy: Lazy): Lazy

The gradient through geluErf: self is x, dy the output's gradient.

Effects: fail, read, write, console, io.

Lazy.on

function Lazy.on(self: Lazy, gpu: Gpu): Lazy

The expression with its inputs moved to gpu, so eval runs it there.

Effects: fail, read, write, console, io.

Lazy.like

function Lazy.like(self: Lazy): Tensor

A tensor of the expression's shape where it is evaluated (a GPU or host memory), with no values: for its shape and place (Tensor.filled, Tensor.gpu), never to read.

Effects: fail.

Lazy.eval

function Lazy.eval(self: Lazy): Tensor

The expression's tensor: one pass, on every core when it is large, or one kernel on the GPU of an input. It is f32 but over a product, where it is of the product's type (productType) when every other input is of that type or has one element, as torch's type promotion; else f32.

Effects: fail, read, write, console, io.

Lazy.evalIn

function Lazy.evalIn(self: Lazy, dtype: i64): Tensor

The expression's tensor stored as dtype (F32, F16, BF16 or FP8) on a GPU, so a cast takes no pass of its own; in host memory it is f32.

Effects: fail, read, write, console, io.

Lazy.evalInto

function Lazy.evalInto(self: Lazy, target: Tensor): void

Stores the expression's values into GPU tensor target (of its shape) as target's type: an update in place, so the kernels of a recording (Gpu.record) that read target read them. Its kernel writes there where nothing it reads can see the write (it reads target only at the element it writes); else it computes a new tensor and copies it.

Effects: fail, read, write, console, io.

Lazy.gluIn

function Lazy.gluIn(self: Lazy, dtype: i64): Tensor

A product chain's rows [m, n] as SwiGLU pairs, dtype elements [m, n / 2]: column c is silu(v[2c]) times v[2c + 1], each after the chain (a gate and an up projection with their rows interleaved). On a GPU the product's store makes them, so C's n columns are never stored.

Effects: fail, read, write, console, io.

Tensor.glu

function Tensor.glu(self: Tensor): Tensor

SwiGLU of the rows [m, n] of a host tensor as pairs, [m, n / 2]: column c is silu(v[2c]) times v[2c + 1], rounded to f32 (Lazy.gluIn on the host).

Effects: fail.

Lazy.gluInto

function Lazy.gluInto(self: Lazy, target: Tensor): void

Lazy.gluIn into GPU tensor target ([m, n / 2]).

Effects: fail, read, write, console, io.

Lazy.reads

function Lazy.reads(self: Lazy, t: Tensor): bool

Whether the expression reads GPU tensor t's memory (an input or a product's operand).

Effects: fail.

Tensor.add

function Tensor.add(self: Tensor, y: Tensor): Tensor

SELF + Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Tensor.sub

function Tensor.sub(self: Tensor, y: Tensor): Tensor

SELF - Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Tensor.mul

function Tensor.mul(self: Tensor, y: Tensor): Tensor

SELF * Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Tensor.div

function Tensor.div(self: Tensor, y: Tensor): Tensor

SELF / Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Tensor.maximum

function Tensor.maximum(self: Tensor, y: Tensor): Tensor

The larger of SELF and Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Tensor.minimum

function Tensor.minimum(self: Tensor, y: Tensor): Tensor

The smaller of SELF and Y, element by element, with broadcasting.

Effects: fail, read, write, console, io.

Tensor.neg

function Tensor.neg(self: Tensor): Tensor

-SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.abs

function Tensor.abs(self: Tensor): Tensor

The absolute value of SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.sqrt

function Tensor.sqrt(self: Tensor): Tensor

The square root of SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.exp

function Tensor.exp(self: Tensor): Tensor

e to the power SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.log

function Tensor.log(self: Tensor): Tensor

The natural logarithm of SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.tanh

function Tensor.tanh(self: Tensor): Tensor

The hyperbolic tangent of SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.sigmoid

function Tensor.sigmoid(self: Tensor): Tensor

The sigmoid 1 / (1 + e^-x) of SELF, element by element.

Effects: fail, read, write, console, io.

Tensor.relu

function Tensor.relu(self: Tensor): Tensor

max(SELF, 0), element by element.

Effects: fail, read, write, console, io.

Tensor.square

function Tensor.square(self: Tensor): Tensor

SELF squared, element by element.

Effects: fail, read, write, console, io.

Tensor.gelu

function Tensor.gelu(self: Tensor): Tensor

GELU of SELF (the tanh approximation), element by element.

Effects: fail, read, write, console, io.

Tensor.geluErf

function Tensor.geluErf(self: Tensor): Tensor

GELU of SELF (the exact erf form), element by element.

Effects: fail, read, write, console, io.

Tensor.expand

function Tensor.expand(self: Tensor, shape: Vec<i64>): Tensor

The tensor broadcast to shape, where it is.

Effects: fail, read, write, console, io.

Tensor.filled

function Tensor.filled(self: Tensor, shape: Vec<i64>, value: f64): Tensor

value in every element of shape, where this tensor is (host memory or its GPU).

Effects: fail, read, write, console, io.

Tensor.item

function Tensor.item(self: Tensor): f64

The value of a one-element tensor.

Effects: fail, read, write, console, io.

Tensor.sum

function Tensor.sum(self: Tensor, policy: Reduction): f64

The sum of all values, in the order POLICY sets (strict: the same on any core count and device).

Effects: fail, read, write, console, io.

Tensor.mean

function Tensor.mean(self: Tensor, policy: Reduction): f64

The mean of all values, summed in the order POLICY sets.

Effects: fail, read, write, console, io.

Tensor.max

function Tensor.max(self: Tensor): f64

The largest value.

Effects: fail, read, write, console, io.

Tensor.transpose

function Tensor.transpose(self: Tensor): Tensor

The 2-d transpose, in bands of 32 output rows on every core (Task.fill); columns past the last whole band one by one.

Effects: fail, read, write, console, io.

Tensor.softmax

function Tensor.softmax(self: Tensor): Tensor

softmax over the last axis: max-subtracted so no exp overflows.

Effects: fail, read, write, console, io.

Tensor.matmul

function Tensor.matmul(self: Tensor, y: Tensor): Tensor

self (..., m, k) times y (k, n): leading axes are rows too.

Effects: fail, read, write, console, io.

Lazy.total

function Lazy.total(self: Lazy, policy: Reduction): Tensor

The sum of the expression's elements (Tensor.total) without its tensor: on a GPU, a strict sum of a small one is one kernel that computes each element as it adds it.

Effects: fail, read, write, console, io.

Lazy.average

function Lazy.average(self: Lazy, policy: Reduction): Tensor

The mean of the expression's elements (Tensor.average), as Lazy.total.

Effects: fail, read, write, console, io.

Tensor.total

function Tensor.total(self: Tensor, policy: Reduction): Tensor

The sum (Tensor.sum) as a tensor of shape [] where this one is: on a GPU it stays there, with no wait.

Effects: fail, read, write, console, io.

Tensor.average

function Tensor.average(self: Tensor, policy: Reduction): Tensor

The mean (Tensor.mean) as a tensor of shape [] where this one is, as Tensor.total.

Effects: fail, read, write, console, io.

Tensor.sumTo

function Tensor.sumTo(self: Tensor, shape: Vec<i64>): Tensor

Summed down to shape over the axes broadcasting stretched (the adjoint of broadcasting; a reduction over any axes that keeps them as 1). Over the last axis only, each row adds as sum(Reduction.strict()) adds a part (f64 lanes); otherwise each element adds its sources in index order from 0 in f32.

Effects: fail, read, write, console, io.

Tensor.maxTo

function Tensor.maxTo(self: Tensor, shape: Vec<i64>): Tensor

The largest element over the axes that shape keeps as 1 (NaN is ignored, as by max; -Infinity where every element is NaN or none is left).

Effects: fail, read, write, console, io.

Tensor.take

function Tensor.take(self: Tensor, ids: Vec<i64>): Tensor

Rows ids of the tensor (its first axis): an embedding lookup.

Effects: fail, read, write, console, io.

Tensor.addRows

function Tensor.addRows(self: Tensor, ids: Vec<i64>, rows: i64): Tensor

The adjoint of take: a (rows, ...) tensor whose row r adds the rows i of this one with ids[i] = r, in increasing i (on a GPU each output element adds its own list, so no two threads write one place and the order holds).

Effects: fail, read, write, console, io.

Tensor.slice

function Tensor.slice(self: Tensor, axis: i64, start: i64, end: i64): Tensor

Elements start..end of axis.

Effects: fail, read, write, console, io.

Tensor.permute

function Tensor.permute(self: Tensor, axes: Vec<i64>): Tensor

The axes reordered: output axis i is input axis axes[i].

Effects: fail, read, write, console, io.

Tensor.rows

function Tensor.rows(self: Tensor, r0: i64, r1: i64): Tensor

Rows [r0, r1) of the tensor ([rows, ...] on a GPU) on the same memory, with no copy (Tensor.slice copies).

Effects: fail.

Tensor.address

function Tensor.address(self: Tensor): u64

The device address of the tensor's first element (a kernel's buffer argument).

Effects: fail.

Stored.upload

function Stored.upload(self: Stored, sys: System, gpu: Gpu): Buffer throws string

The stored tensor's raw bytes in a new buffer on gpu, read from its file straight to the device (Gpu.uploadFile).

Effects: fail, read, write, io.

Constants

F32

const F32 = 0

The element type of 32-bit float GPU tensors (gizhw:kernel's).

F16

const F16 = 4

The element type of IEEE half-precision GPU tensors.

BF16

const BF16 = 5

The element type of bfloat16 GPU tensors.

FP8

const FP8 = 9

FP8 E4M3 (gizhw:kernel E_E4M3): 4 exponent bits, 3 mantissa bits, at most ±448. A storage type: its products are F32 (on matrix cores where the GPU has them, hasFp8Mma) and it takes no scale; scale the values first.

FP4

const FP4 = 11

NVFP4: as compute of lazyMatmulIn, each operand in blocks of 16 along k of FP4 E2M1 values, each block scaled by an E4M3 and the whole operand by an F32 (fp4Blocks). A tensor of this type is such an operand only.

I8

const I8 = 7

Signed 8-bit integers (gizhw:kernel E_I8): a storage type whose products of two such operands are F32 sums of integers on the matrix cores (hasI8Mma; A by rows and B by columns along k). It takes no scale; scale the values first and multiply the product by the scales.

I4

const I4 = 13

Signed 4-bit integers, two a byte (gizhw:kernel E_I4X4 holds four): as I8 on the matrix cores where hasI4Mma, k a multiple of 4. Made by int8Rows with 4 bits; an operand only.

OP_INPUT

const OP_INPUT = 0

The step of a lazy expression that reads one of its inputs.