# Tensors, GPU kernels and AI models

:::status{of="compute"}
:::

The standard library computes on the CPU's vector lanes and cores, and on an NVIDIA or AMD GPU when the machine has
one. The GPU's driver is found when the program runs: nothing is linked at build time, and the same executable runs on
a machine without a GPU.

| Module | What it does |
| --- | --- |
| `gizml:tensor` | n-dimensional arrays, broadcasting, fused elementwise chains, reductions, matrix products, and safetensors files of tensors |
| `gizhw:gpu` | The GPU: device memory, compiled kernels and launches |
| `gizml:tile` | Your own GPU kernels, written as ordinary Gizlang over blocks of a tensor |
| `gizml:grad` | Gradients of tensor programs, in reverse and forward mode |
| `gizml:nn` | Layers, losses and optimizers on top of `gizml:grad` |
| `gizml:quant` | Weight formats: codebooks of 2 to 8 bits, F8, BF16, F16 and H3's W4, the Hadamard rotation they share, and their encoding and decoding |
| `gizml:llm` | Language models: convert a Hugging Face model once, then generate text; BERT-family encoders and cross-encoders |
| `gizml:ai` | Models by name: downloaded and converted once, a chat's answer as it comes, and the OpenAI API server |
| `gizml:diffusion` | Image and video generation: FLUX.3 klein (text to image, edits, prompt upsampler), Z-Image Turbo and MiniMax H3, their text encoders and decoders |

This guide takes about fifteen minutes. Each output on this page is the real output of its example.

## Step 1: Tensors on the CPU

A tensor has a shape and its values in row-major order. A chain of elementwise operations on `T.lazy` values runs as
one pass over the data when you call `eval`.

```gizlang run
import * as T from "gizml:tensor";

function main(): void {
    const a = T.tensor([2, 3], [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
    const b = T.tensor([3, 2], [1.0, 0.0, 0.0, 1.0, 1.0, 1.0]);
    const c = a.matmul(b);
    print(c.text());
    // Elementwise chains run fused, in one pass over the data.
    const d = T.lazy(c).mul(T.lazy(T.scalar(0.5))).relu().eval();
    print(d.text());
}
```

```text output
[[4, 5], [10, 11]]
[[2, 2.5], [5, 5.5]]
```

## Step 2: The same work on a GPU

`Gpu.open` finds a GPU, or gives none. `on(gpu)` copies a tensor to it, and the operations on GPU tensors run there.
`host()` copies a result back. This program prints the same line with or without a GPU.

```gizlang run
import {System} from "gizlang:system";
import * as T from "gizml:tensor";
import * as Gpu from "gizhw:gpu";

function main(sys: System): void {
    const a = T.tensor([2, 3], [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
    const b = T.tensor([3, 2], [1.0, 0.0, 0.0, 1.0, 1.0, 1.0]);
    let c = a.matmul(b);
    match (Gpu.open(sys)) {
        // On a GPU, the same call runs there; host() copies the result back.
        Option.some(gpu) => { c = a.on(gpu).matmul(b.on(gpu)).host(); }
        Option.none() => {}
    }
    print(c.text());
}
```

```text output
[[4, 5], [10, 11]]
```

## Step 3: Write a kernel

A tile program says what one block of the output is. `gizml:tile` decides the rest: which thread holds which value,
workgroup memory, barriers, the masks at the edges, and the matrix cores. The same program runs on host tensors on the
CPU, where `print` and the test runner work, and on GPU tensors on the GPU.

An operator on a tile is its method: `a * s` is `a.mul(s)`. A number next to a named tile becomes a tile of that
value: `t + 1.0` is `t.add(t.k(1.0))`.

```gizlang run
import * as T from "gizml:tensor";
import * as Tile from "gizml:tile";

function main(): void {
    const x = T.tensor([4, 4], [-2.0, -1.0, 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, -3.0, -4.0, 6.0, 7.0, 8.0, -9.0, 1.0, 0.5]);
    const w = T.tensor([1, 4], [1.0, 2.0, 3.0, 4.0]);
    const y = Tile.run([1], (p: Tile.Program): void => {
        const z = p.int(0);
        const a = p.read(x, [4, 4]).at([z, z]);
        const s = p.read(w, [1, 4]).at([z, z]);
        // One block of the output: max(a * s + 1, 0). Rows broadcast against s.
        const t = a * s;
        p.write(T.zeros([4, 4]), [4, 4]).put((t + 1.0).relu(), [z, z]);
    })[0];
    print(y.text());
}
```

```text output
[[0, 0, 1, 5], [3, 7, 13, 21], [0, 0, 19, 29], [9, 0, 4, 3]]
```

For a real kernel, the grid has one point for each output block, `p.id(axis)` is the block of this point, and
`p.fold` loops over the blocks of a reduction. A matrix product of 128 x 128 blocks:

```gizlang
import * as T from "gizml:tensor";
import * as Tile from "gizml:tile";

/** The product of a ([m, k]) and b ([k, n]). */
export function product(a: T.Tensor, b: T.Tensor, m: i64, n: i64): T.Tensor {
    return Tile.run([Tile.cdiv(m, 128), Tile.cdiv(n, 128)], (p: Tile.Program): void => {
        const i = p.id(0); const j = p.id(1);
        const A = p.read(a, [128, 32]); const B = p.read(b, [32, 128]);
        const acc = p.fold(p.int(0), A.blocks(1), p.zeros([128, 128]),
            (k: Tile.Index, acc: Tile.Tile): Tile.Tile => acc + A.at([i, k]).dot(B.at([k, j])));
        p.write(T.zeros([m, n]), [128, 128]).put(acc, [i, j]);
    })[0];
}
```

- `Tile.ir(...)` prints the program as the compiler sees it, with the layout of each value.
- `p.stages(n)` sets how many blocks of a product's operands are read ahead (1 to 4).
- `Tile.tune` runs each configuration you give it (block sizes, warps) on the GPU and keeps the fastest for each
  tensor shape.

## Step 4: Gradients

A `Graph` records the operations on its values. `backward` gives the gradient of a one-element result against every
input. A kernel of your own joins a graph through `custom`, with the kernels of its gradient.

```gizlang run
import * as T from "gizml:tensor";
import {Tensor} from "gizml:tensor";
import * as G from "gizml:grad";
import * as Tile from "gizml:tile";

/** x * y elementwise, written as a tile program. */
function product(x: Tensor, y: Tensor): Tensor {
    return Tile.run([1], (p: Tile.Program): void => {
        const z = p.int(0);
        const a = p.read(x, [4, 4]).at([z, z]);
        const b = p.read(y, [4, 4]).at([z, z]);
        p.write(T.zeros(x.shape), [4, 4]).put(a * b, [z, z]);
    })[0];
}

function main(): void {
    const a = T.tensor([2, 2], [1.0, 2.0, 3.0, 4.0]);
    const b = T.tensor([2, 2], [5.0, 6.0, 7.0, 8.0]);
    // Built-in operations: the gradient of sum(a * b) is b for a and a for b.
    let g = G.graph();
    const x = g.input(a); const y = g.input(b);
    const grads = g.backward(g.sum(g.mul(x, y)));
    print(grads.of(x).text());
    // Your own kernel with its own gradient kernels is one operation of the graph.
    let h = G.graph();
    const hx = h.input(a); const hy = h.input(b);
    const xy = h.custom([hx, hy], product(a, b), (d: Tensor): Vec<Tensor> => [product(d, b), product(d, a)]);
    print(h.backward(h.sum(xy)).of(hy).text());
}
```

```text output
[[5, 6], [7, 8]]
[[1, 2], [3, 4]]
```

## Step 5: Run a language model

To run a model by name, downloaded and converted on its first use, see [Run AI models on your computer](ai.md).
This step shows the parts under it.

`gizml:llm` runs Qwen 3.5 (and later Qwen versions of that architecture), Qwen 3, the text model of Qwen3-VL, Qwen 2,
Llama and Mistral. Convert a Hugging Face folder, or a GGUF file of Llama, Qwen2 or Qwen3 (llama.cpp's format, any of
its Q2_K to Q8_0 block types: `L.convert(sys, "Qwen3-8B-Q4_K_M.gguf", ...)`), once. A PEFT LoRA adapter folder as
`convert`'s last argument is merged into the weights before they are quantized. Each linear layer is stored in 4 bits and the output layer in 6
bits; other widths are a choice of `Bits`. Then load it on the best device the machine has and generate.

```gizlang
import {System} from "gizlang:system";
import * as L from "gizml:llm";

function main(sys: System): void {
    L.convert(sys, "Qwen3-8B", "qwen3-8b-q4", { layers: 4, head: 6 })!;
    const m = L.load(sys, "qwen3-8b-q4")!;
    const prompt = m.tokenizer.tokens(m.chatPrompt([{ role: "user", content: "Name three rivers." }], false));
    let e = m.engine(1, prompt.length + 200)!;
    const out = e.generate(sys, [{ prompt: prompt, limit: 200, sampling: L.greedy() }])!;
    print(m.tokenizer.decode(out[0], true));
}
```

- One `Engine` serves many sequences at once: pass more requests to `generate`, and give `engine` more slots.
- A request's `Sampling` picks its tokens: `L.greedy()`, or a temperature with `topK`, `topP` and a `seed`. Its optional
  fields shape the logits first: `presence`, `frequency` and `repetition` penalties, a `grammar` of `gizml:grammar`
  (`anyObject()` or `fromSchema(schema)`; from the token after `after` on), and `logprobs` (each `Progress` then has the
  tokens' log probabilities and that many of the most likely).
- `Engine.add`, `advance` and `cancel` run requests in turn: a request joins while others run, and its tokens come step
  by step. A later request reads only what its slot does not hold. `Model.chatTokens` marks where a chat's messages end;
  the engine keeps the recurrent states there, so the next turn of the chat reads only its new messages. When a request
  takes a slot from another chat, the slot's start goes to host memory (at most as much as stays free), and the next
  turn of that chat takes it back.
- `Model.chatTokens` also marks the head: the system turn with the tools. `Engine.add(request, mark, head)` makes a head
  of at least 256 tokens a shared start: later prompts that start with it read only the rest, and on a GPU every slot
  reads one copy of its keys and values in the KV pool. Agents whose turns start with one long system prompt so take
  its memory and its reading once.
- While sequences generate, a step reads at most the prompt rows that add as much time as the step of those sequences
  alone (both measured as the engine runs). A long prompt then slows the others to half their pace at most.
- `gizml:ai`'s `serve` serves engines with the OpenAI API on a `Listener`: `POST /v1/chat/completions` and
  `POST /v1/completions`, whole or streamed, `POST /v1/images/generations` and `GET /v1/models`. Each engine has a name; a request names its model
  (`model`), and the first is the default. Make the engines of the smaller models first: an engine's KV pool takes the
  memory its slots can use. A request's `reasoning_effort` (low, medium, high) selects the thinking and the template's
  instruction for that effort. `gizlang:hub`'s `ModelApi` and other OpenAI clients can use it. A model whose template
  writes tool calls (`Model.calls`: Qwen's `<tool_call><function=...>` format) takes `tools` and answers with
  `tool_calls` (finish reason `tool_calls`); the assistant's `tool_calls` and `reasoning_content` and the `tool`
  messages of the next request go back into the prompt as the template writes them. `L.toolCalls` reads the calls in
  a model's text.
- Call `Engine.warm` (or make the engine with `Ai.ready`) before `serve` takes requests: the first run of each step
  shape compiles, tunes and records its kernels (seconds for a large model), so warming moves that wait to the
  server's start.
- A model with a multi-token prediction layer drafts tokens ahead and keeps the ones the model agrees with. The tokens
  follow the same distribution as without drafts, and greedy output is the same. A draft reads an eighth of the output
  layer: the rows of frequent tokens and of the tokens the engine read last, so the drafts follow the prompt's language.
- The engine keeps keys and values in 8 bits, else in 4, else keys in 4 and values in 2: the first in which the
  slots' whole contexts fit. 8 bits give the output of 16 bits; 4 bits hold twice the context. Set `kv` of `engine`
  to choose: `"16"`, `"8"`, `"4"`, `"2"`, or keys and values apart, as `"4/2"`. Each head is coded over the range
  that gives it back with the least squared error.
- `L.load(sys, folder, true, layers)` loads only the first `layers` layers. `Engine.hidden` then gives the hidden states
  of each token after them, for example as the text encoder of an image model.
- A model whose weights leave less than 1 GiB of its GPU's memory free loads across the machine's other NVIDIA GPUs too
  (those `CUDA_VISIBLE_DEVICES` shows): each GPU holds a stage of consecutive layers, in proportion to its free memory,
  and a step runs the stages in turn. `L.loadAcross(sys, folder, gpus, counts)` chooses the GPUs and the layer counts.
  A model in stages gives logits only; its last stage holds the MTP layer and drafts. It reuses a chat's start and a
  shared head as one GPU does: each stage keeps its own KV pages and states of them.

## Step 6: Generate images and video

To make an image with one command, see [Run AI models on your computer](ai.md). This step shows the parts
under it.

`gizml:diffusion` runs image and video models on the same tensors and kernels: FLUX.3 klein (text to image and image edits),
Z-Image Turbo (text to image) and MiniMax H3 (text to video with sound). They need an NVIDIA GPU with INT8 matrix cores (sm_80 and later); their
attention runs in 8 bits on sm_89 and later. The host runs the same blocks in f32, as the reference of the tests.

FLUX.3 klein takes its text from the first 32 layers of Qwen3-VL-4B, converted once with `gizml:llm`. `D.Klein` is
the whole pipeline (the text encoder, the model and the VAE's encoder and decoder on one GPU), from a prompt, and
pictures to edit, to an image:

```gizlang
import {System} from "gizlang:system";
import * as G from "gizhw:gpu";
import * as D from "gizml:diffusion";
import * as Image from "gizlang:image";

function main(sys: System): void {
    const gpu = G.open(sys)!;
    D.convertTextEncoder(sys, "Qwen3-VL-4B", "flux3-text")!;
    const klein = D.loadKlein(sys, "flux3-klein", "flux2-vae.safetensors", "flux3-text", gpu)!;
    // Text to image: 1024 x 1024 from seed 42's noise (ComfyUI's for that seed).
    const fox = klein.paint(sys, "A red fox in fresh snow", 42u, [], [1024, 1024])!;
    // An edit of one or more pictures; the image has the first one's aspect (here 1 megapixel).
    const photo = Image.decode(sys.readFile("fox.jpg")!, 100000)!;
    const edit = klein.paint(sys, "Make it night", 42u, [photo])!;
    print(`${fox.width} x ${fox.height}, ${edit.width} x ${edit.height} pixels`);
}
```

- `Klein.paint` is `condition` (the references' latents and the text's rows), `sample` and `decode`; a caller that
  times the stages, or runs them apart, calls those. `D.Upsampler.caption` (BFL's prompt upsampler, Qwen3.5-9B) writes a
  fuller prompt from a short one and the pictures; `D.fluxLayout` puts bounding boxes in it.
- The 16 steps are the distilled klein model's. A step's linears run on the INT8 matrix cores after a Hadamard
  rotation of their inputs (ConvRot), and the 3 x 3 convolutions of the VAE read their patches as the product copies
  them (`T.lazyConv`). `D.loadFlux` also reads ComfyUI's INT8 files as stored.
- The step cache (`cache`, 0.3 by default) skips the joint blocks of steps that change little. For an edit, the
  option `refresh` (a number of steps) runs the reference pictures through the blocks only that often: faster, and a
  little different, so it is off by default.
- `D.ZImagePipeline` is Z-Image Turbo the same way, from ComfyUI's files: the model (its linears INT8 as it loads), the
  first 35 layers of Qwen3-4B (`D.convertZImageText`) and FLUX.1's VAE. `condition`, `sampleWith` and `decode` make an
  image in 8 steps of ComfyUI's res_multistep sampler; the same seed gives ComfyUI's noise. On an RTX 4080 SUPER, a
  1024 x 1024 image takes 2.3 s warm: 2.0 s for the 8 steps, 0.3 s to decode.
- `D.load` reads MiniMax H3 as it is stored, including the 4-bit (W4A8) checkpoint with its LoRA and AdaLN map.
  All 50 blocks fit a 16 GB GPU. `Model.prepare(sys, steps, shift)` makes a schedule's AdaLN tables once and keeps
  them, so a long-running service pays for them at start and no request does (339 MB on the GPU for 6 steps).
  `Model.sample` runs the Euler passes, then `Decoder.decode` and `AudioDecoder.decode` give the frames and the sound.
- Kernel tunings are measured on the GPU the first time each shape runs. `G.saveTunings` and `G.loadTunings` keep
  them in a folder for the next run.
- On one RTX 4070 Ti SUPER (16 GB), an H3 request of 832 x 480, 124 frames and 6 steps takes about 31.5 s warm:
  4.29 s a denoising pass, 5.55 s of video decoding and 0.18 s of audio. A FLUX.3 klein step at 1024 x 1024
  takes 0.175 s on an RTX 4080 SUPER. See the [benchmarks](../benchmarks.md).

## Next steps

- The [reference](../reference/sdk.md) lists every function of these modules.
- [Machine learning in GizDB](sql-ml.md) trains and serves models with SQL.