Tensors, GPU kernels and AI models

Preview Runs today. The interface can still change. Roadmap

The standard library computes on the CPU's vector lanes and cores, and on an NVIDIA or AMD GPU when the machine has one. The GPU's driver is found when the program runs: nothing is linked at build time, and the same executable runs on a machine without a GPU.

This guide takes about fifteen minutes. Each output on this page is the real output of its example.

Step 1: Tensors on the CPU

A tensor has a shape and its values in row-major order. A chain of elementwise operations on T.lazy values runs as one pass over the data when you call eval.

import * as T from "gizml:tensor";

function main(): void {
    const a = T.tensor([2, 3], [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
    const b = T.tensor([3, 2], [1.0, 0.0, 0.0, 1.0, 1.0, 1.0]);
    const c = a.matmul(b);
    print(c.text());
    // Elementwise chains run fused, in one pass over the data.
    const d = T.lazy(c).mul(T.lazy(T.scalar(0.5))).relu().eval();
    print(d.text());
}

[[4, 5], [10, 11]]
[[2, 2.5], [5, 5.5]]

Step 2: The same work on a GPU

Gpu.open finds a GPU, or gives none. on(gpu) copies a tensor to it, and the operations on GPU tensors run there. host() copies a result back. This program prints the same line with or without a GPU.

import {System} from "gizlang:system";
import * as T from "gizml:tensor";
import * as Gpu from "gizhw:gpu";

function main(sys: System): void {
    const a = T.tensor([2, 3], [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
    const b = T.tensor([3, 2], [1.0, 0.0, 0.0, 1.0, 1.0, 1.0]);
    let c = a.matmul(b);
    match (Gpu.open(sys)) {
        // On a GPU, the same call runs there; host() copies the result back.
        Option.some(gpu) => { c = a.on(gpu).matmul(b.on(gpu)).host(); }
        Option.none() => {}
    }
    print(c.text());
}

[[4, 5], [10, 11]]

Step 3: Write a kernel

A tile program says what one block of the output is. gizml:tile decides the rest: which thread holds which value, workgroup memory, barriers, the masks at the edges, and the matrix cores. The same program runs on host tensors on the CPU, where print and the test runner work, and on GPU tensors on the GPU.

An operator on a tile is its method: a * s is a.mul(s). A number next to a named tile becomes a tile of that value: t + 1.0 is t.add(t.k(1.0)).

import * as T from "gizml:tensor";
import * as Tile from "gizml:tile";

function main(): void {
    const x = T.tensor([4, 4], [-2.0, -1.0, 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, -3.0, -4.0, 6.0, 7.0, 8.0, -9.0, 1.0, 0.5]);
    const w = T.tensor([1, 4], [1.0, 2.0, 3.0, 4.0]);
    const y = Tile.run([1], (p: Tile.Program): void => {
        const z = p.int(0);
        const a = p.read(x, [4, 4]).at([z, z]);
        const s = p.read(w, [1, 4]).at([z, z]);
        // One block of the output: max(a * s + 1, 0). Rows broadcast against s.
        const t = a * s;
        p.write(T.zeros([4, 4]), [4, 4]).put((t + 1.0).relu(), [z, z]);
    })[0];
    print(y.text());
}

[[0, 0, 1, 5], [3, 7, 13, 21], [0, 0, 19, 29], [9, 0, 4, 3]]

For a real kernel, the grid has one point for each output block, p.id(axis) is the block of this point, and p.fold loops over the blocks of a reduction. A matrix product of 128 x 128 blocks:

import * as T from "gizml:tensor";
import * as Tile from "gizml:tile";

/** The product of a ([m, k]) and b ([k, n]). */
export function product(a: T.Tensor, b: T.Tensor, m: i64, n: i64): T.Tensor {
    return Tile.run([Tile.cdiv(m, 128), Tile.cdiv(n, 128)], (p: Tile.Program): void => {
        const i = p.id(0); const j = p.id(1);
        const A = p.read(a, [128, 32]); const B = p.read(b, [32, 128]);
        const acc = p.fold(p.int(0), A.blocks(1), p.zeros([128, 128]),
            (k: Tile.Index, acc: Tile.Tile): Tile.Tile => acc + A.at([i, k]).dot(B.at([k, j])));
        p.write(T.zeros([m, n]), [128, 128]).put(acc, [i, j]);
    })[0];
}

• Tile.ir(...) prints the program as the compiler sees it, with the layout of each value.
• p.stages(n) sets how many blocks of a product's operands are read ahead (1 to 4).
• Tile.tune runs each configuration you give it (block sizes, warps) on the GPU and keeps the fastest for each tensor shape.

Step 4: Gradients

A Graph records the operations on its values. backward gives the gradient of a one-element result against every input. A kernel of your own joins a graph through custom, with the kernels of its gradient.

import * as T from "gizml:tensor";
import {Tensor} from "gizml:tensor";
import * as G from "gizml:grad";
import * as Tile from "gizml:tile";

/** x * y elementwise, written as a tile program. */
function product(x: Tensor, y: Tensor): Tensor {
    return Tile.run([1], (p: Tile.Program): void => {
        const z = p.int(0);
        const a = p.read(x, [4, 4]).at([z, z]);
        const b = p.read(y, [4, 4]).at([z, z]);
        p.write(T.zeros(x.shape), [4, 4]).put(a * b, [z, z]);
    })[0];
}

function main(): void {
    const a = T.tensor([2, 2], [1.0, 2.0, 3.0, 4.0]);
    const b = T.tensor([2, 2], [5.0, 6.0, 7.0, 8.0]);
    // Built-in operations: the gradient of sum(a * b) is b for a and a for b.
    let g = G.graph();
    const x = g.input(a); const y = g.input(b);
    const grads = g.backward(g.sum(g.mul(x, y)));
    print(grads.of(x).text());
    // Your own kernel with its own gradient kernels is one operation of the graph.
    let h = G.graph();
    const hx = h.input(a); const hy = h.input(b);
    const xy = h.custom([hx, hy], product(a, b), (d: Tensor): Vec<Tensor> => [product(d, b), product(d, a)]);
    print(h.backward(h.sum(xy)).of(hy).text());
}

[[5, 6], [7, 8]]
[[1, 2], [3, 4]]

Step 5: Run a language model

To run a model by name, downloaded and converted on its first use, see Run AI models on your computer. This step shows the parts under it.

gizml:llm runs Qwen 3.5 (and later Qwen versions of that architecture), Qwen 3, the text model of Qwen3-VL, Qwen 2, Llama and Mistral. Convert a Hugging Face folder, or a GGUF file of Llama, Qwen2 or Qwen3 (llama.cpp's format, any of its Q2_K to Q8_0 block types: L.convert(sys, "Qwen3-8B-Q4_K_M.gguf", ...)), once. A PEFT LoRA adapter folder as convert's last argument is merged into the weights before they are quantized. Each linear layer is stored in 4 bits and the output layer in 6 bits; other widths are a choice of Bits. Then load it on the best device the machine has and generate.

import {System} from "gizlang:system";
import * as L from "gizml:llm";

function main(sys: System): void {
    L.convert(sys, "Qwen3-8B", "qwen3-8b-q4", { layers: 4, head: 6 })!;
    const m = L.load(sys, "qwen3-8b-q4")!;
    const prompt = m.tokenizer.tokens(m.chatPrompt([{ role: "user", content: "Name three rivers." }], false));
    let e = m.engine(1, prompt.length + 200)!;
    const out = e.generate(sys, [{ prompt: prompt, limit: 200, sampling: L.greedy() }])!;
    print(m.tokenizer.decode(out[0], true));
}

• One Engine serves many sequences at once: pass more requests to generate, and give engine more slots.
• A request's Sampling picks its tokens: L.greedy(), or a temperature with topK, topP and a seed. Its optional fields shape the logits first: presence, frequency and repetition penalties, a grammar of gizml:grammar (anyObject() or fromSchema(schema); from the token after after on), and logprobs (each Progress then has the tokens' log probabilities and that many of the most likely).
• Engine.add, advance and cancel run requests in turn: a request joins while others run, and its tokens come step by step. A later request reads only what its slot does not hold. Model.chatTokens marks where a chat's messages end; the engine keeps the recurrent states there, so the next turn of the chat reads only its new messages. When a request takes a slot from another chat, the slot's start goes to host memory (at most as much as stays free), and the next turn of that chat takes it back.
• Model.chatTokens also marks the head: the system turn with the tools. Engine.add(request, mark, head) makes a head of at least 256 tokens a shared start: later prompts that start with it read only the rest, and on a GPU every slot reads one copy of its keys and values in the KV pool. Agents whose turns start with one long system prompt so take its memory and its reading once.
• While sequences generate, a step reads at most the prompt rows that add as much time as the step of those sequences alone (both measured as the engine runs). A long prompt then slows the others to half their pace at most.
• gizml:ai's serve serves engines with the OpenAI API on a Listener: POST /v1/chat/completions and POST /v1/completions, whole or streamed, POST /v1/images/generations and GET /v1/models. Each engine has a name; a request names its model (model), and the first is the default. Make the engines of the smaller models first: an engine's KV pool takes the memory its slots can use. A request's reasoning_effort (low, medium, high) selects the thinking and the template's instruction for that effort. gizlang:hub's ModelApi and other OpenAI clients can use it. A model whose template writes tool calls (Model.calls: Qwen's <tool_call><function=...> format) takes tools and answers with tool_calls (finish reason tool_calls); the assistant's tool_calls and reasoning_content and the tool messages of the next request go back into the prompt as the template writes them. L.toolCalls reads the calls in a model's text.
• Call Engine.warm (or make the engine with Ai.ready) before serve takes requests: the first run of each step shape compiles, tunes and records its kernels (seconds for a large model), so warming moves that wait to the server's start.
• A model with a multi-token prediction layer drafts tokens ahead and keeps the ones the model agrees with. The tokens follow the same distribution as without drafts, and greedy output is the same. A draft reads an eighth of the output layer: the rows of frequent tokens and of the tokens the engine read last, so the drafts follow the prompt's language.
• The engine keeps keys and values in 8 bits, else in 4, else keys in 4 and values in 2: the first in which the slots' whole contexts fit. 8 bits give the output of 16 bits; 4 bits hold twice the context. Set kv of engine to choose: "16", "8", "4", "2", or keys and values apart, as "4/2". Each head is coded over the range that gives it back with the least squared error.
• L.load(sys, folder, true, layers) loads only the first layers layers. Engine.hidden then gives the hidden states of each token after them, for example as the text encoder of an image model.
• A model whose weights leave less than 1 GiB of its GPU's memory free loads across the machine's other NVIDIA GPUs too (those CUDA_VISIBLE_DEVICES shows): each GPU holds a stage of consecutive layers, in proportion to its free memory, and a step runs the stages in turn. L.loadAcross(sys, folder, gpus, counts) chooses the GPUs and the layer counts. A model in stages gives logits only; its last stage holds the MTP layer and drafts. It reuses a chat's start and a shared head as one GPU does: each stage keeps its own KV pages and states of them.

Step 6: Generate images and video

To make an image with one command, see Run AI models on your computer. This step shows the parts under it.

gizml:diffusion runs image and video models on the same tensors and kernels: FLUX.3 klein (text to image and image edits), Z-Image Turbo (text to image) and MiniMax H3 (text to video with sound). They need an NVIDIA GPU with INT8 matrix cores (sm_80 and later); their attention runs in 8 bits on sm_89 and later. The host runs the same blocks in f32, as the reference of the tests.

FLUX.3 klein takes its text from the first 32 layers of Qwen3-VL-4B, converted once with gizml:llm. D.Klein is the whole pipeline (the text encoder, the model and the VAE's encoder and decoder on one GPU), from a prompt, and pictures to edit, to an image:

import {System} from "gizlang:system";
import * as G from "gizhw:gpu";
import * as D from "gizml:diffusion";
import * as Image from "gizlang:image";

function main(sys: System): void {
    const gpu = G.open(sys)!;
    D.convertTextEncoder(sys, "Qwen3-VL-4B", "flux3-text")!;
    const klein = D.loadKlein(sys, "flux3-klein", "flux2-vae.safetensors", "flux3-text", gpu)!;
    // Text to image: 1024 x 1024 from seed 42's noise (ComfyUI's for that seed).
    const fox = klein.paint(sys, "A red fox in fresh snow", 42u, [], [1024, 1024])!;
    // An edit of one or more pictures; the image has the first one's aspect (here 1 megapixel).
    const photo = Image.decode(sys.readFile("fox.jpg")!, 100000)!;
    const edit = klein.paint(sys, "Make it night", 42u, [photo])!;
    print(`${fox.width} x ${fox.height}, ${edit.width} x ${edit.height} pixels`);
}

• Klein.paint is condition (the references' latents and the text's rows), sample and decode; a caller that times the stages, or runs them apart, calls those. D.Upsampler.caption (BFL's prompt upsampler, Qwen3.5-9B) writes a fuller prompt from a short one and the pictures; D.fluxLayout puts bounding boxes in it.
• The 16 steps are the distilled klein model's. A step's linears run on the INT8 matrix cores after a Hadamard rotation of their inputs (ConvRot), and the 3 x 3 convolutions of the VAE read their patches as the product copies them (T.lazyConv). D.loadFlux also reads ComfyUI's INT8 files as stored.
• The step cache (cache, 0.3 by default) skips the joint blocks of steps that change little. For an edit, the option refresh (a number of steps) runs the reference pictures through the blocks only that often: faster, and a little different, so it is off by default.
• D.ZImagePipeline is Z-Image Turbo the same way, from ComfyUI's files: the model (its linears INT8 as it loads), the first 35 layers of Qwen3-4B (D.convertZImageText) and FLUX.1's VAE. condition, sampleWith and decode make an image in 8 steps of ComfyUI's res_multistep sampler; the same seed gives ComfyUI's noise. On an RTX 4080 SUPER, a 1024 x 1024 image takes 2.3 s warm: 2.0 s for the 8 steps, 0.3 s to decode.
• D.load reads MiniMax H3 as it is stored, including the 4-bit (W4A8) checkpoint with its LoRA and AdaLN map. All 50 blocks fit a 16 GB GPU. Model.prepare(sys, steps, shift) makes a schedule's AdaLN tables once and keeps them, so a long-running service pays for them at start and no request does (339 MB on the GPU for 6 steps). Model.sample runs the Euler passes, then Decoder.decode and AudioDecoder.decode give the frames and the sound.
• Kernel tunings are measured on the GPU the first time each shape runs. G.saveTunings and G.loadTunings keep them in a folder for the next run.
• On one RTX 4070 Ti SUPER (16 GB), an H3 request of 832 x 480, 124 frames and 6 steps takes about 31.5 s warm: 4.29 s a denoising pass, 5.55 s of video decoding and 0.18 s of audio. A FLUX.3 klein step at 1024 x 1024 takes 0.175 s on an RTX 4080 SUPER. See the benchmarks.

Next steps

• The reference lists every function of these modules.
• Machine learning in GizDB trains and serves models with SQL.