# Language models

`gizai/llm` runs language models on NVIDIA and AMD GPUs, Apple GPUs, the CPU and an AMD NPU. One engine serves the
command line, the OpenAI-compatible server and your programs. This guide is the one reference for it: every
option, what it changes, what it costs in quality, and how to measure it.

| Your task | Section |
| --- | --- |
| Chat with a model on this computer | [Chat](#chat) |
| Give an OpenAI client a local model | [Serve the OpenAI API](#serve-the-openai-api) |
| Find a model or add one | [Choose a model](#choose-a-model) |
| Make a model folder in the weight format you want | [Convert a model](#convert-a-model) |
| Trade speed, memory and quality | [Engine options](#engine-options) |
| Know what the fast settings cost | [Quality](#quality) |
| Measure speed and quality on your hardware | [Measure](#measure) |
| Call a model from a program | [Use a model in a program](#use-a-model-in-a-program) |
| Change the engine | [How it works](#how-it-works) |
| Train a model from nothing, on one or many GPUs | [Training language models](training.md) |

## Before you start

[Install Gizlang](../quickstart.md) and run `giz doctor`. A model downloads and converts once, on its first use: check
its license, disk space, memory and GPU memory first. A gated repository needs its license accepted on Hugging Face
and `HF_TOKEN`; `HF_ENDPOINT` names a mirror.

| Device | What runs |
| --- | --- |
| NVIDIA GPU (CUDA), sm_80 and later | Every model and option. INT8 and BF16 tensor-core products; attention by TMA on sm_90 and later. |
| NVIDIA sm_89 and later | Also FP8 products (FP8 weights, 8-bit attention values). |
| NVIDIA sm_120 and sm_121 (RTX 50, GB10) | Also NVFP4 products: NV4 weights with their inputs rounded to NVFP4 (W4A4). |
| AMD GPU (HIP) | Matrix-core products on gfx11 and gfx120, portable kernels elsewhere. |
| Apple GPU (Metal) | Matrix products; measured on an M5 Pro ([bench/ai/llm](https://github.com/gizai/gizlang/blob/main/bench/ai/llm/README.md)). |
| CPU | Every model but Qwen3.8 Flash, slower. Keys and values in 8 bits, 1 draft a step by default. |
| AMD NPU (XDNA 2) | Gemma 4 E2B and Qwen3.5-9B (including Ornith 1.5 9B), from `model.q4nx` and `xclbins/`: one chat, at most 4096 positions. |

`GIZLANG_DEVICE` chooses the device ([settings](../reference/settings.md)); `--cpu` runs a command's chat models on the
CPU for `model.safetensors` folders. A `model.q4nx` folder runs on the NPU.

`gizai/llm/convert`'s `convertQwenNpu(from, to)` converts a bf16 Qwen3.5-9B Hugging Face folder for the NPU:
Q4K projections, q8 head and alpha/beta projections, bf16 embeddings. Supply the unmodified FastFlowLM 1.0.9
Qwen3.5-9B `layer.xclbin` and `lm_head.xclbin` in `to/xclbins/`, with their `LICENSE_BINARY.txt`. Its host library is
not needed. The public FastFlowLM model's `main` branch has older Q41 blocks; its `flm_q4k_high_precision` revision
has the format these kernels take. Qwen prompts currently run a token at a time; MTP and vision are not implemented
on this path. A read appends to the recurrent state or restarts at zero; a partial rewind needs a replay.

## Chat

```sh
giz ai chat qwen3.5-4b "Why is the sky blue? Two sentences."
```

```text
The sky appears blue because sunlight is scattered by the Earth's atmosphere, and shorter blue wavelengths are scattered
more effectively than longer red wavelengths. This phenomenon, known as Rayleigh scattering, causes the blue light to
reach our eyes from all directions in the sky.
```

- The answer goes to standard output as it comes. The first use reports the download and conversion on standard error.
- Without a prompt, `chat` reads your messages line by line and keeps the chat. End it with Ctrl-D.
- Without a model, `giz ai chat` chats with this computer's default chat model ([choose a model](#choose-a-model)).
- `--think` lets a thinking model think first, at effort `high`; `--think=low`, `medium` or `xhigh` give it 2048, 8192
  or 32768 tokens instead of 16384. `giz env -w AI_EFFORT=LEVEL` makes a level the default. The thinking goes to
  standard error, so the answer stays clean.
- `--max N` limits an answer to N tokens. `--context N` sets the positions of the chat (32768 by default).
- `--temperature T`, `--top-p P`, `--top-k K`, `--min-p P` and `--seed N` replace the model's own sampling (its
  `generation_config.json`; [sampling](#sampling)). Temperature 0 takes the most likely token each step.
- Every [engine option](#engine-options) applies, for example `--kv 8` or `--draft mtp`.
- `chat --cloud MODEL` chats with any model of GizStack's AI Gateway instead, billed to the account that
  `giz auth login` signed in.

## Serve the OpenAI API

```sh
giz ai serve qwen3.8-27b --port 8000
```

```text
serving qwen3.8-27b at http://127.0.0.1:8000/v1
```

Any OpenAI client can use it. Give it the address and the model's name:

```sh
curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model": "qwen3.8-27b", "messages": [{"role": "user", "content": "Hello"}], "stream": true}'
```

- The server answers `POST /v1/chat/completions` (whole or streamed, with tools and `reasoning_effort`),
  `POST /v1/completions` (a text `prompt`), `POST /v1/images/generations` and `/v1/images/edits` (image models,
  [compute guide](compute.md#make-an-image)), `POST /v1/embeddings`, `POST /v1/rerank`, `GET /v1/models` and
  `GET /metrics` (Prometheus text: requests running and waiting, KV pages, steps, tokens given out and read, drafts
  kept, and the GPU memory its engines hold and the card uses).
- A thinking model thinks at most 2048, 8192, 16384 or 32768 tokens at `reasoning_effort` `low`, `medium`, `high` or
  `xhigh`, and at most half of `max_tokens`. At that budget the server closes the thinking, and the model goes on.
- A request takes every [sampling field](#sampling) of OpenAI and vLLM, `n` (up to 16 choices) and `stop` (strings).
  The sampling flags (`--temperature`, `--top-p`, `--top-k`, `--min-p`) set the defaults that a request's own fields
  replace.
- `response_format` `{"type": "json_object"}` or `{"type": "json_schema", "json_schema": {"schema": ...}}` lets the
  model write only JSON of that schema (after its thinking): `gizai/llm/grammar` masks each token that cannot continue
  it. A schema's `type`, `properties`, `required`, `additionalProperties` (false when not given), `items`, `minItems`,
  `maxItems`, `minLength`, `maxLength`, `enum`, `const`, `anyOf`, `oneOf` and `$ref` hold; `minimum`, `maximum`,
  `exclusiveMinimum` and `exclusiveMaximum` on a plain `integer` or `number` (up to 10^18; a bounded number has no
  exponent, and a fraction only where every fraction of its whole part is inside); and `pattern` on a plain `string`:
  `^` and `$`, characters, `.`, classes of ASCII characters with ranges and `\d` `\w` `\s`, and the quantifiers `*`
  `+` `?` `{n,m}` (no groups, alternatives or back references; no `\u` escape in the string). A `oneOf` needs choices
  that no value matches twice (other kinds, other `const`/`enum` values, or a required property of other values). A
  schema with a keyword the grammar does not enforce (`format`, a pattern outside that subset, ...) is refused with
  status 400.
- A message's content parts `image_url` and `video_url` (a data URL each: `data:...;base64,...`) show a picture or a
  video to a model with a vision tower (Qwen3.5, Qwen3-VL: `visual.safetensors` in its folder). A picture keeps its
  aspect in sides of multiples of 32, from 65,536 to 1,048,576 pixels (up to 1024 prompt tokens). A video gives 2
  frames a second, at most 32 over the whole video, each at most 200,704 pixels; `ffmpeg` on the PATH decodes it.
  Another model refuses such a request. When the GPU has no room for a picture's pass now, the answer is 503 with
  `Retry-After`, also for a stream (its status waits for the first piece), so a client can go to another server.
- `/v1/embeddings` gives the embeddings of an `embed` model: `input` is a text or a list of texts, and
  `encoding_format` is `float` (the default) or `base64` (the f32 values, little-endian).
- `/v1/rerank` (and `/v2/rerank`, the Jina and Cohere API) ranks `documents` for a `query` with a `rank` model:
  `results` come by `relevance_score`, the highest first. `top_n` keeps that many, and `return_documents: false`
  leaves out the texts.
- Name more than one model to serve them together, for example `giz ai serve qwen3.5-4b flux3-klein e5-small`. The
  first chat model is the default, and its keys and values take the GPU memory the others leave. One image or one
  encoding request runs between two steps of the chats.
- `--slots N` sets the chats that run together (4 by default) and `--context N` the positions of each. Without
  `--context`, a chat may take every position the GPU holds in 4 bits, up to the model's context: the chats share
  them, and one that finds no room waits at most 10 s for the chat that ran longest to step aside. A
  chat's next request reads only its new messages: the engine keeps the rest. `--host 0.0.0.0` listens on every
  address. The server starts after it has compiled and tuned every step shape (`Engine.warm`), so no request waits for
  that.
- `ALIAS=NAME` serves model NAME under the name ALIAS, as `/v1/models` lists it and requests give it:
  `giz ai serve coder=/models/q38 --port 8000`. Without an alias, a model folder is served by its folder's name.
- `--lora NAME=PATH[:STRENGTH][,PATH...]` serves PEFT LoRA adapters (folders or `.safetensors` files) over the
  model's own weights, without converting it again; give it once for each. A request whose `model` is NAME runs with
  that LoRA, and `/v1/models` lists it. Requests of many LoRAs and of none run in the same step: each row reads its own
  adapter's A and B beside the shared quantized product. A chat's keys and values are reused only under the LoRA that
  made them. LoRAs run on a GPU, on every linear (the head and the MTP layer too). `giz ai chat --lora PATH[:STRENGTH]` chats with
  one, or several stacked.
- `--api-key KEY`, or `GIZ_AI_API_KEY` in the environment (which keeps it out of the process list), makes each request
  to `/v1` and `/v2` give `Authorization: Bearer KEY`; other requests get status 401. `/metrics` stays open.
- Without a model, `giz ai serve` serves this computer's default chat model ([choose a model](#choose-a-model)).
- For a coding agent, set its OpenAI base URL to `http://127.0.0.1:8000/v1` and its model to the name you served.

## Choose a model

```sh
giz ai models
```

The output starts with this computer's GPU and its default chat model, then the catalog: each model's name, kind
(`text`, `embed`, `rank`, `image`, ...), whether it is ready here and whether it is its kind's default, its repository
and a short text.

- **The default chat model** is the one `giz ai chat` and `giz ai serve` run when you name none. On a GPU of 16 GB or
  less, or with no GPU, it is Ornith 1.5 9B (`ornith-1.5-9b`); on a larger GPU, Swift 1.5 (`swift-1.5`), a Qwen3.8 27B
  tuned for coding agents. Swift 1.5 has its own license (Swift Open License 1.0): read its terms before commercial
  use. `giz env -w GIZ_AI_MODEL=NAME` saves another default.
- **The formats follow the GPU.** A chat model of 4-bit linears converts them to NVFP4 on an NVIDIA Blackwell GPU (its
  FP4 matrix products), and to 4-bit codebooks on other GPUs and the CPU. Each format is converted once and kept.
  `NAME:FORMAT` names a format yourself: `giz ai chat qwen3.8-27b:q8`.
- **A drafter** comes with the 27B models (Qwen3.8 27B and its fine-tunes): the DFlash 2 block drafter converts beside the model on its first
  use, and drafts tokens on one GPU ([drafts](#drafts)).

A name of the catalog is an entry of a model family, `ai/models/NAME/family.giz`; a new family is a new folder and
`giz ai registry`
([ai/README](https://github.com/gizai/gizlang/blob/main/ai/README.md)).

Three other names run too:

- **A Hugging Face repository** (`owner/name`, or its address), run as its files make it: `config.json` of an
  architecture below (`giz ai chat Qwen/Qwen3-8B "Hello"`), converted with 4-bit layers and a 6-bit output layer; a
  BERT, RoBERTa or XLM-RoBERTa encoder, an embedding model or a reranker (`...ForSequenceClassification`); or GGUF
  files, of which the one named by its address (`.../blob/main/FILE.gguf`) or else the `Q4_K_M` file. A diffusers
  pipeline runs only by a family of the catalog.
  A repository that holds a model already in this library's format (a `giz ai convert` folder uploaded as it is)
  downloads and runs as it is, with its `drafter/` and `tune/` folders: nothing is converted.
- **A model folder**, by its absolute path: `giz ai serve /models/qwen3-8b-nv4`. Make one with
  [`giz ai convert`](#convert-a-model) to choose its formats.
- **`pull NAME`** downloads and converts a model ahead, so its first chat starts at once.

The architectures that `gizai/llm/recipes` reads, by `model_type` of `config.json` (or, for a renamed `model_type`
without code of its own, by the transformers class its `architectures` names, as `Qwen3ForCausalLM`):

| Family | `model_type` |
| --- | --- |
| Llama, InternLM 3, Seed-OSS | `llama`, `internlm3`, `seed_oss` |
| Mistral | `mistral` |
| Qwen 2, 2.5 and their vision models | `qwen2`, `qwen2_vl`, `qwen2_5_vl` |
| Qwen 3 | `qwen3` |
| Qwen3-VL (vision) | `qwen3_vl` |
| Qwen 3.5 and Qwen3.8 (delta-rule and attention layers, an MTP layer, vision) | `qwen3_5` |
| Qwen3.8 Flash Next (experts between hyper-connections; one GPU) | `qwen4_exp` |
| Gemma, Gemma 3, Gemma 4 | `gemma`, `gemma3`, `gemma4`, `gemma4_unified` |
| Phi-3, Granite, GLM-4, SmolLM3, ERNIE 4.5, EXAONE 4 | `phi3`, `granite`, `glm4`, `smollm3`, `ernie4_5`, `exaone4` |
| LFM2 and LFM2.5 (short convolution and attention layers) | `lfm2` |
| BERT, RoBERTa, XLM-RoBERTa encoders (`embed`, `rank`) | `gizai/llm/encoders` |

A supported architecture does not prove every variant or size; the catalog lists the models that are checked. A new
architecture is one recipe: how its `config.json` reads, and the names of its tensors.

## Convert a model

A model runs from a folder in this library's format, or from a Hugging Face folder as it is stored: `LlmModel.load`
reads its safetensors (one file or shards) with no conversion step. Its float linears stay BF16 (F16 when the source is
F16), and its FP8 and NVFP4 linears keep their codes. Each linear is laid out in memory as `convert` would write it,
just before it goes to the device. A model with experts or n-gram embeddings, and a vision tower, still load converted.
A catalog name, a repository and a GGUF file are converted on
their first use (`giz ai chat model.gguf "Hello"`, `giz ai serve Qwen3-8B-Q4_K_M.gguf`) and kept
([the conversion cache](#the-conversion-cache)). `giz ai convert` makes a folder you place yourself, from a Hugging
Face folder or repository, or a GGUF file:

```sh
giz ai convert Qwen/Qwen3-8B /models/qwen3-8b                       # 4-bit layers, 6-bit head
giz ai convert /hf/Qwen3-8B /models/qwen3-8b-q8 --weights 8 --head 8
giz ai convert /hf/Qwen3.8-27B-NVFP4 /models/q27-nv4 --weights nv4   # keeps the checkpoint's NVFP4 values
giz ai convert Qwen3-8B-Q4_K_M.gguf /models/qwen3-8b-gguf
giz ai convert /hf/base /models/tuned --lora /hf/my-adapter
giz ai convert /hf/drafter /models/q27-nv4/drafter                  # a block drafter beside its model
giz ai serve /models/q27-nv4
```

| Flag | Values | Default |
| --- | --- | --- |
| `--weights FORMAT` | The linear layers' format (below). | `4` |
| `--head FORMAT` | The output layer's format. | `6` |
| `--sensitive BITS` | Codebook bits of the linears nearest the ends of the layer stack, one weight in eight ([mixed precision](#mixed-precision)). | none |
| `--layers N` | Only the first N layers, without the head: a text encoder (`Engine.hidden`). | all |
| `--lora DIR` | A PEFT LoRA adapter merged into the weights before they are quantized: W + alpha / r · B A (alpha / sqrt(r) with rsLoRA). `giz ai train --lora` trains one ([Training](training.md#train-a-lora-adapter)). | none |
| `--checkpoint nvfp4` | Write a standard checkpoint, not a model folder: the float model's layer linears as compressed-tensors stores NVFP4 (`NAME_packed`, `NAME_scale`, `NAME_global_scale`: one scale a tensor), the rest as it is. Convert the checkpoint again, without this flag, to run it: its values are kept. It writes no `quantization_config` into `config.json` yet. | none |

| Format | What a weight holds | Products on a GPU |
| --- | --- | --- |
| `2` to `8` | A code of 2 to 8 bits of a codebook, after a Hadamard rotation of each block of 128 inputs. | Decoded in the kernel. |
| `bf16`, `f16` | The value. | BF16 or F16 tensor cores. |
| `f8` | FP8 E4M3 with a scale per row and block. | FP8 tensor cores on sm_89 and later. |
| `nv4` | NVFP4: E2M1 with an E4M3 scale per 16 inputs. | NVFP4 tensor cores on sm_120 and sm_121, inputs rounded to NVFP4. |

### One path for every source

Safetensors and GGUF, float, FP8, NVFP4 and GGUF blocks enter by one path (`gizai/llm/plan`):

| Step | What it is |
| --- | --- |
| `WeightView` | One source tensor, whatever its container: name, shape, dtype, its own quantization and scales, its place in its file. |
| The planner | For each linear, the `QuantSpec` it is stored as, and how: its codes laid out again with every value kept (`relayout`), or its values rounded to the format (`requantize`). A folder already in this library's format is used as it is. |
| `QuantSpec` | One record of how a linear is stored: format, bits, the inputs a scale covers, the scales' kind, the codebook's size, the rotation, the order of the codes, the version. The file names each linear's spec in its metadata. |
| The products | The same linears and calls on the CPU, NVIDIA, AMD and Metal. A device's matrix cores choose only the kernel. |

| Source | Stored as | Values |
| --- | --- | --- |
| Float weights (F32, F16, BF16) | The asked format, after the rotation. | Rounded once. |
| NVFP4 (NVIDIA Model Optimizer, compressed-tensors) | `nv4` | Kept. |
| FP8 with a scale a row or a tensor | `f8` | Kept. |
| FP8 with 128 × 128 block scales | `f8` | Rounded to F8 again. |
| GGUF Q4_K, Q6_K (inputs a multiple of 256) | `k4`, `k6`: the blocks in the lane order. | Kept. |
| GGUF Q2_K to Q8_0, Q8_K, IQ1_S to IQ4_XS | Decoded, then as float weights. | Rounded once more. |

- A source of quantized values (NVFP4, FP8, K blocks) gives a model of unrotated inputs: rotating the inputs of a
  weight needs its values rounded again. Its other linears are stored in `f8`. No linear is then turned back where it
  runs. With `--lora`, a GGUF file keeps the rotation (the merged linears take a codebook), and its untouched K blocks
  read their inputs turned back.
- A loader does not guess a format. It reads each linear's spec, refuses a spec of another version, and checks the
  dtype, shape and size of every tensor against the spec before anything runs. A file of the earlier format (no specs
  in its metadata) still loads.
- A GGUF file of Llama, Qwen2, Qwen3, Qwen3.5 or Gemma 4 is read, with its tokenizer and chat template. Other
  architectures are refused by name. A Qwen3.5 file holds no MTP layer, so it runs without MTP drafts.
- Measured on the CPU with Qwen3.5 0.8B, 1024 tokens of this guide, keys and values in 16 bits: the `Q4_K_M` file's
  conversion (6 s) has -log p 2.981, the original's conversion at `--weights 4` 3.159. Against the original at 8 bits
  the file's next-token KL is 0.00 to 0.12 a position on three short texts.

### Mixed precision

`--sensitive 6` with `--weights 4` gives 6 bits to the linears nearest the ends of the layer stack, layer by layer
from both ends inward, until they hold one weight in eight: 4.25 bits a weight. The first layers' errors pass through
every later layer, and the last layers' go straight to the logits. It is a rule of position, not a measurement: after
the rotation, every linear's codebook error is the same within 5%, so the weights alone rank nothing. Its gain in
-log p is not measured yet.

A codebook is fitted on 256 rows of its linear, one from each of 256 equal runs of rows, each giving every fourth of
its blocks of 128 inputs (a linear of fewer rows gives more of each row's blocks). Against 64 whole rows at even steps (the earlier sample), on 16 linears of Qwen3.5 0.8B the
mean error is the same and the worst linear's error over a fit on all its values falls from 7.2% to 2.0% at 4 bits
(1.2% to 0.8% at 3 bits).

### The conversion cache

| What names a conversion | Why |
| --- | --- |
| The source's revision (a repository's pinned commit; a GGUF file's path, size and time of change) | A new source converts again. |
| The formats asked (`--weights`, `--head`, `--sensitive`) and the layers kept | Each setting has its own folder. |
| The conversion's algorithm and the layout's version | A new version of this library converts again. |

- The folder holds that key in `conversion.json`, written last, with the size of `model.safetensors`.
- A folder is used only when its key is the one asked and the file has that size. A folder of an earlier version, of
  another key, or cut short is converted again in its place.
- A conversion is written to a partial folder that takes its place by a rename. One lock a folder: a second process
  waits for the first.
- A vision tower (`model.visual.*`) goes to `visual.safetensors` and stays in host memory; a picture moves it over the
  bus.
- A model converted from a catalog name or a repository is kept in the [models folder](#where-the-files-are); a folder
  you convert is yours to place.

### Measure the KV costs of a model

The engine chooses a format for each attention layer's keys and values by its cost
([KV formats](#keys-and-values-kv)). `giz ai kvcosts` measures that cost once and keeps it in the folder:

```sh
giz ai kvcosts /models/q27-nv4 sample.txt --tokens 1024
```

It prints and writes `FOLDER/kv-costs`: for each attention layer, the mean KL of the next token when that layer alone
keeps its keys and values in each format (8, 8/4, 4, 4/2) and every other layer in 16 bits, on an exact engine, as
AutoQuant KV scores them. Without the file, every layer takes the costs measured on Qwen3.8-27B. Use a text like your
workload.

## Engine options

One record, `EngineOptions`, holds every option of an engine. `giz ai chat`, `giz ai serve`, the benchmark
`bench/ai/llm/run.giz` and your program (`Model.engine`, `Ai.ready`) read the same flags. A flag left out takes the
best for the model and the device, so the defaults are the tuned setting; set a flag only to trade one thing for
another.

| Flag | `EngineOptions` | Values | Default | What it changes |
| --- | --- | --- | --- | --- |
| `--slots N` | `slots` | 1 or more | 1 (`serve`: 4) | Sequences that run together. More slots: more tokens a second in total, less KV room each. |
| `--context N` | `context` | positions | the trained context, on a GPU at most what its memory holds in 4 bits (`chat`: 32768) | Positions a sequence. Past the trained context, the rotary table stretches (YaRN). |
| `--kv FORMAT` | `kv` | `16`, `8`, `4`, `2`, `nv4`, `nv4/VALUES`, `KEYS/VALUES`, or one a layer | each layer's of least cost that fits | Bytes a position, so contexts that fit, and a small error ([KV](#keys-and-values-kv)). |
| `--kv-codes KINDS` | `kvCodes` | `all`, `nv4`, `uniform` | `all` | The codes the default choice takes: `nv4` drops 4-bit keys at even steps, `uniform` drops NV4 keys ([KV](#keys-and-values-kv)). |
| `--fuse KINDS` | `fuse` | `none`, or `conv`, `nv` comma-separated | `none` | Kernel fusions that were not faster on the RTX 5090: `conv` convolves inside the delta rule's chunk kernel, `nv` writes gate_up's activations as the down GEMM's NVFP4 input. |
| `--drafts N` | `drafts` | 0 or more | 32 on a GPU, 1 on the CPU | The most drafts a sequence and step. 0: no drafts. The engine measures and uses the fastest count up to it. |
| `--draft SOURCE` | `draft` | `block`, `mtp`, `lookup` | the first the model has | Where drafts come from ([drafts](#drafts)). |
| `--lookup N` | `lookup` | 1 or more | 2 | The tokens a lookup draft matches before it. More: fewer drafts, kept more often (vLLM's `prompt_lookup_min`). |
| `--recall N` | `recall` | 0 or more | 8 | The tokens a recalled draft matches before it ([drafts](#drafts)). 0: no recall. |
| `--rows N` | `rows` | rows | 2048 on a GPU (512 on one below 12 GiB), 256 on the CPU | The most rows a step reads: the prompt chunk. More rows: a faster prefill on a large GPU, more work memory. |
| `--prefix on\|off` | `prefix` | `on`, `off` | `on` | Whether a prompt reuses the keys and values of a start that an earlier prompt had. |
| `--reserve BYTES` | `reserve` | bytes | 134217728 (128 MiB) | GPU memory the engine leaves free for other programs. |
| `--exact` | `exact` | flag | off | The linears take their inputs as computed: a slow reference ([precision](#precision)). |

A flag given more than once sweeps in the benchmark: `--kv 8 --kv 4 --drafts 0 --drafts 8` runs all four
combinations and marks the best ([measure](#measure)).

### Keys and values (KV)

Each attention layer keeps the keys and values of every position in pages of 16 positions. A format is the bits of a
key and of a value:

| Format | Size against 16 bits | Use |
| --- | --- | --- |
| `16` | 1 | F16: the reference. |
| `8` | about 1/2 | The output of 16 bits (KL 0.00003, [quality](#quality)). |
| `4` | about 1/4 | Twice the context of `8`. |
| `4/2` | about 1/5 | Keys in 4 bits, values in 2: the most context. |
| `nv4/4`, `nv4/8` | about 1/4, 3/8 | Keys in NVFP4: FP4 products with the queries on RTX 50 GPUs (sm_120). |
| `nv4` | about 1/4 | Keys and values in NVFP4: FP4 products with the queries and with the weights (sm_120, heads of 256 dims). |
| `8,8/4,4,...` | per layer | One format for each pool layer, in order. |

- A code holds `2^bits` even steps over each head's range, after a rotation of each block of the head; the range is
  the one that gives the head back with the least squared error. A head of a size other than 64, 128, 256 or 512
  stays in F16.
- `nv4` keys are NVFP4: an E2M1 code a value of the rotated head, an E4M3 scale a block of 16 dims and an F32
  scale a head. On sm_120 the matrix cores multiply the codes as they are with NVFP4 queries (FP4 products, twice
  the FP8 rate). Elsewhere attention decodes them, so the default choice takes them only on sm_120.
- `nv4` values (with `nv4` keys, heads of 256 dims; other heads take 4 bits) are E2M1 codes of the rotated head and
  an F32 scale a head. On sm_120 the weights of a row become NVFP4 too, and the matrix cores multiply them with the
  codes as they are: no warp decodes keys or values.
- Values lose bits first: a key's error moves every score of its position, a value's only its share of the sum.
- By default the engine gives each layer the format of least summed cost (the folder's `kv-costs`) in which the most
  whole contexts of the slots fit at 4 bits or more; only where none fits it allows 2 bits. A per-layer list needs one
  format for each pool layer: the attention layers, and the MTP layer when it drafts.
- With codes, attention reads 8-bit queries against the key codes in integer products (NV4 keys: FP4 products) and
  the values in FP8; with
  `16` it runs in F16.
- The CPU keeps 8 bits by default.

### Drafts

A step can read drafts after each sequence's token: guesses of the next tokens. The model checks them all in the same
step and keeps those it agrees with, so a step gives several tokens. The tokens follow the same distribution as
without drafts. Greedy output can still differ in a rare place, because a step of more rows sums in another order.

| `--draft` | Source | Needs |
| --- | --- | --- |
| `block` | A block drafter (DFlash, DFlash 2): a small model that drafts a whole block in one pass from the target's hidden states of a few layers over the last 2048 positions. Its selector picks each draft from the 16 most likely tokens. | `drafter/` in the model's folder ([convert](#convert-a-model)), the model on one GPU. |
| `mtp` | The model's multi-token prediction layer: one draft a pass of it. A draft reads an eighth of the output layer, the rows of frequent tokens and of the tokens read last. | A model with an MTP layer (Qwen 3.5, Qwen 3.8). |
| `lookup` | The tokens that came after the sequence's last 2 tokens (`--lookup`) where they came before in its own text. | Nothing. |

- By default the engine takes `block` where the folder has a drafter, else `mtp` where the model has the layer, else
  `lookup`. The model loads only the weights of the source you name: `--draft lookup` or `--drafts 0` leaves the MTP
  layer and the drafter out of GPU memory, so more KV fits.
- With `block` and `mtp`, recall adds drafts: where a sequence's last 8 tokens or more (`--recall`; 0 turns recall
  off) came before in its text (a code edit, a quote) and the source's first draft agrees, or the match is 32 tokens or more, the tokens after the match
  take the whole depth.
- The engine drafts the count that gives the most tokens a second: the tokens a step it expects over the time a step
  of that count took, for each load (sequences generating) and context length. It judges every draft a source made
  against the tokens that follow, also those the step did not read, so it knows what a longer count would keep. A
  count it has not measured runs once. `--drafts N` caps the count.
- On Qwen3.8-27B (RTX 5090) a step of 7 block drafts gives about 6.4 tokens, against about 4.6 for the MTP layer.
  [A 27B model on one RTX 5090](../labs/qwen-27b-rtx5090.md) has the whole setup and its numbers.
- A request whose logits are shaped ([sampling](#sampling)) makes no drafts.

### Prefix reuse and shared heads

- A chat's next request reads only its new messages: `Model.chatTokens` marks where each message ends, and the slot
  keeps its keys, values and recurrent states there. When another chat takes the slot, the old start stays in the KV
  pool until the pool needs the room, then in host memory, and the chat's next turn takes it back. Past the host
  memory free, the oldest starts of a GPU model go to files in `GIZLANG_CACHE/kv` (one folder for each model folder
  and the KV page layout of its stages, a model in stages filing each stage's pages beside the first's; the pages of all of them share `GIZLANG_KV_GB`, 16 GiB by default, the oldest out first), and
  a later start of the server takes them too. A filed start comes back from its page files straight to the card, the
  pages read together (`Gpu.readFiles`).
- A Flash (MoE from the disk) model keeps a chat's start too, with its PLE states; it is taken back whole, as on a
  model with recurrent layers, and slots do not share heads in the pool.
- A system prompt with tools of 256 tokens or more is a shared head: every slot reads one copy of its keys and values.
- `--prefix off` turns all of this off: each prompt is computed whole, as a server without a prefix cache.

### Precision

- On sm_120 and sm_121, NV4 linears round their inputs to NVFP4 by row and run NVFP4 products (W4A4), as standard
  NVFP4 serving does. FP8 linears run FP8 products on sm_89 and later. Codebook linears decode their codes in the
  kernel.
- `--exact` gives the linears their inputs as computed (F32) and runs NV4 and FP8 as matrix-vector products: the
  reference of [quality](#quality), many times slower. There is no faster high-precision mode yet.

### Long contexts

- `--context` past the trained context stretches the rotary table by YaRN for that engine. On 2K tokens, a factor of
  2 cost KL 0.0025 and a factor of 4 cost 0.0068 (Qwen3.8-27B).
- A step reads at most `--rows` prompt rows. While other sequences generate, it reads at most the prompt rows that add
  as much time as their step alone: a long prompt slows them to half their pace at most.

### Many GPUs

A model that fits one GPU serves more requests on several: `giz ai serve qwen3.8-27b --gpus 0,1,2,3 --port 8000` runs a
server of the models on each GPU (its `CUDA_VISIBLE_DEVICES`, at the ports after `--port` on 127.0.0.1) and answers
at `--port` once each listens. A request goes to the replica with the fewest requests in flight; a conversation's
next request goes back to the replica that had it, whose engine keeps its keys and values, while that one holds at
most 4 requests more than the least busy. Answers stream through as they come, `/metrics` gives every replica's
samples with a `replica` label, and a replica that ends ends the server.

A model whose weights leave less than 1 GiB of its GPU free loads across the other NVIDIA GPUs that
`CUDA_VISIBLE_DEVICES` shows: each GPU holds a stage of consecutive layers, in proportion to its free memory, and a
step runs the stages in turn. The last stage holds the head and the MTP layer and drafts; a block drafter needs the
model on one GPU. Each stage keeps its own KV pages. On two RTX 4080s, a decode hop of 4 rows took 10–40 µs and a
2048-row prefill hop 3–4 ms. `loadAcross(folder, gpus, counts)` chooses the GPUs and the layer counts.

### Qwen3.8 Flash

Qwen3.8 Flash (experts and hyper-connection streams) runs on one GPU. Its experts are read from the model file as
steps need them into an expert cache, which takes the GPU memory left after the work buffers, `--reserve` and 2 GiB.

## Sampling

Each request picks its tokens by its own sampling. A model's `generation_config.json` gives the defaults; a model
without that file takes its family's recommended sampling (Qwen3.5: temperature 1, top-k 20, top-p 0.95), or the most
likely token where the family has none. The `chat` and `serve` flags replace them, and a request's fields replace
those. One table names each control in all
three places:

| Control | Flag (`chat`, `serve`) | API field | `Sampling` field | Shapes the logits |
| --- | --- | --- | --- | --- |
| Temperature (0: the most likely token) | `--temperature T` | `temperature` | `temperature` | no |
| Keep the K most likely (0: all) | `--top-k K` | `top_k` | `topK` | no |
| Keep the most likely whose probabilities reach P | `--top-p P` | `top_p` | `topP` | no |
| Keep the tokens at least P times as likely as the most likely | `--min-p P` | `min_p` | `minP` | no |
| Random seed | `--seed N` (`chat`) | `seed` | `seed` | no |
| Less likely each time the answer has the token | | `frequency_penalty` | `frequency` | yes |
| Less likely once the answer has the token | | `presence_penalty` | `presence` | yes |
| Divide the logit of a token the prompt or the answer has | | `repetition_penalty` | `repetition` | yes |
| Add to a token's logit (-100 bans it) | | `logit_bias` (`{"TOKEN": VALUE}`) | `bias` | yes |
| No stop token before N tokens | | `min_tokens` | `minTokens` | yes |
| Only JSON, or JSON of a schema | | `response_format` | `grammar` | yes |
| Log probabilities and the most likely tokens | | `logprobs`, `top_logprobs` | `logprobs` | yes |
| More tokens that end the answer | | `stop_token_ids` | `stopTokens` | no |
| The model's own stop tokens do not end the answer (a fixed length) | | `ignore_eos` | `ignoreEos` | no |
| A program's own rule over the logits | | | `shape` | yes |

- `top_k`, `min_p`, `repetition_penalty`, `min_tokens`, `stop_token_ids` and `ignore_eos` are vLLM's fields; the
  rest are OpenAI's. A value out of range (`min_p` above 1, a `logit_bias` token the model lacks) gets status 400.
- Temperature, top-k, top-p and min-p choose from the model's distribution, so a request with only these still drafts.
  A request that shapes the logits makes no drafts: each step reads its token alone.

## Quality

What the fast settings cost, against the same NVFP4 weights run exactly (`--exact`, 16-bit keys and values).
Qwen3.8-27B NVFP4, an RTX 5090, an English text, 2026-11-08 (`run.giz quality`; KL of the next-token distribution,
top 1 agreement, mean -log p of the text's next tokens):

| Setting | Tokens | KL | Top 1 | -log p (exact → setting) |
| --- | --- | --- | --- | --- |
| Keys and values in 8 bits, exact linears | 4K | 0.00003 | 99% | 1.974 → 1.974 |
| The tuned KV plan per layer, exact linears | 4K | 0.0030 | 97% | 1.974 → 1.978 |
| NVFP4 inputs to the linears (W4A4), 16-bit KV | 4K | 0.0319 | 92% | 1.974 → 1.989 |
| NVFP4 inputs to the linears (W4A4), 16-bit KV | 16K | 0.0339 | 91% | 1.569 → 1.583 |
| W4A4, the KV plan (the default `--kv`), 2048-row steps | 4K | 0.0340 | 88% | 1.974 → 2.001 |
| W4A4, the KV plan, 2048-row steps | 16K | 0.0340 | 91% | 1.569 → 1.547 |
| W4A4, the KV plan, 16-row steps (as decode) | 4K | 0.0272 | 92% | 1.979 → 1.965 |
| W4A4, `--kv nv4`, 2048-row steps (the fastest on sm_120) | 4K | 0.0393 | 90% | 1.974 → 1.962 |
| W4A4, `--kv nv4`, 2048-row steps | 16K | 0.0406 | 91% | 1.569 → 1.577 |

- Nearly all of the error is the NVFP4 rounding of the linears' inputs, the same as any W4A4 NVFP4 server. The KV
  plan adds KL 0.002 at 4K and nothing measurable at 16K; `nv4` keys and values add about 0.007. Windows of 127 rows
  move by about 0.003 between runs.
- A fact at 10%, 50% and 90% of 32K tokens and at 50% of 100K was answered right with 16-bit keys and values, with
  the plan and with `nv4` (12 of 12).

## Measure

`bench/ai/llm/run.giz` measures a model folder. Build it once and run it on the GPU host
([bench/ai/llm](https://github.com/gizai/gizlang/blob/main/bench/ai/llm/README.md) has the published results):

```sh
bin/gizlang build bench/ai/llm/run.giz -o llmrun
./llmrun speed /models/q27-nv4 1 8                    # prefill of 1K tokens, decode of 1 and 8 sequences
./llmrun speed /models/q27-nv4 1 8 --drafts 0 --drafts 32   # a sweep: each, then a table
./llmrun chat /models/q27-nv4 "Explain rotary embeddings." 256 --profile
./llmrun prefill /models/q27-nv4 2048 4 32768        # a 2048-row step after 32K positions, by kernel
./llmrun quality /models/q27-nv4 text.txt 4096 --kv 8 --exact
./llmrun needle /models/q27-nv4 text.txt 32768 50    # a fact at 50% of 32K tokens, then the answer
./llmrun load http://127.0.0.1:8000/v1 8 64          # 8 chats at once against a server
```

| Command | What it prints |
| --- | --- |
| `check FOLDER [TOKENS]` | The GPU's logits against the CPU's after a prompt. |
| `chat FOLDER TEXT\|@FILE [TOKENS [TEMPERATURE]]` | A reply, its speed and the drafts kept; `--profile`: its kernels a step. |
| `speed FOLDER [BATCH...]` | Prefill tokens/s of a 1024-token prompt, then decode tokens/s per batch size. |
| `profile FOLDER [BATCH [STEPS [AFTER]]]` | The GPU time of decode steps by kernel. |
| `prefill FOLDER ROWS [STEPS [AFTER]]` | The GPU time of a prompt step of ROWS rows after AFTER positions, by kernel. |
| `quality FOLDER FILE [TOKENS]` | The engine of the flags against the exact one: KL, top 1, -log p, by window and in all. `--rows` reads that many rows a step. |
| `needle FOLDER FILE TOKENS [DEPTH]` | KV positions, prefill and decode speed at that length, and the answer about a fact at DEPTH %. |
| `load URL CLIENTS REQUESTS [TOKENS [PROMPT]]` | Tokens a second, the median first-token time and the share of later prompts read from the server's cache. |

- Every command takes the engine flags. `speed` turns prefix reuse off, so a repeated prompt is computed whole.
- Every run keeps the kernel tunings it measured in `FOLDER/tune`; later runs and the server take them.
- `GIZLANG_LLM_PROBE` set prints each step's rows, drafts, tokens and time, and each stage's hop
  ([settings](../reference/settings.md)).

## Use a model in a program

`gizai/ai` gives a program the models by name. `Ai.models` opens the models folder, `Ai.text` loads a chat model,
`Ai.ready` makes its engine warm, and `Ai.reply` gives the answer piece by piece:

```typescript gizlang
import {Message} from "gizlang/openai";
import * as Ai from "gizai/ai";
import {EngineOptions} from "gizai/llm/engine";

function main(): void {
    const hub = Ai.models()!;
    const options: EngineOptions = { context: 8192, kv: "8" };
    const m = Ai.text(hub, "qwen3.5-4b", true, options)!;
    let e = Ai.ready(m, options)!;
    const asked: Vec<Message> = [{ role: "user", content: "Name three rivers." }];
    const answer = Ai.reply(e, asked, Ai.ask(), (p: Ai.Piece): bool => true)!;
    console.log(answer.text);
}
```

- Pass the same `EngineOptions` to `Ai.text` (it loads only the named draft source's weights) and to `Ai.ready`.
- `Ai.Ask` holds what a reply may do: its most tokens, stop strings, reasoning effort, tools (OpenAI's tool objects)
  and `sampling`. A `Piece` holds the text, the reasoning, tool calls, log probabilities and why it ended.
- `Ai.encoder(hub, name)` loads an `embed` or `rank` model; `gizai/serve`'s `Serve.serve(listener, models,
  images, encoders, key)` is the [server](#serve-the-openai-api), with the API key `key` ("" for none).
- `gizdesk/studio` (GizDesk's, AGPL-3.0) connects local models and model APIs to agents ([agent guide](agent.md#choose-a-model-and-connect-tools)).

`gizai/llm` is the layer below: convert, load, and run an engine of many sequences:

```typescript gizlang
import {convert} from "gizai/llm/convert";
import {load} from "gizai/llm/model";
import {greedy} from "gizai/llm/sample";

function main(): void {
    convert("Qwen3-8B", "qwen3-8b-q4", { layers: 4, head: 6 })!;
    const m = load("qwen3-8b-q4")!;
    const prompt = m.tokenizer.tokens(m.chatPrompt([{ role: "user", content: "Name three rivers." }], false));
    let e = m.engine({ context: prompt.length + 200 })!;
    const out = e.generate([{ prompt: prompt, limit: 200, sampling: greedy() }])!;
    console.log(m.tokenizer.decode(out[0], true));
}
```

- `load(folder, onGpu, layers, options)` loads on the best device; with `layers`, only the first layers, and
  `Engine.hidden` gives each token's hidden state after them (a text encoder of an image model).
- One `Engine` serves many sequences: pass more requests to `generate`, or run them in turn with `Engine.add`,
  `advance` and `cancel` (a request joins while others run, and its tokens come step by step).
  `Engine.add(request, mark, head)` keeps a chat's start and a shared head ([prefix reuse](#prefix-reuse-and-shared-heads)).
- A request's `Sampling` picks its tokens: `greedy()`, or `temperature` with `topK`, `topP`, `minP` and `seed`. Its
  optional fields are the rest of the [sampling table](#sampling). A `grammar` of `gizai/llm/grammar` is
  `anyObject()` or `fromSchema(schema)`, from the token after `after` on; with `logprobs` each `Progress` has the
  tokens' log probabilities and that many of the most likely.
- `shape` is your own rule: a function that changes the logits of the whole vocabulary, given the prompt and the
  answer so far. It runs after the other controls, before the token is picked:

```typescript gizlang
import {Model} from "gizai/llm/state";
import {Sampling, greedy} from "gizai/llm/sample";

/** Sampling that gives only digits and the stop tokens: an answer that is a number. */
export function digitsOnly(m: Model): Sampling {
    let allowed: Vec<bool> = new Array(m.arch.vocab).fill(false);
    const pieces = m.tokenizer.tokenBytes();
    for (const t of range(0, pieces.length)) {
        let digits = pieces[t].length > 0;
        for (const b of pieces[t]) { if (b < 48u || b > 57u) { digits = false; } }
        allowed[t] = digits;
    }
    for (const t of m.stops) { allowed[t] = true; }
    let s = greedy();
    s.shape = (logits: mut Vec<f32>, tokens: Vec<i64>): void => {
        for (const t of range(0, logits.length)) { if (!allowed[t]) { logits[t] = -1.0e30; } }
    };
    return s;
}
```

- Call `Engine.warm` (or use `Ai.ready`) before requests: the first run of each step shape compiles, tunes and
  records its kernels, and `Model.saveTunings` keeps the tunings for the next start.
- `engineFlags(args)` reads `EngineOptions` from command-line flags, and `ENGINE_FLAGS` lists them.

## Extend

Each kind of change has one place:

| To add | Change | Then |
| --- | --- | --- |
| A sampling rule of your own | `Sampling.shape` in your program ([above](#use-a-model-in-a-program)) | Nothing else: it runs on every engine. |
| A model to the catalog, by name | An entry of a family in `ai/models/NAME/family.giz` ([gizai README](https://github.com/gizai/gizlang/blob/main/ai/README.md#add-a-model-family)) | `giz ai registry`; `models`, `pull`, `chat` and `serve` find it. |
| An architecture (a new `model_type`) | A recipe in `gizai/llm/recipes`: it reads `config.json` into the features of an `Arch` (`gizai/llm/shape`) | A tiny model of it in `tests/ai/llm/tiny.giz`, checked against the f32 reference. |
| A weight format | `gizai/quant` (`parseFormat`) and its products in `gizai/llm/gemm` | `--weights NAME` of `convert` takes it. |
| A KV format | `kvFormats` in `gizai/llm/kvformat` and its attention in `gizai/llm/kv` | `--kv` and `kvcosts` take it. |
| An engine option | A field of `EngineOptions` and its flag in `ENGINE_FLAGS` (`gizai/llm/engine`) | `chat`, `serve`, the benchmark and programs read it; list it in `giz help ai` (`tools/cli/commands.giz`). |
| A draft source | `DRAFT_SOURCES` in `gizai/llm/engine`, its weights in `LlmModel.load` and its drafts in `gizai/llm/generate` | `--draft NAME`; the `quality` row of the benchmark shows that it keeps the distribution. |

A model whose architecture a recipe reads needs no code: `giz ai convert` and `giz ai chat /path/to/folder` run it.

## Where the files are

The models folder is `$GIZLANG_CACHE/huggingface`, else `~/.cache/gizlang/huggingface`.

| Path | What it holds |
| --- | --- |
| `OWNER/NAME/` | The downloaded files: JSON, safetensors, tokenizer and template files. Never code. |
| `.pins/OWNER/NAME.pin` | The commit of the downloads and the SHA-256 of each file. A later download of the same model takes the same commit. |
| `.gizlang-llm/OWNER/NAME/ID/qL-H/` | The model converted with L bits for its layers and H for its output layer (`qL-H-sS` with sensitive bits). ID names the pinned commit. `conversion.json` holds the key that says whether the folder is complete and current ([the conversion cache](#the-conversion-cache)). |
| `.gizlang-llm/files/ID/NAME.gguf.qL-H/` | A GGUF file converted on its first use. ID names its path, size and time of change. |

A model folder holds:

| File | What it holds |
| --- | --- |
| `model.safetensors` | The linears in their formats, the norms and small tensors in F32, the embedding as its source held it. Its metadata names the format (`gizlang.llm.2`) and each linear's `QuantSpec`. |
| `config.json`, `tokenizer.json`, `generation_config.json`, chat template | The source's files: architecture, tokens, sampling, chat format. |
| `visual.safetensors` | The vision tower, where the model has one. |
| `drafter/` | A converted block drafter. |
| `kv-costs` | Each attention layer's measured cost in each KV format (`giz ai kvcosts`). |
| `tune/` | The kernel tunings measured on this machine. |

- A download is checked against the SHA-256 that Hugging Face states, and a stopped download leaves no file.
- To remove a model, remove its folders. Its next use downloads it again.

## How it works

A step reads rows of many sequences at once: prompt chunks and decode rows with their drafts. Each layer runs as GPU
kernels on all the rows; attention reads each sequence's paged keys and values. The engine then samples or checks
each sequence's tokens, keeps the drafts it agrees with, and plans the next step.

| Module | Owns |
| --- | --- |
| `gizai/llm/recipes`, `arch`, `shape` | Each family's `config.json` read into an `Arch`: layers, heads, norms, windows, rotary settings, tensor names. |
| `gizai/llm/plan` | `WeightView` (a source tensor), the planner (each linear's `QuantSpec` and whether its values are kept), the kernel family a device takes. |
| `gizai/llm/convert`, `visionconvert` | Conversion to the folder format as the plan says: rotation, codebooks, mixed precision, LoRA, drafters, vision towers. |
| `gizai/llm/model` | Loading a folder on one or more devices by its specs, the conversion cache, the chat template, tunings and KV costs. |
| `gizai/llm/state` | The records: `Model`, `Engine`, requests, drafts, slots. It imports no other part. |
| `gizai/llm/engine` | `EngineOptions`, `Model.engine`, a step, the KV choice and `kvMeasured`. |
| `gizai/llm/generate` | Requests in progress, the prefix cache, draft counts and recall, warming. |
| `gizai/llm/gpu`, `gemm` | A step's buffers and launches on a GPU; the products of each weight format, tuned per shape. |
| `gizai/llm/kv`, `kvformat` | Paged attention over coded keys and values; the page layout and the per-layer choice. |
| `gizai/llm/delta` | The gated delta rule (Qwen 3.5 and 3.8 recurrent layers) and its replay of kept drafts. |
| `gizai/llm/block` | The block drafter on a GPU. |
| `gizai/llm/sample`, `grammar` | Sampling, logit shaping, the check of drafts, JSON-schema grammars. |
| `gizai/llm/encoders` | BERT-family encoders. |
| `gizai/llm/vision`, `flash`, `npu` | Vision towers, Qwen3.8 Flash, the NPU path. |
| `gizai/llm/host` | The CPU path of every operation: the reference of the GPU path. |

The products, attention and normalization kernels are written in Gizlang with `gizhw/kernel` and compiled for each
device at the first run of a shape ([compute internals](../internals/compute.md)). A change to the engine keeps the
output distribution: measure it with `quality` and `needle` against the previous build on the same base.
