# gizml:llm

language models, fast on any hardware.

A model family is a recipe: its config.json read as an Arch, and the names of its tensors. Every family runs on the same operations, so a new family is one recipe. `convert` turns a Hugging Face folder into this module's format once: each linear is stored in a format (Formats), after a Hadamard rotation of each block of 128 inputs, in the order the GPU reads it. `load` opens a converted model on the best device it finds, and its Engine runs steps of many sequences at once.

```gizlang
import * as Llm from "gizml:llm";
```

## Types

### `Message`

```gizlang
type Message = { role: string; content: string; reasoning?: string; calls?: Vec<Hub.Call>; }
```

A message of a chat: its role ("system", "user", "assistant" or "tool") and text; an assistant's reasoning before its
answer and the tools it called.

- `role`: `string`
- `content`: `string`
- `reasoning`: `_gizlang_sdk_core.Option<string>`
- `calls`: `_gizlang_sdk_core.Option<Vec<gizlang_hub.Call>>`

### `Arch`

```gizlang
type Arch = { family: string; vocab: i64; dim: i64; ffn: i64; layers: i64; kinds: Vec<i64>; eps: f64; heads: i64; kv: i64; headDim: i64; rotary: i64; rope: Vec<f64>; qkNorm: bool; gated: bool; bias: bool; kHeads: i64; vHeads: i64; kDim: i64; vDim: i64; conv: i64; tied: bool; mtp: i64; context: i64; prefix: string; sections: Vec<i64>; hc: i64; hcRank: i64; experts: i64; topK: i64; ple: i64; eos: i64; sigmoidGate: bool; }
```

What a model computes, as its family's recipe reads config.json. Attention: `heads` query heads of `headDim` over
`kv` key-value heads, rotary positions on the first `rotary` dimensions of a head (`rope` their inverse
frequencies), RMS norms of q and k (`qkNorm`), a sigmoid gate on the output (`gated`), biases on q, k and v.
Recurrent layers: `kHeads` query-key heads of `kDim`, `vHeads` value heads of `vDim`, a causal convolution of
`conv` taps. `mtp` layers predict a further token.

- `family`: `string`
- `vocab`: `i64`
- `dim`: `i64`
- `ffn`: `i64`
- `layers`: `i64`
- `kinds`: `Vec<i64>`
- `eps`: `f64`
- `heads`: `i64`
- `kv`: `i64`
- `headDim`: `i64`
- `rotary`: `i64`
- `rope`: `Vec<f64>`
- `qkNorm`: `bool`
- `gated`: `bool`
- `bias`: `bool`
- `kHeads`: `i64`
- `vHeads`: `i64`
- `kDim`: `i64`
- `vDim`: `i64`
- `conv`: `i64`
- `tied`: `bool`
- `mtp`: `i64`
- `context`: `i64`
- `prefix`: `string`
- `sections`: `Vec<i64>`
- `hc`: `i64`
- `hcRank`: `i64`
- `experts`: `i64`
- `topK`: `i64`
- `ple`: `i64`
- `eos`: `i64`
- `sigmoidGate`: `bool`

### `Bits`

```gizlang
type Bits = { layers: i64; head: i64; }
```

The format of each kind of linear (Formats): `layers` for the layers, `head` for the output head.

- `layers`: `i64`
- `head`: `i64`

### `Model`

```gizlang
type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec<i64>; template: string; system: string; think: bool; calls: bool; efforts: Vec<string>; sampling: Sampling; embed: Stored; layers: Vec<Layer>; norm: i64; head: i64; mtp: Option<Mtp>; mixer: Vec<i64>; linears: Vec<Linear>; small: Vec<Vec<f32>>; gpu: Option<Gpu>; device: Vec<DeviceLinear>; deviceSmall: Vec<Buffer>; signs: Vec<Buffer>; files: FlashFiles; vision?: Vision; stages: Vec<Model>; npu?: NpuLlm.Model; }
```

A converted model, on the GPU that gizhw:gpu opens (GIZLANG_DEVICE) when there is one (`device`), else on the
host (`linears`); the small tensors are on both. The embedding stays in the file: a step reads its tokens' rows.
`template` is the chat layout ("chatml", "llama3" or ""), `system` the system turn it writes when a chat has none,
`think` whether its assistant turn opens with a thinking
block, `calls` whether it writes tool calls as <tool_call><function=NAME><parameter=P>...; `efforts` the instructions
its template gives a thinking turn at low, medium and high reasoning effort ("": none); `stops` are the tokens that end
a generation and `sampling` the model's own settings. `mtp` is the MTP layer on a GPU when the model has one, `vision`
its vision tower when its folder has one (convertVision), `stages` the models of its later layers on other GPUs
(loadAcross).

- `arch`: `M23.Arch`
- `folder`: `string`
- `tokenizer`: `M34.Tokenizer`
- `stops`: `Vec<i64>`
- `template`: `string`
- `system`: `string`
- `think`: `bool`
- `calls`: `bool`
- `efforts`: `Vec<string>`
- `sampling`: `M23.Sampling`
- `embed`: `gizlang_safetensors.Stored`
- `layers`: `Vec<M23.Layer>`
- `norm`: `i64`
- `head`: `i64`
- `mtp`: `_gizlang_sdk_core.Option<M23.Mtp>`
- `mixer`: `Vec<i64>`
- `linears`: `Vec<M23.Linear>`
- `small`: `Vec<Vec<f32>>`
- `gpu`: `_gizlang_sdk_core.Option<M10.Gpu>`
- `device`: `Vec<M23.DeviceLinear>`
- `deviceSmall`: `Vec<M10.Buffer>`
- `signs`: `Vec<M10.Buffer>`
- `files`: `M23.FlashFiles`
- `vision`: `_gizlang_sdk_core.Option<M23.Vision>`
- `stages`: `Vec<M23.Model>`
- `npu`: `_gizlang_sdk_core.Option<M28.Model>`

### `ChatTokens`

```gizlang
type ChatTokens = { tokens: Vec<i64>; mark: i64; head: i64; }
```

A chat's prompt as tokens (Model.chatPrompt), the mark of Engine.add (where its messages end, before the assistant's
turn starts: the chat's next request starts with these messages, so its slot reads only the rest) and its head (where
the system turn with the tools ends: the chats with the same system turn share its keys and values).

- `tokens`: `Vec<i64>`
- `mark`: `i64`
- `head`: `i64`

### `Engine`

```gizlang
type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec<i64>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; convs: Vec<Vec<f32>>; states: Vec<Vec<f32>>; replay: Vec<Vec<f32>>; hidden: Vec<f32>; work: Vec<Buffer>; pages: Vec<Vec<i64>>; table: Vec<f32>; free: Vec<i64>; maxPages: i64; seen: Vec<string>; shapes: Vec<string>; recordings: Vec<G.Recording>; depth: i64; pending: Vec<i64>; drafts: i64; kept: i64; draftBlocks: Vec<i64>; blockEntry: Vec<i64>; nextEntry: i64; paces: Vec<f64>; stepNs: f64; rowNs: f64; kv: KvFormat; waiting: Vec<Turn>; turns: Vec<Turn>; next: Vec<i64>; guesses: Vec<Vec<Guess>>; held: Vec<Vec<i64>>; freed: Vec<i64>; saved: Vec<Saved>; pieces: Vec<bytes>; shared: Vec<Shared>; borrowed: Vec<i64>; stored: Vec<Prefix>; ids: i64; steps: i64; reuse: bool; flash: Flash; stages: Vec<Engine>; }
```

The engine of a model: `slots` sequences of at most `context` positions, steps of at most `rows` rows.

- `model`: `M23.Model`
- `slots`: `i64`
- `context`: `i64`
- `rows`: `i64`
- `lengths`: `Vec<i64>`
- `keys`: `Vec<Vec<f32>>`
- `values`: `Vec<Vec<f32>>`
- `convs`: `Vec<Vec<f32>>`
- `states`: `Vec<Vec<f32>>`
- `replay`: `Vec<Vec<f32>>`
- `hidden`: `Vec<f32>`
- `work`: `Vec<M10.Buffer>`
- `pages`: `Vec<Vec<i64>>`
- `table`: `Vec<f32>`
- `free`: `Vec<i64>`
- `maxPages`: `i64`
- `seen`: `Vec<string>`
- `shapes`: `Vec<string>`
- `recordings`: `Vec<M10.Recording>`
- `depth`: `i64`
- `pending`: `Vec<i64>`
- `drafts`: `i64`
- `kept`: `i64`
- `draftBlocks`: `Vec<i64>`
- `blockEntry`: `Vec<i64>`
- `nextEntry`: `i64`
- `paces`: `Vec<f64>`
- `stepNs`: `f64`
- `rowNs`: `f64`
- `kv`: `M23.KvFormat`
- `waiting`: `Vec<M23.Turn>`
- `turns`: `Vec<M23.Turn>`
- `next`: `Vec<i64>`
- `guesses`: `Vec<Vec<M23.Guess>>`
- `held`: `Vec<Vec<i64>>`
- `freed`: `Vec<i64>`
- `saved`: `Vec<M23.Saved>`
- `pieces`: `Vec<bytes>` — The bytes of each token (Tokenizer.tokenBytes), read at the first request with a grammar.
- `shared`: `Vec<M23.Shared>`
- `borrowed`: `Vec<i64>`
- `stored`: `Vec<M23.Prefix>`
- `ids`: `i64`
- `steps`: `i64`
- `reuse`: `bool`
- `flash`: `M23.Flash`
- `stages`: `Vec<M23.Engine>`

### `Visual`

```gizlang
type Visual = { rows: Vec<i64>; embeds: Vec<f32>; deep: Vec<Vec<f32>>; grid: Vec<i64>; }
```

Rows of a step that hold no token's embedding (Qwen3-VL's pictures): `rows` of the step, their embeddings (dim
floats a row, in turn), `deep` the features added to them after each of the first layers (Qwen3-VL's deepstack, rows
as embeds), and `grid` every row's three rotary positions (time, height, width; a text row's three are its
position: see ropeTable).

- `rows`: `Vec<i64>`
- `embeds`: `Vec<f32>`
- `deep`: `Vec<Vec<f32>>`
- `grid`: `Vec<i64>`

### `KvFormat`

```gizlang
type KvFormat = { keys: i64; values: i64; }
```

How an engine keeps keys and values: the bits of a key and of a value. 16: F16 values; 8, 4 or 2: codes of the
head rotated by blocks of kvBlock, at 2^bits even steps (kvStore).

- `keys`: `i64`
- `values`: `i64`

### `ExpertCache`

```gizlang
type ExpertCache = { slots: i64; hits: i64; misses: i64; readMs: f64; }
```

The expert cache of an engine: its slots, the experts found in it and read into it, and the ms the reads took.

- `slots`: `i64`
- `hits`: `i64`
- `misses`: `i64`
- `readMs`: `f64`

### `Guess`

```gizlang
type Guess = { token: i64; q: Dist; }
```

A draft per sequence: its token and the distribution it was drawn from (all on the token when it is the most likely).

- `token`: `i64`
- `q`: `M23.Dist`

### `Vision`

```gizlang
type Vision = { named: Map<string, Tensor>; gpu: Option<Gpu>; deep: Vec<i64>; depth: i64; width: i64; heads: i64; side: i64; }
```

A Qwen vision tower (loadVision): its weights by name where it runs (linears BF16, the rest f32; the patch embedding
f32 [width, 768], the position table on the host), its blocks, width and heads, the side of its position table, and
the blocks after which a deepstack merger reads the stream.

- `named`: `_gizlang_sdk_collections.Map<string, M32.Tensor>`
- `gpu`: `_gizlang_sdk_core.Option<M10.Gpu>`
- `deep`: `Vec<i64>`
- `depth`: `i64`
- `width`: `i64`
- `heads`: `i64`
- `side`: `i64`

### `Stream`

```gizlang
type Stream = { ids: Vec<i64>; tags: Vec<i64>; grid: Vec<i64>; visual: Visual; shift: i64; }
```

A prompt as a language model with a vision tower reads it: its token ids, each with three rotary positions (`grid`:
its index plus `shift`, which a picture's rows lower, as their grid is smaller than their count) and a `tag` of the
caller's, and the pictures' rows (`visual`: the tower's rows at their places).

- `ids`: `Vec<i64>`
- `tags`: `Vec<i64>`
- `grid`: `Vec<i64>`
- `visual`: `M23.Visual`
- `shift`: `i64`

### `Shown`

```gizlang
type Shown = { picture: Tensor; times: Vec<f64>; }
```

What a chat shows (Vision.chat): a picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32), or a
video's frames ([n, 3, h, w]) with each frame's second in `times`.

- `picture`: `M32.Tensor`
- `times`: `Vec<f64>`

### `Seen`

```gizlang
type Seen = { chat: ChatTokens; visual: Visual; }
```

A chat's prompt with what it shows (Vision.chat): its tokens, mark and head, and the Visual of the whole prompt
(Request.visual).

- `chat`: `M23.ChatTokens`
- `visual`: `M23.Visual`

### `Sampling`

```gizlang
type Sampling = { temperature: f64; topK: i64; topP: f64; seed: u64; presence?: f64; frequency?: f64; repetition?: f64; grammar?: Grammar; after?: i64; logprobs?: i64; }
```

How tokens are picked: temperature 0 the most likely, else one of the topK most likely (0: any) within topP of
their probability, at temperature. Before that the logits are shaped, as OpenAI and vLLM shape them: less by
`presence` for a token the answer has and by `frequency` for each time it has it; divided by `repetition` (above 0,
times it below 0) for a token the prompt or the answer has. `grammar`: only the tokens it allows (gizml:grammar),
from the token after `after` on (a reasoning model's end of thinking; none: from the first); a stop token only where
the text is whole. `logprobs`: each token's log probability and the most likely ones (that many) are given, of the
model's logits before shaping. A shaped request makes no drafts.

- `temperature`: `f64`
- `topK`: `i64`
- `topP`: `f64`
- `seed`: `u64`
- `presence`: `_gizlang_sdk_core.Option<f64>`
- `frequency`: `_gizlang_sdk_core.Option<f64>`
- `repetition`: `_gizlang_sdk_core.Option<f64>`
- `grammar`: `_gizlang_sdk_core.Option<M22.Grammar>`
- `after`: `_gizlang_sdk_core.Option<i64>`
- `logprobs`: `_gizlang_sdk_core.Option<i64>`

### `Dist`

```gizlang
type Dist = { ids: Vec<i64>; weights: Vec<f64>; }
```

What a sampled token is drawn from: tokens, most likely first, and their weights (a token's probability is its
weight over their sum).

- `ids`: `Vec<i64>`
- `weights`: `Vec<f64>`

### `Request`

```gizlang
type Request = { prompt: Vec<i64>; limit: i64; sampling: Sampling; visual?: Visual; }
```

A prompt to continue: at most `limit` new tokens. With `visual` some of its rows are pictures' (Vision.chat): their
embeddings replace their tokens', and its rows take their three rotary positions from visual.grid (three for each of
its first rows, the last a text row's), the rows after them their position plus that last row's shift.

- `prompt`: `Vec<i64>`
- `limit`: `i64`
- `sampling`: `M23.Sampling`
- `visual`: `_gizlang_sdk_core.Option<M23.Visual>`

### `Logprob`

```gizlang
type Logprob = { token: i64; logprob: f64; top: Vec<i64>; tops: Vec<f64>; }
```

A token's log probability and the `top` most likely tokens with theirs, of the model's logits.

- `token`: `i64`
- `logprob`: `f64`
- `top`: `Vec<i64>`
- `tops`: `Vec<f64>`

### `Progress`

```gizlang
type Progress = { id: i64; tokens: Vec<i64>; cached: i64; end: string; logprobs: Vec<Logprob>; }
```

What a step gave a request (Engine.advance): its new tokens (without a stop token), the tokens at the start of its
prompt that its slot held already (not read again), and why it ended: "stop" (a stop token), "length" (its limit, the
context's end, or a KV cache too small for it alone) or "" (it goes on).

- `id`: `i64`
- `tokens`: `Vec<i64>`
- `cached`: `i64`
- `end`: `string`
- `logprobs`: `Vec<M23.Logprob>`

### `Turn`

```gizlang
type Turn = { id: i64; request: Request; random: Vec<u64>; made: i64; mark: i64; head: i64; cached: i64; reading?: Recognizer; }
```

A request in an engine's turn: its id, its random state, the tokens it gave, the prompt position after which its
slot saves the recurrent states (Engine.add), the position after which its prompt's start
is shared (Engine.add; 0: none), the prompt tokens its slot held already and its grammar's recognizer while the
grammar holds (Sampling).

- `id`: `i64`
- `request`: `M23.Request`
- `random`: `Vec<u64>`
- `made`: `i64`
- `mark`: `i64`
- `head`: `i64`
- `cached`: `i64`
- `reading`: `_gizlang_sdk_core.Option<M22.Recognizer>`

### `Saved`

```gizlang
type Saved = { at: i64; conv: Vec<f32>; state: Vec<f32>; }
```

A slot's recurrent states after its first `at` tokens (0: none): the convolutions' last inputs and the delta rules'
states, on the host (from a GPU the states' BF16, two to a float).

- `at`: `i64`
- `conv`: `Vec<f32>`
- `state`: `Vec<f32>`

### `Prefix`

```gizlang
type Prefix = { tokens: Vec<i64>; saved: Saved; pages: Vec<bytes>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Prefix>; }
```

The start of a chat that left its slot for another (see admit): its tokens, the recurrent states after them, and
their keys and values (on a GPU the bytes of their KV pages, on the host per pool layer); `used` is the step it was
stored at. A later prompt that starts with the tokens takes them back into a slot and reads only the rest. On a
model in stages, `stages` holds each later stage's pages and states (Engine.stages).

- `tokens`: `Vec<i64>`
- `saved`: `M23.Saved`
- `pages`: `Vec<bytes>`
- `keys`: `Vec<Vec<f32>>`
- `values`: `Vec<Vec<f32>>`
- `used`: `i64`
- `stages`: `Vec<M23.Prefix>`

### `Shared`

```gizlang
type Shared = { tokens: Vec<i64>; saved: Saved; pages: Vec<i64>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Shared>; }
```

A start that many prompts share (as the system turn and tools of an agent's chats, Model.chatTokens): its tokens
(whole KV pages), the recurrent states after them, and their keys and values: on a GPU the KV pages of the pool that
hold them, which the slots that start with them read in place (Engine.borrowed), on the host a copy per pool layer;
`used` is the step a slot last took it. On a model in stages, `stages` holds each later stage's pages and states.

- `tokens`: `Vec<i64>`
- `saved`: `M23.Saved`
- `pages`: `Vec<i64>`
- `keys`: `Vec<Vec<f32>>`
- `values`: `Vec<Vec<f32>>`
- `used`: `i64`
- `stages`: `Vec<M23.Shared>`

### `BertConfig`

```gizlang
type BertConfig = { kind: string; layers: i64; heads: i64; dim: i64; eps: f64; act: string; positions: i64; offset: i64; }
```

A BERT-family encoder as a Hugging Face config.json gives it: BERT, RoBERTa and XLM-RoBERTa have one layout and
differ in where positions start (`offset`: 0 for BERT, pad_token_id + 1 for RoBERTa). act is hidden_act: "gelu"
(the erf form), "relu" or a tanh form ("gelu_new", "gelu_pytorch_tanh").

- `kind`: `string`
- `layers`: `i64`
- `heads`: `i64`
- `dim`: `i64`
- `eps`: `f64`
- `act`: `string`
- `positions`: `i64`
- `offset`: `i64`

### `ModelFiles`

```gizlang
type ModelFiles = (path: string) => Result<Option<bytes>, string>
```

Reads the file at a path of a model's repository: none when the model has no such file.

### `Encoder`

```gizlang
type Encoder = { config: BertConfig; weights: Map<string, Tensor>; tokenizer: Tokenizer; pooling: string; normalize: bool; limit: i64; }
```

A sentence encoder as sentence-transformers 3 builds it from a model's files: an encoder, then the pooling of its
token states ("mean", "cls", "max", "mean_sqrt_len" or "last"), then L2 normalization when the model has that
module.

- `config`: `M23.BertConfig`
- `weights`: `_gizlang_sdk_collections.Map<string, M32.Tensor>`
- `tokenizer`: `M34.Tokenizer`
- `pooling`: `string`
- `normalize`: `bool`
- `limit`: `i64`

### `CrossEncoder`

```gizlang
type CrossEncoder = { config: BertConfig; weights: Map<string, Tensor>; tokenizer: Tokenizer; head: string; labels: i64; activation: string; limit: i64; }
```

A cross-encoder as sentence-transformers' CrossEncoder builds it: a sequence classification model that reads a
pair of texts (a query and a document) and gives the scores of its labels. `head` is "bert" (BertForSequence-
Classification: the pooler's tanh on the first token, then the classifier) or "roberta" (RobertaFor- and
XLMRobertaForSequenceClassification: classifier.dense, tanh, classifier.out_proj). `activation` is the function on
the scores: "sigmoid" (the default with one label), "identity" (the default with more) or "tanh".

- `config`: `M23.BertConfig`
- `weights`: `_gizlang_sdk_collections.Map<string, M32.Tensor>`
- `tokenizer`: `M34.Tokenizer`
- `head`: `string`
- `labels`: `i64`
- `activation`: `string`
- `limit`: `i64`

### `Ranked`

```gizlang
type Ranked = { corpus: i64; score: f64; }
```

A document of a ranking: its index in the documents and its score.

- `corpus`: `i64`
- `score`: `f64`

## Functions

### `archOf`

```gizlang
function archOf(config: JsonValue): Arch throws string
```

The Arch of a model by its config.json; an error names what this module does not run yet.

Effects: fail.

### `convert`

```gizlang
function convert(sys: System, from: string, to: string, bits: Bits, layers: i64 = 0, lora: string = ""): i64 throws string
```

Converts the Hugging Face model in folder `from` (config.json, safetensors, tokenizer), or the GGUF file `from`
(gizlang:gguf: Llama, Qwen2 and Qwen3, its block types read as they are; the embedding keeps its block type), to this module's format in
folder `to`: model.safetensors with every linear quantized (see Quantization), the norms and small tensors as f32
and the embedding as its source holds it, with the source's config, tokenizer and generation files. With `layers`
less than the model's, only its first `layers` layers, without the final norm, the head and the MTP layer (a
source cut after them, as MiniMax H3's text encoder, holds no more): load it with the same `layers`. A cut Qwen3.8 Flash keeps its mixer and head (a test of its first layers). A
vision tower (model.visual.*) goes to visual.safetensors (convertVision). Returns the bytes written. With `lora`, the PEFT LoRA adapter in that folder (adapter_config.json,
adapter_model.safetensors) is merged into the linears it adapts before they are quantized: W + alpha / r B A
(alpha / sqrt(r) with rsLoRA).

Effects: fail, read, write, console, io.

### `load`

```gizlang
function load(sys: System, folder: string, onGpu: bool = true, layers: i64 = 0): Model throws string
```

The converted model in `folder`, on the GPU when `onGpu` and there is one, and across the driver's other GPUs too
when its weights do not leave SPLIT_ROOM free on that one (loadAcross). With `layers` less than the model's, only its
first `layers` layers, without the final norm, the head and the MTP layer: an engine of it gives hidden states
(Engine.hidden), not logits. A gizml:npullm folder (model.q4nx) runs on this host's NPU (Model.npu).

Effects: fail, read, write, console, io.

### `loadAcross`

```gizlang
function loadAcross(sys: System, folder: string, gpus: Vec<Gpu>, counts: Vec<i64> = []): Model throws string
```

The converted model in `folder` across `gpus` (Gpu.others), in stages of consecutive layers: the first GPU holds the
first stage (and the vision tower), the last the last stage with the final norm and the head (Model.stages). A
stage's layer count is counts[i], by default in proportion to its GPU's free memory. A step runs the stages in turn,
each on its rows' hidden states after the one before (Engine). Without the MTP layer, and not for Qwen3.8 Flash
yet. The first GPU is current at the end.

Effects: fail, read, write, console, io.

### `loadOn`

```gizlang
function loadOn(sys: System, folder: string, gpu: Option<Gpu>, layers: i64 = 0, from: i64 = 0, streamed: bool = false): Result<Model, string>
```

`load` on the GPU `gpu` (none: the host). With `from`, the `layers` layers from layer `from` on (a stage of
loadAcross): the norm, the head and the MTP layer only where they are the model's last. `streamed`: the vision
tower's linears stay in host memory (loadVision).

Effects: fail, read, write, console, io.

### `converted`

```gizlang
function converted(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string throws string
```

The folder of the model in folder `folder` ("" for its top) of Hugging Face repository `repo` ("Qwen/Qwen3.5-4B")
converted with `bits` (and `layers`, as `convert` cuts), for `load` with the same `layers`: the folder is downloaded
through `hub` once (Hub.snapshot) and converted once into `.gizlang-llm/REPO[/FOLDER]/BITS` in the hub's folder. A
conversion is written to a partial folder and renamed when it is complete, so a stopped one is done again, never
loaded.

Effects: fail, read, write, console, io.

### `convertedFolder`

```gizlang
function convertedFolder(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string
```

Where `converted` puts a conversion; the folder holds model.safetensors once the conversion is complete.

Effects: fail.

### `toolCalls`

```gizlang
function toolCalls(text: string, tools: Vec<JsonValue>, first: i64 = 0): Vec<Hub.Call>
```

The tool calls in the text a model wrote after its answer (Model.calls: <tool_call><function=NAME><parameter=P>
VALUE</parameter>...), each value as JSON by the type its tool gives the parameter: a string stays text, another
type is read as JSON (text when it is not). Their ids are call_FIRST_K.

Effects: fail.

### `kvFormat`

```gizlang
function kvFormat(name: string): KvFormat throws string
```

The format a name gives: BITS for keys and values, or KEYS/VALUES ("4/2"); "" is the engine's choice (keys 0).

Effects: fail.

### `kvName`

```gizlang
function kvName(f: KvFormat): string
```

The name of a format, as kvFormat reads it.

Effects: fail.

### `convertVision`

```gizlang
function convertVision(sys: System, stored: Vec<Stored>, config: JsonValue, to: string): i64 throws string
```

Writes the vision tower of the Hugging Face folder `from` (model.visual.*, as stored) to `to`/visual.safetensors,
with its heads and deepstack blocks (config.json's vision_config) in the header; returns its bytes.

Effects: fail, io.

### `hasVision`

```gizlang
function hasVision(sys: System, folder: string): bool
```

Whether `folder` holds a vision tower (convertVision's).

Effects: fail, io.

### `loadVision`

```gizlang
function loadVision(sys: System, folder: string, gpu: Option<Gpu>, streamed: bool = false): Vision throws string
```

The vision tower of `folder`/visual.safetensors (convertVision's) on `gpu` when there is one. A file of format 1
(before 10/22: Qwen3-VL-32B's for H3, Qwen3.5-9B's for FLUX.3's upsampler) has 16 heads, and its deepstack mergers
read blocks 8, 16 and 24. With `streamed`, its linears' weights stay in host memory and a pass uploads each as it
reads it (Vision.w): the card keeps their bytes free between pictures, and a pass moves them over the bus.

Effects: fail, read, write, console, io.

### `stream`

```gizlang
function stream(deep: i64): Stream
```

An empty stream whose pictures keep `deep` deepstack features each (Vision.seen).

Effects: fail.

### `visionSize`

```gizlang
function visionSize(width: i64, height: i64, least: i64, most: i64): Vec<i64>
```

The size [w, h] at which Qwen's processors show a picture of `width` x `height` (smart_resize): each side a multiple
of 32 (2 x 2 patches of 16), the area from `least` to `most` pixels, the aspect kept as near as they allow.

Effects: fail.

### `greedy`

```gizlang
function greedy(): Sampling
```

The most likely token at each step: the same tokens on each run.

### `pick`

```gizlang
function pick(logits: Vec<f32>, at: i64, vocab: i64, s: Sampling, state: mut Vec<u64>): i64
```

A token from logits[at, at + vocab); `state` holds the random state.

Effects: fail.

### `bertWeights`

```gizlang
function bertWeights(tensors: Vec<Named>): Map<string, Tensor>
```

The tensors of a BertModel or RobertaModel checkpoint by their names, without the "bert." or "roberta." prefix of a
task model, and with LayerNorm.gamma and beta (older checkpoints) as weight and bias.

Effects: fail.

### `bert`

```gizlang
function bert(c: BertConfig, w: Map<string, Tensor>, ids: Vec<i64>, types: Vec<i64>, sequences: i64): Tensor throws string
```

The last hidden states (sequences tokens, dim) of the encoder for `sequences` sequences of one length: their token
ids and token types one sequence after another, every token attended (no padding). The weights may be on a GPU
(Encoder.on); the states are where the weights are.

Effects: fail, read, write, console, io.

### `encoderOf`

```gizlang
function encoderOf(files: ModelFiles): Encoder throws string
```

The encoder of a model by the files it reads from its repository: config.json, tokenizer.json, model.safetensors
and, for a sentence-transformers model, modules.json, sentence_bert_config.json and the pooling module's
config.json. Without modules.json the tokens are mean-pooled and not normalized, as sentence-transformers does.

Effects: fail, read, write, console, io.

### `crossEncoderOf`

```gizlang
function crossEncoderOf(files: ModelFiles): CrossEncoder throws string
```

The cross-encoder of a model by its repository's files: config.json (its sentence_transformers.activation_fn, when
present, names the activation), tokenizer.json and model.safetensors.

Effects: fail, read, write, console, io.

## Methods

### `Arch.qRows`

```gizlang
function Arch.qRows(self: Arch): i64
```

Values of a query (and of its output gate when the model has one) per token.

Effects: fail.

### `Arch.qkvRows`

```gizlang
function Arch.qkvRows(self: Arch): i64
```

Rows of the fused q, k, v projection.

Effects: fail.

### `Arch.convDim`

```gizlang
function Arch.convDim(self: Arch): i64
```

Channels of a recurrent layer's convolution: q, k and v.

Effects: fail.

### `Arch.inRows`

```gizlang
function Arch.inRows(self: Arch): i64
```

Rows of a recurrent layer's fused input projection: q, k, v, the gate z, then b and a (one a value head).

Effects: fail.

### `Model.chatPrompt`

```gizlang
function Model.chatPrompt(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): string
```

The prompt of a chat as the model's template writes it, ending with the start of the assistant's turn (with an
empty thinking block unless `thinking`), a thinking turn with the instruction of its reasoning `effort` (low, medium or
high: Model.efforts), and the tools it may call (OpenAI's tool objects, for a model whose
template writes tool calls: Model.calls).

Effects: fail.

### `Model.chatTokens`

```gizlang
function Model.chatTokens(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): ChatTokens
```

The tokens of `messages` in the model's chat template, up to where the assistant's turn starts (ChatTokens).

Effects: fail.

### `Model.engine`

```gizlang
function Model.engine(self: Model, slots: i64, context: i64, reserve: i64 = 134_217_728, drafts: i64 = 3, rows: i64 = 0, kv: string = ""): Result<Engine, string>
```

An engine for `slots` sequences of up to `context` positions, reading up to `rows` rows a step (0: 256; at least
1 + drafts rows a slot, 2 without drafts). On a GPU the KV pages take the device memory left after the work buffers
and recurrent states, at most what the slots can hold, keeping `reserve` bytes free. Keys and values take the format
`kv` names (kvFormat; F16 where the heads take no codes, see kvWidth), by default the first of kvFormats in which
the slots' whole contexts fit above 4 bits, or one whole context at most 4 bits, else the last (on the host 8 bits). An engine reads up to `drafts` drafts a sequence and step (0: none; 3 is
DRAFTS), and with the model's MTP layer Engine.advance drafts them where that gives tokens faster (EXPLORE), on a GPU
or the host (where 1 draft is the fastest: its 4-bit layers take two rows in the time of one, and each further draft
costs a step of the MTP layer).

Effects: fail, read, write, console, io.

### `Engine.reset`

```gizlang
function Engine.reset(self: mut Engine, slot: i64): void
```

Empties a slot for a new sequence.

Effects: fail, read, write, console, io.

### `Engine.length`

```gizlang
function Engine.length(self: Engine, slot: i64): i64
```

Positions held by a slot.

Effects: fail.

### `Engine.step`

```gizlang
function Engine.step(self: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, wanted: Vec<i64>, drafts: Vec<i64> = [], visual: Option<Visual> = undefined): Vec<f32> throws string
```

Reads `tokens` into their slots (the rows of a slot together and in order, after what the slot holds) and
returns the logits of the rows `wanted` (vocab floats each, in that order). On an engine that drafts (Engine.depth),
a slot's last drafts[slot] rows (none when drafts is empty) are drafts of the tokens after its row before them:
they wait for Engine.accept, which the slot's next step needs first. With `visual`, some rows are a picture's
(Visual, as Engine.hidden), and every row takes its three rotary positions from it.

Effects: fail, read, write, console, io.

### `Engine.accept`

```gizlang
function Engine.accept(self: mut Engine, slot: i64, kept: i64): bool throws string
```

Keeps the first `kept` of a slot's drafts of its last step and drops the others.

Effects: fail, read, write, console, io.

### `Engine.hidden`

```gizlang
function Engine.hidden(self: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, layers: Vec<i64> = [], visual: Option<Visual> = undefined): Vec<f32> throws string
```

Reads `tokens` into their slots as Engine.step does and returns the residual stream of each row after each of
`layers` (counted from 1; none: the model's last), dim floats a row and layer: Hugging Face's hidden_states[l] for
each l, before the final norm (Hugging Face norms the whole model's last hidden state). With `visual`, some rows are a
picture's (Visual), not their tokens' embeddings.

Effects: fail, read, write, console, io.

### `Engine.padding`

```gizlang
function Engine.padding(self: mut Engine, sys: System, slot: i64, token: i64, position: i64, layers: Vec<i64> = []): Vec<f32> throws string
```

The hidden states (as Engine.hidden) of one row of `token` at rotary position `position` that sees the keys of the
slot's tokens but not its own, and leaves the slot as it was: a padding row of a Hugging Face batch, whose mask
hides the padding keys from every row. Every padding row of a sequence is this one row (FLUX.3 reads Qwen3-VL's,
at position 1). Models of attention layers only.

Effects: fail, read, write, console, io.

### `Engine.expertCache`

```gizlang
function Engine.expertCache(self: Engine): ExpertCache
```

The expert cache counts of the engine.

### `Vision.seen`

```gizlang
function Vision.seen(self: Vision, picture: Tensor, later: Vec<Tensor> = []): Vec<Tensor>
```

The vision tower's rows of a picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32): its merged rows
([h w / 1024, the language model's width]) and the deepstack features after each of its `deep` blocks (rows as the
merged), f32 on the host. A video's pair of frames is `picture` and `later` (one frame).

Effects: fail, read, write, console, io.

### `Stream.words`

```gizlang
function Stream.words(self: mut Stream, tokens: Vec<i64>, tag: i64): void
```

The tokens `tokens` of `tag`.

Effects: fail.

### `Stream.picture`

```gizlang
function Stream.picture(self: mut Stream, vision: Vision, picture: Tensor, start: i64, pad: i64, end: i64, later: Vec<Tensor> = []): void
```

Picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32) as `vision` reads it: the token `start`, a
`pad` token for each row of the tower's merged rows (on the rotary positions of its grid), then `end`. A video's pair of
frames is `picture` and `later` (one frame; the tower's temporal patch of two).

Effects: fail, read, write, console, io.

### `Stream.video`

```gizlang
function Stream.video(self: mut Stream, vision: Vision, tok: Tokenizer, clip: Tensor, times: Vec<f64>, start: i64, pad: i64, end: i64, tag: i64): bool throws string
```

A video ([n, 3, h, w] f32 in [0, 1] on the host, h and w multiples of 32; `times`: each frame's second) as Qwen3-VL
reads it: its frames in pairs (the last repeated when n is odd), each pair the text "<t seconds>" (`tok`'s tokens, of
`tag`; t the second of its middle) and then the pair as Stream.picture shows it.

Effects: fail, read, write, console, io.

### `Stream.part`

```gizlang
function Stream.part(self: Stream, at: i64, end: i64, dim: i64): Option<Visual>
```

The pictures' rows that fall in tokens `at` to `end`, as the step over them takes them (rows `dim` wide), or none
when there are no pictures.

Effects: fail.

### `Vision.chat`

```gizlang
function Vision.chat(self: Vision, tok: Tokenizer, chat: ChatTokens, shown: Vec<Shown>): Seen throws string
```

The prompt `chat` (Model.chatTokens) with what it shows: in its text each picture is Qwen's
"<|vision_start|><|image_pad|><|vision_end|>" and each video "<|vision_start|><|video_pad|><|vision_end|>", in the
order of `shown`; each becomes the tower's rows (Stream.picture, Stream.video), and the mark and head move past the
rows before them.

Effects: fail, read, write, console, io.

### `Engine.generate`

```gizlang
function Engine.generate(self: mut Engine, sys: System, requests: Vec<Request>): Vec<Vec<i64>> throws string
```

Generates for every request and returns each one's new tokens (without the stop token): Engine.add of each, then
Engine.advance until none is in turn.

Effects: fail, read, write, console, io.

### `Engine.warm`

```gizlang
function Engine.warm(self: mut Engine, sys: System): bool throws string
```

Runs once each kind of step a server meets first: a long prompt read in parts, then 1 to `slots` sequences that read
a prompt and write tokens. The first run of a step shape compiles, tunes and records its kernels (seconds for a large
model), so a server warms before it takes requests.

Effects: fail, read, write, console, io.

### `Engine.add`

```gizlang
function Engine.add(self: mut Engine, request0: Request, mark: i64 = 0, head: i64 = 0): i64 throws string
```

Puts a request in turn and returns its id, which Engine.advance's results name. A free slot that holds the start of
its prompt (an earlier request's prompt and answer) takes it and reads only the rest; so does a slot that takes back
such a start from the host, where a slot keeps what it held when another request takes it (Prefix). After `mark`
tokens of the prompt (0: none) its slot also keeps the recurrent states, so a later prompt that starts with those
tokens but not with all the slot read reads only the rest too; for a chat that is where its messages end
(Model.chatTokens). The prompt's first `head` tokens (0: none; for a chat its system turn with
the tools), cut to whole KV pages and when at least SHARED_MIN, become a start that later prompts share (Shared):
a slot whose prompt starts with it reads only the rest, and on a GPU all such slots read one copy of its keys and
values.

Effects: fail.

### `Engine.cancel`

```gizlang
function Engine.cancel(self: mut Engine, id: i64): bool
```

Takes a request out of turn; false when it is not in turn (it ended). Its slot keeps what it read.

Effects: fail.

### `Engine.busy`

```gizlang
function Engine.busy(self: Engine): bool
```

Whether requests are in turn.

Effects: fail.

### `Engine.advance`

```gizlang
function Engine.advance(self: mut Engine, sys: System): Vec<Progress> throws string
```

One step of the requests in turn, and what each gained. First free slots take waiting requests (Engine.add). The
step reads the next token of every generating sequence and, in the rows left, the next piece of the prompts not
read yet. When the KV pool has no room for the step, a free slot gives its pages, else the request that came last
waits again from where it is (its prompt and answer so far).

A drafting engine (Engine.depth) also reads each generating sequence's drafts of the tokens after that: each row
judges the draft after it (kept when it is the token picked; when sampling, kept with its probability and otherwise
a token other than it is picked, which gives the same distribution as without drafts) up to the first rejected,
and when all are kept the last row gives one more token, so a step reading weights once can give 1 + depth tokens.
The step reads them as drafts (Engine.step) and keeps those judged right (Engine.accept). Then the MTP layer reads
every row kept, with the token after it, and where drafts gave tokens faster for as many generating sequences
(EXPLORE), drafts after each sequence's last; it reads each draft again, with its own hidden state for it, for the next.

Effects: fail, read, write, console, io.

### `Engine.complete`

```gizlang
function Engine.complete(self: mut Engine, sys: System, prompt: string, limit: i64, s: Sampling): string throws string
```

The text the model writes after `prompt` (special tokens in it read as such), at most `limit` tokens.

Effects: fail, read, write, console, io.

### `Encoder.on`

```gizlang
function Encoder.on(self: Encoder, gpu: Gpu): Encoder
```

The encoder with its weights on `gpu`: it encodes there and gives the same embeddings to f32 precision.

Effects: fail, read, write, console, io.

### `Encoder.embed`

```gizlang
function Encoder.embed(self: Encoder, text: string, normalize: bool): Vec<f64> throws string
```

The embedding of a text (Encoder.embedBatch of one).

Effects: fail, read, write, console, io.

### `Encoder.embedBatch`

```gizlang
function Encoder.embedBatch(self: Encoder, texts: Vec<string>, normalize: bool, batch: i64): Vec<Vec<f64>> throws string
```

The embeddings of texts: their token states pooled and, when the model says so or `normalize`, of length 1. The
texts of one token count go through the encoder together, at most `batch` of them at a time, so no token is padding.

Effects: fail, read, write, console, io.

### `CrossEncoder.on`

```gizlang
function CrossEncoder.on(self: CrossEncoder, gpu: Gpu): CrossEncoder
```

The cross-encoder with its weights on `gpu`.

Effects: fail, read, write, console, io.

### `CrossEncoder.scores`

```gizlang
function CrossEncoder.scores(self: CrossEncoder, query: string, documents: Vec<string>, batch: i64): Vec<Vec<f64>> throws string
```

The scores of the labels of each pair (query, documents[i]), after the activation: CrossEncoder.predict. The pairs
of one token count go through the encoder together, at most `batch` at a time.

Effects: fail, read, write, console, io.

### `CrossEncoder.rank`

```gizlang
function CrossEncoder.rank(self: CrossEncoder, query: string, documents: Vec<string>, top: i64, batch: i64): Vec<Ranked> throws string
```

The documents by the score of (query, document), the highest first, at most `top` of them (0: all): CrossEncoder.
rank. Equal scores keep the documents' order. The model must have one label.

Effects: fail, read, write, console, io.

## Constants

### `ATTENTION`

```gizlang
const ATTENTION = 0
```

A layer of softmax attention over a KV cache.

### `RECURRENT`

```gizlang
const RECURRENT = 1
```

A layer of gated delta-rule linear attention over a recurrent state.

### `CALL`

```gizlang
const CALL = "<tool_call>"
```

Where a model that writes tool calls (Model.calls) starts them.

### `DRAFTS`

```gizlang
const DRAFTS = 3
```

The tokens an engine drafts per sequence and step by default: the n-th draft is kept less often than the one before,
and each costs a step of the MTP layer and the head.

gizml:llm

language models, fast on any hardware.

A model family is a recipe: its config.json read as an Arch, and the names of its tensors. Every family runs on the same operations, so a new family is one recipe. convert turns a Hugging Face folder into this module's format once: each linear is stored in a format (Formats), after a Hadamard rotation of each block of 128 inputs, in the order the GPU reads it. load opens a converted model on the best device it finds, and its Engine runs steps of many sequences at once.

gizlang

import * as Llm from "gizml:llm";

Types

`Message`

gizlang

type Message = { role: string; content: string; reasoning?: string; calls?: Vec<Hub.Call>; }

A message of a chat: its role ("system", "user", "assistant" or "tool") and text; an assistant's reasoning before its answer and the tools it called.

role: string
content: string
reasoning: _gizlang_sdk_core.Option<string>
calls: _gizlang_sdk_core.Option<Vec<gizlang_hub.Call>>

`Arch`

gizlang

type Arch = { family: string; vocab: i64; dim: i64; ffn: i64; layers: i64; kinds: Vec<i64>; eps: f64; heads: i64; kv: i64; headDim: i64; rotary: i64; rope: Vec<f64>; qkNorm: bool; gated: bool; bias: bool; kHeads: i64; vHeads: i64; kDim: i64; vDim: i64; conv: i64; tied: bool; mtp: i64; context: i64; prefix: string; sections: Vec<i64>; hc: i64; hcRank: i64; experts: i64; topK: i64; ple: i64; eos: i64; sigmoidGate: bool; }

What a model computes, as its family's recipe reads config.json. Attention: heads query heads of headDim over kv key-value heads, rotary positions on the first rotary dimensions of a head (rope their inverse frequencies), RMS norms of q and k (qkNorm), a sigmoid gate on the output (gated), biases on q, k and v. Recurrent layers: kHeads query-key heads of kDim, vHeads value heads of vDim, a causal convolution of conv taps. mtp layers predict a further token.

family: string
vocab: i64
dim: i64
ffn: i64
layers: i64
kinds: Vec<i64>
eps: f64
heads: i64
kv: i64
headDim: i64
rotary: i64
rope: Vec<f64>
qkNorm: bool
gated: bool
bias: bool
kHeads: i64
vHeads: i64
kDim: i64
vDim: i64
conv: i64
tied: bool
mtp: i64
context: i64
prefix: string
sections: Vec<i64>
hc: i64
hcRank: i64
experts: i64
topK: i64
ple: i64
eos: i64
sigmoidGate: bool

`Bits`

gizlang

type Bits = { layers: i64; head: i64; }

The format of each kind of linear (Formats): layers for the layers, head for the output head.

layers: i64
head: i64

`Model`

gizlang

type Model = { arch: Arch; folder: string; tokenizer: Tokenizer; stops: Vec<i64>; template: string; system: string; think: bool; calls: bool; efforts: Vec<string>; sampling: Sampling; embed: Stored; layers: Vec<Layer>; norm: i64; head: i64; mtp: Option<Mtp>; mixer: Vec<i64>; linears: Vec<Linear>; small: Vec<Vec<f32>>; gpu: Option<Gpu>; device: Vec<DeviceLinear>; deviceSmall: Vec<Buffer>; signs: Vec<Buffer>; files: FlashFiles; vision?: Vision; stages: Vec<Model>; npu?: NpuLlm.Model; }

A converted model, on the GPU that gizhw:gpu opens (GIZLANG_DEVICE) when there is one (device), else on the host (linears); the small tensors are on both. The embedding stays in the file: a step reads its tokens' rows. template is the chat layout ("chatml", "llama3" or ""), system the system turn it writes when a chat has none, think whether its assistant turn opens with a thinking block, calls whether it writes tool calls as <tool_call><function=NAME><parameter=P>...; efforts the instructions its template gives a thinking turn at low, medium and high reasoning effort ("": none); stops are the tokens that end a generation and sampling the model's own settings. mtp is the MTP layer on a GPU when the model has one, vision its vision tower when its folder has one (convertVision), stages the models of its later layers on other GPUs (loadAcross).

arch: M23.Arch
folder: string
tokenizer: M34.Tokenizer
stops: Vec<i64>
template: string
system: string
think: bool
calls: bool
efforts: Vec<string>
sampling: M23.Sampling
embed: gizlang_safetensors.Stored
layers: Vec<M23.Layer>
norm: i64
head: i64
mtp: _gizlang_sdk_core.Option<M23.Mtp>
mixer: Vec<i64>
linears: Vec<M23.Linear>
small: Vec<Vec<f32>>
gpu: _gizlang_sdk_core.Option<M10.Gpu>
device: Vec<M23.DeviceLinear>
deviceSmall: Vec<M10.Buffer>
signs: Vec<M10.Buffer>
files: M23.FlashFiles
vision: _gizlang_sdk_core.Option<M23.Vision>
stages: Vec<M23.Model>
npu: _gizlang_sdk_core.Option<M28.Model>

`ChatTokens`

gizlang

type ChatTokens = { tokens: Vec<i64>; mark: i64; head: i64; }

A chat's prompt as tokens (Model.chatPrompt), the mark of Engine.add (where its messages end, before the assistant's turn starts: the chat's next request starts with these messages, so its slot reads only the rest) and its head (where the system turn with the tools ends: the chats with the same system turn share its keys and values).

tokens: Vec<i64>
mark: i64
head: i64

`Engine`

gizlang

type Engine = { model: Model; slots: i64; context: i64; rows: i64; lengths: Vec<i64>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; convs: Vec<Vec<f32>>; states: Vec<Vec<f32>>; replay: Vec<Vec<f32>>; hidden: Vec<f32>; work: Vec<Buffer>; pages: Vec<Vec<i64>>; table: Vec<f32>; free: Vec<i64>; maxPages: i64; seen: Vec<string>; shapes: Vec<string>; recordings: Vec<G.Recording>; depth: i64; pending: Vec<i64>; drafts: i64; kept: i64; draftBlocks: Vec<i64>; blockEntry: Vec<i64>; nextEntry: i64; paces: Vec<f64>; stepNs: f64; rowNs: f64; kv: KvFormat; waiting: Vec<Turn>; turns: Vec<Turn>; next: Vec<i64>; guesses: Vec<Vec<Guess>>; held: Vec<Vec<i64>>; freed: Vec<i64>; saved: Vec<Saved>; pieces: Vec<bytes>; shared: Vec<Shared>; borrowed: Vec<i64>; stored: Vec<Prefix>; ids: i64; steps: i64; reuse: bool; flash: Flash; stages: Vec<Engine>; }

The engine of a model: slots sequences of at most context positions, steps of at most rows rows.

model: M23.Model
slots: i64
context: i64
rows: i64
lengths: Vec<i64>
keys: Vec<Vec<f32>>
values: Vec<Vec<f32>>
convs: Vec<Vec<f32>>
states: Vec<Vec<f32>>
replay: Vec<Vec<f32>>
hidden: Vec<f32>
work: Vec<M10.Buffer>
pages: Vec<Vec<i64>>
table: Vec<f32>
free: Vec<i64>
maxPages: i64
seen: Vec<string>
shapes: Vec<string>
recordings: Vec<M10.Recording>
depth: i64
pending: Vec<i64>
drafts: i64
kept: i64
draftBlocks: Vec<i64>
blockEntry: Vec<i64>
nextEntry: i64
paces: Vec<f64>
stepNs: f64
rowNs: f64
kv: M23.KvFormat
waiting: Vec<M23.Turn>
turns: Vec<M23.Turn>
next: Vec<i64>
guesses: Vec<Vec<M23.Guess>>
held: Vec<Vec<i64>>
freed: Vec<i64>
saved: Vec<M23.Saved>
pieces: Vec<bytes> — The bytes of each token (Tokenizer.tokenBytes), read at the first request with a grammar.
shared: Vec<M23.Shared>
borrowed: Vec<i64>
stored: Vec<M23.Prefix>
ids: i64
steps: i64
reuse: bool
flash: M23.Flash
stages: Vec<M23.Engine>

`Visual`

gizlang

type Visual = { rows: Vec<i64>; embeds: Vec<f32>; deep: Vec<Vec<f32>>; grid: Vec<i64>; }

Rows of a step that hold no token's embedding (Qwen3-VL's pictures): rows of the step, their embeddings (dim floats a row, in turn), deep the features added to them after each of the first layers (Qwen3-VL's deepstack, rows as embeds), and grid every row's three rotary positions (time, height, width; a text row's three are its position: see ropeTable).

rows: Vec<i64>
embeds: Vec<f32>
deep: Vec<Vec<f32>>
grid: Vec<i64>

`KvFormat`

gizlang

type KvFormat = { keys: i64; values: i64; }

How an engine keeps keys and values: the bits of a key and of a value. 16: F16 values; 8, 4 or 2: codes of the head rotated by blocks of kvBlock, at 2^bits even steps (kvStore).

keys: i64
values: i64

`ExpertCache`

gizlang

type ExpertCache = { slots: i64; hits: i64; misses: i64; readMs: f64; }

The expert cache of an engine: its slots, the experts found in it and read into it, and the ms the reads took.

slots: i64
hits: i64
misses: i64
readMs: f64

`Guess`

gizlang

type Guess = { token: i64; q: Dist; }

A draft per sequence: its token and the distribution it was drawn from (all on the token when it is the most likely).

token: i64
q: M23.Dist

`Vision`

gizlang

type Vision = { named: Map<string, Tensor>; gpu: Option<Gpu>; deep: Vec<i64>; depth: i64; width: i64; heads: i64; side: i64; }

A Qwen vision tower (loadVision): its weights by name where it runs (linears BF16, the rest f32; the patch embedding f32 [width, 768], the position table on the host), its blocks, width and heads, the side of its position table, and the blocks after which a deepstack merger reads the stream.

named: _gizlang_sdk_collections.Map<string, M32.Tensor>
gpu: _gizlang_sdk_core.Option<M10.Gpu>
deep: Vec<i64>
depth: i64
width: i64
heads: i64
side: i64

`Stream`

gizlang

type Stream = { ids: Vec<i64>; tags: Vec<i64>; grid: Vec<i64>; visual: Visual; shift: i64; }

A prompt as a language model with a vision tower reads it: its token ids, each with three rotary positions (grid: its index plus shift, which a picture's rows lower, as their grid is smaller than their count) and a tag of the caller's, and the pictures' rows (visual: the tower's rows at their places).

ids: Vec<i64>
tags: Vec<i64>
grid: Vec<i64>
visual: M23.Visual
shift: i64

`Shown`

gizlang

type Shown = { picture: Tensor; times: Vec<f64>; }

What a chat shows (Vision.chat): a picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32), or a video's frames ([n, 3, h, w]) with each frame's second in times.

picture: M32.Tensor
times: Vec<f64>

`Seen`

gizlang

type Seen = { chat: ChatTokens; visual: Visual; }

A chat's prompt with what it shows (Vision.chat): its tokens, mark and head, and the Visual of the whole prompt (Request.visual).

chat: M23.ChatTokens
visual: M23.Visual

`Sampling`

gizlang

type Sampling = { temperature: f64; topK: i64; topP: f64; seed: u64; presence?: f64; frequency?: f64; repetition?: f64; grammar?: Grammar; after?: i64; logprobs?: i64; }

How tokens are picked: temperature 0 the most likely, else one of the topK most likely (0: any) within topP of their probability, at temperature. Before that the logits are shaped, as OpenAI and vLLM shape them: less by presence for a token the answer has and by frequency for each time it has it; divided by repetition (above 0, times it below 0) for a token the prompt or the answer has. grammar: only the tokens it allows (gizml:grammar), from the token after after on (a reasoning model's end of thinking; none: from the first); a stop token only where the text is whole. logprobs: each token's log probability and the most likely ones (that many) are given, of the model's logits before shaping. A shaped request makes no drafts.

temperature: f64
topK: i64
topP: f64
seed: u64
presence: _gizlang_sdk_core.Option<f64>
frequency: _gizlang_sdk_core.Option<f64>
repetition: _gizlang_sdk_core.Option<f64>
grammar: _gizlang_sdk_core.Option<M22.Grammar>
after: _gizlang_sdk_core.Option<i64>
logprobs: _gizlang_sdk_core.Option<i64>

`Dist`

gizlang

type Dist = { ids: Vec<i64>; weights: Vec<f64>; }

What a sampled token is drawn from: tokens, most likely first, and their weights (a token's probability is its weight over their sum).

ids: Vec<i64>
weights: Vec<f64>

`Request`

gizlang

type Request = { prompt: Vec<i64>; limit: i64; sampling: Sampling; visual?: Visual; }

A prompt to continue: at most limit new tokens. With visual some of its rows are pictures' (Vision.chat): their embeddings replace their tokens', and its rows take their three rotary positions from visual.grid (three for each of its first rows, the last a text row's), the rows after them their position plus that last row's shift.

prompt: Vec<i64>
limit: i64
sampling: M23.Sampling
visual: _gizlang_sdk_core.Option<M23.Visual>

`Logprob`

gizlang

type Logprob = { token: i64; logprob: f64; top: Vec<i64>; tops: Vec<f64>; }

A token's log probability and the top most likely tokens with theirs, of the model's logits.

token: i64
logprob: f64
top: Vec<i64>
tops: Vec<f64>

`Progress`

gizlang

type Progress = { id: i64; tokens: Vec<i64>; cached: i64; end: string; logprobs: Vec<Logprob>; }

What a step gave a request (Engine.advance): its new tokens (without a stop token), the tokens at the start of its prompt that its slot held already (not read again), and why it ended: "stop" (a stop token), "length" (its limit, the context's end, or a KV cache too small for it alone) or "" (it goes on).

id: i64
tokens: Vec<i64>
cached: i64
end: string
logprobs: Vec<M23.Logprob>

`Turn`

gizlang

type Turn = { id: i64; request: Request; random: Vec<u64>; made: i64; mark: i64; head: i64; cached: i64; reading?: Recognizer; }

A request in an engine's turn: its id, its random state, the tokens it gave, the prompt position after which its slot saves the recurrent states (Engine.add), the position after which its prompt's start is shared (Engine.add; 0: none), the prompt tokens its slot held already and its grammar's recognizer while the grammar holds (Sampling).

id: i64
request: M23.Request
random: Vec<u64>
made: i64
mark: i64
head: i64
cached: i64
reading: _gizlang_sdk_core.Option<M22.Recognizer>

`Saved`

gizlang

type Saved = { at: i64; conv: Vec<f32>; state: Vec<f32>; }

A slot's recurrent states after its first at tokens (0: none): the convolutions' last inputs and the delta rules' states, on the host (from a GPU the states' BF16, two to a float).

at: i64
conv: Vec<f32>
state: Vec<f32>

`Prefix`

gizlang

type Prefix = { tokens: Vec<i64>; saved: Saved; pages: Vec<bytes>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Prefix>; }

The start of a chat that left its slot for another (see admit): its tokens, the recurrent states after them, and their keys and values (on a GPU the bytes of their KV pages, on the host per pool layer); used is the step it was stored at. A later prompt that starts with the tokens takes them back into a slot and reads only the rest. On a model in stages, stages holds each later stage's pages and states (Engine.stages).

tokens: Vec<i64>
saved: M23.Saved
pages: Vec<bytes>
keys: Vec<Vec<f32>>
values: Vec<Vec<f32>>
used: i64
stages: Vec<M23.Prefix>

`Shared`

gizlang

type Shared = { tokens: Vec<i64>; saved: Saved; pages: Vec<i64>; keys: Vec<Vec<f32>>; values: Vec<Vec<f32>>; used: i64; stages: Vec<Shared>; }

A start that many prompts share (as the system turn and tools of an agent's chats, Model.chatTokens): its tokens (whole KV pages), the recurrent states after them, and their keys and values: on a GPU the KV pages of the pool that hold them, which the slots that start with them read in place (Engine.borrowed), on the host a copy per pool layer; used is the step a slot last took it. On a model in stages, stages holds each later stage's pages and states.

tokens: Vec<i64>
saved: M23.Saved
pages: Vec<i64>
keys: Vec<Vec<f32>>
values: Vec<Vec<f32>>
used: i64
stages: Vec<M23.Shared>

`BertConfig`

gizlang

type BertConfig = { kind: string; layers: i64; heads: i64; dim: i64; eps: f64; act: string; positions: i64; offset: i64; }

A BERT-family encoder as a Hugging Face config.json gives it: BERT, RoBERTa and XLM-RoBERTa have one layout and differ in where positions start (offset: 0 for BERT, pad_token_id + 1 for RoBERTa). act is hidden_act: "gelu" (the erf form), "relu" or a tanh form ("gelu_new", "gelu_pytorch_tanh").

kind: string
layers: i64
heads: i64
dim: i64
eps: f64
act: string
positions: i64
offset: i64

`ModelFiles`

gizlang

type ModelFiles = (path: string) => Result<Option<bytes>, string>

Reads the file at a path of a model's repository: none when the model has no such file.

`Encoder`

gizlang

type Encoder = { config: BertConfig; weights: Map<string, Tensor>; tokenizer: Tokenizer; pooling: string; normalize: bool; limit: i64; }

A sentence encoder as sentence-transformers 3 builds it from a model's files: an encoder, then the pooling of its token states ("mean", "cls", "max", "mean_sqrt_len" or "last"), then L2 normalization when the model has that module.

config: M23.BertConfig
weights: _gizlang_sdk_collections.Map<string, M32.Tensor>
tokenizer: M34.Tokenizer
pooling: string
normalize: bool
limit: i64

`CrossEncoder`

gizlang

type CrossEncoder = { config: BertConfig; weights: Map<string, Tensor>; tokenizer: Tokenizer; head: string; labels: i64; activation: string; limit: i64; }

A cross-encoder as sentence-transformers' CrossEncoder builds it: a sequence classification model that reads a pair of texts (a query and a document) and gives the scores of its labels. head is "bert" (BertForSequence- Classification: the pooler's tanh on the first token, then the classifier) or "roberta" (RobertaFor- and XLMRobertaForSequenceClassification: classifier.dense, tanh, classifier.out_proj). activation is the function on the scores: "sigmoid" (the default with one label), "identity" (the default with more) or "tanh".

config: M23.BertConfig
weights: _gizlang_sdk_collections.Map<string, M32.Tensor>
tokenizer: M34.Tokenizer
head: string
labels: i64
activation: string
limit: i64

`Ranked`

gizlang

type Ranked = { corpus: i64; score: f64; }

A document of a ranking: its index in the documents and its score.

corpus: i64
score: f64

Functions

`archOf`

gizlang

function archOf(config: JsonValue): Arch throws string

The Arch of a model by its config.json; an error names what this module does not run yet.

Effects: fail.

`convert`

gizlang

function convert(sys: System, from: string, to: string, bits: Bits, layers: i64 = 0, lora: string = ""): i64 throws string

Converts the Hugging Face model in folder from (config.json, safetensors, tokenizer), or the GGUF file from (gizlang:gguf: Llama, Qwen2 and Qwen3, its block types read as they are; the embedding keeps its block type), to this module's format in folder to: model.safetensors with every linear quantized (see Quantization), the norms and small tensors as f32 and the embedding as its source holds it, with the source's config, tokenizer and generation files. With layers less than the model's, only its first layers layers, without the final norm, the head and the MTP layer (a source cut after them, as MiniMax H3's text encoder, holds no more): load it with the same layers. A cut Qwen3.8 Flash keeps its mixer and head (a test of its first layers). A vision tower (model.visual.*) goes to visual.safetensors (convertVision). Returns the bytes written. With lora, the PEFT LoRA adapter in that folder (adapter_config.json, adapter_model.safetensors) is merged into the linears it adapts before they are quantized: W + alpha / r B A (alpha / sqrt(r) with rsLoRA).

Effects: fail, read, write, console, io.

`load`

gizlang

function load(sys: System, folder: string, onGpu: bool = true, layers: i64 = 0): Model throws string

The converted model in folder, on the GPU when onGpu and there is one, and across the driver's other GPUs too when its weights do not leave SPLIT_ROOM free on that one (loadAcross). With layers less than the model's, only its first layers layers, without the final norm, the head and the MTP layer: an engine of it gives hidden states (Engine.hidden), not logits. A gizml:npullm folder (model.q4nx) runs on this host's NPU (Model.npu).

Effects: fail, read, write, console, io.

`loadAcross`

gizlang

function loadAcross(sys: System, folder: string, gpus: Vec<Gpu>, counts: Vec<i64> = []): Model throws string

The converted model in folder across gpus (Gpu.others), in stages of consecutive layers: the first GPU holds the first stage (and the vision tower), the last the last stage with the final norm and the head (Model.stages). A stage's layer count is counts[i], by default in proportion to its GPU's free memory. A step runs the stages in turn, each on its rows' hidden states after the one before (Engine). Without the MTP layer, and not for Qwen3.8 Flash yet. The first GPU is current at the end.

Effects: fail, read, write, console, io.

`loadOn`

gizlang

function loadOn(sys: System, folder: string, gpu: Option<Gpu>, layers: i64 = 0, from: i64 = 0, streamed: bool = false): Result<Model, string>

load on the GPU gpu (none: the host). With from, the layers layers from layer from on (a stage of loadAcross): the norm, the head and the MTP layer only where they are the model's last. streamed: the vision tower's linears stay in host memory (loadVision).

Effects: fail, read, write, console, io.

`converted`

gizlang

function converted(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string throws string

The folder of the model in folder folder ("" for its top) of Hugging Face repository repo ("Qwen/Qwen3.5-4B") converted with bits (and layers, as convert cuts), for load with the same layers: the folder is downloaded through hub once (Hub.snapshot) and converted once into .gizlang-llm/REPO[/FOLDER]/BITS in the hub's folder. A conversion is written to a partial folder and renamed when it is complete, so a stopped one is done again, never loaded.

Effects: fail, read, write, console, io.

`convertedFolder`

gizlang

function convertedFolder(hub: Hub.Hub, repo: string, folder: string, bits: Bits, layers: i64 = 0): string

Where converted puts a conversion; the folder holds model.safetensors once the conversion is complete.

Effects: fail.

`toolCalls`

gizlang

function toolCalls(text: string, tools: Vec<JsonValue>, first: i64 = 0): Vec<Hub.Call>

The tool calls in the text a model wrote after its answer (Model.calls: <tool_call><function=NAME><parameter=P> VALUE</parameter>...), each value as JSON by the type its tool gives the parameter: a string stays text, another type is read as JSON (text when it is not). Their ids are call_FIRST_K.

Effects: fail.

`kvFormat`

gizlang

function kvFormat(name: string): KvFormat throws string

The format a name gives: BITS for keys and values, or KEYS/VALUES ("4/2"); "" is the engine's choice (keys 0).

Effects: fail.

`kvName`

gizlang

function kvName(f: KvFormat): string

The name of a format, as kvFormat reads it.

Effects: fail.

`convertVision`

gizlang

function convertVision(sys: System, stored: Vec<Stored>, config: JsonValue, to: string): i64 throws string

Writes the vision tower of the Hugging Face folder from (model.visual.*, as stored) to to/visual.safetensors, with its heads and deepstack blocks (config.json's vision_config) in the header; returns its bytes.

Effects: fail, io.

`hasVision`

gizlang

function hasVision(sys: System, folder: string): bool

Whether folder holds a vision tower (convertVision's).

Effects: fail, io.

`loadVision`

gizlang

function loadVision(sys: System, folder: string, gpu: Option<Gpu>, streamed: bool = false): Vision throws string

The vision tower of folder/visual.safetensors (convertVision's) on gpu when there is one. A file of format 1 (before 10/22: Qwen3-VL-32B's for H3, Qwen3.5-9B's for FLUX.3's upsampler) has 16 heads, and its deepstack mergers read blocks 8, 16 and 24. With streamed, its linears' weights stay in host memory and a pass uploads each as it reads it (Vision.w): the card keeps their bytes free between pictures, and a pass moves them over the bus.

Effects: fail, read, write, console, io.

`stream`

gizlang

function stream(deep: i64): Stream

An empty stream whose pictures keep deep deepstack features each (Vision.seen).

Effects: fail.

`visionSize`

gizlang

function visionSize(width: i64, height: i64, least: i64, most: i64): Vec<i64>

The size [w, h] at which Qwen's processors show a picture of width x height (smart_resize): each side a multiple of 32 (2 x 2 patches of 16), the area from least to most pixels, the aspect kept as near as they allow.

Effects: fail.

`greedy`

gizlang

function greedy(): Sampling

The most likely token at each step: the same tokens on each run.

`pick`

gizlang

function pick(logits: Vec<f32>, at: i64, vocab: i64, s: Sampling, state: mut Vec<u64>): i64

A token from logits[at, at + vocab); state holds the random state.

Effects: fail.

`bertWeights`

gizlang

function bertWeights(tensors: Vec<Named>): Map<string, Tensor>

The tensors of a BertModel or RobertaModel checkpoint by their names, without the "bert." or "roberta." prefix of a task model, and with LayerNorm.gamma and beta (older checkpoints) as weight and bias.

Effects: fail.

`bert`

gizlang

function bert(c: BertConfig, w: Map<string, Tensor>, ids: Vec<i64>, types: Vec<i64>, sequences: i64): Tensor throws string

The last hidden states (sequences tokens, dim) of the encoder for sequences sequences of one length: their token ids and token types one sequence after another, every token attended (no padding). The weights may be on a GPU (Encoder.on); the states are where the weights are.

Effects: fail, read, write, console, io.

`encoderOf`

gizlang

function encoderOf(files: ModelFiles): Encoder throws string

The encoder of a model by the files it reads from its repository: config.json, tokenizer.json, model.safetensors and, for a sentence-transformers model, modules.json, sentence_bert_config.json and the pooling module's config.json. Without modules.json the tokens are mean-pooled and not normalized, as sentence-transformers does.

Effects: fail, read, write, console, io.

`crossEncoderOf`

gizlang

function crossEncoderOf(files: ModelFiles): CrossEncoder throws string

The cross-encoder of a model by its repository's files: config.json (its sentence_transformers.activation_fn, when present, names the activation), tokenizer.json and model.safetensors.

Effects: fail, read, write, console, io.

Methods

`Arch.qRows`

gizlang

function Arch.qRows(self: Arch): i64

Values of a query (and of its output gate when the model has one) per token.

Effects: fail.

`Arch.qkvRows`

gizlang

function Arch.qkvRows(self: Arch): i64

Rows of the fused q, k, v projection.

Effects: fail.

`Arch.convDim`

gizlang

function Arch.convDim(self: Arch): i64

Channels of a recurrent layer's convolution: q, k and v.

Effects: fail.

`Arch.inRows`

gizlang

function Arch.inRows(self: Arch): i64

Rows of a recurrent layer's fused input projection: q, k, v, the gate z, then b and a (one a value head).

Effects: fail.

`Model.chatPrompt`

gizlang

function Model.chatPrompt(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): string

The prompt of a chat as the model's template writes it, ending with the start of the assistant's turn (with an empty thinking block unless thinking), a thinking turn with the instruction of its reasoning effort (low, medium or high: Model.efforts), and the tools it may call (OpenAI's tool objects, for a model whose template writes tool calls: Model.calls).

Effects: fail.

`Model.chatTokens`

gizlang

function Model.chatTokens(self: Model, messages: Vec<Message>, thinking: bool, tools: Vec<JsonValue> = [], effort: string = "high"): ChatTokens

The tokens of messages in the model's chat template, up to where the assistant's turn starts (ChatTokens).

Effects: fail.

`Model.engine`

gizlang

function Model.engine(self: Model, slots: i64, context: i64, reserve: i64 = 134_217_728, drafts: i64 = 3, rows: i64 = 0, kv: string = ""): Result<Engine, string>

An engine for slots sequences of up to context positions, reading up to rows rows a step (0: 256; at least 1 + drafts rows a slot, 2 without drafts). On a GPU the KV pages take the device memory left after the work buffers and recurrent states, at most what the slots can hold, keeping reserve bytes free. Keys and values take the format kv names (kvFormat; F16 where the heads take no codes, see kvWidth), by default the first of kvFormats in which the slots' whole contexts fit above 4 bits, or one whole context at most 4 bits, else the last (on the host 8 bits). An engine reads up to drafts drafts a sequence and step (0: none; 3 is DRAFTS), and with the model's MTP layer Engine.advance drafts them where that gives tokens faster (EXPLORE), on a GPU or the host (where 1 draft is the fastest: its 4-bit layers take two rows in the time of one, and each further draft costs a step of the MTP layer).

Effects: fail, read, write, console, io.

`Engine.reset`

gizlang

function Engine.reset(self: mut Engine, slot: i64): void

Empties a slot for a new sequence.

Effects: fail, read, write, console, io.

`Engine.length`

gizlang

function Engine.length(self: Engine, slot: i64): i64

Positions held by a slot.

Effects: fail.

`Engine.step`

gizlang

function Engine.step(self: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, wanted: Vec<i64>, drafts: Vec<i64> = [], visual: Option<Visual> = undefined): Vec<f32> throws string

Reads tokens into their slots (the rows of a slot together and in order, after what the slot holds) and returns the logits of the rows wanted (vocab floats each, in that order). On an engine that drafts (Engine.depth), a slot's last drafts[slot] rows (none when drafts is empty) are drafts of the tokens after its row before them: they wait for Engine.accept, which the slot's next step needs first. With visual, some rows are a picture's (Visual, as Engine.hidden), and every row takes its three rotary positions from it.

Effects: fail, read, write, console, io.

`Engine.accept`

gizlang

function Engine.accept(self: mut Engine, slot: i64, kept: i64): bool throws string

Keeps the first kept of a slot's drafts of its last step and drops the others.

Effects: fail, read, write, console, io.

`Engine.hidden`

gizlang

function Engine.hidden(self: mut Engine, sys: System, tokens: Vec<i64>, slots: Vec<i64>, layers: Vec<i64> = [], visual: Option<Visual> = undefined): Vec<f32> throws string

Reads tokens into their slots as Engine.step does and returns the residual stream of each row after each of layers (counted from 1; none: the model's last), dim floats a row and layer: Hugging Face's hidden_states[l] for each l, before the final norm (Hugging Face norms the whole model's last hidden state). With visual, some rows are a picture's (Visual), not their tokens' embeddings.

Effects: fail, read, write, console, io.

`Engine.padding`

gizlang

function Engine.padding(self: mut Engine, sys: System, slot: i64, token: i64, position: i64, layers: Vec<i64> = []): Vec<f32> throws string

The hidden states (as Engine.hidden) of one row of token at rotary position position that sees the keys of the slot's tokens but not its own, and leaves the slot as it was: a padding row of a Hugging Face batch, whose mask hides the padding keys from every row. Every padding row of a sequence is this one row (FLUX.3 reads Qwen3-VL's, at position 1). Models of attention layers only.

Effects: fail, read, write, console, io.

`Engine.expertCache`

gizlang

function Engine.expertCache(self: Engine): ExpertCache

The expert cache counts of the engine.

`Vision.seen`

gizlang

function Vision.seen(self: Vision, picture: Tensor, later: Vec<Tensor> = []): Vec<Tensor>

The vision tower's rows of a picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32): its merged rows ([h w / 1024, the language model's width]) and the deepstack features after each of its deep blocks (rows as the merged), f32 on the host. A video's pair of frames is picture and later (one frame).

Effects: fail, read, write, console, io.

`Stream.words`

gizlang

function Stream.words(self: mut Stream, tokens: Vec<i64>, tag: i64): void

The tokens tokens of tag.

Effects: fail.

`Stream.picture`

gizlang

function Stream.picture(self: mut Stream, vision: Vision, picture: Tensor, start: i64, pad: i64, end: i64, later: Vec<Tensor> = []): void

Picture ([3, h, w] f32 in [0, 1] on the host, h and w multiples of 32) as vision reads it: the token start, a pad token for each row of the tower's merged rows (on the rotary positions of its grid), then end. A video's pair of frames is picture and later (one frame; the tower's temporal patch of two).

Effects: fail, read, write, console, io.

`Stream.video`

gizlang

function Stream.video(self: mut Stream, vision: Vision, tok: Tokenizer, clip: Tensor, times: Vec<f64>, start: i64, pad: i64, end: i64, tag: i64): bool throws string

A video ([n, 3, h, w] f32 in [0, 1] on the host, h and w multiples of 32; times: each frame's second) as Qwen3-VL reads it: its frames in pairs (the last repeated when n is odd), each pair the text "<t seconds>" (tok's tokens, of tag; t the second of its middle) and then the pair as Stream.picture shows it.

Effects: fail, read, write, console, io.

`Stream.part`

gizlang

function Stream.part(self: Stream, at: i64, end: i64, dim: i64): Option<Visual>

The pictures' rows that fall in tokens at to end, as the step over them takes them (rows dim wide), or none when there are no pictures.

Effects: fail.

`Vision.chat`

gizlang

function Vision.chat(self: Vision, tok: Tokenizer, chat: ChatTokens, shown: Vec<Shown>): Seen throws string

Effects: fail, read, write, console, io.

`Engine.generate`

gizlang

function Engine.generate(self: mut Engine, sys: System, requests: Vec<Request>): Vec<Vec<i64>> throws string

Generates for every request and returns each one's new tokens (without the stop token): Engine.add of each, then Engine.advance until none is in turn.

Effects: fail, read, write, console, io.

`Engine.warm`

gizlang

function Engine.warm(self: mut Engine, sys: System): bool throws string

Runs once each kind of step a server meets first: a long prompt read in parts, then 1 to slots sequences that read a prompt and write tokens. The first run of a step shape compiles, tunes and records its kernels (seconds for a large model), so a server warms before it takes requests.

Effects: fail, read, write, console, io.

`Engine.add`

gizlang

function Engine.add(self: mut Engine, request0: Request, mark: i64 = 0, head: i64 = 0): i64 throws string

Puts a request in turn and returns its id, which Engine.advance's results name. A free slot that holds the start of its prompt (an earlier request's prompt and answer) takes it and reads only the rest; so does a slot that takes back such a start from the host, where a slot keeps what it held when another request takes it (Prefix). After mark tokens of the prompt (0: none) its slot also keeps the recurrent states, so a later prompt that starts with those tokens but not with all the slot read reads only the rest too; for a chat that is where its messages end (Model.chatTokens). The prompt's first head tokens (0: none; for a chat its system turn with the tools), cut to whole KV pages and when at least SHARED_MIN, become a start that later prompts share (Shared): a slot whose prompt starts with it reads only the rest, and on a GPU all such slots read one copy of its keys and values.

Effects: fail.

`Engine.cancel`

gizlang

function Engine.cancel(self: mut Engine, id: i64): bool

Takes a request out of turn; false when it is not in turn (it ended). Its slot keeps what it read.

Effects: fail.

`Engine.busy`

gizlang

function Engine.busy(self: Engine): bool

Whether requests are in turn.

Effects: fail.

`Engine.advance`

gizlang

function Engine.advance(self: mut Engine, sys: System): Vec<Progress> throws string

One step of the requests in turn, and what each gained. First free slots take waiting requests (Engine.add). The step reads the next token of every generating sequence and, in the rows left, the next piece of the prompts not read yet. When the KV pool has no room for the step, a free slot gives its pages, else the request that came last waits again from where it is (its prompt and answer so far).

A drafting engine (Engine.depth) also reads each generating sequence's drafts of the tokens after that: each row judges the draft after it (kept when it is the token picked; when sampling, kept with its probability and otherwise a token other than it is picked, which gives the same distribution as without drafts) up to the first rejected, and when all are kept the last row gives one more token, so a step reading weights once can give 1 + depth tokens. The step reads them as drafts (Engine.step) and keeps those judged right (Engine.accept). Then the MTP layer reads every row kept, with the token after it, and where drafts gave tokens faster for as many generating sequences (EXPLORE), drafts after each sequence's last; it reads each draft again, with its own hidden state for it, for the next.

Effects: fail, read, write, console, io.

`Engine.complete`

gizlang

function Engine.complete(self: mut Engine, sys: System, prompt: string, limit: i64, s: Sampling): string throws string

The text the model writes after prompt (special tokens in it read as such), at most limit tokens.

Effects: fail, read, write, console, io.

`Encoder.on`

gizlang

function Encoder.on(self: Encoder, gpu: Gpu): Encoder

The encoder with its weights on gpu: it encodes there and gives the same embeddings to f32 precision.

Effects: fail, read, write, console, io.

`Encoder.embed`

gizlang

function Encoder.embed(self: Encoder, text: string, normalize: bool): Vec<f64> throws string

The embedding of a text (Encoder.embedBatch of one).

Effects: fail, read, write, console, io.

`Encoder.embedBatch`

gizlang

function Encoder.embedBatch(self: Encoder, texts: Vec<string>, normalize: bool, batch: i64): Vec<Vec<f64>> throws string

The embeddings of texts: their token states pooled and, when the model says so or normalize, of length 1. The texts of one token count go through the encoder together, at most batch of them at a time, so no token is padding.

Effects: fail, read, write, console, io.

`CrossEncoder.on`

gizlang

function CrossEncoder.on(self: CrossEncoder, gpu: Gpu): CrossEncoder

The cross-encoder with its weights on gpu.

Effects: fail, read, write, console, io.

`CrossEncoder.scores`

gizlang

function CrossEncoder.scores(self: CrossEncoder, query: string, documents: Vec<string>, batch: i64): Vec<Vec<f64>> throws string

The scores of the labels of each pair (query, documents[i]), after the activation: CrossEncoder.predict. The pairs of one token count go through the encoder together, at most batch at a time.

Effects: fail, read, write, console, io.

`CrossEncoder.rank`

gizlang

function CrossEncoder.rank(self: CrossEncoder, query: string, documents: Vec<string>, top: i64, batch: i64): Vec<Ranked> throws string

The documents by the score of (query, document), the highest first, at most top of them (0: all): CrossEncoder. rank. Equal scores keep the documents' order. The model must have one label.

Effects: fail, read, write, console, io.

Constants

`ATTENTION`

gizlang

const ATTENTION = 0

A layer of softmax attention over a KV cache.

`RECURRENT`

gizlang

const RECURRENT = 1

A layer of gated delta-rule linear attention over a recurrent state.

`CALL`

gizlang

const CALL = "<tool_call>"

Where a model that writes tool calls (Model.calls) starts them.

`DRAFTS`

gizlang

const DRAFTS = 3

The tokens an engine drafts per sequence and step by default: the n-th draft is kept less often than the one before, and each costs a step of the MTP layer and the head.